Hadoop2.0 YARN cloudra4.4.0安装配置

一时失言乱红尘 2022-09-18 09:55 97阅读 0赞

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.137.2 yard02
192.168.137.3 yard03
192.168.137.4 yard04
192.168.137.5 yard05
hadoop@hadoop-virtual-machine:~$
</pre> </td> 
 </tr> 
 </tbody> 
</table>

core-site.xml —-conf/hadoop

hdfs-site.xml

yarn-site.xml

<table style="margin:0px; padding:0px; border-spacing:0px"> 
 <tbody style="margin:0px; padding:0px"> 
 <tr style="margin:0px; padding:0px"> 
 <td style="margin:0px; padding:0px 15px 0px 0px; color:rgb(153,153,153); border-right-width:1px; border-right-style:solid; border-right-color:rgb(221,221,221); text-align:right"> <pre style="margin-top:0px; margin-bottom:0px; padding:0px; font-family:Monaco,Menlo,Consolas,'Courier New',monospace; background-color:rgb(238,238,238); border:none; overflow:auto">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
</pre> </td> 
 <td style="margin:0px; padding:0px 0px 0px 15px; border-left-width:1px; border-left-style:solid; border-left-color:rgb(255,255,255); color:rgb(102,102,102)"> <pre style="margin-top:0px; margin-bottom:0px; padding:0px; font-family:Monaco,Menlo,Consolas,'Courier New',monospace; background-color:rgb(238,238,238); border:none; overflow:auto">&lt;configuration&gt;
&lt;!-- Site specific YARN configuration properties --&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.resourcemanager.resource-tracker.address&lt;/name&gt;
 &lt;value&gt;yard02:8031&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.resourcemanager.address&lt;/name&gt;
 &lt;value&gt;yard02:8032&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.resourcemanager.scheduler.address&lt;/name&gt;
 &lt;value&gt;yard02:8030&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.resourcemanager.admin.address&lt;/name&gt;
 &lt;value&gt;yard02:8033&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.resourcemanager.webapp.address&lt;/name&gt;
 &lt;value&gt;yard02:8088&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;description&gt;Classpath for typical applications.&lt;/description&gt;
 &lt;name&gt;yarn.application.classpath&lt;/name&gt;
 &lt;value&gt;$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,
 $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
 $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
 $YARN_HOME/share/hadoop/yarn/*,$YARN_HOME/share/hadoop/yarn/lib/*,
 $YARN_HOME/share/hadoop/mapreduce/*,$YARN_HOME/share/hadoop/mapreduce/lib/*&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt;
 &lt;value&gt;mapreduce.shuffle&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.nodemanager.aux-services.mapreduce.shuffle.class&lt;/name&gt;
 &lt;value&gt;org.apache.hadoop.mapred.ShuffleHandler&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.nodemanager.local-dirs&lt;/name&gt;
 &lt;value&gt;/home/hadoop/bigdata/hadoop-2.0.0-cdh4.4.0/data/yarn/local&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.nodemanager.log-dirs&lt;/name&gt;
 &lt;value&gt;/home/hadoop/bigdata/hadoop-2.0.0-cdh4.4.0/data/yarn/logs&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;description&gt;Where to aggregate logs&lt;/description&gt;
 &lt;name&gt;yarn.nodemanager.remote-app-log-dir&lt;/name&gt;
 &lt;value&gt;/home/hadoop/bigdata/hadoop-2.0.0-cdh4.4.0/data/yarn/logs&lt;/value&gt;
 &lt;/property&gt;
 &lt;property&gt;
 &lt;name&gt;yarn.app.mapreduce.am.staging-dir&lt;/name&gt;
 &lt;value&gt;/home/hadoop/bigdata/hadoop-2.0.0-cdh4.4.0&lt;/value&gt;
&lt;/property&gt;
&lt;/configuration&gt;
</pre> </td> 
 </tr> 
 </tbody> 
</table>

mapred-site.xml

masters

slaves

.bashrc

同步个配置文件

scp /home/hadoop/.bashrc hadoop@yard03:/home/hadoop
scp /home/hadoop/.bashrc hadoop@yard04:/home/hadoop
scp /home/hadoop/.bashrc hadoop@yard05:/home/hadoop
</pre> </td> 
 </tr> 
 </tbody> 
</table>

关于ssh无密码还是这个屡试不爽

scp authorized_keys hadoop@yard03:/home/hadoop/.ssh
scp authorized_keys hadoop@yard04:/home/hadoop/.ssh
scp authorized_keys hadoop@yard05:/home/hadoop/.ssh
</pre> </td> 
 </tr> 
 </tbody> 
</table>

这个启动基本在sbin目录了，  
格式化：

启动

启动hdfs：

启动mapreduce

启动historyserver

WordCount 测试  
./hadoop fs -put /etc/profile /user/hadoop/input  
./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar wordcount input output  
输出结果  
访问：  
[http://yard02:50070/][http_yard02_50070]  
[http://yard03:50090][http_yard03_50090]  
[http://yard02:8088/cluster][http_yard02_8088_cluster]  
[http://yard02:8088/cluster/apps][http_yard02_8088_cluster_apps]  
[http://yard02:8088/cluster/nodes][http_yard02_8088_cluster_nodes]  
[http://yard02:8088/cluster/scheduler][http_yard02_8088_cluster_scheduler]

#### 参考链接 ####

[http://blog.csdn.net/qiaochao911/article/details/9143303][http_blog.csdn.net_qiaochao911_article_details_9143303]

[http://blog.javachen.com/hadoop/2013/03/24/manual-install-Cloudera-Hadoop-CDH/][http_blog.javachen.com_hadoop_2013_03_24_manual-install-Cloudera-Hadoop-CDH]

[http://archive.cloudera.com/cdh4/cdh/4/][http_archive.cloudera.com_cdh4_cdh_4]

[http_yard02_50070]: http://yard02:50070/
[http_yard03_50090]: http://yard03:50090/
[http_yard02_8088_cluster]: http://yard02:8088/cluster
[http_yard02_8088_cluster_apps]: http://yard02:8088/cluster/apps
[http_yard02_8088_cluster_nodes]: http://yard02:8088/cluster/nodes
[http_yard02_8088_cluster_scheduler]: http://yard02:8088/cluster/scheduler
[http_blog.csdn.net_qiaochao911_article_details_9143303]: http://blog.csdn.net/qiaochao911/article/details/9143303
[http_blog.javachen.com_hadoop_2013_03_24_manual-install-Cloudera-Hadoop-CDH]: http://blog.javachen.com/hadoop/2013/03/24/manual-install-Cloudera-Hadoop-CDH/
[http_archive.cloudera.com_cdh4_cdh_4]: http://archive.cloudera.com/cdh4/cdh/4/