Hadoop完全分布式环境

谁借莪１个温暖的怀抱￠ 2022-05-15 06:55 251阅读 0赞

# **Hadoop完全分布式环境** #

### **前提条件** ###

*  三台虚拟机centos01、centos02、centos03【可以先将一台虚拟机配置起来，然后克隆，克隆遇到的网卡问题参考[https://blog.csdn.net/eieiei438/article/details/80644261][https_blog.csdn.net_eieiei438_article_details_80644261]】

### **分布式结构说明** ###

<table style="margin-left:0px;"> 
 <thead style="margin-left:0px;"> 
  <tr style="margin-left:0px;"> 
   <th style="margin-left:0px;"><span style="color:#333333;">主机名称</span></th> 
   <th style="margin-left:0px;"><span style="color:#333333;">角色</span></th> 
   <th style="margin-left:0px;"><span style="color:#333333;">备注</span></th> 
  </tr> 
 </thead> 
 <tbody style="margin-left:0px;"> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">centos01</span></td> 
   <td style="margin-left:0px;"><br><span style="color:#333333;"><span style="color:#ff0000;">NameNode</span><br><br> DataNode<br><br><span style="color:#ff0000;">ResourceManager</span><br><br> NodeManager</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">主节点</span></td> 
  </tr> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">centos02</span></td> 
   <td style="margin-left:0px;"><br><span style="color:#333333;">DataNode<br><br> NodeManager</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">从节点</span></td> 
  </tr> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">centos03</span></td> 
   <td style="margin-left:0px;"><br><span style="color:#333333;">DataNode<br><br> NodeManager</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">从节点</span></td> 
  </tr> 
 </tbody> 
</table>

### **环境说明** ###

<table style="margin-left:0px;"> 
 <thead style="margin-left:0px;"> 
  <tr style="margin-left:0px;"> 
   <th style="margin-left:0px;"><span style="color:#333333;">名称</span></th> 
   <th style="margin-left:0px;"><span style="color:#333333;">版本</span></th> 
   <th style="margin-left:0px;"><span style="color:#333333;">备注</span></th> 
  </tr> 
 </thead> 
 <tbody style="margin-left:0px;"> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">宿主系统</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">Win7【64位】</span></td> 
   <td style="margin-left:0px;">&nbsp;</td> 
  </tr> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">VMware</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">12</span></td> 
   <td style="margin-left:0px;">&nbsp;</td> 
  </tr> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">虚拟机镜像</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">CentOS-6.5-x86_64-minimal.iso</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">下载地址（不同版本）：<a href="http://vault.centos.org/" style="margin-left:0px;" rel="nofollow">http://vault.centos.org/</a></span></td> 
  </tr> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">jdk</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">jdk-8u65-linux-x64.tar.gz</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">linux版</span></td> 
  </tr> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">hadoop</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">hadoop-2.6.0-cdh5.7.0.tar.gz</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">linux版</span></td> 
  </tr> 
 </tbody> 
</table>

### **软件安装路径** ###

<table style="margin-left:0px;"> 
 <thead style="margin-left:0px;"> 
  <tr style="margin-left:0px;"> 
   <th style="margin-left:0px;"><span style="color:#333333;">软件名称</span></th> 
   <th style="margin-left:0px;"><span style="color:#333333;">路径</span></th> 
  </tr> 
 </thead> 
 <tbody style="margin-left:0px;"> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">jdk</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">/software/jdk/jdk8</span></td> 
  </tr> 
  <tr style="margin-left:0px;"> 
   <td style="margin-left:0px;"><span style="color:#333333;">hadoop</span></td> 
   <td style="margin-left:0px;"><span style="color:#333333;">/software/hadoop/hadoop</span></td> 
  </tr> 
 </tbody> 
</table>

### **主机名称** ###

*  centos01【主】
 *  centos02【从】
 *  centos03【从】

### **主机网卡连接方式** ###

*  NAT 静态固定IP
 *  IP：
    
     *  192.168.66.66【主】
     *  192.168.66.77【从】
     *  192.168.66.88【从】

### **免密登录** ###

*  已经免密
 *  免密步骤
    
    1.  ssh-keygen
    2.  cat ~/.ssh/ad\_rsa.pub >> authorized\_keys

### **环境关键配置文件** ###

*  /etc/hosts
    
     
    
        192.168.66.66 centos01 192.168.66.77 centos02 192.168.66.88 centos03
 *  /etc/profile
    
     
    
        #jdk JAVA_HOME=/software/jdk/jdk8 PATH=$JAVA_HOME/bin:$PATH CLASSPATH=.:$JAVA_HOME/lb/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME PATH CLASSPATH #hadoop export HADOOP_HOME=/software/hadoop/hadoop export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
 *  /software/hadoop/hadoop/etc/hadoop/hadoop-env.sh【将export JAVA\_HOME=$\{JAVA\_HOME在的地方进行修改\}】
    
     
    
        。。。 export JAVA_HOME=/software/jdk/jdk8 。。。
 *  /software/hadoop/hadoop/etc/hadoop/core-site.xml
    
     
    
        <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration>  <property> <name>fs.defaultFS</name> <value>hdfs://centos01:8020</value> </property>  <property> <name>hadoop.tmp.dir</name> <value>/software/hadoop/tmp</value> </property> </configuration>
 *  /software/hadoop/hadoop/etc/hadoop/hdfs-site.xml 注意副本的数量（默认为3）
    
     
    
        <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration>  <property> <name>dfs.replication</name> <value>3</value> </property>  <property> <name>dfs.namenode.name.dir</name> <value>file:/software/hadoop/tmp/dfs/name</value> </property>  <property> <name>dfs.namenode.data.dir</name> <value>file:/software/hadoop/tmp/dfs/data</value> </property> </configuration>
 *  /software/hadoop/hadoop/etc/hadoop/slaves 这里添加节点
    
     
    
        centos01 centos02 centos03
 *  /software/hadoop/hadoop/etc/hadoop/mapred-site.xml
    
     
    
        <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration>  <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>  <property> <name>mapreduce.jobhistory.address</name> <value>centos01:10020</value> <description>MapReduce JobHistory Server IPC host:port</description> </property>  <property> <name>mapreduce.jobhistory.webapp.address</name> <value>centos01:19888</value> <description>MapReduce JobHistory Server Web UI host:port</description> </property>  <property> <name>mapreduce.jobhistory.done-dir</name> <value>/history/done</value> </property>  <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/history/done_intermediate</value> </property> </configuration>
 *  /software/hadoop/hadoop/etc/hadoop/yarn-site.xml 这里添加resourcemanager
    
     
    
        <?xml version="1.0"?> <configuration>  <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property>  <property> <name>yarn.resourcemanager.hostname</name> <value>centos01</value> </property>  <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> </configuration>

### **启动** ###

*  命令
    
     *  sh /software/hadoop/hadoop/sbin/start-all.sh
     *  jps
 *  执行结果
    
     *  ![7PyhiLd.png][]
     *  ![d6Ms0lC.png][]
     *  ![UwHyFH0.png][]
     *  ![3acFhtT.png][]

[https_blog.csdn.net_eieiei438_article_details_80644261]: https://blog.csdn.net/eieiei438/article/details/80644261
[7PyhiLd.png]: https://i.imgur.com/7PyhiLd.png
[d6Ms0lC.png]: https://i.imgur.com/d6Ms0lC.png
[UwHyFH0.png]: https://i.imgur.com/UwHyFH0.png
[3acFhtT.png]: https://i.imgur.com/3acFhtT.png