hadoop2.6完全分布式安装

忘是亡心i 2022-06-14 09:57 154阅读 0赞

192.168.0.110 master

192.168.0.111 slave1

192.168.0.112 slave2

1.配置jdk

另一博客里，三台都要配。

2.添加用户hadoop

groupadd hadoop useradd -g hadoop hadoop passwd hadoop vi /etc/sudoers 配置hadoop ALL=(ALL) NOPASSWD: ALL

3.ssh无密码登录

ssh localhost cd ~ ssh-keygen -t rsa -P '' -f .ssh/id_rsa cat .ssh/id_rsa.pub >> .ssh/authorized_keys chmod 600 .ssh/authorized_keys

把master上的id\_rsa.pub传到slave

scp id_rsa.pub hadoop@slave1:/home/hadoop/.ssh

cat id_rsa.pub >> ~/.ssh/authorized_keys

slave中.ssh的权限为700，authorized\_keys的权限为744.

4.配置hadoop集群

修改slaves文件

slave1 slave2

core-site.xml

<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://master:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/hadoop/tmp</value> </property> </configuration>

hdfs-site.xml

<configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>master:50090</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/usr/hadoop/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/usr/hadoop/dfs/data</value> </property> </configuration>

mapred-site.xml     没有的话把mapred-site.xml.template改为mapred-site.xml

<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>master:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>master:19888</value> </property> </configuration>

yarn-site.xml

<configuration>  <property> <name>yarn.resourcemanager.hostname</name> <value>master</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>

压缩hadoop文件夹传到slave1和slave2中

5.启动

hdfs namenode -format

在master中启动

start-dfs.sh start-yarn.sh mr-jobhistory-daemon.sh start historyserver

用jps查看

[hadoop@master sbin]$ jps
    6208 JobHistoryServer
    5617 NameNode
    6242 Jps
    5930 ResourceManager
    5791 SecondaryNameNode

[hadoop@slave1 hadoop]$ jps
    3616 DataNode
    3715 NodeManager
    3811 Jps

[hadoop@slave2 hadoop]$ jps
    5216 DataNode
    5315 NodeManager
    5412 Jps

用hdfs dfsadmin -report查看

[hadoop@master sbin]$ hdfs dfsadmin -report
    Configured Capacity: 37492883456 (34.92 GB)
    Present Capacity: 32955469824 (30.69 GB)
    DFS Remaining: 32955461632 (30.69 GB)
    DFS Used: 8192 (8 KB)
    DFS Used%: 0.00%
    Under replicated blocks: 0
    Blocks with corrupt replicas: 0
    Missing blocks: 0
    
    -------------------------------------------------
    Live datanodes (2):
    
    Name: 192.168.0.111:50010 (slave1)
    Hostname: slave1
    Decommission Status : Normal
    Configured Capacity: 18746441728 (17.46 GB)
    DFS Used: 4096 (4 KB)
    Non DFS Used: 2447355904 (2.28 GB)
    DFS Remaining: 16299081728 (15.18 GB)
    DFS Used%: 0.00%
    DFS Remaining%: 86.94%
    Configured Cache Capacity: 0 (0 B)
    Cache Used: 0 (0 B)
    Cache Remaining: 0 (0 B)
    Cache Used%: 100.00%
    Cache Remaining%: 0.00%
    Xceivers: 1
    Last contact: Fri Jun 16 21:58:20 CST 2017
    
    
    Name: 192.168.0.112:50010 (slave2)
    Hostname: slave2
    Decommission Status : Normal
    Configured Capacity: 18746441728 (17.46 GB)
    DFS Used: 4096 (4 KB)
    Non DFS Used: 2090057728 (1.95 GB)
    DFS Remaining: 16656379904 (15.51 GB)
    DFS Used%: 0.00%
    DFS Remaining%: 88.85%
    Configured Cache Capacity: 0 (0 B)
    Cache Used: 0 (0 B)
    Cache Remaining: 0 (0 B)
    Cache Used%: 100.00%
    Cache Remaining%: 0.00%
    Xceivers: 1
    Last contact: Fri Jun 16 21:58:22 CST 2017

这样完全分布式就搭好了