hadoop平台搭建-多节点详细教程

向右看齐 2022-10-02 10:42 182阅读 0赞

Hadoop平台搭建说明

# 1. Hadoop节点规划 #

本次安装规划使用三个节点，每个节点都使用centos系统。

三个节点的hostname分别规划为：centoshadoop1、centoshadoop2、centoshadoop3（此处为本教程参数，可根据实际环境情况修改）

三个节点的ip地址分别规划为：192.168.65.57、192.168.65.58、192.168.65.59（此处为本教程参数，根据实际环境情况修改）

# 2. 平台搭建使用的软件 #

下载如下软件

操作系统安装包：Centos6.3\_x64

Jdk安装包：jdk-6u37-linux-x64.bin

Hadoop安装包：hadoop-1.1.2.tar.gz

# 3. 安装centos操作系统 #

安装三个节点的操作系统，安装过程省略。

# 4. 配置centoshadoop1节点 #

## 4.1. 修改节点hostname ##

\[root@localhost ~\]\# vi /etc/sysconfig/network

HOSTNAME=centoshadoop1

\[root@localhost ~\]\# vi /etc/hosts

……

192.168.65.57 centoshadoop1

192.168.65.58 centoshadoop2

192.168.65.59 centoshadoop3

\[root@localhost ~\]\#reboot

## 4.2. 关闭iptables防火墙 ##

\[root@ centoshadoop1~\]\#service iptables stop

注意每次操作系统重启后都要操作

## 4.3. 建立无ssh密码登陆 ##

生成签名文件

\[root@ centoshadoop1~\]\#cd /root

\[root@ centoshadoop1~\]\#ssh-keygen -t dsa -P '' -f ~/.ssh/id\_dsa

\[root@ centoshadoop1~\]\#cat ~/.ssh/id\_dsa.pub >> ~/.ssh/authorized\_keys

\[root@ centoshadoop1~\]\#

测试本地SSH无密码登录

\[root@ centoshadoop1~\]\#sshcentoshadoop1

## 4.4. 安装jdk ##

上传jdk-6u37-linux-x64.bin到/root目录下

\[root@ centoshadoop1~\]\#chmod 777 jdk-6u37-linux-x64.bin

\[root@ centoshadoop1~\]\#./jdk-6u37-linux-x64.bin

\[root@ centoshadoop1~\]\#ll

查看生成jdk-6u37-linux-x64目录

## 4.5. 安装hadoop软件 ##

上传hadoop-1.1.2.tar.gz到/root目录下

\[root@ centoshadoop1~\]\#tar -zvxf hadoop-1.1.2.tar.gz

\[root@ centoshadoop1~\]\#ll

查看生成hadoop-1.1.2目录

\[root@ centoshadoop1~\]\#vi /conf/core-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>fs.default.name</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/root/hadoop-1.1.2/tmp</value>

</property>

</configuration>

\[root@ centoshadoop1~\]\#vi hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>dfs.replication</name>

</property>

</configuration>

\[root@ centoshadoop1~\]\#vi mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>mapred.job.tracker</name>

</property>

</configuration>

## 4.6. 配置hadoop数据节点 ##

\[root@ centoshadoop1~\]\#vi /root/hadoop-1.1.2/conf/masters

192.168.65.57 \#secondaryNameNode

\[root@ centoshadoop1~\]\#vi /root/hadoop-1.1.2/conf/slaves

192.168.65.58 \#datanode

192.168.65.59 \#datanode

## 4.7. 添加环境变量 ##

\[root@ centoshadoop1~\]\#cd /root

\[root@ centoshadoop1~\]\#vi /etc/profile

export JAVA\_HOME=/root/jdk1.6.0\_37

export JRE\_HOME=/root/jdk1.6.0\_37/jre

export HADOOP\_HOME=/root/hadoop-1.1.2

export HADOOP\_HOME\_WARN\_SUPPRESS=1

export CLASSPATH=.:$JAVA\_HOME/lib:$JRE\_HOME/lib:$HADOOP\_HOME/lib:$CLASSPATH

export PATH=$JAVA\_HOME/bin:$JRE\_HOME/bin:$HADOOP\_HOME/bin:$PATH

\[root@ centoshadoop1~\]\#vi ~/.bashrc

export JAVA\_HOME=/root/jdk1.6.0\_37

export JRE\_HOME=/root/jdk1.6.0\_37/jre

export HADOOP\_HOME=/root/hadoop-1.1.2

export HADOOP\_DEV\_HOME=/root/hadoop-1.1.2

export HADOOP\_COMMON\_HOME=/root/hadoop-1.1.2

export HADOOP\_HDFS\_HOME=/root/hadoop-1.1.2

export HADOOP\_CONF\_DIR=/root/hadoop-1.1.2/conf

\[root@ centoshadoop1~\]\#vi /root/hadoop-1.1.2/conf/hadoop-env.sh

export JAVA\_HOME=/root/jdk1.6.0\_37

export HADOOP\_HOME\_WARN\_SUPPRESS=1

export HADOOP\_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib

# 5. 配置centoshadoop2节点 #

## 5.1. 修改节点hostname ##

\[root@localhost ~\]\# vi /etc/sysconfig/network

HOSTNAME=centoshadoop2

\[root@localhost ~\]\# vi /etc/hosts

……

192.168.65.57 centoshadoop1

192.168.65.58 centoshadoop2

192.168.65.59 centoshadoop3

\[root@localhost ~\]\#reboot

## 5.2. 关闭iptables防火墙 ##

\[root@ centoshadoop1~\]\#service iptables stop

注意每次操作系统重启后都要操作

## 5.3. 建立无ssh密码登陆 ##

将centoshadoop1节点上/root/.ssh目录下的authorized\_keys、known\_hosts 、id\_rsa.pub、id\_rsa四个文件拷贝到本机的/root/.ssh目录下

## 5.4. 安装jdk ##

上传jdk-6u37-linux-x64.bin到/root目录下

\[root@ centoshadoop1~\]\#chmod 777 jdk-6u37-linux-x64.bin

\[root@ centoshadoop1~\]\#./jdk-6u37-linux-x64.bin

\[root@ centoshadoop1~\]\#ll

查看生成jdk-6u37-linux-x64目录

## 5.5. 安装hadoop软件 ##

上传hadoop-1.1.2.tar.gz到/root目录下

\[root@ centoshadoop1~\]\#tar -zvxf hadoop-1.1.2.tar.gz

\[root@ centoshadoop1~\]\#ll

查看生成hadoop-1.1.2目录

\[root@ centoshadoop1~\]\#vi /conf/core-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>fs.default.name</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/root/hadoop-1.1.2/tmp</value>

</property>

</configuration>

\[root@ centoshadoop1~\]\#vi hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>dfs.replication</name>

</property>

</configuration>

\[root@ centoshadoop1~\]\#vi mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<name>mapred.job.tracker</name>

</property>

</configuration>

## 5.6. 添加环境变量 ##

\[root@ centoshadoop1~\]\#cd /root

\[root@ centoshadoop1~\]\#vi /etc/profile

export JAVA\_HOME=/root/jdk1.6.0\_37

export JRE\_HOME=/root/jdk1.6.0\_37/jre

export HADOOP\_HOME=/root/hadoop-1.1.2

export HADOOP\_HOME\_WARN\_SUPPRESS=1

export CLASSPATH=.:$JAVA\_HOME/lib:$JRE\_HOME/lib:$HADOOP\_HOME/lib:$CLASSPATH

export PATH=$JAVA\_HOME/bin:$JRE\_HOME/bin:$HADOOP\_HOME/bin:$PATH

\[root@ centoshadoop1~\]\#vi ~/.bashrc

export JAVA\_HOME=/root/jdk1.6.0\_37

export JRE\_HOME=/root/jdk1.6.0\_37/jre

export HADOOP\_HOME=/root/hadoop-1.1.2

export HADOOP\_DEV\_HOME=/root/hadoop-1.1.2

export HADOOP\_COMMON\_HOME=/root/hadoop-1.1.2

export HADOOP\_HDFS\_HOME=/root/hadoop-1.1.2

export HADOOP\_CONF\_DIR=/root/hadoop-1.1.2/conf

\[root@ centoshadoop1~\]\#vi /root/hadoop-1.1.2/conf/hadoop-env.sh

export JAVA\_HOME=/root/jdk1.6.0\_37

export HADOOP\_HOME\_WARN\_SUPPRESS=1

export HADOOP\_CLASSPATH=/root/hadoop-1.1.2:/root/hadoop-1.1.2/lib

# 6. 配置centoshadoop3节点 #

## 6.1. 修改节点hostname ##

\[root@localhost ~\]\# vi /etc/sysconfig/network

HOSTNAME=centoshadoop3

\[root@localhost ~\]\# vi /etc/hosts

……

192.168.65.57 centoshadoop1

192.168.65.58 centoshadoop2

192.168.65.59 centoshadoop3

\[root@localhost ~\]\#reboot

关闭iptables防火墙、建立无ssh密码登陆、安装jdk、安装hadoop软件、添加环境变量等操作跟“配置centoshadoop2节点”完全相同

# 7. 启动运行hadoop平台 #

## 7.1. 首先执行格式化 ##

\[root@ centoshadoop1~\]\#hadoop namenode –format

注意：core-site.xml、hdfs-site.xml、mapred-site.xml等配置文件变化后都需要执行格式化

## 7.2. 启动hadoop ##

\[root@ centoshadoop1~\]\#start-all.sh

## 7.3. 停止hadoop ##

\[root@ centoshadoop1~\]\#stop-all.sh

## 7.4. 查看hadoop运行状态 ##

查看hadoop启动的进程

\[root@ centoshadoop1~\]\#jps

查看hadoop集群状态

\[root@ centoshadoop1~\]\#hadoop dfsadmin -report

查看hadoop中的目录情况

\[root@ centoshadoop1~\]\#hadoop fs –ls /

转载于:https://blog.51cto.com/hdfantasy/1403999

hadoop平台搭建-多节点详细教程

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关 docker搭建Hadoop 3节点及N节点集群完整教程

相关 docker 搭建 hadoop 集群平台

相关基于Docker快速搭建多节点Hadoop集群

相关 Hadoop集群搭建教程

相关 hadoop平台搭建-多节点详细教程

相关 Hadoop/Spark平台搭建

相关详细解读如何搭建hadoop平台

相关 Hadoop单节点示例搭建

相关十分详细的Hadoop环境搭建

相关 Spark、Hadoop大数据平台搭建