docker搭建hadoop集群

朱雀 2022-10-05 09:53 168阅读 0赞

**Hadoop简介**

Hadoop是一个由Apache基金会所开发的[分布式系统][Link 1]基础架构。用户可以在不了解分布式底层细节的情况下，开发分布式程序。充分利用集群的威力进行高速运算和存储。Hadoop实现了一个[分布式文件系统][Link 2]（ Distributed File System），其中一个组件是HDFS（Hadoop Distributed File System）。HDFS有高[容错性][Link 3]的特点，并且设计用来部署在低廉的（low-cost）硬件上；而且它提供高吞吐量（high throughput）来访问[应用程序][Link 4]的数据，适合那些有着超大数据集（large data set）的应用程序。HDFS放宽了（relax）[POSIX][]的要求，可以以流的形式访问（streaming access）文件系统中的数据。Hadoop的框架最核心的设计就是：[HDFS][]和[MapReduce][]。HDFS为海量的数据提供了存储，而MapReduce则为海量的数据提供了计算  。

**docker搭建hadoop集群**

**1.拉取hadoop镜像**

**docker pull docker.io/sequenceiq/hadoop-docker**

**2.查看镜像**

**docker images**

![20210608144328291.png][]

**3.创建hadoop容器**

**创建master节点  
`docker run --name hadoop1 -d `\-p 50070:50070 -p 9000:9000 -p 8088:8088 -p 8040:8040 -p 8042:8042  -p 49707:49707  -p 50010:50010  -p 50075:50075  -p 50090:50090`-h master docker.io/sequenceiq/hadoop-docker`**

**创建slave1节点**

**`docker run --name hadoop2 -d -h`**slave1**`docker.io/sequenceiq/hadoop-docker`**

**创建slave2节点**

**`docker run --name hadoop3 -d -h`**slave2**` docker.io/sequenceiq/hadoop-docker`**

**查看容器**

**docker ps -a**

![20210608144616903.png][]

**4.配置ssh生成秘钥，所有的节点都要配置**

**首先配置master节点，进入master容器**

**docker exec -it hadoop1 bash**

**启动ssh  
`/etc/init.d/sshd start`**

**生成秘钥，生成的秘钥在容器/root/.ssh目录下  
`ssh-keygen -t rsa`**

**进入.ssh目录**

**cd  /root/.ssh**

**ls**

![20210608145258507.png][]

** 将公钥导入authorized\_keys文件**

**cat id\_rsa.pub > authorized\_keys**

**cat authorized\_keys**

![20210608145400432.png][]

**退出master容器**

**exit**

**同样操作配置slave1节点**

**docker exec -it hadoop2 bash  
/etc/init.d/sshd start  
ssh-keygen -t rsa  
cd /root/.ssh  
cat id\_rsa.pub > authorized\_keys  
exit**

**配置slave2节点**

**docker exec -it hadoop3 bash  
/etc/init.d/sshd start  
ssh-keygen -t rsa  
cd /root/.ssh  
cat id\_rsa.pub > authorized\_keys  
exit**

**将容器中这3个节点的公钥也都复制到authorized\_keys，也就是说每个authorized\_keys 文件中存储的公钥都是3个而且是一样的。**

**创建文件夹**

**mkdir -p /home/hadoop**

**将容器中的公钥复制到centos本地**

**docker cp hadoop1:/root/.ssh/authorized\_keys /home/hadoop/authorized\_keys\_master**

**docker cp hadoop2:/root/.ssh/authorized\_keys /home/hadoop/slave1**

**docker cp hadoop3:/root/.ssh/authorized\_keys /home/hadoop/slave2**

**将这三个公钥合并成一个文件**

**cd /home/hadoop**

**cat authorized\_keys\_master slave1 slave2 > authorized\_keys**

**cat authorized\_keys**

**![20210608150200546.png][]**

**将centos本地的文件复制到容器**

**docker cp /home/hadoop/authorized\_keys hadoop1:/root/.ssh/authorized\_keys  
docker cp /home/hadoop/authorized\_keys hadoop2:/root/.ssh/authorized\_keys  
docker cp /home/hadoop/authorized\_keys hadoop3:/root/.ssh/authorized\_keys**

**5.分别为每个节点配置ip地址**

**分别进入master、salve1、slave2容器查看ip**

**docker exec -it hadoop1 bash**

**ifconfig**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xldHRlcnNz_size_16_color_FFFFFF_t_70][]

** 为每个容器设置地址，master、slave1、slave2都要配置**

**`vi /etc/hosts` **

![20210608150517395.png][]

**在容器内ssh master 测试一下 ，测试成功**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xldHRlcnNz_size_16_color_FFFFFF_t_70 1][]

**6.配置hadoop(配置文件的目录一般都在/usr/local/hadoop-2.7.0/etc/hadoop下面) **

**docker exec -it hadoop1 bash**

**cd /usr/local/hadoop-2.7.0/etc/hadoop/**

**ls**

![20210608151018153.png][]

vi core-site.xml ,加入配置

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://Master:9000</value>
      </property>
        <property>
          <name>hadoop.tmp.dir</name>
          <value>/hadoop/tmp</value>
        </property>

配置yarn-site.xml，加入配置

vi yarn-site.xml

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xldHRlcnNz_size_16_color_FFFFFF_t_70 2][]

<property>
    		<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    		<value>org.apache.hadoop.mapred.ShuffleHandler</value>
    	</property>
    	<property>
    		<name>yarn.resourcemanager.address</name>
    		<value>bigdata:8032</value>
    	</property>
    	<property>
    		<name>yarn.resourcemanager.scheduler.address</name>
    		<value>bigdata:8030</value>
    	</property>
    	<property>
    		<name>yarn.resourcemanager.resource-tracker.address</name>
    		<value>bigdata:8031</value>
    	</property>
    	<property>
    		<name>yarn.resourcemanager.admin.address</name>
    		<value>bigdata:8033</value>
    	</property>
    	<property>
    		<name>yarn.resourcemanager.webapp.address</name>
    		<value>bigdata:8089</value>
    	</property>

**将这些参数发送到其它节点 **

** scp /usr/local/hadoop-2.7.0/etc/hadoop/yarn-site.xml slave1:/usr/local/hadoop-2.7.0/etc/hadoop/**

**scp /usr/local/hadoop-2.7.0/etc/hadoop/yarn-site.xml slave2:/usr/local/hadoop-2.7.0/etc/hadoop/**

**7.运行hadoop**

**在master上格式化namenode**

**docker exec -it hadoop1 bash**

**`cd /usr/local/hadoop-2.7.0/bin/`  
`./hadoop namenode -format` **

**在master上启动集群**

**`cd /usr/local/hadoop-2.7.0/sbin/`  
`./start-all.sh`  
如果报错告诉端口被占用  
那么就先停止,然后再启动  
`./stop-all.sh`  
`./start-all.sh`**

**jps 查看进程，查看到说明已启动**

![20210608151553433.png][]

[Link 1]: https://baike.baidu.com/item/%E5%88%86%E5%B8%83%E5%BC%8F%E7%B3%BB%E7%BB%9F/4905336
[Link 2]: https://baike.baidu.com/item/%E5%88%86%E5%B8%83%E5%BC%8F%E6%96%87%E4%BB%B6%E7%B3%BB%E7%BB%9F/1250388
[Link 3]: https://baike.baidu.com/item/%E5%AE%B9%E9%94%99%E6%80%A7/9131391
[Link 4]: https://baike.baidu.com/item/%E5%BA%94%E7%94%A8%E7%A8%8B%E5%BA%8F/5985445
[POSIX]: https://baike.baidu.com/item/POSIX/3792413
[HDFS]: https://baike.baidu.com/item/HDFS/4836121
[MapReduce]: https://baike.baidu.com/item/MapReduce/133425
[20210608144328291.png]: /images/20221005/252c9404dfad44a38e226d8a1dadbe63.png
[20210608144616903.png]: /images/20221005/39340ff0e11842748e494e7953ac42eb.png
[20210608145258507.png]: /images/20221005/4c701aa229d24c2f931ff4a5b693eeee.png
[20210608145400432.png]: /images/20221005/b8f61519703240e088884b54f0de16d9.png
[20210608150200546.png]: /images/20221005/241a25364a484c4daa933e51a89e1d46.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xldHRlcnNz_size_16_color_FFFFFF_t_70]: /images/20221005/304e19bc280a4f9686452d2e5d34e732.png
[20210608150517395.png]: /images/20221005/0076a2ca23f44f029ecbe98cb78da598.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xldHRlcnNz_size_16_color_FFFFFF_t_70 1]: /images/20221005/121dfc723f6c4110a080d87c68729175.png
[20210608151018153.png]: /images/20221005/3a0bbe23acfb434cb13e990a5aabb677.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xldHRlcnNz_size_16_color_FFFFFF_t_70 2]: /images/20221005/825ee0a58a2b4e0f8bc8a36628bfcf59.png
[20210608151553433.png]: /images/20221005/4ec991ff0dd84e26a65318d12b8f0e97.png