Hadoop分布式集群安装（非虚拟机）

た 入场券 2023-02-18 01:58 93阅读 0赞
  1. 机器

准备三台服务器，网络要通。配置大致如下

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0hlbGxvV29ybGRZYW5nU29uZw_size_16_color_FFFFFF_t_70

  1. 安装jdk

比较简单，略过了

  1. 新增Linux用户hadoop

    useradd hadoop

    useradd hadoop

输入密码即可

授予root权限

  1. vim /etc/sudoers

增加：

2020061514525481.png

3个节点都执行一遍如上操作

  1. 免密登入

节点名字：master、slave1、slave2

4.1 3个节点切换到hadoop用户

  1. su hadoop

4.2 分别执行，一直回车

  1. ssh-keygen -t rsa

并将公钥复制为authorized_keys，3个节点都执行

  1. cp ~/.ssh/id_rsa  ~/.ssh/authorized_keys

4.3 slave1和slave2中通过scp拷贝 ~/.ssh/authorized_keys到master

如：

  1. scp ~/.ssh/authorized_keys zhangc@master:~/.ssh/authorized_keys_slave1

4.4 在master节点将上边的 authorized_keys_slave1和authorized_keys_slave2 追加到authorized_keys

如：

  1. cd ~/.ssh/
  2. cat authorized_keys_slave1 >> authorized_keys
  3. cat authorized_keys_slave2 >> authorized_keys

4.5 将master中的~/.ssh/authorized_keys分发到slave1、slave2节点

4.6 然后在slave节点执行

  1. scp ~/.ssh/authorized_keys zhangc@slave1:~/.ssh/authorized_keys
  3. scp ~/.ssh/authorized_keys zhangc@slave2:~/.ssh/authorized_keys

参考：https://www.cnblogs.com/20kuaiqian/p/11202330.html

  1. 下载Hadoop安装包

https://hadoop.apache.org/releases.html

or

https://www.apache.org/dyn/closer.cgi/hadoop/common

解压

  1. tar -zxvf hadoop-3.2.1.tar.gz -C /usr/local/bigdata/hadoop

环境变量

  1. vim ~/.bashrc
  3. export PATH=$PATH:/usr/local/bigdata/hadoop/hadoop-3.2.1/bin:/usr/local/bigdata/hadoop/hadoop-3.2.1/sbin
  5. source ~/.bashrc

5.1 配置hadoop-env.sh，内容如下

  1. export JAVA_HOME=/usr/local/jdk
  3. export HDFS_NAMENODE_USER="hadoop"
  4. export HDFS_DATANODE_USER="hadoop"
  5. export HDFS_SECONDARYNAMENODE_USER="hadoop"
  6. export YARN_RESOURCEMANAGER_USER="hadoop"
  7. export YARN_NODEMANAGER_USER="hadoop"

5.2 配置core-site.xml

  1. <configuration>
  2.         <property>
  3.                 <name>fs.defaultFS</name>
  4.                 <value>hdfs://master:9000</value>
  5.         </property>
  6.         <property>
  7.                 <name>hadoop.tmp.dir</name>
  8.                 <value>file:/usr/local/bigdata/hadoop/tmp</value>
  9.                 <description>Abase for other temporary directories.</description>
  10.         </property>
  11. </configuration>

其中master可以自行改成指定的ip地址,hadoop.tmp.dir记得新建

5.3 配置hdfs-site.xml

  1. <configuration>
  2.         <property>
  3.                 <name>dfs.namenode.secondary.http-address</name>
  4.                 <value>master:50090</value>
  5.         </property>
  6.         <property>
  7.                 <name>dfs.replication</name>
  8.                 <value>1</value>
  9.         </property>
  10.         <property>
  11.                 <name>dfs.namenode.name.dir</name>
  12.                 <value>file:/usr/local/bigdata/hadoop/hdfs/name</value>
  13.         </property>
  14.         <property>
  15.                 <name>dfs.datanode.data.dir</name>
  16.                 <value>file:/usr/local/bigdata/hadoop/hdfs/data</value>
  17.         </property>
  18. </configuration>

其中，file路径记得新建好

5.4 配置mapred-site.xml

  1. <configuration>
  2.         <property>
  3.                 <name>mapreduce.framework.name</name>
  4.                 <value>yarn</value>
  5.         </property>
  6.         <property>
  7.                 <name>mapreduce.jobhistory.address</name>
  8.                 <value>master:10020</value>
  9.         </property>
  10.         <property>
  11.                 <name>mapreduce.jobhistory.webapp.address</name>
  12.                 <value>master:19888</value>
  13.         </property>
  14. </configuration>

5.5 配置yarn-site.xml

  1. <configuration>
  2.         <property>
  3.                 <name>yarn.resourcemanager.hostname</name>
  4.                 <value>master</value>
  5.         </property>
  6.         <property>
  7.                 <name>yarn.nodemanager.aux-services</name>
  8.                 <value>mapreduce_shuffle</value>
  9.         </property>
  10. </configuration>

注：master根据具体情况配置，配置ip也可以

5.6 配置workers

  1. slave1
  2. slave2

注：非虚拟机用两个真实ip

  1. 复制hadoop文件夹到其他节点

    cd ~
    sudo tar -zxvf hadoop.master.tar.gz -C /usr/local/

进入 maser的/user/local执行以下命令压缩文件夹

  1. tar -zcf ~/hadoop.master.tar.gz ./bigdata/

进入路径

  1. cd ~

20200617093827792.png

拷贝到slave1和slave2节点

  1. scp ./hadoop.master.tar.gz slave1:/home/hadoop/
  2. 和
  3. scp ./hadoop.master.tar.gz slave2:/home/hadoop/

20200617094033577.png

  1. 首次启动先格式化HDFS，一定要在master节点执行

    hdfs namenode -format

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0hlbGxvV29ybGRZYW5nU29uZw_size_16_color_FFFFFF_t_70 1

  1. 启动

启动前关闭防火墙

进入/usr/local/bigdata/hadoop/hadoop-3.2.1/sbin执行

  1. ./start-dfs.sh

master：

20200617135650576.png

slave

20200617135709894.png

  1. ./start-yarn.sh

master

20200617135912305.png

slave

20200617135939392.png

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0hlbGxvV29ybGRZYW5nU29uZw_size_16_color_FFFFFF_t_70 2

安装成功。。。

查看端口

  1. netstat -nltp

各个端口访问地址https://blog.csdn.net/qq_41851454/article/details/79938811

  1. 1、HDFS页面：50070
  3. 2、YARN的管理界面：8088
  5. 3、HistoryServer的管理界面：19888
  7. 4、Zookeeper的服务端口号：2181
  9. 5、Mysql的服务端口号：3306
  11. 6、Hive.server1=10000
  13. 7、Kafka的服务端口号：9092
  15. 8、azkaban界面：8443
  17. 9、Hbase界面：16010,60010
  19. 10、Spark的界面：8080
  21. 11、Spark的URL：7077
  1. 异常解决

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0hlbGxvV29ybGRZYW5nU29uZw_size_16_color_FFFFFF_t_70 3

  1. java.net.BindException: Port in use: master:8088
  2.     at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1218)
  3.     at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1240)
  4.     at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1299)
  5.     at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1154)
  6.     at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:439)
  7.     at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1231)
  8.     at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1340)
  9.     at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  10.     at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1535)
  11. Caused by: java.net.BindException: Cannot assign requested address
  12.     at sun.nio.ch.Net.bind0(Native Method)
  13.     at sun.nio.ch.Net.bind(Net.java:436)
  14.     at sun.nio.ch.Net.bind(Net.java:428)
  15.     at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
  16.     at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
  17.     at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:351)
  18.     at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:319)
  19.     at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1205)
  20.     at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1236)
  21.     ... 7 more
  22. 2020-06-17 13:28:17,806 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state STARTED
  23. org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
  24.     at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:443)
  25.     at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1231)
  26.     at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1340)
  27.     at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  28.     at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1535)
  29. Caused by: java.net.BindException: Port in use: master:8088
  30.     at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1218)
  31.     at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1240)
  32.     at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1299)
  33.     at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1154)
  34.     at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:439)
  35.     ... 4 more
  36. Caused by: java.net.BindException: Cannot assign requested address
  37.     at sun.nio.ch.Net.bind0(Native Method)
  38.     at sun.nio.ch.Net.bind(Net.java:436)
  39.     at sun.nio.ch.Net.bind(Net.java:428)
  40.     at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
  41.     at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
  42.     at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:351)
  43.     at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:319)
  44.     at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1205)
  45.     at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1236)
  46.     ... 7 more
  47. 2020-06-17 13:28:17,810 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to standby state
  48. 2020-06-17 13:28:17,810 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state
  49. 2020-06-17 13:28:17,811 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
  50. org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
  51.     at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:443)
  52.     at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1231)
  53.     at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1340)
  54.     at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  55.     at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1535)
  56. Caused by: java.net.BindException: Port in use: master:8088
  57.     at org.apache.hadoop.http.HttpServer2.constructBindException(HttpServer2.java:1218)
  58.     at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1240)
  59.     at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:1299)
  60.     at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1154)
  61.     at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:439)
  62.     ... 4 more
  63. Caused by: java.net.BindException: Cannot assign requested address
  64.     at sun.nio.ch.Net.bind0(Native Method)
  65.     at sun.nio.ch.Net.bind(Net.java:436)
  66.     at sun.nio.ch.Net.bind(Net.java:428)
  67.     at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
  68.     at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
  69.     at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:351)
  70.     at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:319)
  71.     at org.apache.hadoop.http.HttpServer2.bindListener(HttpServer2.java:1205)
  72.     at org.apache.hadoop.http.HttpServer2.bindForSinglePort(HttpServer2.java:1236)

这个异常多半是/etc/hosts下的配置有问题，我是把master这个域名直接指定到127.0.0.1就好了，其他的实事求是的看看。

20200617135422765.png

参考：

https://blog.csdn.net/dream_an/article/details/80258283

https://www.cnblogs.com/20kuaiqian/p/11202330.html

http://dblab.xmu.edu.cn/blog/install-hadoop-cluster/

发表评论

表情：
评论列表 （有 0 条评论，93人围观）

还没有评论，来说两句吧...

相关阅读

    相关 hadoop分布式集群安装

    搭建分布式集群的前提安装步骤需要每一台机器都要执行（jdk安装，ip配置，主机名设置，hosts文件配置，关闭防火墙，免密登录） 免密登录注意，至少确保主节点可以免密登录到

    秒速五厘米秒速五厘米/ 0 赞/ 169 阅读

    相关 hadoop完全分布式集群安装

    因为机器学习，接触到了数据挖掘；因为数据挖掘，接触到了大数据；因为大数据，接触到了hadoop。之前有过hadoop的简单了解，但都是基于别人提供的hadoop来学习和使用，虽

    清疚清疚/ 0 赞/ 399 阅读