Yarn集群搭建

￡神魔★判官ぃ 2023-07-21 06:14 83阅读 0赞

## 原理 ##

搭建Yarn集群，首先要知道它的集群的原理，现在我有一台破机器，那么如何把它加入到我现在的yarn单机集群里面呢？

我现在的部署模式是单机部署

Yarn下面的执行逻辑如下：

![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-rsCVDy3y-1585835287363)(docs/images/image-20200401211916461.png)\]][img-rsCVDy3y-1585835287363_docs_images_image-20200401211916461.png]  
我只需要把我的破机器连接到原来的resourceManager就好了。

## 配置 ##

假设现在node1已经部署好一套了，破机器node2需要加进来

1.  设置免密登录

生成rsa
    ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    
    copy 公钥 到其它机器 免密登陆
     ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver1

1.  配置slaves
    
    `etc/hadoop/slaves`

node1
    node2

1.  配置yarn （两台机器的要保持一致）
    
    `etc/hadoop/yarn-site.xml`

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>node1</value>
    </property>

1.  启动

`sbin/start-all.sh`

1.  查看

Yarn

![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-FjkwCf4e-1585835287365)(docs/images/image-20200401221452965.png)\]][img-FjkwCf4e-1585835287365_docs_images_image-20200401221452965.png]

## QA ##

### 虚拟内存超出限制 ###

Application application_1585750131577_0002 failed 1 times due to AM Container for appattempt_1585750131577_0002_000001 exited with exitCode: -103
    For more detailed output, check application tracking page:http://flinkhadoop:8088/cluster/app/application_1585750131577_0002Then, click on links to logs of each attempt.
    Diagnostics: Container [pid=16530,containerID=container_1585750131577_0002_01_000001] is running beyond virtual memory limits. Current usage: 318.6 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.

https://blog.csdn.net/wq123123423432/article/details/83043091

关闭虚拟内存

<property>
      <name>yarn.nodemanager.vmem-check-enabled</name>
      <value>false</value>
      <description>Whether virtual memory limits will be enforced for containers.</description>
    </property>

如果任务所占用的内存太过离谱，更多考虑的应该是程序是否有内存泄漏，是否存在数据倾斜等，优先程序解决此类问题

[img-rsCVDy3y-1585835287363_docs_images_image-20200401211916461.png]: /images/20230528/f09fef240f4f43d183a4fab8d2644e3e.png
[img-FjkwCf4e-1585835287365_docs_images_image-20200401221452965.png]: /images/20230528/71615c134a2c439bacaf36dc44c378eb.png