解决yarn上的MapReduce作业失败问题container XXXX is running beyond physical memory limits

阳光穿透心脏的1/2处 2022-05-14 07:20 166阅读 0赞

【问题】  
今天突然发现有多个报表作业失败，涉及到查询有关报活的表。联想到最近几天有ETL同学的作业从老集群迁移到这个新集群上运行，推测可能是资源不够用导致的作业失败。  
错误日志摘要：

Container [pid=31300,containerID=container_e80_1535989582384_2041_01_000210] is running beyond physical memory limits.
    Current usage: 2.2 GB of 2 GB physical memory used; 4.0 GB of 4.2 GB virtual memory used. Killing container.

【日志解读】  
running beyond physical memory limits 意思是容器运行时超出了物理内存限制。

【问题分析】  
Cloudera的有关介绍：  
The setting mapreduce.map.memory.mb will set the physical memory size of the container running the mapper (mapreduce.reduce.memory.mb will do the same for the reducer container).  
Besure that you adjust the heap value as well. In newer version of YARN/MRv2 the setting mapreduce.job.heap.memory-mb.ratio can be used to have it auto-adjust. The default is .8, so 80% of whatever the container size is will be allocated as the heap. Otherwise, adjust manually using mapreduce.map.java.opts.max.heap and mapreduce.reduce.java.opts.max.heap settings.  
BTW, I believe that 1 GB is the default and it is quite low. I recommend reading the below link. It provides a good understanding of YARN and MR memory setting, how they relate, and how to set some baseline settings based on the cluster node size (disk, memory, and cores).  
可以分析得出原因：我的集群中每个map任务(或者reduce任务)的物理内存2G(或者4G)不够用了，那么对症下药，调大一点就可以了。但是不建议粗暴地一下子从2G double到4G这样调大，而是平缓地调大。调的时候也要同时调大JVM堆内存大小，一般是物理内存的80~90%.

【解决】  
调大物理内存——分别把 mapreduce.map.memory.mb 和 mapreduce.reduce.memory.mb 调大到3个G、5个G，

<property>
        <name>mapreduce.map.memory.mb</name>
        
        <value>3072</value>
    </property>
    
    
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        
    <value>5120</value>
    </property>

调大jvm堆内存——分别把 mapreduce.map.java.opts 和 mapreduce.reduce.java.opts 调大到2.5个G、4.5个G

<property>
            <name>mapreduce.map.java.opts</name>
            
            <value>-Xmx2560m</value>
        </property>
    
    <property>
        <name>mapreduce.reduce.java.opts</name>
        
        <value>-Xmx4608m</value>
    </property>

同步此配置文件mapred-site.xml到本集群的其他节点，无需重启yarn组件即可生效！

参考Cloudera的官网推荐：  
[https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh\_ig\_yarn\_tuning.html][https_www.cloudera.com_documentation_enterprise_5-4-x_topics_cdh_ig_yarn_tuning.html]

[https_www.cloudera.com_documentation_enterprise_5-4-x_topics_cdh_ig_yarn_tuning.html]: https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_ig_yarn_tuning.html