CDH 集群日常管理&调优

- 日理万妓 2023-02-19 13:25 18阅读 0赞

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70]  
调优无非从如上几个方面

*  硬件
 *  Linux
 *  JVM
 *  Hadoop
 *

### 文章目录 ###

*  1 调优
 *   *  1.1 Linux调优
     *  1.2 JVM参数的调优
     *  1.3 Hadoop
 *  2 CDH 集群管理
 *   *  2.1 配置参数
     *  2.2 监控
     *  2.3 慢磁盘 监控

# 1 调优 #

## 1.1 Linux调优 ##

**调整Linux的最大文件打开数和最大进程数**

vi /etc/security/limits.conf
    
    	* soft nofile 65535
    	* hard nofile 65535
    	* soft nproc 65535
    	* hard nproc 65535

**网络参数**

more /etc/sysctl.conf | grep net.core.somanconn\
    
    [root@localhost security]# sysctl -w net.core.somaxconn=32768
    net.core.somaxconn = 32768
    
    
    [root@localhost security]# echo net.core.somaxconn=32768 >> /etc/sysctl.conf
    [root@localhost security]# more /etc/sysctl.conf | grep net.core.somaxconn

**网络参数txqueuelen**

ifconfig
    ens5f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 10.2.0.6  netmask 255.255.255.0  broadcast 10.2.0.255
            inet6 fe80::bb3:69b7:6a55:e0b6  prefixlen 64  scopeid 0x20<link>
            ether a0:36:9f:fa:4b:04  txqueuelen 1000  (Ethernet)
            RX packets 11132379  bytes 2037240247 (1.8 GiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 17525655  bytes 4102894410 (3.8 GiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    ### 发现txquenelen 值为1000 调整为4096

**调整SWAP**  
设置SWAP为0 ，不是禁用 而是更多不用SWAP

more /etc/sysctl.conf | grep vm.swappiness
    echo vm.swappiness=0 >> /etc/sysctl.conf

## 1.2 JVM参数的调优 ##

Xms10240m -Xmx10240m
    -XX:+UseParNewGC
    -XX:+UseConcMarkSweePGC
    -XX:-CmSConcurrentMTEnabled
    -XX:CMSInitiatingOccupancyFration=70
    -XX:+CMSParalleelRemarkEnabled

## 1.3 Hadoop ##

# 2 CDH 集群管理 #

## 2.1 配置参数 ##

## 2.2 监控 ##

## 2.3 慢磁盘 监控 ##

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200620204216641.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTM4MTgzMw==,size_16,color_FFFFFF,t_70