Centos7 Jenkins 日志记录爆满磁盘问题

电玩女神 2021-10-19 23:28 257阅读 0赞

问题描述：上周准备线上上Jenkens,然后配置好之后......

今天还在睡懒觉，一通电话惊醒，服务器有部分服务不能用了...

### 1、分析问题 ###

服务一直很稳定，再测试一把出现问题的服务。发现问题，连运行日志都没有了，默默的猜服务器是不是文件哪里挂了，直接上杀手：

第一步：清除nginx日志

第二步：删除服务日志

第三步：重启服务

结果：日志照样没有

### 2、只能是服务器的磁盘爆满了 ###

第一步：df -sh

果然是磁盘爆了。

第二步：找服务器磁盘哪里被爆了

首先看哪个文件夹爆了-这个是我清理后的结果
    
    [root@iZ23evimvf8Z ~]# df -lh
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/xvda1       40G   39G   0     100% /
    devtmpfs        7.8G     0  7.8G   0% /dev
    tmpfs           7.7G   44K  7.7G   1% /dev/shm
    tmpfs           7.7G  384K  7.7G   1% /run
    tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
    tmpfs           1.6G     0  1.6G   0% /run/user/1000
    tmpfs           1.6G     0  1.6G   0% /run/user/0

[root@iZ23evimvf8Z ~]# du -sh /*
    4.0K	/1
    0	/bin
    103M	/boot
    624K	/C:
    24K	/deployer
    44K	/dev
    143M	/etc
    52K	/file:
    4.0G	/ftpDir
    19M	/home
    0	/lib
    0	/lib64
    16K	/lost+found
    4.0K	/media
    4.0K	/mnt
    116M	/opt
    ...........................

[root@iZ23evimvf8Z ~]# du -sh /var/log/*
    36K	/var/log/agent.log
    16K	/var/log/aliyun-util.log
    4.0K	/var/log/anaconda
    4.0K	/var/log/audit
    8.0K	/var/log/boot.log
    4.0K	/var/log/btmp
    56K	/var/log/btmp-20190801
    16K	/var/log/cron
    100K	/var/log/cron-20190729
    76K	/var/log/cron-20190804
    88K	/var/log/cron-20190811
    84K	/var/log/cron-20190818
    36K	/var/log/dmesg
    36K	/var/log/dmesg.old
    0	/var/log/grubby
    24K	/var/log/gshell.log
    1.9G	/var/log/journal

找到最终的文件，然后干掉：

[root@iZ23evimvf8Z log]# rm -rf /var/log/jenkins

然后看看我们的服务器：

[root@iZ23evimvf8Z log]# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/xvda1       40G   39G     0 100% /
    devtmpfs        7.8G     0  7.8G   0% /dev
    tmpfs           7.7G   44K  7.7G   1% /dev/shm
    tmpfs           7.7G  356K  7.7G   1% /run
    tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
    tmpfs           1.6G     0  1.6G   0% /run/user/1000
    tmpfs           1.6G     0  1.6G   0% /run/user/0

原因：我们彻底删除了是文件的磁盘，但是实际Jenkins还在运行中，还有占用文件：于是我们这么做：

[root@iZ23evimvf8Z log]# ps aux | grep jenkins
    root      2468  6.6 10.2 7796180 1634752 ?     Ssl  10:50   0:47 /usr/local/www/zcsjw/jdk1.8.0_172/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/libjenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --daemon --httpPort=9000 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
    root     10184  0.0  0.0 112652   956 pts/2    S+   11:02   0:00 grep --color=auto jenkins
    [root@iZ23evimvf8Z log]# kill -9 2468

彻底干掉Jenkins就好了：

[root@iZ23evimvf8Z log]# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/xvda1       40G   23G   15G  61% /
    devtmpfs        7.8G     0  7.8G   0% /dev
    tmpfs           7.7G   44K  7.7G   1% /dev/shm
    tmpfs           7.7G  356K  7.7G   1% /run
    tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
    tmpfs           1.6G     0  1.6G   0% /run/user/1000
    tmpfs           1.6G     0  1.6G   0% /run/user/0

大功告成。

其他更方便简单的处理方式

第一步：查找系统大文件

\[root@iZ23evimvf8Z log\]\# find / -type f -size +100M  -print0 | xargs -0 du -h | sort -nr

[root@iZ23evimvf8Z log]# find / -type f -size +100M  -print0 | xargs -0 du -h | sort -nr
    find: ‘/proc/24277/task/24277/fdinfo/6’: No such file or directory
    find: ‘/proc/24277/fdinfo/6’: No such file or directory
    131M	/ftpDir/sl/tjsljs.flv
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-00000000004e09fc-00058f83c0d11719.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-00000000004b5d72-00058e8ce84241fd.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-000000000048bfe4-00058db36555cbcc.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-0000000000462262-00058ca95dffa790.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-000000000042e589-00058b7a553abdb1.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-0000000000402986-000589de23e00a9d.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-00000000003b49e7-00057a645c72469f.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-000000000038cb1a-00057a5bac12ab59.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-0000000000364c51-00057a52fbda482e.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-000000000033cd8c-00057a4a4b4daa56.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-0000000000314ec9-00057a419adf7984.journal
    129M	/var/log/journal/7d26c16f128042a684ea474c9e2c240f/system@d403eec4743545a882d3e33e80c18095-00000000002ecfee-00057a38ea9ac8e7.journal
    102M	/usr/lib/locale/locale-archive
    1.4G	/usr/local/csjw/elasticsearch-6.3.5-node2/bin/java_pid1453.hprof
    1.4G	/usr/local/csjw/elasticsearch-6.3.5-node2/bin/java_pid1215.hprof
    1.4G	/usr/local/csjw/elasticsearch-6.3.5-node1/bin/java_pid1857.hprof
    1.4G	/usr/local/csjw/elasticsearch-6.3.5-node1/bin/java_pid1388.hprof
    1.4G	/usr/local/csjw/elasticsearch-6.3.5-node1/bin/java_pid1223.hprof
    1.4G	/usr/local/csjw/elasticsearch-6.3.5-node1/bin/java_pid1188.hprof

第二步：删除查找到的大文件，然后干掉。部分步骤如上面处理。

### 3、Jenkins设置日志 ###

jenkins 设置 -》 系统日志 -》日志级别 -》配置级别

javax.jmdns off

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lleGlhb21vZGVtbw_size_16_color_FFFFFF_t_70][]

设置完了后，重启即可。

参考来源：

[https://blog.csdn.net/u012075238/article/details/84592327][https_blog.csdn.net_u012075238_article_details_84592327]

[https://www.jianshu.com/p/fd2eee044d62][https_www.jianshu.com_p_fd2eee044d62]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lleGlhb21vZGVtbw_size_16_color_FFFFFF_t_70]: /images/20211019/660d9485681045df9bacdfe6ea4fc1d4.png
[https_blog.csdn.net_u012075238_article_details_84592327]: https://blog.csdn.net/u012075238/article/details/84592327
[https_www.jianshu.com_p_fd2eee044d62]: https://www.jianshu.com/p/fd2eee044d62