Linux服务器缓存总是不够用

小鱼儿 2022-09-01 07:57 148阅读 0赞

# **一：问题描述** #

服务的 Buffer/Cache 总是在高位运行，可用的缓存一直不够用

![20210727091928151.png][]

# **二：问题分析** #

这个缓存，我用了定时清理方式，但是治标不治本，用户量大了后，还是经常不够用，怎么办嘞？

那就看一看这个 Buffer/Cache 里面到底存了什么数据？

想要看 Buffer/Cache 我们要安装一个神器
    [root@iZbp1gp1t778obaz5m8vk8Z ~]# wget https://silenceshell-1255345740.cos.ap-shanghai.myqcloud.com/hcache
    [root@iZbp1gp1t778obaz5m8vk8Z ~]# chmod +x hcache
    [root@iZbp1gp1t778obaz5m8vk8Z ~]# mv hcache /usr/local/bin/
    [root@iZbp1gp1t778obaz5m8vk8Z ~]#

查看前十个占用缓存最大的文件
    [root@iZbp1gp1t778obaz5m8vk8Z ~]# hcache -top 10
    +-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
    | Name                                                                                                                                | Size (bytes)   | Pages      | Cached    | Percent |
    |-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001847356-0005c705938eb664.journal | 117440512      | 28672      | 16792     | 058.566 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001804329-0005c6c0d699f6b5.journal | 117440512      | 28672      | 15558     | 054.262 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001913bd6-0005c7b5d2a0a068.journal | 117440512      | 28672      | 14873     | 051.873 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001830cc0-0005c6f1d16335e1.journal | 117440512      | 28672      | 14501     | 050.575 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001709205-0005c5ec8d98c3bd.journal | 117440512      | 28672      | 14401     | 050.227 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001695bce-0005c591f4f9b6c2.journal | 117440512      | 28672      | 13798     | 048.124 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000167ef8b-0005c5801512bfe8.journal | 117440512      | 28672      | 13798     | 048.124 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000018b8036-0005c76fcfab5444.journal | 117440512      | 28672      | 13027     | 045.435 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000177c190-0005c64db1792194.journal | 117440512      | 28672      | 12802     | 044.650 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001941e20-0005c7dd75f05239.journal | 117440512      | 28672      | 12740     | 044.434 |
    +-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+

分析得出是因为我们的日志文件大量占用缓存

CentOS系统中有两个日志服务，分别是传统的 rsyslog 和 systemd-journal

systemd-journald是一个改进型日志管理服务，可以收集来自内核、系统早期启动阶段的日志、系统守护进程在启动和运行中的标准输出和错误信息，还有syslog的日志。

该日志服务仅仅把日志集中保存在单一结构的日志文件/run/log中，由于日志是经历过压缩和格式化的二进制数据，所以在查看和定位的时候很迅速。

默认情况下并不会持久化保存日志，只会保留一个月的日志。另外，一些rsyslog无法收集的日志也会被journal记录到。

rsyslog作为传统的系统日志服务，把所有收集到的日志都记录到/var/log/目录下的各个日志文件中。

原来是系统日志在作祟，那就去看看。

**二.一 细化分析**

[root@iZbp1gp1t778obaz5m8vk8Z journal]# ll
    total 16
    drwxr-sr-x+ 2 root systemd-journal 12288 Jul 27 02:49 20190711105006363114529432776998
    [root@iZbp1gp1t778obaz5m8vk8Z journal]# 
    [root@iZbp1gp1t778obaz5m8vk8Z journal]# du -sh
    3.9G	.

# **三：处理问题** #

大气出奇迹，开干，把系统日志的最大控制在500M，物理文件被删除 3.3G

[root@iZbp1gp1t778obaz5m8vk8Z journal]# journalctl --vacuum-size=500M
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000165276b-0005c559b355a8c7.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001668bbf-0005c56d9fdc61f3.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000167ef8b-0005c5801512bfe8.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001695bce-0005c591f4f9b6c2.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000016ad657-0005c5a0f50b116c.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000016c3cf5-0005c5bf172b8aca.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000016da438-0005c5d49cd6b6fd.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000016f11aa-0005c5e198551ec4.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001709205-0005c5ec8d98c3bd.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001720988-0005c5f7f7f78e0f.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017383da-0005c607981114f1.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000174f435-0005c61abbac5d31.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001765e21-0005c631203d3e02.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000177c190-0005c64db1792194.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017927bc-0005c65ff00bbafb.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017a968d-0005c67116a09e54.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017bfd07-0005c685c9cf1389.journal (112.0M).
    Deleted archived journal /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-00000000017d6d81-0005c698038e2ad2.journal (112.0M).
    Deleted archived journal 
    ...................................................................
    Vacuuming done, freed 3.3G of archived journals on disk.
    [root@iZbp1gp1t778obaz5m8vk8Z journal]#

再看一下我们的物理路径下面，以及我们的内存状态，唉 物理路径下只有500M文件了，而我们的缓存处给我空出来 1.5G 可用内存！！！

[root@iZbp1gp1t778obaz5m8vk8Z journal]# 
    [root@iZbp1gp1t778obaz5m8vk8Z journal]# du -sh
    497M	.
    [root@iZbp1gp1t778obaz5m8vk8Z journal]# 
    [root@iZbp1gp1t778obaz5m8vk8Z journal]# 
    [root@iZbp1gp1t778obaz5m8vk8Z journal]# free -h
                  total        used        free      shared  buff/cache   available
    Mem:            15G        9.6G        4.6G        1.5M        831M        5.0G
    Swap:            0B          0B          0B
    [root@iZbp1gp1t778obaz5m8vk8Z journal]#

再看一下我们的缓存占用情况，结果日志文件还是瑶瑶领先，但是终于看到其他的服务了，先这样OK运行。

[root@iZbp1gp1t778obaz5m8vk8Z journal]# 
    [root@iZbp1gp1t778obaz5m8vk8Z journal]# hcache -top 10
    +-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
    | Name                                                                                                                                | Size (bytes)   | Pages      | Cached    | Percent |
    |-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001913bd6-0005c7b5d2a0a068.journal | 117440512      | 28672      | 14873     | 051.873 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001941e20-0005c7dd75f05239.journal | 117440512      | 28672      | 12740     | 044.434 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-0000000001958066-0005c7face4cfe97.journal | 117440512      | 28672      | 12522     | 043.673 |
    | /var/log/journal/20190711105006363114529432776998/system@00000000000000000000000000000000-000000000192aebd-0005c7c5f63c62af.journal | 117440512      | 28672      | 9982      | 034.814 |
    | /var/log/journal/20190711105006363114529432776998/system.journal                                                                    | 25165824       | 6144       | 6144      | 100.000 |
    | /usr/bin/dockerd-current                                                                                                            | 33420192       | 8160       | 3934      | 048.211 |
    | /usr/local/jdk1.8.0_172/jre/lib/amd64/server/libjvm.so                                                                              | 17067833       | 4167       | 2507      | 060.163 |
    | /usr/local/share/aliyun-assist/2.2.3.221/aliyun-service                                                                             | 15611914       | 3812       | 1952      | 051.207 |
    | /usr/bin/docker-containerd-current                                                                                                  | 10806584       | 2639       | 1217      | 046.116 |
    | /usr/local/nginx/sbin/nginx                                                                                                         | 6099568        | 1490       | 1024      | 068.725 |
    +-------------------------------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+

# 四：总结 #

从运行方向来看，我这台阿里云服务器的缓存，大部分可用内存均是系统日志占用，但是这个系统日志占用了超大量的内存，这一点阿里云需要改善，毕竟访问量大就日志疯狂的存储占用可用内存，这有点做得可不够好。

然后大家如果遇到同样的问题，可以根据自己的情况适当的配置日志可占用的大小情况。

希望跟大家一起进步！！！

参考资料：

《[Linux中清理/var/log/journal/垃圾日志][Linux_var_log_journal]》

# 五：补充学习 #

[root@iZbp1f0xuq9rc41s6gdvfyZ log]# 
    [root@iZbp1f0xuq9rc41s6gdvfyZ log]# ps aux|grep tomcat
    root     16583  0.0  0.0 112708   984 pts/1    S+   09:47   0:00 grep --color=auto tomcat
    zcsjw    30375  2.5 40.4 11693112 6576120 ?    Sl   Jun28 1072:50 /usr/local/jdk1.8.0_172/jre/bin/java -Djava.util.logging.config.file=/usr/local/www/apache-tomcat-8.5.43/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -XX:PermSize=1024M -XX:MaxPermSize=4096m -Xms2048M -Xmx4096M -XX:MaxNewSize=1024m -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -server -Xms2048m -Xmx4096m -Dignore.endorsed.dirs= -classpath /usr/local/www/apache-tomcat-8.5.43/bin/bootstrap.jar:/usr/local/www/apache-tomcat-8.5.43/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/www/apache-tomcat-8.5.43 -Dcatalina.home=/usr/local/www/apache-tomcat-8.5.43 -Djava.io.tmpdir=/usr/local/www/apache-tomcat-8.5.43/temp org.apache.catalina.startup.Bootstrap start
    [root@iZbp1f0xuq9rc41s6gdvfyZ log]# 
    [root@iZbp1f0xuq9rc41s6gdvfyZ log]# 
    [root@iZbp1f0xuq9rc41s6gdvfyZ log]# hcache -pid 30375
    +---------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------+
    | Name                                                                                                          | Size (bytes)   | Pages      | Cached    | Percent |
    |---------------------------------------------------------------------------------------------------------------+----------------+------------+-----------+---------|
    | /usr/local/jdk1.8.0_172/jre/lib/amd64/libnet.so                                                               | 115485         | 29         | 18        | 062.069 |
    | /usr/local/www/apache-tomcat-8.5.43/webapps/guides/WEB-INF/lib/aopalliance-1.0.jar                      | 4467           | 2          | 2         | 100.000 |
    | /usr/local/www/apache-tomcat-8.5.43/webapps/guides/WEB-INF/lib/commons-httpclient-3.1.jar               | 305001         | 75         | 75        | 100.000 |
    | /usr/local/www/apache-tomcat-8.5.43/webapps/guides/WEB-INF/lib/mybatis-spring-1.2.2.jar                 | 48783          | 12         | 12        | 100.000 |
    | /usr/lib/fontconfig/cache/83bf7a3f-cf3f-4a35-a43e-c013d1e66acb-le64.cache-7                                   | 20592          | 6          | 6         | 100.000 |
    | /usr/local/www/apache-tomcat-8.5.43/webapps/guides/WEB-INF/lib/codejson-1.1.1.jar                       | 102621         | 26         | 26        | 100.000 |
    | /usr/lib64/libresolv-2.17.so                                                                                  | 105824         | 26         | 26        | 100.000 |
    .......................................................

[20210727091928151.png]: /images/20220829/d777428e4a4a44198cf11b4c7250e10b.png
[Linux_var_log_journal]: http://blog.majiameng.com/article/2767.html