atop和audit对服务器进行监控

朴灿烈づ我的快乐病毒、 2023-01-07 07:28 118阅读 0赞

出现这个问题`-bash: [: : integer expression expected`，没有找到真正原因是什么，因为从监控没看到来自外部的攻击，下面的脚本已经验证过，也发现什么异常，但实地问题就出现了.ls、cat、vi、ll等命令全都执行不了  
![1][]  
跟踪发现环境变量中文件被删掉了  
![2][]  
如果同一个目录都是空的，还可以理解，但是/usr/bin目录下并不是所有都是空的  
![3][]  
云厂商工程师，先将系统盘卸载掉，挂靠到没有问题的服务器上，进行跟进。到了新服务器上，这个磁盘就是数据盘了，里面很多东西就可以读写了。的确是个好的办法，之前没接触过，确实是个很好的经验。

由于主机重启进程和链接未见异常，对主机下面的目录进行检查，这个也是个很好的经验，可以看出高级运维工程师的工作思路

crontab -l 
    var/spool/cron 
    /etc/init.d 
    /bin 
    /usr/bin
    /sbin 
    /etc/password
    /usr/local/bin/ 
    /root 
    /tmp 
    /var/tmp

文件检查，未见异常，查看主机日志未见异常  
bin目录文件被修改时间是15号11:21:24。history没有该时段操作记录，无法判断是如何被修改的  
工程师的建议方案是安装atop和audit，进行监控。当下情况只能这样，发现问题并不可怕，因为他会让你成长，你不可能天生什么都会，那么从比你厉害的人身上可以学到丰富的经验。  
**1 atop**  
`yum install -y atop`安装很是简单,这里参考[Linux atop 监控系统状态][Linux atop]对atop命令的结果做一些说明  
它的数据采集主要包括：CPU、内存、磁盘、网络、进程等，并且内容非常的详细，特别是当那一部分存在压力它会以特殊的颜色进行展示，如果颜色是红色那么说明已经非常严重了

当前情况，一切正常，就没什么可看的，以后有问题再追加。

[root@sp5 ~]# atop
    ATOP - sp5                                            2021/01/16  12:22:28                                            ----x---------                                             7d16h41m46s elapsed
    PRC | sys   55m35s  | user  10h20m  |               | #proc    147  | #trun      1  | #tslpi   273 |               |  #tslpu     0 |  #zombie    0 |  clones 104e5 |               |  #exit      0 |
    CPU | sys       2%  | user      7%  | irq       0%  | idle    790%  | wait      0%  | steal     0% |  guest     0% |               |  ipc notavail |  cycl unknown |  curf 2.60GHz |  curscal   ?% |
    cpu | sys       0%  | user      1%  | irq       0%  | idle     99%  | cpu007 w  0%  | steal     0% |  guest     0% |               |  ipc notavail |  cycl unknown |  curf 2.60GHz |  curscal   ?% |
    cpu | sys       0%  | user      1%  | irq       0%  | idle     99%  | cpu001 w  0%  | steal     0% |  guest     0% |               |  ipc notavail |  cycl unknown |  curf 2.60GHz |  curscal   ?% |
    cpu | sys       0%  | user      1%  | irq       0%  | idle     99%  | cpu003 w  0%  | steal     0% |  guest     0% |               |  ipc notavail |  cycl unknown |  curf 2.60GHz |  curscal   ?% |
    cpu | sys       0%  | user      1%  | irq       0%  | idle     99%  | cpu004 w  0%  | steal     0% |  guest     0% |               |  ipc notavail |  cycl unknown |  curf 2.60GHz |  curscal   ?% |
    cpu | sys       0%  | user      1%  | irq       0%  | idle     99%  | cpu000 w  0%  | steal     0% |  guest     0% |               |  ipc notavail |  cycl unknown |  curf 2.60GHz |  curscal   ?% |
    cpu | sys       0%  | user      1%  | irq       0%  | idle     99%  | cpu005 w  0%  | steal     0% |  guest     0% |               |  ipc notavail |  cycl unknown |  curf 2.60GHz |  curscal   ?% |
    cpu | sys       0%  | user      1%  | irq       0%  | idle     99%  | cpu006 w  0%  | steal     0% |  guest     0% |               |  ipc notavail |  cycl unknown |  curf 2.60GHz |  curscal   ?% |
    cpu | sys       0%  | user      1%  | irq       0%  | idle     99%  | cpu002 w  0%  | steal     0% |  guest     0% |               |  ipc notavail |  cycl unknown |  curf 2.60GHz |  curscal   ?% |
    CPL | avg1    0.46  | avg5    0.32  |               | avg15   0.27  |               |              |  csw 466495e3 |  intr 46034e4 |               |               |  numcpu     8 |               |
    MEM | tot    31.3G  | free   22.1G  | cache   6.5G  | dirty   0.4M  | buff  199.7M  | slab  327.3M |  slrec 291.5M |  shmem  16.6M |  shrss   0.0M |  vmbal   0.0M |  hptot   0.0M |  hpuse   0.0M |
    SWP | tot     0.0M  | free    0.0M  |               |               |               |              |               |               |               |               |  vmcom   2.8G |  vmlim  15.6G |
    LVM | gdata-lvData  | busy      0%  | read     269  |               | write  42349  | KiB/r     21 |  KiB/w     79 |  MBr/s    0.0 |  MBw/s    0.0 |               |  avq   108.14 |  avio 0.33 ms |
    DSK |          vda  | busy      0%  | read   16139  |               | write 1088e3  | KiB/r     25 |  KiB/w     16 |  MBr/s    0.0 |  MBw/s    0.0 |               |  avq     4.25 |  avio 0.67 ms |
    DSK |          vdb  | busy      0%  | read     535  |               | write   8394  | KiB/r     21 |  KiB/w    400 |  MBr/s    0.0 |  MBw/s    0.0 |               |  avq     7.81 |  avio 1.60 ms |
    NET | transport     | tcpi 37938e3  | tcpo 52709e3  | udpi  135222  | udpo  135233  | tcpao 320192 |  tcppo   1953 |  tcprs  36385 |  tcpie     59 |  tcpor   2598 |  udpnp     10 |  udpie      0 |
    NET | network       | ipi 39069675  | ipo 53290068  |               | ipfrw      0  | deliv 3907e4 |               |               |               |               |  icmpi 996440 |  icmpo      8 |
    NET | eth0    ----  | pcki 39367e3  | pcko 53292e3  | sp    0 Mbps  | si  165 Kbps  | so  133 Kbps |  coll       0 |  mlti       0 |  erri       0 |  erro       0 |  drpi       0 |  drpo       0 |
    NET | lo      ----  | pcki    7452  | pcko    7452  | sp    0 Mbps  | si    0 Kbps  | so    0 Kbps |  coll       0 |  mlti       0 |  erri       0 |  erro       0 |  drpi       0 |  drpo       0 |
                                                                               *** system and process activity since boot ***
      PID       SYSCPU       USRCPU        VGROW        RGROW        RDDSK        WRDSK       RUID           EUID           ST       EXC        THR       S       CPUNR        CPU       CMD         1/5
    21902        7m58s        7h47m       616.6M       61012K           0K       576.8M       root           root           N-         -          5       S           4         4%       python3.8
     4692        1m50s       91m00s       404.1M       69204K           0K       707.3M       root           root           N-         -          2       S           0         1%       python3.8
     1552       17m12s       14m39s         1.2G       13440K        1144K       51072K       root           root           N-         -         17       S           2         0%       hostguard
     1151       48.10s       12m49s       581.6M       99560K           0K       349.2M       root           root           N-         -          4       S           4         0%       python3.8
      999        5m33s        6m25s        11.1G       200.6M       16544K       190.1M       root           root           N-         -         24       S           5         0%       java
      746        4m16s        2m08s       119.3M        1436K           8K          40K       root           root           N-         -          2       S           2         0%       wrapper
        9        5m32s       21.45s           0K           0K           0K           0K       root           root           N-         -          1       S           7         0%       rcu_sched
     2407       22.51s        4m41s       575.2M       93052K           0K       29448K       root           root           N-         -          4       S           6         0%       python3.8
    24587       12.62s        3m02s       549.3M       67576K           0K       33244K       root           root           N-         -          4       S           0         0%       python3.8
      486        2m07s       59.96s       55532K        1080K          72K         1.6G       root           root           N-         -          2       S           1         0%       auditd
      658       76.05s       66.91s         1.5G       21852K        7984K       62260K       root           root           N-         -         24       S           4         0%       uniagent
    28779       17.82s        1m55s       293.4M       46480K           0K         1.3G       root           root           N-         -          1       S           6         0%       scrapyd
      148        1m54s        0.00s           0K           0K           0K           0K       root           root           N-         -          1       S           2         0%       kauditd
     3422       11.92s       85.06s       560.7M       77796K           0K       12444K       root           root           N-         -          4       S           2         0%       python3.8
     1856       11.21s       80.11s       558.3M       75352K           0K       11084K       root           root           N-         -          4       S           2         0%       python3.8
      984       11.01s       72.93s       549.7M       67780K           0K       13836K       root           root           N-         -          4       S           1         0%       python3.8
    10347       12.02s       67.66s       636.0M       84040K           0K       10916K       root           root           N-         -          5       S           6         0%       python3.8
    23080       12.03s       67.47s       549.6M       67460K           0K       11516K       root           root           N-         -          4       S           0         0%       python3.8
    22214       11.51s       60.57s       633.1M       78388K           0K       11796K       root           root           N-         -          5       S           4         0%       python3.8
    24172       11.13s       60.22s       548.4M       64368K           0K       10940K       root           root           N-         -          4       S           4         0%       python3.8
     4237       11.17s       60.06s       556.1M       72568K           0K       11820K       root           root           N-         -          4       S           1         0%       python3.8
     1550       31.73s       38.57s       49044K        2052K           8K        2216K       root           root           N-         -          1       S           4         0%       hostguard
    32598       11.07s       57.56s       550.1M       68408K           0K       10956K       root           root           N-         -          4       S           0         0%       python3.8
     2627       11.20s       55.45s       550.7M       67152K           0K       10424K       root           root           N-         -          4       S           6         0%       python3.8
     4344       11.06s       55.03s       550.1M       66776K           0K       10476K       root           root           N-         -          4       S           2         0%       python3.8
      743        9.28s       48.73s       560.8M       17524K        7828K          16K       root           root           N-         -          5       S           2         0%       tuned
      833       16.23s       11.98s       221.4M        8136K         840K        5296K       root           root           N-         -          3       S           0         0%       rsyslogd
      652       21.28s        4.48s       21672K        1252K         104K           0K       root           root           N-         -          1       S           2         0%       irqbalance
      319       20.73s        0.00s           0K           0K           0K         2.3G       root           root           N-         -          1       S           6         0%       jbd2/vda1-8
       33       20.09s        0.00s           0K           0K           0K           0K       root           root           N-         -          1       S           5         0%       migration/5
       23       20.02s        0.00s           0K           0K           0K           0K       root           root           N-         -          1       S           3         0%       migration/3
       43       19.99s        0.00s           0K           0K           0K           0K       root           root           N-         -          1       S           7         0%       migration/7
       13       19.82s        0.00s           0K           0K           0K           0K       root           root           N-         -          1       S           1         0%       migration/1

**2 audit**  
因为遇到过命令没法用的，故而先将重要的目录做一下审计，这样可以跟踪什么被删除了。  
[使用 auditd 监控目录变化][auditd]

# 一开始，你可以看到，什么规则都没有
    [root@sp1 ~]# auditctl -l
    No rules
    # 这个目录下面有很多常规的命令，添加到审计中
    auditctl -w /usr/bin -p wxa -k watch_usr_bin
    auditctl -w /usr/sbin -p wxa -k wath_usr_sbin
    
    # 查看规则添加进去了
    [root@sp1 sbin]# auditctl -l
    -w /usr/bin -p wxa -k watch_usr_bin
    -w /usr/sbin -p wxa -k wath_usr_sbin
    
    # 查看日志
    ausearch -k watch_usr_bin
    ausearch -k wath_usr_sbin
    #

[1]: /images/20221119/d345a7a135284f57aeb4fbce5d049aba.png
[2]: /images/20221119/53febe3d10184294869df2bc140d570a.png
[3]: /images/20221119/40a06dd02fc24d8ab00f01ea6ad9b99f.png
[Linux atop]: https://www.cnblogs.com/DataArt/p/10344438.html
[auditd]: https://www.aikaiyuan.com/12902.html