记一次解决被恶意植入挖矿程序kdevtmpfsi的过程

淡淡的烟草味﹌ 2022-10-06 00:40 345阅读 0赞

## 起因 ##

收到一封腾讯云的邮件，内容如下  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3draF9fXw_size_16_color_FFFFFF_t_70][]

> 一开始没在意，以为是腾讯云误判(还是我太年轻)，毕竟我也没调用任何别人6379端口 的业务，俺自己的redis端口也改了不是这个，所以有点莫名其妙，就没理。  
>   
>   
> 谁知道当天就出现业务异常卡顿，网络通信存在异常延迟等问题，顺带查看了一下腾讯云自带的免费云监控(没有广告费)，7天网络如下图  
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3draF9fXw_size_16_color_FFFFFF_t_70 1][]  
> 最高的时候到了15M，感觉有点小DDOS的意思了…难怪腾讯云以为我攻击别人，现在才引起我注意，连上服务器敲一个top，就看到了挖矿程序：kdevtmpfsi  
> ![20210610104040498.png][]

## kdevtmpfsi特点 ##

> 含有守护进程并带有定时任务，所以往往清理后，隔几天又会出现

## 解决过程 ##

**1.查询kdevtmpfsi进程id**

[root@VM-0-2-centos ~]# ps -ef | grep kdevtmpfsi
    root     11451  1286 99 09:58 ?        01:50:07 /tmp/kdevtmpfsi
    root     14267 10669  0 11:02 pts/2    00:00:00 grep --color=auto kdevtmpfsi

**2.根据进程id查询守护进程**

[root@VM-0-2-centos ~]# systemctl status 11451
    # 阿哦~ 系统指令也都被黑了，所有关键系统文件的权限均被修改
    -bash: /usr/bin/systemctl: Permission denied
    # 赋予文件755权限
    [root@VM-0-2-centos ~]# chmod 755 /usr/bin/systemctl
    [root@VM-0-2-centos ~]# systemctl status 11451
    Failed to get unit for PID 11451: PID 11451 does not belong to any loaded unit.

> 这里进程11451已经被我刚刚kill了，我想试试kill指令能不能用来着，所以这里显示id不存在

**3.删除kdevtmpfsi进程**

# 挖矿程序
    kill kdevtmpfsi
    # 挖矿程序的守护进程
    kill kinsing
    # 验证是否已杀死
    ps -aux | grep kdevtmpfsi
    ps -aux | grep kinsing

**4.删除kdevtmpfsi相关文件夹**

解决步骤较多，都在下面了，具体解释看注释

[root@VM-0-2-centos tmp]# cd /tmp
    # 试图删除kdevtmpfsi文件夹，结果被锁了
    [root@VM-0-2-centos tmp]# rm -rf kdevtmpfsi
    rm: cannot remove ‘kdevtmpfsi’: Operation not permitted
    # 发现kdevtmpfsi文件夹被加了i属性，具体含义参考：https://www.runoob.com/linux/linux-comm-chattr.html
    [root@VM-0-2-centos tmp]# lsattr
    -------------e-- ./cpuidle_support.log
    -------------e-- ./net_affinity.log
    ----ia-------e-- ./svcupdates
    -------------e-- ./yum_save_tx.2021-06-10.10-47.RIGnYJ.yumtx
    -------------e-- ./yum_save_tx.2021-06-09.14-42.9567MU.yumtx
    ----ia-------e-- ./svcguard
    -------------e-- ./setRps.log
    -------------e-- ./nv_gpu_conf.log
    -------------e-- ./yum_save_tx.2021-06-09.14-45.ewXMhx.yumtx
    -------------e-- ./yum_save_tx.2021-06-08.20-18.o5LMma.yumtx
    -------------e-- ./yum_save_tx.2021-06-08.22-39.fMM654.yumtx
    -------------e-- ./yum_save_tx.2021-06-08.18-50.nzlZMH.yumtx
    ----i--------e-- ./redis2
    -------------e-- ./crontab.PqH9Tb
    -------------e-- ./yum_save_tx.2021-06-08.18-50.C9KsvO.yumtx
    -------------e-- ./yum_save_tx.2021-06-08.21-53.5ay7xI.yumtx
    -------------e-- ./virtio_blk_affinity.log
    -------------e-- ./virtio_blk_affinity_udev.log
    ----i--------e-- ./kdevtmpfsi
    -------------e-- ./yum_save_tx.2021-06-09.15-03.7KNsei.yumtx
    -------------e-- ./crontab.7lC9aC
    -------------e-- ./systemd-private-85826f165bb14fc9b36ced7d99e7989b-ntpd.service-aDaYPL
    ----ia-------e-- ./svcupdate
    ----ia-------e-- ./newsvc.sh
    -------------e-- ./yum_save_tx.2021-06-09.15-33.pDiw4o.yumtx
    -------------e-- ./yum_save_tx.2021-06-08.18-50.uPB19d.yumtx
    ----ia-------e-- ./svcworkmanager
    # 用chattr命令来删除i属性，果然不让我删
    [root@VM-0-2-centos tmp]# chattr -i kdevtmpfsi
    -bash: /usr/bin/chattr: Permission denied
    [root@VM-0-2-centos tmp]# cd /usr/bin/
    # 复制chattr来替换使用
    [root@VM-0-2-centos bin]# ls -lh chattr;lsattr chattr
    -rw-r--r-- 1 root root 12K Apr  1  2020 chattr
    ----i--------e-- chattr
    [root@VM-0-2-centos bin]# cp chattr chattr.new
    [root@VM-0-2-centos bin]# chmod a+x chattr.new
    [root@VM-0-2-centos bin]# chattr.new -i chattr
    [root@VM-0-2-centos bin]# rm -f chattr.new
    [root@VM-0-2-centos bin]# chmod a+x chattr
    [root@VM-0-2-centos bin]# ls -lh chattr;lsattr chattr
    -rwxr-xr-x 1 root root 12K Apr  1  2020 chattr
    -------------e-- chattr
    [root@VM-0-2-centos bin]# cd /tmp/
    # 删除kdevtmpfsi文件夹的i属性
    [root@VM-0-2-centos tmp]# chattr -i kdevtmpfsi
    [root@VM-0-2-centos bin]#
    [root@VM-0-2-centos bin]#
    [root@VM-0-2-centos bin]#
    [root@VM-0-2-centos bin]#
    [root@VM-0-2-centos bin]#
    [root@VM-0-2-centos tmp]# cd /var/spool
    [root@VM-0-2-centos spool]# lsattr
    -------------e-- ./at
    -------------e-- ./mail
    -------------e-- ./plymouth
    -------------e-- ./postfix
    ----ia-------e-- ./cron
    -------------e-- ./anacron
    -------------e-- ./lpd
    -------------e-- ./abrt
    -------------e-- ./abrt-upload
    # 删除定时任务文件的i属性
    [root@VM-0-2-centos spool]# chattr -ai /var/spool/cron
    [root@VM-0-2-centos spool]# lsattr
    -------------e-- ./at
    -------------e-- ./mail
    -------------e-- ./plymouth
    -------------e-- ./postfix
    -------------e-- ./cron
    -------------e-- ./anacron
    -------------e-- ./lpd
    -------------e-- ./abrt
    -------------e-- ./abrt-upload
    [root@VM-0-2-centos spool]#

**5.删除kdevtmpfsi相关文件夹**

rm -rf /tmp/kdevtmpfsi
    rm -rf /var/tmp/kinsing
    # 查看是否有残留，查到就rm -rf
    find / -name "*kdevtmpfsi*"
    find / -name "*kinsing*"

**6.清理定时任务**

# 查看当前已有的定时任务
    crontab -l
    # 清理所有定时任务
    crontab -r

**7.删除免密登录ssh**

[root@VM-0-2-centos .ssh]# cd /root/.ssh
    # 发现有个authorized_keys2，以及当前目录全部锁了i
    [root@VM-0-2-centos .ssh]# lsattr -a
    -------------e-- ./..
    ----ia-------e-- ./authorized_keys2
    ----ia---------- ./authorized_keys
    ----i----------- ./.
    [root@VM-0-2-centos .ssh]# chattr -i ./
    [root@VM-0-2-centos .ssh]# chattr -ia authorized_keys
    [root@VM-0-2-centos .ssh]# chattr -ia authorized_keys2
    # 删除所有密钥文件
    [root@VM-0-2-centos .ssh]# rm -rf ./*
    # 创建空的私钥文件
    [root@VM-0-2-centos .ssh]# touch authorized_keys
    # 修改ssh配置文件。确保PubkeyAuthentication为no或者已被注释掉即可
    [root@VM-0-2-centos .ssh]# vim /etc/ssh/sshd_config

## 后续处理 ##

1.防火墙该开开  
2.安全组该开开  
3.端口该关关  
过几天再看看会不会卷土重来

流量监控等也均恢复正常啦，满血复活！  
![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3draF9fXw_size_16_color_FFFFFF_t_70 2][]

--------------------

## 时隔三天 ##

> 2021-06-13再次收到腾讯云的警告邮件，内容一致，还是说我在攻击其他人6379端口，看来之前清理的kdevtmpfsi还是有残留，这里就接着对剩余残留处理。

[root@VM-0-2-centos ~]# netstat -an
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
    tcp        0      1 172.17.0.2:47660        65.5.49.179:6379        SYN_SENT
    tcp        0      1 172.17.0.2:50264        65.5.50.177:6379        SYN_SENT
    tcp        0      0 172.19.0.1:44890        172.19.0.3:80           TIME_WAIT
    tcp        0      0 172.19.0.1:43788        172.19.0.3:80           TIME_WAIT
    tcp        0      0 172.19.0.1:55582        172.19.0.3:80           TIME_WAIT
    tcp        0      1 172.17.0.2:44704        65.5.48.59:6379         SYN_SENT
    tcp        0      1 172.17.0.2:41186        65.5.51.15:6379         SYN_SENT
    tcp        0      0 172.19.0.1:47904        172.19.0.3:80           TIME_WAIT
    tcp        0      0 172.19.0.1:56860        172.19.0.3:80           TIME_WAIT
    tcp        0      1 172.17.0.2:44702        65.5.51.173:6379        SYN_SENT
    tcp        0      1 172.17.0.2:57440        65.5.50.50:6379         SYN_SENT
    tcp        0      1 172.17.0.2:40746        65.5.50.104:6379        SYN_SENT
    tcp        0      1 172.17.0.2:58656        65.5.51.254:6379        SYN_SENT
    tcp        0      1 172.17.0.2:60436        65.5.51.171:6379        SYN_SENT

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3draF9fXw_size_16_color_FFFFFF_t_70 3][]  
可以看到频繁调用美国ip服务的6379端口，猜测应该是这病毒想办法在传播，利用redis的6379端口。

## 开始搞它！ ##

# 发现病毒进程：1584/pnscan
    [root@VM-0-2-centos ~]# netstat -antp |grep 1
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      6971/sshd
    tcp        0      1 172.17.0.2:33416        65.141.50.158:6379      SYN_SENT    1584/pnscan
    tcp        0      0 172.19.0.1:36736        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:58506        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:41298        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:45006        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:51746        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:34208        172.19.0.3:80           TIME_WAIT   -
    tcp        0      1 172.17.0.2:45738        65.141.50.77:6379       SYN_SENT    1584/pnscan
    tcp        0      0 172.19.0.1:45406        172.19.0.3:80           TIME_WAIT   -
    tcp        0      1 172.17.0.2:59096        65.141.49.180:6379      SYN_SENT    1584/pnscan
    tcp        0      0 172.19.0.1:47682        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:55756        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:57948        172.19.0.3:80           ESTABLISHED 853/docker-proxy
    tcp        0      0 172.19.0.1:46024        172.19.0.3:80           ESTABLISHED 853/docker-proxy
    tcp        0      0 172.19.0.1:51486        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:47870        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:36458        172.19.0.3:80           TIME_WAIT   -
    tcp        0      1 172.17.0.2:46668        65.141.49.81:6379       SYN_SENT    1584/pnscan
    tcp        0      1 172.17.0.2:55490        65.141.48.147:6379      SYN_SENT    1584/pnscan
    # 查找病毒pnscan所在路径
    [root@VM-0-2-centos ~]# find / -name "*pnscan*"
    /usr/local/bin/pnscan
    /usr/local/share/man/man1/pnscan.1.gz
    # 杀它！！！！
    [root@VM-0-2-centos ~]# rm -rf /usr/local/bin/pnscan /usr/local/share/man/man1/pnscan.1.gz
    # 再次验证是否有发起6379的网络请求，可以看到没有6379的请求端口了，但是！
    [root@VM-0-2-centos ~]# netstat -anptl
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      6971/sshd
    tcp        0      0 172.19.0.1:49710        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:51082        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:53556        172.19.0.3:80           ESTABLISHED 853/docker-proxy
    tcp        0      0 172.19.0.1:49298        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:47786        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:50184        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:53496        172.19.0.3:80           ESTABLISHED 853/docker-proxy
    tcp        0      0 172.19.0.1:53818        172.19.0.3:80           ESTABLISHED 853/docker-proxy
    tcp        0      0 172.19.0.1:52173        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:53024        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:59698        172.19.0.3:80           CLOSE_WAIT  853/docker-proxy
    tcp        0      0 172.19.0.1:50394        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:52694        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:50382        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:48692        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:56954        172.19.0.3:80           CLOSE_WAIT  853/docker-proxy
    tcp        0      0 172.19.0.1:50710        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:49884        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:51434        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:52858        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:33880        172.19.0.3:80           CLOSE_WAIT  853/docker-proxy
    tcp        0      0 172.19.0.1:52416        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:49504        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:51296        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:50628        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:48666        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:42912        172.19.0.3:80           CLOSE_WAIT  853/docker-proxy
    tcp        0      0 172.19.0.1:53812        172.19.0.3:80           ESTABLISHED 853/docker-proxy
    tcp        0      0 172.19.0.1:50402        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:52370        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.19.0.1:52538        172.19.0.3:80           TIME_WAIT   -
    tcp        0      0 172.17.0.2:60168        139.99.102.70:14444     ESTABLISHED 683/[crypto]
    tcp        0    280 172.17.0.2:22           45.55.134.210:33818     ESTABLISHED 27443/sshd: [accept
    # 着重看末尾两个；ESTABLISHED 27443/sshd: [accept应该是黑客在尝试登录，输入一下who看到登录用户只有我自己
    # ESTABLISHED 683/[crypto]：又发现一个病毒，在请求139.99.102.70的ip
    [root@VM-0-2-centos ~]# lsof -i
    COMMAND     PID USER   FD   TYPE     DEVICE SIZE/OFF NODE NAME
    [crypto]    683 root   13u  IPv4 1200665664      0t0  TCP VM-0-2-centos:60168->70.ip-139-99-102.eu:14444 (ESTABLISHED)
    # 搞它！
    [root@VM-0-2-centos ~]# kill -9 683
    [root@VM-0-2-centos ~]# find / -name "*\\[crypto\\]*"
    /usr/share/[crypto].log
    /usr/share/[crypto]
    /usr/share/[crypto].sh
    /usr/share/[crypto].pid
    # 又是执行脚本、log文件七七八八的，直接删它！！！！
    [root@VM-0-2-centos ~]# rm -rf /usr/share/[crypto].log /usr/share/[crypto] /usr/share/[crypto].sh /usr/share/[crypto].pid

最后再查看一下top以及网络请求等等，目前为止都正常了…如果又冒出来此文再接着更新。— 2021-06-13

码字不易，对你有帮助的话，点个赞呗~

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3draF9fXw_size_16_color_FFFFFF_t_70]: /images/20221005/8d1dcba8d3eb4b67adae25819a068173.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3draF9fXw_size_16_color_FFFFFF_t_70 1]: /images/20221005/a29ebd3ae4eb4d49b679d6dc1ce5caed.png
[20210610104040498.png]: /images/20221005/217195e01e9247da8303ddb3c8e1f299.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3draF9fXw_size_16_color_FFFFFF_t_70 2]: /images/20221005/26602c239ef544788167c20ef4d77718.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3draF9fXw_size_16_color_FFFFFF_t_70 3]: /images/20221005/e104ad233d024783905e10112badd31a.png