docker 解决 stop kill 假死和docker-compose的 network has active endpoints 问题

亦凉 2022-12-20 14:59 258阅读 0赞

## 问题 ##

使用`docker-compose down`的时候发现`network`不能停止

[root@176 website-cdn]# docker-compose down
    Stopping hw_nginx  ... done
    Stopping hw_django ... done
    Stopping hw_mysql  ... done
    Removing hw_django ... done
    Removing hw_mysql  ... done
    Removing network website-cdn_django_network
    ERROR: error while removing network: network website-cdn_django_network id 0f44a37851fb3e4160676d1e3620fd56ace3bc098686851ce5ca772c128f0559 has active endpoints
    [root@176 website-cdn]# docker-compose ps

## 解决 ##

1.  `docker network inspect {network}`获得`{endpoint-name}`，即`docker-compse.yml`中的`container_name`
2.  `docker network disconnect -f {network} {endpoint-name}`
3.  重启`docker-compose up -d`
4.  关闭`docker-compose down`
5.  估计有些人的问题已经解决，但是我发现进程依然存在，已经造成假死
6.  `docker`进行`stop`，失败
7.  容器挂载对应目录`/var/lib/docker/containers`进行root删除，提示设备或资源忙，失败
8.  取消各种挂载`umount /data/sys/var/docker/containers/`，失败
9.  相关的进程号`lsof |grep "挂载目录"`直接`kill -9`进程号失败
10. 最终，重启docker解决`sudo service docker restart`， \* .\*!!!

[root@176 ~]# cd /opt/website-cdn/
    [root@176 website-cdn]# docker-compose down
    Stopping hw_nginx  ... done
    Stopping hw_django ... done
    Stopping hw_mysql  ... done
    Removing hw_django ... done
    Removing hw_mysql  ... done
    Removing network website-cdn_django_network
    ERROR: error while removing network: network website-cdn_django_network id 0f44a37851fb3e4160676d1e3620fd56ace3bc098686851ce5ca772c128f0559 has active endpoints
    [root@176 website-cdn]# docker-compose ps
      Name                Command               State                    Ports                  
    --------------------------------------------------------------------------------------------
    hw_nginx   /docker-entrypoint.sh ngin ...   Up      0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
    [root@176 website-cdn]# docker-compose down
    Stopping hw_nginx ... done
    Removing network website-cdn_django_network
    ERROR: error while removing network: network website-cdn_django_network id 0f44a37851fb3e4160676d1e3620fd56ace3bc098686851ce5ca772c128f0559 has active endpoints
    [root@176 website-cdn]# docker network inspect website-cdn_django_network
    [
        { 
            "Name": "website-cdn_django_network",
            "Id": "0f44a37851fb3e4160676d1e3620fd56ace3bc098686851ce5ca772c128f0559",
            "Created": "2020-11-06T12:41:06.634617381-05:00",
            "Scope": "local",
            "Driver": "bridge",
            "EnableIPv6": false,
            "IPAM": { 
                "Driver": "default",
                "Options": null,
                "Config": [
                    { 
                        "Subnet": "192.168.112.0/20",
                        "Gateway": "192.168.112.1"
                    }
                ]
            },
            "Internal": false,
            "Attachable": true,
            "Ingress": false,
            "ConfigFrom": { 
                "Network": ""
            },
            "ConfigOnly": false,
            "Containers": { 
                "83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571": { 
                    "Name": "hw_nginx",
                    "EndpointID": "15d1d549645a496ddb75c336a9abed8f3b598ede440f13e67560300ecf01b33b",
                    "MacAddress": "02:42:c0:a8:70:04",
                    "IPv4Address": "192.168.112.4/20",
                    "IPv6Address": ""
                }
            },
            "Options": { },
            "Labels": { 
                "com.docker.compose.network": "django_network",
                "com.docker.compose.project": "website-cdn",
                "com.docker.compose.version": "1.25.0"
            }
        }
    ]
    [root@176 website-cdn]# docker network disconnect -f website-cdn_django_network hw_nginx
    [root@176 website-cdn]# docker-compose ps
      Name                Command               State   Ports
    ---------------------------------------------------------
    hw_nginx   /docker-entrypoint.sh ngin ...   Up           
    [root@176 website-cdn]# docker-compose up -d
    Creating hw_mysql ... done
    Creating hw_django ... done
    hw_nginx is up-to-date
    [root@176 website-cdn]# docker-compose down
    Stopping hw_django ... done
    Stopping hw_mysql  ... done
    Stopping hw_nginx  ... done
    Removing hw_django ... done
    Removing hw_mysql  ... done
    Removing network website-cdn_django_network
    [root@176 website-cdn]# 
    [root@176 website-cdn]# docker-compose ps
      Name                Command               State   Ports
    ---------------------------------------------------------
    hw_nginx   /docker-entrypoint.sh ngin ...   Up           
    [root@176 website-cdn]# docker-compose down
    Stopping hw_nginx ... done
    Removing network website-cdn_django_network
    WARNING: Network website-cdn_django_network not found.
    [root@176 website-cdn]# docker-compose ps
      Name                Command               State   Ports
    ---------------------------------------------------------
    hw_nginx   /docker-entrypoint.sh ngin ...   Up           
    [root@176 website-cdn]# docker ps
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
    83967693c041        nginx:latest        "/docker-entrypoint.…"   5 days ago          Up 5 days                               hw_nginx
    [root@176 website-cdn]# docker stop 83967693c041
    83967693c041
    [root@176 website-cdn]# docker ps
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
    83967693c041        nginx:latest        "/docker-entrypoint.…"   5 days ago          Up 5 days                               hw_nginx
    [root@176 website-cdn]# docker ps -a
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                   PORTS                                      NAMES
    83967693c041        nginx:latest        "/docker-entrypoint.…"   5 days ago          Up 5 days                                                           hw_nginx
    7542244e5ff7        fb5bf11d85f3        "/bin/sh -c 'apk upd…"   8 weeks ago         Exited (6) 8 weeks ago                                              brave_kowalevski
    01cdd076b16f        fb5bf11d85f3        "/bin/sh -c 'apk upd…"   8 weeks ago         Exited (6) 8 weeks ago                                              objective_bose
    c942777c683f        nginx:alpine        "nginx -g 'daemon of…"   7 months ago        Created                  0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   nginx_proxy
    [root@176 website-cdn]# ll /var/lib/docker/container
    ls: 无法访问/var/lib/docker/container: 没有那个文件或目录
    [root@176 website-cdn]# ll /var/lib/docker/containers
    总用量 16
    drwx------ 4 root root 4096 9月  17 11:42 01cdd076b16f1749f3a376d61aaf9edc1dafa642437cac545ef21a53f36a3ea8
    drwx------ 4 root root 4096 9月  17 11:42 7542244e5ff788305cf06e5367cf59fda16e06aff81b66083d16f6ad6224ae32
    drwx------ 4 root root 4096 11月 12 11:12 83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    drwx------ 4 root root 4096 9月  17 11:42 c942777c683fe74ef3a2f161f4a0e4214e82529f22025c6878eb3d3879ad5fc2
    [root@176 website-cdn]# rm -rf /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/
    rm: 无法删除"/var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/mounts/shm": 设备或资源忙
    [root@176 website-cdn]# sudo rm -rf /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/
    rm: 无法删除"/var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/mounts/shm": 设备或资源忙
    [root@176 website-cdn]# cat /proc/mounts | grep "83967693c041"
    shm /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/mounts/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
    [root@176 website-cdn]# lsof | grep "83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571"
    -bash: lsof: 未找到命令
    [root@176 website-cdn]# sudo yum install lsof
    已加载插件：fastestmirror
    Determining fastest mirrors
     * base: repos.lax.quadranet.com
     * elrepo-kernel: repos.lax-noc.com
     * extras: centos.mirror.ndchost.com
     * updates: mirror.hostduplex.com
    base                                                                                                                                                                                                                                                                                                                         | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                                                                                             | 3.5 kB  00:00:00     
    elrepo-kernel                                                                                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    extras                                                                                                                                                                                                                                                                                                                       | 2.9 kB  00:00:00     
    updates                                                                                                                                                                                                                                                                                                                      | 2.9 kB  00:00:00     
    (1/5): extras/7/x86_64/primary_db                                                                                                                                                                                                                                                                                            | 220 kB  00:00:00     
    (2/5): updates/7/x86_64/primary_db                                                                                                                                                                                                                                                                                           | 2.5 MB  00:00:00     
    (3/5): docker-ce-stable/x86_64/primary_db                                                                                                                                                                                                                                                                                    |  46 kB  00:00:00     
    (4/5): elrepo-kernel/primary_db                                                                                                                                                                                                                                                                                              | 1.9 MB  00:00:00     
    (5/5): base/7/x86_64/primary_db                                                                                                                                                                                                                                                                                              | 6.1 MB  00:00:01     
    正在解决依赖关系
    --> 正在检查事务
    ---> 软件包 lsof.x86_64.0.4.87-6.el7 将被 安装
    --> 解决依赖关系完成
    
    依赖关系解决
    
    ====================================================================================================================================================================================================================================================================================================================================================
     Package                                                                          架构                                                                               版本                                                                                    源                                                                                大小
    ====================================================================================================================================================================================================================================================================================================================================================
    正在安装:
     lsof                                                                             x86_64                                                                             4.87-6.el7                                                                              base                                                                             331 k
    
    事务概要
    ====================================================================================================================================================================================================================================================================================================================================================
    安装  1 软件包
    
    总下载量：331 k
    安装大小：927 k
    Is this ok [y/d/N]: y
    Downloading packages:
    lsof-4.87-6.el7.x86_64.rpm                                                                                                                                                                                                                                                                                                   | 331 kB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      正在安装    : lsof-4.87-6.el7.x86_64                                                                                                                                                                                                                                                                                                          1/1 
      验证中      : lsof-4.87-6.el7.x86_64                                                                                                                                                                                                                                                                                                          1/1 
    
    已安装:
      lsof.x86_64 0:4.87-6.el7                                                                                                                                                                                                                                                                                                                          
    
    完毕！
    [root@176 website-cdn]# lsof | grep "83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571"
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    dockerd   26304           root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26313     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26314     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26315     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26317     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26318     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26319     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26320     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26321     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26322     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26323     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26324     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    dockerd   26304 26325     root   64w      REG                8,2    296748     266484 /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571-json.log (deleted)
    container 27095           root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095           root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095           root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095           root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095           root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095           root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095           root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27096     root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095 27096     root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27096     root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27096     root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27096     root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27096     root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27096     root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27097     root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095 27097     root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27097     root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27097     root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27097     root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27097     root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27097     root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27098     root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095 27098     root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27098     root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27098     root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27098     root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27098     root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27098     root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27099     root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095 27099     root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27099     root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27099     root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27099     root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27099     root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27099     root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27100     root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095 27100     root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27100     root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27100     root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27100     root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27100     root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27100     root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27101     root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095 27101     root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27101     root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27101     root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27101     root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27101     root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27101     root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27102     root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095 27102     root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27102     root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27102     root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27102     root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27102     root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27102     root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27104     root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095 27104     root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27104     root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27104     root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27104     root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27104     root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27104     root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27186     root  cwd       DIR               0,22       120   10916380 /run/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    container 27095 27186     root    3u     unix 0xffff88000e73d800       0t0   10916404 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    container 27095 27186     root    4u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27186     root    6u     FIFO                8,2       0t0     269609 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stdout.log
    container 27095 27186     root    7u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27186     root    8u     FIFO                8,2       0t0     269610 /var/lib/containerd/io.containerd.runtime.v1.linux/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.stderr.log
    container 27095 27186     root   10u     unix 0xffff8800035db800       0t0   10916385 @/containerd-shim/moby/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/shim.sock@
    [root@176 website-cdn]# kill -9 26304
    [root@176 website-cdn]# kill -9 27095
    [root@176 website-cdn]# lsof | grep "83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571"
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    lsof: no pwd entry for UID 101
    [root@176 website-cdn]# rm -rf /var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/
    rm: 无法删除"/var/lib/docker/containers/83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571/mounts/shm": 设备或资源忙
    [root@176 website-cdn]# ll /var/lib/docker/containers
    总用量 16
    drwx------ 4 root root 4096 11月 12 11:22 01cdd076b16f1749f3a376d61aaf9edc1dafa642437cac545ef21a53f36a3ea8
    drwx------ 4 root root 4096 11月 12 11:22 7542244e5ff788305cf06e5367cf59fda16e06aff81b66083d16f6ad6224ae32
    drwx------ 3 root root 4096 11月 12 11:15 83967693c041acbf8f6467d21bf422fd7ffee29954d2063e0474a63f15b24571
    drwx------ 4 root root 4096 11月 12 11:22 c942777c683fe74ef3a2f161f4a0e4214e82529f22025c6878eb3d3879ad5fc2
    您在 /var/spool/mail/root 中有新邮件
    [root@176 website-cdn]# sudo service docker restart
    Redirecting to /bin/systemctl restart docker.service
    [root@176 website-cdn]# docker ps
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
    [root@176 website-cdn]#

参考：

*  [docker 解决network has active endpoints][docker _network has active endpoints]
 *  [解决Docker不能stop或者kill一个容器方法][Docker_stop_kill]
 *  [解决Docker假死问题(stop或kill不掉容器)][Docker_stop_kill 1]

[docker _network has active endpoints]: https://blog.csdn.net/weixin_30686845/article/details/94908202
[Docker_stop_kill]: https://blog.csdn.net/alxe_made/article/details/89246245
[Docker_stop_kill 1]: https://www.huolg.net/devops/698