[问题未处理]-docker报错stream copy error- reading from a closed fifo导致服务器oom并cpu飙升后无响应

素颜马尾好姑娘i 2022-12-11 06:26 173阅读 0赞

## 问题及原因 ##

告警触发 服务器探活失败 最后宕机告警。

![dc13ab9c42c8c6e787bd04bd3f5f8fd5.png][]

![81cf8d66e22b9be5a2097b6b4cfaa9d6.png][]

![79d3f7fd3651a3b8f4a3ca985440d3e0.png][]

##### 现象：系统oom cpu负载爆高 服务器连接不上 读写等爆炸 阿里云控制台vnc进入也显示超时 #####

总共想通业务的机子有5台 其余4台服务器都没有问题。

![0f910b4eb814ec3480fb1a60564d31a3.png][]

##### 关键错误日志 #####

Sep 27 08:37:54 docker005 dockerd: time="2020-09-27T08:37:54.525450264+08:00" level=error msg="stream copy error: reading from a closed fifo"
    Sep 27 08:37:54 docker005 dockerd: time="2020-09-27T08:37:54.525450309+08:00" level=error msg="stream copy error: reading from a closed fifo"
    Sep 27 08:37:55 docker005 dockerd: time="2020-09-27T08:37:55.652626099+08:00" level=warning msg="Health check for container 9b351669f228259d22736b4a1097ddbfc26225ae2978a702a4b5149ff6e30f7d error: context deadline exceeded: unknown"
    Sep 27 08:39:03 docker005 dockerd: time="2020-09-27T08:39:03.580218917+08:00" level=warning msg="Health check for container 9b351669f228259d22736b4a1097ddbfc26225ae2978a702a4b5149ff6e30f7d error: context deadline exceeded: unknown"
    Sep 27 08:39:03 docker005 dockerd: time="2020-09-27T08:39:03.580277907+08:00" level=error msg="exit event" container=9b351669f228259d22736b4a1097ddbfc26225ae2978a702a4b5149ff6e30f7d error="no such process" module=libcontainerd namespace=moby process=fa63ec2efcfb6f5d03069b5f4775e08da32274272b1f35bc646dbfd84f2d70de
    Sep 27 08:39:36 docker005 dockerd: time="2020-09-27T08:39:36.605897377+08:00" level=error msg="stream copy error: reading from a closed fifo"
    Sep 27 08:39:36 docker005 dockerd: time="2020-09-27T08:39:36.605898957+08:00" level=error msg="stream copy error: reading from a closed fifo"

暂时未确认引起宕机的具体原因 猜测是绑定在这台机子上的定时任务引起的 不排除阿里云的问题。

类似issues

https://github.com/docker/for-linux/issues/402

[dc13ab9c42c8c6e787bd04bd3f5f8fd5.png]: /images/20221123/b5ba2fab17804be8a555b26202e50802.png
[81cf8d66e22b9be5a2097b6b4cfaa9d6.png]: /images/20221123/9e6c4bd662014b989f20b8d88cbd8a86.png
[79d3f7fd3651a3b8f4a3ca985440d3e0.png]: /images/20221123/242911392f5d424f95fba20be3860708.png
[0f910b4eb814ec3480fb1a60564d31a3.png]: /images/20221123/3f214c12e6c74c1cbd1ec833d684c64d.png