kubernetes 创建pod /merged/dev/shm: invalid argument

妖狐艹你老母 2022-05-15 08:10 179阅读 0赞

**问题描述**  
系统

[root@master-47-35 ~]# cat /etc/redhat-release 
    CentOS Linux release 7.5.1804 (Core)

kubelet 创建服务fluentd 创建失败，报`/merged/dev/shm: invalid argument`错误  
相关日志

Warning  Failed  19s               kubelet, slave-47-32  Error: Error response from daemon: mkdir /var/lib/docker/overlay2/38b9c31a3b4b9876fc215126016d03aa9009f90a082d03973aa9e1cb8aad7b44-init/merged/dev/shm: invalid argument
      Warning  Failed  18s               kubelet, slave-47-32  Error: Error response from daemon: mkdir /var/lib/docker/overlay2/ef5371fb7ef7b45997c786a3dc9b270aafdc413455284cf1744c67dbffaca900-init/merged/dev/shm: invalid argument
      Normal   Pulled  5s (x3 over 19s)  kubelet, slave-47-32  Container image "harbor.enncloud.cn/paas/fluentd-elk:v2.2.3-test" already present on machine
      Warning  Failed  5s                kubelet, slave-47-32  Error: Error response from daemon: mkdir /var/lib/docker/overlay2/d471e42ae8ae97b287acef9707c23d06835d25ca2e7d99a48459161f427d4639-init/merged/dev/shm: invalid argument

相关文档[issues/32651][issues_32651]

主要是存储驱动的问题，改为`devicemapper`就搞定了

**解决办法**

修改文件`/etc/docker/daemon.json`

cat /etc/docker/daemon.json
    
    { 
      "registry-mirrors": ["http://579fe187.m.daocloud.io","https://pee6w651.mirror.aliyuncs.com"],
      "insecure-registries": ["harbor.enncloud.cn"],
      "hosts": ["unix:///var/run/docker.sock"],
       "storage-driver": "devicemapper",
      "storage-opts": ["dm.basesize=10G"],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "20m",
        "max-file": "10"
      }
    
    }

修改文件`/etc/sysconfig/docker`

[root@slave-47-32 ~]# cat /etc/sysconfig/docker
    # /etc/sysconfig/docker
    
    # Modify these options if you want to change the way the docker daemon runs
    OPTIONS='--selinux-enabled --signature-verification=false'
    if [ -z "${DOCKER_CERT_PATH}" ]; then
        DOCKER_CERT_PATH=/etc/docker
    fi
    
    # Do not add registries in this file anymore. Use /etc/containers/registries.conf
    # instead. For more information reference the registries.conf(5) man page.
    
    # Location used for temporary files, such as those created by
    # docker load and build operations. Default is /var/lib/docker/tmp
    # Can be overriden by setting the following environment variable.
    # DOCKER_TMPDIR=/var/tmp
    
    # Controls the /etc/cron.daily/docker-logrotate cron job status.
    # To disable, uncomment the line below.
    # LOGROTATE=false
    
    # docker-latest daemon can be used by starting the docker-latest unitfile.
    # To use docker-latest client, uncomment below lines
    #DOCKERBINARY=/usr/bin/docker-latest
    #DOCKERDBINARY=/usr/bin/dockerd-latest
    #DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
    #DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest

去掉启动选择存储驱动的启动参数,修改完成如下

[root@slave-47-32 ~]# cat /usr/lib/systemd/system/docker.service 
    [Unit]
    Description=Docker Application Container Engine
    Documentation=http://docs.docker.com
    After=network.target rhel-push-plugin.socket registries.service
    Wants=docker-storage-setup.service
    Requires=docker-cleanup.timer
    
    [Service]
    Type=notify
    NotifyAccess=main
    EnvironmentFile=-/run/containers/registries.conf
    EnvironmentFile=-/etc/sysconfig/docker
    EnvironmentFile=-/etc/sysconfig/docker-network
    Environment=GOTRACEBACK=crash
    Environment=DOCKER_HTTP_HOST_COMPAT=1
    Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
    ExecStart=/usr/bin/dockerd-current \
              --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \
              --default-runtime=docker-runc \
              --exec-opt native.cgroupdriver=systemd \
              --userland-proxy-path=/usr/libexec/docker/docker-proxy-current \
              --init-path=/usr/libexec/docker/docker-init-current \
              --seccomp-profile=/etc/docker/seccomp.json \
              $OPTIONS \
              $DOCKER_NETWORK_OPTIONS \
              $ADD_REGISTRY \
              $BLOCK_REGISTRY \
              $INSECURE_REGISTRY \
          $REGISTRIES
    ExecReload=/bin/kill -s HUP $MAINPID
    LimitNOFILE=1048576
    LimitNPROC=1048576
    LimitCORE=infinity
    TimeoutStartSec=0
    Restart=on-abnormal
    KillMode=process
    
    [Install]
    WantedBy=multi-user.target

[issues_32651]: https://github.com/moby/moby/issues/32651