kubernetes ceph is locked by other nodes 错误解决方案

快来打我* 2022-06-01 12:17 133阅读 0赞

#### 1.问题描述 ####

pod的状态一直处于ContainerCreating状态

[root@master ~]# kubectl get pods -n wangweidd | grep -v Running
    NAME         READY     STATUS              RESTARTS   AGE
    enncloud-1   0/1       ContainerCreating   0          14d

[root@master ~]# kubectl describe pod enncloud-1 -n wangweidd
    ...
    Events:
      FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason      Message
      --------- --------    -----   ----            -------------   --------    ------      -------
      14d       1m      10013   kubelet, test           Warning     FailedMount MountVolume.SetUp failed for volume "wangweidd.datadir-enncloud-2" : rbd: image wangweidd.CID-516874818ed4.datadir-enncloud-2 is locked by other nodes
      14d       54s     8993    kubelet, test           Warning     FailedMount Unable to mount volumes for pod "enncloud-1_wangweidd(69b763cd-f9a3-11e7-9d36-5254b24cbf5e)": timeout expired waiting for volumes to attach/mount for pod "wangweidd"/"enncloud-1". list of unattached/unmounted volumes=[datadir]
      14d       54s     8993    kubelet, test           Warning     FailedSync  Error syncing pod

有错误的信息

rbd: image wangweidd.CID-516874818ed4.datadir-enncloud-2 is locked by other nodes

#### 2.解决问题 ####

1. 登录任何一个ceph 节点上 
      执行：
      rbd -p test-pool status  enngastest.CID-516874818ed4.datadir-mi-es-pro-1   
      Watchers:
               watcher=10.39.1.95:0/3703494529 client.104824 cookie=2
     查看的信息如下: 
     挂载在10.39.1.95 机器上 
    
    2. rbd  showmapped  查询这台机器上挂载的rbd 对应的设备文件
        rbd showmapped 
        2017-07-13 14:10:15.704627 7f824e7657c0 -1 asok(0x3791960) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-clients/ceph-client.admin.8255.58268576.asok': (2) No such file or directory
       id pool      image                                           snap device    
         0  tenx-pool lixueyang.CID-516874818ed4.datadir-iauc-crm-0   -    /dev/rbd0 
         1  tenx-pool enngastest.CID-516874818ed4.datadir-mi-es-pro-1 -   /dev/rbd1 
         2  tenx-pool ceres-prod.CID-516874818ed4.ceres-etcd-0        -      /dev/rbd2 
    查出
    rbd lock list tenx-pool/kubernetes-dynamic-pvc-d1043bbb-d5e4-11e7-a259-5254b24cbf5e Locker ID Address
    client.811885 kubelet_lock_magic_paas-slave-20-48 10.39.20.48:0/2805197474
    
    解锁 rbd unmap /dev/rbd2

--------------------

#### ceph rbd 扩容 ####

rbd showmapped   
    df -h   查看/dev/rbdx   的大小 
    扩容  
    rbd  resize --size    1024   foo   
    rbd resize --size  1024  liuguanghuif.CID-516874818ed4.datadir-lk-test-1  -p tenx-pool 
    df -Th   查看大小是否变化
    如果文件系统还未增大
    blockdev --getsize64 /dev/rbd4 
    resize2fs /dev/rbd4 
    
    查看大小ok！ 
    
    注意:对于xfs 要在resize 之后执行   xfs_growfs /mnt