swap未禁用导致的k8s NotReady

浅浅的花香味﹌ 2022-09-06 15:25 170阅读 0赞

查看node节点发现node3是NotReady状态

[root@k8s-node01 ~]# kubectl  get nodes
    NAME         STATUS     ROLES                  AGE   VERSION
    k8s-node01   Ready      control-plane,master   28d   v1.21.2
    k8s-node02   Ready      <none>                 28d   v1.21.2
    k8s-node03   NotReady   <none>                 28d   v1.21.2
    k8s-node04   Ready      <none>                 28d   v1.21.2
    k8s-node05   Ready      <none>                 28d   v1.21.2

查看日志发现是swap没关

[root@k8s-node03 ~]# journalctl -f -u kubelet
    -- Logs begin at 日 2021-05-23 00:20:29 CST. --
    8月 10 01:05:08 k8s-node03 kubelet[4408]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
    8月 10 01:05:08 k8s-node03 kubelet[4408]: I0810 01:05:08.774223    4408 server.go:440] "Kubelet version" kubeletVersion="v1.21.2"
    8月 10 01:05:08 k8s-node03 kubelet[4408]: I0810 01:05:08.775361    4408 server.go:851] "Client rotation is on, will bootstrap in background"
    8月 10 01:05:08 k8s-node03 kubelet[4408]: I0810 01:05:08.843965    4408 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
    8月 10 01:05:08 k8s-node03 kubelet[4408]: I0810 01:05:08.848600    4408 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
    8月 10 01:05:09 k8s-node03 kubelet[4408]: I0810 01:05:09.157192    4408 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
    8月 10 01:05:09 k8s-node03 kubelet[4408]: E0810 01:05:09.157586    4408 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename\t\t\t\tType\t\tSize\tUsed\tPriority /dev/dm-1                               partition\t2097148\t1280\t-2]"
    8月 10 01:05:09 k8s-node03 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
    8月 10 01:05:09 k8s-node03 systemd[1]: Unit kubelet.service entered failed state.
    8月 10 01:05:09 k8s-node03 systemd[1]: kubelet.service failed.
    8月 10 01:05:19 k8s-node03 systemd[1]: kubelet.service holdoff time over, scheduling restart.
    8月 10 01:05:19 k8s-node03 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
    8月 10 01:05:19 k8s-node03 systemd[1]: Started kubelet: The Kubernetes Node Agent.
    8月 10 01:05:19 k8s-node03 kubelet[4480]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
    8月 10 01:05:19 k8s-node03 kubelet[4480]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
    8月 10 01:05:19 k8s-node03 kubelet[4480]: I0810 01:05:19.486095    4480 server.go:440] "Kubelet version" kubeletVersion="v1.21.2"
    8月 10 01:05:19 k8s-node03 kubelet[4480]: I0810 01:05:19.486746    4480 server.go:851] "Client rotation is on, will bootstrap in background"
    8月 10 01:05:19 k8s-node03 kubelet[4480]: I0810 01:05:19.517364    4480 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
    8月 10 01:05:19 k8s-node03 kubelet[4480]: I0810 01:05:19.532697    4480 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt
    8月 10 01:05:19 k8s-node03 kubelet[4480]: I0810 01:05:19.782954    4480 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
    8月 10 01:05:19 k8s-node03 kubelet[4480]: E0810 01:05:19.783965    4480 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename\t\t\t\tType\t\tSize\tUsed\tPriority /dev/dm-1                               partition\t2097148\t1280\t-2]"
    8月 10 01:05:19 k8s-node03 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
    8月 10 01:05:19 k8s-node03 systemd[1]: Unit kubelet.service entered failed state.
    8月 10 01:05:19 k8s-node03 systemd[1]: kubelet.service failed.
    8月 10 01:05:29 k8s-node03 systemd[1]: kubelet.service holdoff time over, scheduling restart.
    8月 10 01:05:29 k8s-node03 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
    8月 10 01:05:29 k8s-node03 systemd[1]: Started kubelet: The Kubernetes Node Agent.

关闭swap

[root@k8s-node03 ~]# swapoff -a

再次查看节点状态

\[root@k8s-node01 ~\]\# kubectl get nodes  
NAME STATUS ROLES AGE VERSION  
k8s-node01 Ready control-plane,master 28d v1.21.2  
k8s-node02 Ready 28d v1.21.2  
k8s-node03 Ready 28d v1.21.2  
k8s-node04 Ready 28d v1.21.2  
k8s-node05 Ready 28d v1.21.2