[K8S] Pod的健康检查

本是古典何须时尚 2022-09-06 15:20 176阅读 0赞

健康检查由kubelet操作，包括  
1）存活检查(livenessProbe): 如果检查失败 -> 杀死容器 并根据Pod的restartPolicy来操作  
2）就绪检查(readinessProbe): 如果检查失败 -> 把Pod从service endpoints中剔除

@重启策略(restartPolicy)  
Never：容器终止退出不重启容器  
**Alway**s：容器终止退出后重启容器(**默认**策略)  
OnFailure：容器异常退出(退出状态码非0)才重启容器

@检查方法  
exec: 执行Shell命令 -> 返回状态码是0为成功  
httpGet: 发送HTTP请求 -> 返回200-400范围状态码为成功  
tcpSocket: 发起TCP Socket建立成功

【例】httpGet

生成一个deployment的yaml，kubectl create deployment test-probe --image=nginx --dry-run=client -o yaml > test-probe.yaml，编辑test-probe.yaml的内容

apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: test-probe
      name: test-probe
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: test-probe
      strategy: {}
      template:
        metadata:
          labels:
            app: test-probe
        spec:
          containers:
          - image: nginx
            name: nginx
            resources: {}
            livenessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 30
              periodSeconds: 10
            readinessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 30
              periodSeconds: 10

说明：

livenessProbe和readinessProbe与containers -> image 对齐  
initialDelaySeconds: 30 \#启动容器后30秒健康检查  
periodSeconds: 10 \#以后每间隔10秒检查一次

创建deployment，可以查看到3个pod

[root@k8s-master ~]# kubectl apply -f test-probe.yaml
    deployment.apps/test-probe created
    [root@k8s-master ~]#
    [root@k8s-master ~]# kubectl get pod
    NAME                         READY   STATUS    RESTARTS   AGE
    test-probe-766bcd6cf-2qpz4   1/1     Running   0          130m
    test-probe-766bcd6cf-p9f8p   1/1     Running   0          130m
    test-probe-766bcd6cf-tx52m   1/1     Running   0          130m
    [root@k8s-master ~]#

例如查看第一个pod的日志，kubectl logs test-probe-766bcd6cf-2qpz4 -f，每隔10秒会打印两条日志，分别对应存活检查和就绪检查

[root@k8s-master ~]# kubectl logs test-probe-766bcd6cf-2qpz4 -f
    ……
    192.168.231.123 - - [21/Aug/2021:06:07:45 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "kube-probe/1.20" "-"
    192.168.231.123 - - [21/Aug/2021:06:07:47 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "kube-probe/1.20" "-"
    192.168.231.123 - - [21/Aug/2021:06:07:55 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "kube-probe/1.20" "-"
    192.168.231.123 - - [21/Aug/2021:06:07:57 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "kube-probe/1.20" "-"
    192.168.231.123 - - [21/Aug/2021:06:08:05 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "kube-probe/1.20" "-"
    192.168.231.123 - - [21/Aug/2021:06:08:07 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "kube-probe/1.20" "-"
    192.168.231.123 - - [21/Aug/2021:06:08:15 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "kube-probe/1.20" "-"
    192.168.231.123 - - [21/Aug/2021:06:08:17 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "kube-probe/1.20" "-"
    ……

@存活检查测试  
第一个窗口执行 kubectl get pods -w  
第二个窗口进入pod，删掉主页

[root@k8s-master ~]# kubectl exec -it test-probe-766bcd6cf-2qpz4 -- bash
    root@test-probe-766bcd6cf-2qpz4:/# cd /usr/share/nginx/html/
    root@test-probe-766bcd6cf-2qpz4:/usr/share/nginx/html# ls
    50x.html  index.html
    root@test-probe-766bcd6cf-2qpz4:/usr/share/nginx/html# rm -rf index.html
    root@test-probe-766bcd6cf-2qpz4:/usr/share/nginx/html# command terminated with exit code 137
    [root@k8s-master ~]#

在第一个窗口可以观察到该pod被重启了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70][]

kubectl describe pod test-probe-766bcd6cf-2qpz4 可以看到检测失败的记录，可以注意到，存活探测和就绪探测都失败了，然后重新拉起了该pod

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70 1][]

@就绪检查测试  
生成一个svc的yaml，kubectl expose deployment test-probe --port=90 --target-port=80 --type=NodePort --dry-run=client -o yaml > test-probe-svc.yaml，编辑test-probe-svc.yaml的内容

apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: test-probe
      name: test-probe
    spec:
      ports:
      - port: 90
        protocol: TCP
        targetPort: 80
      selector:
        app: test-probe
      type: NodePort

查看endpoint

[root@k8s-master ~]# kubectl apply -f test-probe-svc.yaml
    service/test-probe created
    [root@k8s-master ~]# kubectl get svc
    NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
    kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        26d
    test-probe   NodePort    10.110.173.120   <none>        90:30070/TCP   9s
    [root@k8s-master ~]# kubectl get ep
    NAME         ENDPOINTS                                           AGE
    kubernetes   192.168.231.121:6443                                26d
    test-probe   10.244.169.141:80,10.244.36.65:80,10.244.36.67:80   12s
    [root@k8s-master ~]#

同样打开两个窗  
第一个窗口执行 kubectl get ep -w  
第二个窗口进入pod，删掉主页

在第一个窗口可以观察到endpoint中，这个节点被剔除，稍后又加回去了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70 2][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70]: /images/20220829/39323afd39d94a119f6503872e4bcaca.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70 1]: /images/20220829/f5c8cc0f43174da8bd12cf264a7b5b56.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70 2]: /images/20220829/e45a8bb671e840ad995758d822cca2d3.png