kubenetes配置vip的一些问题处理

矫情吗；* 2022-11-07 11:36 130阅读 0赞

按照[kubeadm部署k8s高可用集群][kubeadm_k8s], 执行`kubeadm init --config kubeadm-config.yaml --v=5`,初始化失败，添加`--v=5`是为了看更详细的日志信息  
[kubernetes之kubeadm 安装kubernetes 高可用集群][kubernetes_kubeadm _kubernetes]  
根据指示，跟进日志

Unfortunately, an error has occurred:
                    timed out waiting for the condition
    
            This error is likely caused by:
                    - The kubelet is not running
                    - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
    
            If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
                    - 'systemctl status kubelet'
                    - 'journalctl -xeu kubelet'
    
            Additionally, a control plane component may have crashed or exited when started by the container runtime.
            To troubleshoot, list all containers using your preferred container runtimes CLI.
    
            Here is one example how you may list all Kubernetes containers running in docker:
                    - 'docker ps -a | grep kube | grep -v pause'
                    Once you have found the failing container, you can inspect its logs with:
                    - 'docker logs CONTAINERID'
     couldn't initialize a Kubernetes cluster
    k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:114
    k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
    k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
    k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
    k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:147
    k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826
    k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
    k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
    k8s.io/kubernetes/cmd/kubeadm/app.Run
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
    main.main
            _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
    runtime.main
            /usr/local/go/src/runtime/proc.go:203
    runtime.goexit
            /usr/local/go/src/runtime/asm_amd64.s:1357
    error execution phase wait-control-plane
    k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
    k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
    k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
    k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:147
    k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826
    k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914
    k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
    k8s.io/kubernetes/cmd/kubeadm/app.Run
            /workspace/anago-v1.18.9-rc.0.79+8147d851af540a/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
    main.main
            _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
    runtime.main
            /usr/local/go/src/runtime/proc.go:203
    runtime.goexit
            /usr/local/go/src/runtime/asm_amd64.s:1357

执行`journalctl -xeu kubelet`,跟进日志，异常如下:

Mar 11 17:59:33 master1 kubelet[6879]: I0311 17:59:33.594045    6879 kubelet_node_status.go:294] Setting node annotation to enable volume controller att
    Mar 11 17:59:33 master1 kubelet[6879]: E0311 17:59:33.613681    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:33 master1 kubelet[6879]: I0311 17:59:33.621491    6879 kubelet_node_status.go:70] Attempting to register node master1
    Mar 11 17:59:33 master1 kubelet[6879]: E0311 17:59:33.621978    6879 kubelet_node_status.go:92] Unable to register node "master1" with API server: Post 
    Mar 11 17:59:33 master1 kubelet[6879]: E0311 17:59:33.713811    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:33 master1 kubelet[6879]: E0311 17:59:33.813955    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:33 master1 kubelet[6879]: E0311 17:59:33.914068    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.014206    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.114347    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.214460    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.231304    6879 kubelet.go:2190] Container runtime network not ready: NetworkReady=false reason:Net
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.314587    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.414701    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.514826    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.614937    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.715068    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.815196    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:34 master1 kubelet[6879]: E0311 17:59:34.915320    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.015436    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.115570    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.215691    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.315815    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.415941    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.516064    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.616203    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.716318    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.816447    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:35 master1 kubelet[6879]: E0311 17:59:35.916578    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:36 master1 kubelet[6879]: E0311 17:59:36.016691    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:36 master1 kubelet[6879]: E0311 17:59:36.116824    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:36 master1 kubelet[6879]: E0311 17:59:36.216938    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:36 master1 kubelet[6879]: E0311 17:59:36.317069    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:36 master1 kubelet[6879]: E0311 17:59:36.417197    6879 kubelet.go:2270] node "master1" not found
    Mar 11 17:59:36 master1 kubelet[6879]: W0311 17:59:36.419041    6879 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d

查看[\[kubernates\]Unable to update cni config: no networks found in /etc/cni/net.d][kubernates_Unable to update cni config_ no networks found in _etc_cni_net.d]，执行`kubectl get nodes`，提示异常

Unable to connect to the server: http: server gave HTTP response to HTTPS client

执行`journalctl -f -u kubelet.service`可以看到稍微详细一点的日志

Mar 11 18:05:19 master1 kubelet[6879]: E0311 18:05:19.910903    6879 kubelet.go:2190] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
    Mar 11 18:05:19 master1 kubelet[6879]: E0311 18:05:19.954870    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.055052    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.155194    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.255314    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.355455    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.455611    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.555746    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.655873    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.756003    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.856149    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:20 master1 kubelet[6879]: E0311 18:05:20.956303    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:21 master1 kubelet[6879]: E0311 18:05:21.056424    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:21 master1 kubelet[6879]: E0311 18:05:21.156542    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:21 master1 kubelet[6879]: E0311 18:05:21.256671    6879 kubelet.go:2270] node "master1" not found
    Mar 11 18:05:21 master1 kubelet[6879]: E0311 18:05:21.321245    6879 controller.go:136] failed to ensure node lease exists, will retry in 7s, error: Get https://10.128.4.18:16443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/master1?timeout=10s: http: server gave HTTP response to HTTPS client

执行`docker ps -a | grep kube | grep -v pause`  
在华为云，还需要在子网中配置vpc，将这个ip与keepalived的虚拟ip进行绑定  
![1][]  
通过上面的配置解决了VIP的问题，但是还有问题，就应该顺着排查haproxy了,还是从下方这个异常着手，[haproxy代理https配置方法][haproxy_https]

Unable to connect to the server: http: server gave HTTP response to HTTPS client

既然api-server配置的证书，那么这里采用haproxy只提供代理，由web服务器配置https更合理了。对比[kubeadm部署k8s高可用集群][kubeadm_k8s]发现haproxy的配置多了

stick-table type ip size 200k expire 30m
     stick on src

[haproxy实现会话保持(2):stick table][haproxy_2_stick table]  
完整的`/etc/haproxy/haproxy.cfg`配置如下

#---------------------------------------------------------------------
    # Example configuration for a possible web application.  See the
    # full configuration options online.
    #
    #   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
    #
    #---------------------------------------------------------------------
    
    #---------------------------------------------------------------------
    # Global settings
    #---------------------------------------------------------------------
    global
        # to have these messages end up in /var/log/haproxy.log you will
        # need to:
        #
        # 1) configure syslog to accept network log events.  This is done
        #    by adding the '-r' option to the SYSLOGD_OPTIONS in
        #    /etc/sysconfig/syslog
        #
        # 2) configure local2 events to go to the /var/log/haproxy.log
        #   file. A line like the following can be added to
        #   /etc/sysconfig/syslog
        #
        #    local2.*                       /var/log/haproxy.log
        #
        log         127.0.0.1 local2
    
        chroot      /var/lib/haproxy
        pidfile     /var/run/haproxy.pid
        maxconn     4000
        user        haproxy
        group       haproxy
        daemon
    
        # turn on stats unix socket
        stats socket /var/lib/haproxy/stats
    
    #---------------------------------------------------------------------
    # common defaults that all the 'listen' and 'backend' sections will
    # use if not designated in their block
    #---------------------------------------------------------------------
    defaults
        mode                    http
        log                     global
        option                  httplog
        option                  dontlognull
        option http-server-close
        option forwardfor       except 127.0.0.0/8
        option                  redispatch
        retries                 3
        timeout http-request    10s
        timeout queue           1m
        timeout connect         10s
        timeout client          1m
        timeout server          1m
        timeout http-keep-alive 10s
        timeout check           10s
        maxconn                 3000
    
    #---------------------------------------------------------------------
    # main frontend which proxys to the backends
    #---------------------------------------------------------------------
    frontend  kubernetes-apiserver
        mode                        tcp
        bind                        *:16443
        option                      tcplog
        default_backend             kubernetes-apiserver
    
    #---------------------------------------------------------------------
    # static backend for serving up images, stylesheets and such
    #---------------------------------------------------------------------
    listen stats
        bind            *:1080
        stats auth      admin:awesomePassword
        stats refresh   5s
        stats realm     HAProxy\ Statistics
        stats uri       /admin?stats
    
    #---------------------------------------------------------------------
    # round robin balancing between the various backends
    #---------------------------------------------------------------------
    backend kubernetes-apiserver
        mode        tcp
        balance     roundrobin
        stick-table type ip size 200k expire 30m
        stick on src
        server  master1  10.128.4.164:6443 check
        server  master2  10.128.4.251:6443 check
        server  master3  10.128.4.211:6443 check

[kubeadm_k8s]: https://www.cnblogs.com/lfl17718347843/p/13417304.html
[kubernetes_kubeadm _kubernetes]: https://www.cnblogs.com/cptao/p/10912644.html
[kubernates_Unable to update cni config_ no networks found in _etc_cni_net.d]: https://blog.csdn.net/qq_23146469/article/details/102486500
[1]: /images/20221023/038395ee9caf448aad18113351cd25b3.png
[haproxy_https]: http://www.mamicode.com/info-detail-1539510.html
[haproxy_2_stick table]: https://www.cnblogs.com/f-ck-need-u/p/8558514.html