3. 二进制方式搭建k8s集群

谁借莪１个温暖的怀抱￠ 2023-06-13 11:08 14阅读 0赞

二进制方式搭建集群优缺点：

优点：
        易于维护，灵活，升级方便
        
    缺点：
        安装复杂，没有文档

将master1作为deploy节点，未指定节点时默认在master1上进行操作。

建议deploy节点与其它节点配置ssh免密登录，配置过程参考：[批量实现SSH免密登录][SSH] 。

--------------------

### 环境准备 ###

环境准备工作请在所有节点进行。

*  主机说明：

<table> 
 <thead> 
  <tr> 
   <th>系统</th> 
   <th>ip</th> 
   <th>角色</th> 
   <th>cpu</th> 
   <th>内存</th> 
   <th>hostname</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>CentOS 7.7</td> 
   <td>192.168.100.128</td> 
   <td>master、deploy</td> 
   <td>&gt;=2</td> 
   <td>&gt;=2G</td> 
   <td>master1</td> 
  </tr> 
  <tr> 
   <td>CentOS 7.7</td> 
   <td>192.168.100.129</td> 
   <td>master</td> 
   <td>&gt;=2</td> 
   <td>&gt;=2G</td> 
   <td>master2</td> 
  </tr> 
  <tr> 
   <td>CentOS 7.7</td> 
   <td>192.168.100.130</td> 
   <td>master</td> 
   <td>&gt;=2</td> 
   <td>&gt;=2G</td> 
   <td>master3</td> 
  </tr> 
  <tr> 
   <td>CentOS 7.7</td> 
   <td>192.168.100.131</td> 
   <td>node</td> 
   <td>&gt;=2</td> 
   <td>&gt;=2G</td> 
   <td>node1</td> 
  </tr> 
  <tr> 
   <td>CentOS 7.7</td> 
   <td>192.168.100.132</td> 
   <td>node</td> 
   <td>&gt;=2</td> 
   <td>&gt;=2G</td> 
   <td>node2</td> 
  </tr> 
  <tr> 
   <td>CentOS 7.7</td> 
   <td>192.168.100.133</td> 
   <td>node</td> 
   <td>&gt;=2</td> 
   <td>&gt;=2G</td> 
   <td>node3</td> 
  </tr> 
 </tbody> 
</table>

*  设置主机名：

每个节点的主机名必须不一样，且保证节点之间可以通过hostname互相访问。

以master1为例，

# mkdir /software
    
    # hostnamectl set-hostname master1
    
    # vim /etc/hosts
    
    192.168.100.128 master1
    192.168.100.129 master2
    192.168.100.130 master3
    192.168.100.131 node1
    192.168.100.132 node2
    192.168.100.133 node3

*  安装依赖包：

# yum update -y
    
    # yum install -y curl git iptables conntrack ipvsadm ipset jq sysstat libseccomp

*  关闭防火墙、selinux和swap，重置iptables：

# systemctl stop firewalld && systemctl disable firewalld
    
    # sed -i 's/=enforcing/=disabled/g' /etc/selinux/config && setenforce 0
    
    # iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT
    
    # swapoff -a
    
    # sed -i '/swap/s/^$.*$$/#\1/g' /etc/fstab
    
    # systemctl stop dnsmasq && systemctl disable dnsmasq #否则可能导致docker容器无法解析域名

*  系统参数设置：

# cat > /etc/sysctl.d/kubernetes.conf <<EOF
    net.bridge.bridge-nf-call-iptables=1
    net.bridge.bridge-nf-call-ip6tables=1
    net.ipv4.ip_forward=1
    vm.swappiness=0
    vm.overcommit_memory=1
    vm.panic_on_oom=0
    fs.inotify.max_user_watches=89100
    EOF
    
    # sysctl -p /etc/sysctl.d/kubernetes.conf

`sysctl -p /etc/sysctl.d/kubernetes.conf`这一步如果报错：

sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
    sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory

请这样执行：

# modprobe br_netfilter
    
    # sysctl -p /etc/sysctl.d/kubernetes.conf

*  安装docker：

# curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker.repo
    
    # yum makecache fast
    
    # yum install -y docker-ce
    
    # systemctl start docker && systemctl enable docker
    
    # cat <<EOF > /etc/docker/daemon.json
    { 
     "exec-opts":["native.cgroupdriver=cgroupfs"]
    }
    EOF
    
    # curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
    
    # systemctl restart docker

*  下载二进制文件（deploy节点）：

官方下载地址： https://github.com/kubernetes/kubernetes/releases

百度云地址（推荐）: https://pan.baidu.com/s/1j27LqsSXeNXzu-HtWC6CPg 提取码: ujpo

将下载的tar包传至deploy节点的`/software`目录下，

# cd /software
    
    # tar xf kubernetes-bin-1.14.0.tar.gz
    
    # cd kubernetes-bin-1.14.0/
    
    # tree .
    .
    ├── master
    │   ├── etcd
    │   ├── etcdctl
    │   ├── kubeadm
    │   ├── kube-apiserver
    │   ├── kube-controller-manager
    │   ├── kubectl
    │   └── kube-scheduler
    └── worker
        ├── kubelet
        └── kube-proxy
    
    2 directories, 9 files

分发文件到其它节点，

# ssh root@master1 "mkdir -p /opt/kubernetes/bin"
    
    # ssh root@master2 "mkdir -p /opt/kubernetes/bin"
    
    # ssh root@master3 "mkdir -p /opt/kubernetes/bin"
    
    # ssh root@node1 "mkdir -p /opt/kubernetes/bin"
    
    # ssh root@node2 "mkdir -p /opt/kubernetes/bin"
    
    # ssh root@node3 "mkdir -p /opt/kubernetes/bin"
    
    # scp master/* master1:/opt/kubernetes/bin/
    
    # scp master/* master2:/opt/kubernetes/bin/
    
    # scp master/* master3:/opt/kubernetes/bin/
    
    # scp worker/* node1:/opt/kubernetes/bin/
    
    # scp worker/* node2:/opt/kubernetes/bin/
    
    # scp worker/* node3:/opt/kubernetes/bin/

设置PATH，

# ssh root@master1 "echo 'PATH=/opt/kubernetes/bin:$PATH' >> ~/.bashrc"
    
    # ssh root@master2 "echo 'PATH=/opt/kubernetes/bin:$PATH' >> ~/.bashrc"
    
    # ssh root@master3 "echo 'PATH=/opt/kubernetes/bin:$PATH' >> ~/.bashrc"
    
    # ssh root@node1 "echo 'PATH=/opt/kubernetes/bin:$PATH' >> ~/.bashrc"
    
    # ssh root@node2 "echo 'PATH=/opt/kubernetes/bin:$PATH' >> ~/.bashrc"
    
    # ssh root@node3 "echo 'PATH=/opt/kubernetes/bin:$PATH' >> ~/.bashrc"
    
    # ssh root@master1 "source ~/.bashrc"
    
    # ssh root@master2 "source ~/.bashrc"
    
    # ssh root@master3 "source ~/.bashrc"
    
    # ssh root@node1 "source ~/.bashrc"
    
    # ssh root@node2 "source ~/.bashrc"
    
    # ssh root@node3 "source ~/.bashrc"

*  配置文件（deploy节点）：

git压缩包下载：https://pan.baidu.com/s/1\_jntuO\_LunAV\_NvnbV4NIw 提取码：j2a7

# cd /software && git clone https://git.imooc.com/LZXLINUX/kubernetes-ha-binary.git
    
    # cd kubernetes-ha-binary/ && ls
    addons  configs  global-config.properties  init.sh  LICENSE  pki  README.md  services

文件说明：

addons      kubernetes的插件，比如calico、coredns和dashboard
    
    configs     包含了部署集群过程中用到的配置文件及脚本
    
    pki         各个组件的认证授权相关证书配置
    
    services    所有的kubernetes服务(service)配置文件
    
    global-configs.properties   全局配置，包含各种易变的配置内容
    
    init.sh     初始化脚本，配置好global-config之后，会自动生成所有配置文件

# vim global-config.properties

#3个master节点的ip
    MASTER_0_IP=192.168.100.128
    MASTER_1_IP=192.168.100.129
    MASTER_2_IP=192.168.100.130
    
    #3个master节点的hostname
    MASTER_0_HOSTNAME=master1
    MASTER_1_HOSTNAME=master2
    MASTER_2_HOSTNAME=master3
    
    #api-server的高可用虚拟ip（建议为同网段ip）
    MASTER_VIP=192.168.100.188
    
    #keepalived用到的网卡接口名，一般是eth0
    VIP_IF=ens33
    
    #worker节点的ip列表
    WORKER_IPS=192.168.100.131,192.168.100.132,192.168.100.133
    
    #kubernetes服务ip网段
    SERVICE_CIDR=10.254.0.0/16
    
    #kubernetes的默认服务ip，一般是cidr的第一个
    KUBERNETES_SVC_IP=10.254.0.1
    
    #dns服务的ip地址，一般是cidr的第二个
    CLUSTER_DNS=10.254.0.2
    
    #pod网段
    POD_CIDR=172.10.0.0/16
    
    #NodePort的取值范围
    NODE_PORT_RANGE=8400-8900

# ./init.sh
    
    ====替换变量列表====
    MASTER_0_IP=192.168.100.128
    MASTER_1_IP=192.168.100.129
    MASTER_2_IP=192.168.100.130
    MASTER_0_HOSTNAME=master1
    MASTER_1_HOSTNAME=master2
    MASTER_2_HOSTNAME=master3
    MASTER_VIP=192.168.100.188
    VIP_IF=ens33
    WORKER_IPS=192.168.100.131,192.168.100.132,192.168.100.133
    SERVICE_CIDR=10.254.0.0/16
    KUBERNETES_SVC_IP=10.254.0.1
    CLUSTER_DNS=10.254.0.2
    POD_CIDR=172.10.0.0/16
    NODE_PORT_RANGE=8400-8900
    
    ====替换证书配置文件====
    pki/admin/admin-csr.json
    pki/apiserver/kubernetes-csr.json
    pki/ca-config.json
    pki/ca-csr.json
    pki/controller-manager/controller-manager-csr.json
    pki/etcd/etcd-csr.json
    pki/proxy/kube-proxy-csr.json
    pki/scheduler/scheduler-csr.json
    
    ====替换配置文件====
    configs/check-apiserver.sh
    configs/download-images.sh
    configs/keepalived-backup.conf
    configs/keepalived-master.conf
    configs/kubelet.config.json
    configs/kube-proxy.config.yaml
    addons/calico-rbac-kdd.yaml
    addons/calico.yaml
    addons/coredns.yaml
    addons/dashboard-all.yaml
    worker-192.168.100.131/kubelet.config.json
    worker-192.168.100.131/kubelet.service
    worker-192.168.100.131/kube-proxy.config.yaml
    worker-192.168.100.132/kubelet.config.json
    worker-192.168.100.132/kubelet.service
    worker-192.168.100.132/kube-proxy.config.yaml
    worker-192.168.100.133/kubelet.config.json
    worker-192.168.100.133/kubelet.service
    worker-192.168.100.133/kube-proxy.config.yaml
    
    ====替换service文件====
    192.168.100.128/services/etcd.service
    192.168.100.128/services/kube-apiserver.service
    192.168.100.128/services/kube-controller-manager.service
    192.168.100.128/services/kubelet.service
    192.168.100.128/services/kube-proxy.service
    192.168.100.128/services/kube-scheduler.service
    192.168.100.129/services/etcd.service
    192.168.100.129/services/kube-apiserver.service
    192.168.100.129/services/kube-controller-manager.service
    192.168.100.129/services/kubelet.service
    192.168.100.129/services/kube-proxy.service
    192.168.100.129/services/kube-scheduler.service
    192.168.100.130/services/etcd.service
    192.168.100.130/services/kube-apiserver.service
    192.168.100.130/services/kube-controller-manager.service
    192.168.100.130/services/kubelet.service
    192.168.100.130/services/kube-proxy.service
    192.168.100.130/services/kube-scheduler.service
    services/etcd.service
    services/kube-apiserver.service
    services/kube-controller-manager.service
    services/kubelet.service
    services/kube-proxy.service
    services/kube-scheduler.service
    配置生成成功，位置: /home/kubernetes-ha-binary/target
    
    # find target/ -type f
    
    target/pki/admin/admin-csr.json
    target/pki/apiserver/kubernetes-csr.json
    target/pki/ca-config.json
    target/pki/ca-csr.json
    target/pki/controller-manager/controller-manager-csr.json
    target/pki/etcd/etcd-csr.json
    target/pki/proxy/kube-proxy-csr.json
    target/pki/scheduler/scheduler-csr.json
    target/configs/check-apiserver.sh
    target/configs/download-images.sh
    target/configs/keepalived-backup.conf
    target/configs/keepalived-master.conf
    target/configs/kube-proxy.config.yaml
    target/configs/kubelet.config.json
    target/addons/calico-rbac-kdd.yaml
    target/addons/calico.yaml
    target/addons/coredns.yaml
    target/addons/dashboard-all.yaml
    target/services/etcd.service
    target/services/kube-apiserver.service
    target/services/kube-controller-manager.service
    target/services/kube-proxy.service
    target/services/kube-scheduler.service
    target/services/kubelet.service
    target/worker-192.168.100.131/kubelet.config.json
    target/worker-192.168.100.131/kubelet.service
    target/worker-192.168.100.131/kube-proxy.config.yaml
    target/worker-192.168.100.132/kubelet.config.json
    target/worker-192.168.100.132/kubelet.service
    target/worker-192.168.100.132/kube-proxy.config.yaml
    target/worker-192.168.100.133/kubelet.config.json
    target/worker-192.168.100.133/kubelet.service
    target/worker-192.168.100.133/kube-proxy.config.yaml
    target/192.168.100.128/services/etcd.service
    target/192.168.100.128/services/kube-apiserver.service
    target/192.168.100.128/services/kube-controller-manager.service
    target/192.168.100.128/services/kube-proxy.service
    target/192.168.100.128/services/kube-scheduler.service
    target/192.168.100.128/services/kubelet.service
    target/192.168.100.129/services/etcd.service
    target/192.168.100.129/services/kube-apiserver.service
    target/192.168.100.129/services/kube-controller-manager.service
    target/192.168.100.129/services/kube-proxy.service
    target/192.168.100.129/services/kube-scheduler.service
    target/192.168.100.129/services/kubelet.service
    target/192.168.100.130/services/etcd.service
    target/192.168.100.130/services/kube-apiserver.service
    target/192.168.100.130/services/kube-controller-manager.service
    target/192.168.100.130/services/kube-proxy.service
    target/192.168.100.130/services/kube-scheduler.service
    target/192.168.100.130/services/kubelet.service

--------------------

### 搭建高可用集群 ###

*  生成CA证书：

cfssl是非常好用的CA工具，我们用它来生成证书和秘钥文件。

# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl
    
    # wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson
    
    # chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
    
    # cfssl version

根证书是集群所有节点共享的，只需要创建一个 CA 证书，后续创建的所有证书都由它签名。

# cd /software/kubernetes-ha-binary/target/pki/
    
    # cfssl gencert -initca ca-csr.json | cfssljson -bare ca

分发到每个master节点，

# ssh root@master1 "mkdir -p /etc/kubernetes/pki"
    
    # ssh root@master2 "mkdir -p /etc/kubernetes/pki"
    
    # ssh root@master3 "mkdir -p /etc/kubernetes/pki"
    
    # scp ca*.pem master1:/etc/kubernetes/pki/
    
    # scp ca*.pem master2:/etc/kubernetes/pki/
    
    # scp ca*.pem master3:/etc/kubernetes/pki/

*  部署etcd集群：

如果是从百度云网盘下载的二进制文件可以跳过下载etcd，否则需要先下载etcd。

# wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz

生成证书、私钥，并分发到每个master节点，

# cd /software/kubernetes-ha-binary/target/pki/etcd/
    
    # cfssl gencert -ca=../ca.pem \
      -ca-key=../ca-key.pem \
      -config=../ca-config.json \
      -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
        
    # scp etcd*.pem master1:/etc/kubernetes/pki/
    
    # scp etcd*.pem master2:/etc/kubernetes/pki/
    
    # scp etcd*.pem master3:/etc/kubernetes/pki/

分发etcd服务文件到每个master节点，并创建数据目录，

# cd /software/kubernetes-ha-binary/
    
    # scp target/192.168.100.128/services/etcd.service master1:/etc/systemd/system/
    
    # scp target/192.168.100.129/services/etcd.service master2:/etc/systemd/system/
    
    # scp target/192.168.100.130/services/etcd.service master3:/etc/systemd/system/
    
    # ssh root@master1 "mkdir /var/lib/etcd"
    
    # ssh root@master2 "mkdir /var/lib/etcd"
    
    # ssh root@master3 "mkdir /var/lib/etcd"

master节点启动etcd服务，

# ssh root@master1 "systemctl daemon-reload && systemctl enable etcd && systemctl start etcd"
    
    # ssh root@master2 "systemctl daemon-reload && systemctl enable etcd && systemctl start etcd"
    
    # ssh root@master3 "systemctl daemon-reload && systemctl enable etcd && systemctl start etcd"

# systemctl status etcd
    
    # journalctl -f -u etcd #查看日志

*  部署api-server：

生成证书、私钥，并分发到每个master节点，

# cd /software/kubernetes-ha-binary/target/pki/apiserver/
    
    # cfssl gencert -ca=../ca.pem \
      -ca-key=../ca-key.pem \
      -config=../ca-config.json \
      -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
      
    # scp kubernetes*.pem master1:/etc/kubernetes/pki/
    
    # scp kubernetes*.pem master2:/etc/kubernetes/pki/
    
    # scp kubernetes*.pem master3:/etc/kubernetes/pki/

分发api-server服务文件到每个master节点，并创建日志目录，

# cd /software/kubernetes-ha-binary/
    
    # scp target/192.168.100.128/services/kube-apiserver.service master1:/etc/systemd/system/
    
    # scp target/192.168.100.129/services/kube-apiserver.service master2:/etc/systemd/system/
    
    # scp target/192.168.100.130/services/kube-apiserver.service master3:/etc/systemd/system/
    
    # ssh root@master1 "mkdir /var/log/kubernetes"
    
    # ssh root@master2 "mkdir /var/log/kubernetes"
    
    # ssh root@master3 "mkdir /var/log/kubernetes"

master节点启动api-server，

# ssh root@master1 "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl start kube-apiserver"
    
    # ssh root@master2 "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl start kube-apiserver"
    
    # ssh root@master3 "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl start kube-apiserver"

# systemctl status kube-apiserver
    
    # journalctl -f -u kube-apiserver
    
    # netstat -lntp |grep 6443

*  部署keepalived：

3个master节点安装keepalived，保证master节点的api server进程高可用。

> 注意：云服务器一般不支持自定义虚拟ip，请跳过安装keepalived。高可用可以使用云商的负载均衡服务（比如阿里云的SLB），把backends设置成你的3个master节点，然后虚拟ip就配置成负载均衡的内网ip即可。

安装keepalived，

# ssh root@master1 "yum install -y keepalived"
    
    # ssh root@master2 "yum install -y keepalived"
    
    # ssh root@master3 "yum install -y keepalived"

分发keepalived配置文件到每个master节点，

# cd /software/kubernetes-ha-binary/
    
    # scp target/configs/keepalived-master.conf master1:/etc/keepalived/keepalived.conf
    
    # scp target/configs/keepalived-backup.conf master2:/etc/keepalived/keepalived.conf
    
    # scp target/configs/keepalived-backup.conf master3:/etc/keepalived/keepalived.conf
    
    # scp target/configs/check-apiserver.sh master1:/etc/keepalived/
    
    # scp target/configs/check-apiserver.sh master2:/etc/keepalived/
    
    # scp target/configs/check-apiserver.sh master3:/etc/keepalived/

分别在3个master节点上修改keepalived配置文件，

master1

# vim /etc/keepalived/keepalived.conf
    
    ! Configuration File for keepalived
    global_defs { 
     router_id keepalive-master
    }
    
    vrrp_script check_apiserver { 
     script "/etc/keepalived/check-apiserver.sh"
     interval 3
     weight -3
    }
    
    vrrp_instance VI-kube-master { 
       state MASTER
       interface ens33
       virtual_router_id 68
       priority 100
       dont_track_primary
       advert_int 3
       virtual_ipaddress { 
         192.168.100.188
       }
       track_script { 
           check_apiserver
       }
    }

master2

# vim /etc/keepalived/keepalived.conf
    
    ! Configuration File for keepalived
    global_defs { 
     router_id keepalive-backup1
    }
    
    vrrp_script check_apiserver { 
     script "/etc/keepalived/check-apiserver.sh"
     interval 3
     weight -3
    }
    
    vrrp_instance VI-kube-master { 
       state BACKUP
       interface ens33
       virtual_router_id 68
       priority 99
       dont_track_primary
       advert_int 3
       virtual_ipaddress { 
         192.168.100.188
       }
       track_script { 
           check_apiserver
       }
    }

master3

# vim /etc/keepalived/keepalived.conf
    
    ! Configuration File for keepalived
    global_defs { 
     router_id keepalive-backup2
    }
    
    vrrp_script check_apiserver { 
     script "/etc/keepalived/check-apiserver.sh"
     interval 3
     weight -3
    }
    
    vrrp_instance VI-kube-master { 
       state BACKUP
       interface ens33
       virtual_router_id 68
       priority 98
       dont_track_primary
       advert_int 3
       virtual_ipaddress { 
         192.168.100.188
       }
       track_script { 
           check_apiserver
       }
    }

master节点启动keepalived，

# ssh root@master1 "systemctl enable keepalived && systemctl start keepalived"
    
    # ssh root@master2 "systemctl enable keepalived && systemctl start keepalived"
    
    # ssh root@master3 "systemctl enable keepalived && systemctl start keepalived"

# systemctl status keepalived
    
    # journalctl -f -u keepalived

*  部署kubectl：

kubectl 是 kubernetes 集群的命令行管理工具，它默认从`~/.kube/config`文件读取 kube-apiserver 地址、证书、用户名等信息。

kubectl 与 apiserver https 安全端口通信，apiserver 对提供的证书进行认证和授权。kubectl 作为集群的管理工具，需要被授予最高权限。

生成admin证书、私钥，

# cd /software/kubernetes-ha-binary/target/pki/admin/
    
    # cfssl gencert -ca=../ca.pem \
      -ca-key=../ca-key.pem \
      -config=../ca-config.json \
      -profile=kubernetes admin-csr.json | cfssljson -bare admin

kubeconfig 为 kubectl 的配置文件，包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书。

创建kubeconfig配置文件，

# kubectl config set-cluster kubernetes \
      --certificate-authority=../ca.pem \
      --embed-certs=true \
      --server=https://192.168.100.188:6443 \
      --kubeconfig=kube.config
    
    # kubectl config set-credentials admin \
      --client-certificate=admin.pem \
      --client-key=admin-key.pem \
      --embed-certs=true \
      --kubeconfig=kube.config
    
    # kubectl config set-context kubernetes \
      --cluster=kubernetes \
      --user=admin \
      --kubeconfig=kube.config
      
    # kubectl config use-context kubernetes --kubeconfig=kube.config
    
    # ssh root@master1 "mkdir ~/.kube"
    
    # ssh root@master2 "mkdir ~/.kube"
    
    # ssh root@master3 "mkdir ~/.kube"
    
    # scp kube.config master1:~/.kube/config
    
    # scp kube.config master2:~/.kube/config
    
    # scp kube.config master3:~/.kube/config

在执行 kubectl 命令时，apiserver 会转发到 kubelet。这里定义 RBAC 规则，授权 apiserver 调用 kubelet API。

# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

# kubectl cluster-info
    
    # kubectl get all --all-namespaces
    
    # kubectl get componentstatuses

*  部署controller-manager（master节点）：

controller-manager启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用后，剩余节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。

生成证书、私钥，并分发到每个master节点，

# cd /software/kubernetes-ha-binary/target/pki/controller-manager/
    
    # cfssl gencert -ca=../ca.pem \
      -ca-key=../ca-key.pem \
      -config=../ca-config.json \
      -profile=kubernetes controller-manager-csr.json | cfssljson -bare controller-manager
      
    # scp controller-manager*.pem master1:/etc/kubernetes/pki/
    
    # scp controller-manager*.pem master2:/etc/kubernetes/pki/
    
    # scp controller-manager*.pem master3:/etc/kubernetes/pki/

创建controller-manager的kubeconfig，并分发到每个master节点，

# kubectl config set-cluster kubernetes \
      --certificate-authority=../ca.pem \
      --embed-certs=true \
      --server=https://192.168.100.188:6443 \
      --kubeconfig=controller-manager.kubeconfig
      
    # kubectl config set-credentials system:kube-controller-manager \
      --client-certificate=controller-manager.pem \
      --client-key=controller-manager-key.pem \
      --embed-certs=true \
      --kubeconfig=controller-manager.kubeconfig
      
    # kubectl config set-context system:kube-controller-manager \
      --cluster=kubernetes \
      --user=system:kube-controller-manager \
      --kubeconfig=controller-manager.kubeconfig
      
    # kubectl config use-context system:kube-controller-manager --kubeconfig=controller-manager.kubeconfig
    
    # scp controller-manager.kubeconfig master1:/etc/kubernetes/
    
    # scp controller-manager.kubeconfig master2:/etc/kubernetes/
    
    # scp controller-manager.kubeconfig master3:/etc/kubernetes/

分发controller-manager服务文件到每个master节点，

# cd /software/kubernetes-ha-binary/
    
    # scp target/services/kube-controller-manager.service master1:/etc/systemd/system/
    
    # scp target/services/kube-controller-manager.service master2:/etc/systemd/system/
    
    # scp target/services/kube-controller-manager.service master3:/etc/systemd/system/

master节点启动controller-manager，

# ssh root@master1 "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl start kube-controller-manager"
    
    # ssh root@master2 "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl start kube-controller-manager"
    
    # ssh root@master3 "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl start kube-controller-manager"

# systemctl status kube-controller-manager
    
    # journalctl -f -u kube-controller-manager

*  部署scheduler：

scheduler启动后将通过竞争选举机制产生一个 leader 节点，其它节点为阻塞状态。当 leader 节点不可用后，剩余节点将再次进行选举产生新的 leader 节点，从而保证服务的可用性。

生成证书、私钥，

# cd /software/kubernetes-ha-binary/target/pki/scheduler/
    
    # cfssl gencert -ca=../ca.pem \
      -ca-key=../ca-key.pem \
      -config=../ca-config.json \
      -profile=kubernetes scheduler-csr.json | cfssljson -bare kube-scheduler

创建scheduler的kubeconfig，并分发kubeconfig到每个master节点，

# kubectl config set-cluster kubernetes \
      --certificate-authority=../ca.pem \
      --embed-certs=true \
      --server=https://192.168.100.188:6443 \
      --kubeconfig=kube-scheduler.kubeconfig
      
    # kubectl config set-credentials system:kube-scheduler \
      --client-certificate=kube-scheduler.pem \
      --client-key=kube-scheduler-key.pem \
      --embed-certs=true \
      --kubeconfig=kube-scheduler.kubeconfig
      
    # kubectl config set-context system:kube-scheduler \
      --cluster=kubernetes \
      --user=system:kube-scheduler \
      --kubeconfig=kube-scheduler.kubeconfig
      
    # kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
    
    # scp kube-scheduler.kubeconfig master1:/etc/kubernetes/
    
    # scp kube-scheduler.kubeconfig master2:/etc/kubernetes/
    
    # scp kube-scheduler.kubeconfig master3:/etc/kubernetes/

分发scheduler服务文件到每个master节点，

# cd /software/kubernetes-ha-binary/
    
    # scp target/services/kube-scheduler.service master1:/etc/systemd/system/
    
    # scp target/services/kube-scheduler.service master2:/etc/systemd/system/
    
    # scp target/services/kube-scheduler.service master3:/etc/systemd/system/

master节点启动scheduler，

# ssh root@master1 "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl start kube-scheduler"
    
    # ssh root@master2 "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl start kube-scheduler"
    
    # ssh root@master3 "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl start kube-scheduler"

# systemctl status kube-scheduler
    
    # journalctl -f -u kube-scheduler

*  部署kubelet：

预先下载镜像到每个node节点，

# scp target/configs/download-images.sh node1:/software
    
    # scp target/configs/download-images.sh node2:/software
    
    # scp target/configs/download-images.sh node3:/software
    
    # ssh root@node1 "sh /software/download-images.sh"
    
    # ssh root@node2 "sh /software/download-images.sh"
    
    # ssh root@node3 "sh /software/download-images.sh"

创建bootstrap配置文件，并分发到每个node节点，

# cd /software/kubernetes-ha-binary/target/pki/admin/
    
    # export BOOTSTRAP_TOKEN=$(kubeadm token create \
      --description kubelet-bootstrap-token \
      --groups system:bootstrappers:worker \
      --kubeconfig kube.config)
      
    # kubectl config set-cluster kubernetes \
      --certificate-authority=../ca.pem \
      --embed-certs=true \
      --server=https://192.168.100.188:6443 \
      --kubeconfig=kubelet-bootstrap.kubeconfig
      
    # kubectl config set-credentials kubelet-bootstrap \
      --token=${BOOTSTRAP_TOKEN} \
      --kubeconfig=kubelet-bootstrap.kubeconfig
      
    # kubectl config set-context default \
      --cluster=kubernetes \
      --user=kubelet-bootstrap \
      --kubeconfig=kubelet-bootstrap.kubeconfig
      
    # kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig
    
    # ssh root@node1 "mkdir -p /etc/kubernetes/pki"
    
    # ssh root@node2 "mkdir -p /etc/kubernetes/pki"
    
    # ssh root@node3 "mkdir -p /etc/kubernetes/pki"
    
    # scp kubelet-bootstrap.kubeconfig node1:/etc/kubernetes/
    
    # scp kubelet-bootstrap.kubeconfig node2:/etc/kubernetes/
    
    # scp kubelet-bootstrap.kubeconfig node3:/etc/kubernetes/
    
    # cd /software/kubernetes-ha-binary/
    
    # scp target/pki/ca.pem node1:/etc/kubernetes/pki/
    
    # scp target/pki/ca.pem node2:/etc/kubernetes/pki/
    
    # scp target/pki/ca.pem node3:/etc/kubernetes/pki/

分发kubelet配置文件和服务文件到每个node节点，

# scp target/worker-192.168.100.131/kubelet.config.json node1:/etc/kubernetes/
    
    # scp target/worker-192.168.100.132/kubelet.config.json node2:/etc/kubernetes/
    
    # scp target/worker-192.168.100.133/kubelet.config.json node3:/etc/kubernetes/
    
    # scp target/worker-192.168.100.131/kubelet.service node1:/etc/systemd/system/
    
    # scp target/worker-192.168.100.132/kubelet.service node2:/etc/systemd/system/
    
    # scp target/worker-192.168.100.133/kubelet.service node3:/etc/systemd/system/

bootstrap附权，

# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers

node节点启动kubelet，

# ssh root@node1 "mkdir /var/lib/kubelet"
    
    # ssh root@node2 "mkdir /var/lib/kubelet"
    
    # ssh root@node3 "mkdir /var/lib/kubelet"
    
    # ssh root@node1 "systemctl daemon-reload && systemctl enable kubelet && systemctl start kubelet"
    
    # ssh root@node2 "systemctl daemon-reload && systemctl enable kubelet && systemctl start kubelet"
    
    # ssh root@node3 "systemctl daemon-reload && systemctl enable kubelet && systemctl start kubelet"

同意bootstrap请求，

# kubectl get csr #获取csr name
    
    NAME                                                   AGE   REQUESTOR                 CONDITION
    node-csr-IxZr_CqrO0DyR8EgmEY8QzEKcVIb1fq3CakxzyNMz-E   18s   system:bootstrap:2z33di   Pending
    node-csr-TwfuVIfLWJ21im_p64jTCjnakyIAB3LYtbJ_O0bfYfg   9s    system:bootstrap:2z33di   Pending
    node-csr-oxYYnJ8-WGbxEJ8gL-Q41FHHUM2FOb8dv3g8_GNs7Ok   3s    system:bootstrap:2z33di   Pending
    
    # kubectl certificate approve node-csr-IxZr_CqrO0DyR8EgmEY8QzEKcVIb1fq3CakxzyNMz-E
    
    # kubectl certificate approve node-csr-TwfuVIfLWJ21im_p64jTCjnakyIAB3LYtbJ_O0bfYfg
    
    # kubectl certificate approve node-csr-oxYYnJ8-WGbxEJ8gL-Q41FHHUM2FOb8dv3g8_GNs7Ok

node节点查看服务状态与日志，

# ssh root@node1 "systemctl status kubelet"
    
    # ssh root@node1 "journalctl -f -u kubelet"

*  部署kube-proxy：

生成证书、私钥，

# cd /software/kubernetes-ha-binary/target/pki/proxy/
    
    # cfssl gencert -ca=../ca.pem \
      -ca-key=../ca-key.pem \
      -config=../ca-config.json \
      -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy

创建kubeconfig文件，并分发到每个node节点上，

# kubectl config set-cluster kubernetes \
      --certificate-authority=../ca.pem \
      --embed-certs=true \
      --server=https://192.168.100.188:6443 \
      --kubeconfig=kube-proxy.kubeconfig
    
    # kubectl config set-credentials kube-proxy \
      --client-certificate=kube-proxy.pem \
      --client-key=kube-proxy-key.pem \
      --embed-certs=true \
      --kubeconfig=kube-proxy.kubeconfig
      
    # kubectl config set-context default \
      --cluster=kubernetes \
      --user=kube-proxy \
      --kubeconfig=kube-proxy.kubeconfig
      
    # kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
    
    # scp kube-proxy.kubeconfig node1:/etc/kubernetes/
    
    # scp kube-proxy.kubeconfig node2:/etc/kubernetes/
    
    # scp kube-proxy.kubeconfig node3:/etc/kubernetes/

分发kube-proxy配置文件和服务文件到每个node节点，

# cd /software/kubernetes-ha-binary/
    
    # scp target/worker-192.168.100.131/kube-proxy.config.yaml node1:/etc/kubernetes/
    
    # scp target/worker-192.168.100.132/kube-proxy.config.yaml node2:/etc/kubernetes/
    
    # scp target/worker-192.168.100.133/kube-proxy.config.yaml node3:/etc/kubernetes/
    
    # scp target/services/kube-proxy.service node1:/etc/systemd/system/
    
    # scp target/services/kube-proxy.service node2:/etc/systemd/system/
    
    # scp target/services/kube-proxy.service node3:/etc/systemd/system/

node节点启动kube-proxy，

# ssh root@node1 "mkdir /var/lib/kube-proxy && mkdir /var/log/kubernetes"
    
    # ssh root@node2 "mkdir /var/lib/kube-proxy && mkdir /var/log/kubernetes"
    
    # ssh root@node3 "mkdir /var/lib/kube-proxy && mkdir /var/log/kubernetes"
    
    # ssh root@node1 "systemctl daemon-reload && systemctl enable kube-proxy && systemctl start kube-proxy"
    
    # ssh root@node2 "systemctl daemon-reload && systemctl enable kube-proxy && systemctl start kube-proxy"
    
    # ssh root@node3 "systemctl daemon-reload && systemctl enable kube-proxy && systemctl start kube-proxy"

# ssh root@node1 "systemctl status kube-proxy"
    
    # ssh root@node1 "journalctl -f -u kube-proxy"

*  部署calico：

# mkdir /etc/kubernetes/addons
    
    # cd /software/kubernetes-ha-binary/
    
    # scp target/addons/calico* master1:/etc/kubernetes/addons/
    
    # kubectl create -f /etc/kubernetes/addons/calico-rbac-kdd.yaml
    
    # kubectl create -f /etc/kubernetes/addons/calico.yaml
    
    # kubectl get pods -n kube-system #查看pod状态

*  部署DNS插件coredns：

# cd /software/kubernetes-ha-binary/
    
    # scp target/addons/coredns.yaml master1:/etc/kubernetes/addons/
    
    # kubectl create -f /etc/kubernetes/addons/coredns.yaml

--------------------

### 可用性测试 ###

集群已经初步搭建起来，下面进行集群的可用性测试。

一个DaemonSet对象能确保其创建的Pod在集群中的每一台（或指定）Node上都运行一个副本。如果集群中动态加入了新的Node，DaemonSet中的Pod也会被添加在新加入的Node上运行。删除一个DaemonSet也会级联删除所有其创建的Pod。

因此，创建一个DaemonSet对象来测试可用性比较合适。

*  创建nginx daemonset：

# cd /software && vim nginx-ds.yaml

apiVersion: v1
    kind: Service
    metadata:
      name: nginx-ds
      labels:
        app: nginx-ds
    spec:
      type: NodePort
      selector:
        app: nginx-ds
      ports:
      - name: http
        port: 80
        targetPort: 80
    ---
    apiVersion: extensions/v1beta1
    kind: DaemonSet
    metadata:
      name: nginx-ds
      labels:
        addonmanager.kubernetes.io/mode: Reconcile
    spec:
      template:
        metadata:
          labels:
            app: nginx-ds
        spec:
          containers:
          - name: my-nginx
            image: nginx:1.14.0
            ports:
            - containerPort: 80

# kubectl create -f nginx-ds.yaml
    service/nginx-ds created
    daemonset.extensions/nginx-ds created

*  检查ip连通性：

# kubectl get pods -o wide
    
    NAME             READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
    nginx-ds-8tjhm   1/1     Running   0          42s   172.10.0.2   node1   <none>           <none>
    nginx-ds-jdhmw   1/1     Running   0          42s   172.10.1.2   node2   <none>           <none>
    nginx-ds-s2kwt   1/1     Running   0          42s   172.10.2.3   node3   <none>           <none>
    
    # kubectl get svc
    
    NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)       AGE
    kubernetes   ClusterIP   10.254.0.1      <none>        443/TCP       126m
    nginx-ds     NodePort    10.254.119.71   <none>        80:8610/TCP   66s

在每个node节点上ping pod ip，同时访问服务ip及其端口，在每个node节点检查node-port可用性。

*  检查dns可用性：

# vim pod-nginx.yaml

apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.0
        ports:
        - containerPort: 80

# kubectl create -f pod-nginx.yaml
    pod/nginx created
    
    # kubectl exec -it nginx bash
    
    root@nginx:/# apt-get update
    
    root@nginx:/# apt install -y iputils-ping
    
    root@nginx:/# ping nginx-ds
    PING nginx-ds.default.svc.cluster.local (10.254.119.71) 56(84) bytes of data.
    
    # kubectl get svc
    
    NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)       AGE
    kubernetes   ClusterIP   10.254.0.1      <none>        443/TCP       144m
    nginx-ds     NodePort    10.254.119.71   <none>        80:8610/TCP   19m

可以看到，在nginx pod中ping nginx-ds时dns解析没问题，返回的是nginx-ds的cluster-ip。这说明之前搭建的集群正常可用。

--------------------

### 部署dashboard ###

*  部署dashboard：

# cd /software/kubernetes-ha-binary/
    
    # scp target/addons/dashboard-all.yaml master1:/etc/kubernetes/addons/
    
    # kubectl apply -f /etc/kubernetes/addons/dashboard-all.yaml
    
    # kubectl get deploy kubernetes-dashboard -n kube-system
    
    NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
    kubernetes-dashboard   1/1     1            1           8s
    
    # kubectl get svc kubernetes-dashboard -n kube-system
    
    NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
    kubernetes-dashboard   NodePort   10.254.39.199   <none>        443:8401/TCP   24s

*  访问dashboard：

dashboard只允许通过https访问，不过使用nodeport的方式暴露服务后，可以使用`https://NodeIP:NodePort`地址访问。

关于自定义证书，默认dashboard的证书是自动生成的非安全的证书。如果有域名和对应的安全证书可以自行替换，使用安全的域名方式访问dashboard。

在`dashboard-all.yaml`中增加dashboard启动参数，可以指定证书文件，其中证书文件是通过secret注进来的。

- –tls-cert-file 
    - dashboard.cer 
    - –tls-key-file 
    - dashboard.key

*  登录dashboard：

dashboard默认只支持token认证，所以如果使用KubeConfig文件，需要在该文件中指定token，这里使用token的方式登录。

# kubectl create sa dashboard-admin -n kube-system
    
    # kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
    
    # ADMIN_SECRET=$(kubectl get secrets -n kube-system | grep dashboard-admin | awk '{print $1}')
    
    # kubectl describe secret -n kube-system ${ADMIN_SECRET} | grep -E '^token' | awk '{print $2}' #打印token

使用打印的token登录dashboard，

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70]

使用谷歌浏览器无法访问，不过可以使用火狐浏览器访问，这里省略该步骤。

如果想要使用谷歌浏览器访问，可以这样做，

# mkdir /software/key && cd /software/key
    
    # openssl genrsa -out dashboard.key 2048
    
    # openssl req -new -out dashboard.csr -key dashboard.key -subj '/CN=192.168.100.131'
    
    # openssl x509 -req -in dashboard.csr -signkey dashboard.key -out dashboard.crt
    
    # kubectl delete secret kubernetes-dashboard-certs -n kube-system
    
    # kubectl create secret generic kubernetes-dashboard-certs --from-file=dashboard.key --from-file=dashboard.crt -n kube-system
    
    # kubectl get pod -n kube-system |grep dashboard
    kubernetes-dashboard-6f4595c9c9-bj8s6   1/1     Running   0          14m
    
    # kubectl delete pod kubernetes-dashboard-6f4595c9c9-bj8s6 -n kube-system

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 1]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 2]

kubernetes dashboard部署完成，kubernetes的高可用集群到此也搭建完成。

--------------------

[SSH]: https://blog.csdn.net/miss1181248983/article/details/89182939
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191116155517803.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191116155539440.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191116155548150.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L21pc3MxMTgxMjQ4OTgz,size_16,color_FFFFFF,t_70