kubeadm高可用集群安装k8s1.8(二)

r囧r小猫 2022-12-13 01:48 223阅读 0赞

1.查看系统版本要求

*  Ubuntu 16.04+
 *  Debian 9+
 *  CentOS 7
 *  Red Hat Enterprise Linux (RHEL) 7
 *  Fedora 25+
 *  HypriotOS v1.0.1+
 *  Container Linux (测试 1800.6.0 版本)

要求：cpu>2核，内存>2G

生产上禁用交换分区

如果iptables使用了nftables，应切回旧版

2.检查端口

控制平面

<table> 
 <tbody> 
 <tr> 
 <td> 协议 </td> 
 <td> 方向 </td> 
 <td> 端口范围 </td> 
 <td> 作用 </td> 
 <td> 使用者 </td> 
 </tr> 
 <tr> 
 <td> TCP </td> 
 <td> 入站 </td> 
 <td> 6443* </td> 
 <td> Kubernetes API 服务器 </td> 
 <td> 所有组件 </td> 
 </tr> 
 <tr> 
 <td> TCP </td> 
 <td> 入站 </td> 
 <td> 2379-2380 </td> 
 <td> etcd server client API </td> 
 <td> kube-apiserver, etcd </td> 
 </tr> 
 <tr> 
 <td> TCP </td> 
 <td> 入站 </td> 
 <td> 10250 </td> 
 <td> Kubelet API </td> 
 <td> kubelet 自身、控制平面组件 </td> 
 </tr> 
 <tr> 
 <td> TCP </td> 
 <td> 入站 </td> 
 <td> 10251 </td> 
 <td> kube-scheduler </td> 
 <td> kube-scheduler 自身 </td> 
 </tr> 
 <tr> 
 <td> TCP </td> 
 <td> 入站 </td> 
 <td> 10252 </td> 
 <td> kube-controller-manager </td> 
 <td> kube-controller-manager 自身 </td> 
 </tr> 
 </tbody> 
</table>

工作节点

3.pod 网络插件

开启特定端口

常用CNI（container network 插件）

*  flannel   ---不支持网络策略
 *  calico    ---支持地址分配和网络策略
 *  canel     --- flannel+calico
 *  kube-router...

4.重置iptables规则

iptables -P INPUT ACCEPT

iptables -F

iptables -L 查看

5.安装runtime(容器)

检查域套接字

检查路径

<table> 
 <tbody> 
 <tr> 
 <td> 运行时 </td> 
 <td> 域套接字 </td> 
 </tr> 
 <tr> 
 <td> Docker </td> 
 <td> /var/run/docker.sock </td> 
 </tr> 
 <tr> 
 <td> containerd </td> 
 <td> /run/containerd/containerd.sock </td> 
 </tr> 
 <tr> 
 <td> CRI-O </td> 
 <td> /var/run/crio/crio.sock </td> 
 </tr> 
 </tbody> 
</table>

如同时检查到docker和containerd，则优先使用docker

docker 18.09 附带了 containerd 并且两者都是可以检测到的

6.安装docker(如安装请忽略)

a.配置docker的yum仓库（这里使用阿里云仓库）

yum -y install yum-utils device-mapper-persistent-data lvm2

yum-config-manager --add-repo [https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo][https_mirrors.aliyun.com_docker-ce_linux_centos_docker-ce.repo]

b.安装docker

yum -y install docker-ce-18.09.7 docker-ce-cli-18.09.7 containerd.io

c.修改docker cgroup driver为systemd(驱动程序)

vim /etc/docker/daemon.json \#如果不存在则创建 \{

"exec-opts":\["native.cgroupdriver=systemd"\]

d.启动docker

systemctl daemon-reload

systemctl restart docker \#启动docker systemctl enable docker (开机启动)

e.docker info |grep Cgroup

Cgroup Driver: systemd

7.设置主机名:

hostnamectl set-hostname kube-master

vim /etc/hosts

192.168.9.104 k8s-master1

192.168.9.105 k8s-master2

192.168.9.100 k8s-node1

8.配置免密

ssh-keygen -t rsa (生成公钥)

ssh-copy-id -i ~/.ssh/id\_rsa.pub root@192.168.9.100 (拷贝公钥)

9.同步时间

ntpdate 0.rhel.pool.ntp.org

10.将 SELinux 设置为 permissive 模式(禁用)

SELinux作用：限制进程访问资源

getenforce:获取当前状态

setenforce 0 sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

目的：允许容器访问主机文件系统所必须的

11.设置内核参数net.bridge.bridge-nf-call-iptables

将桥接的IPv4流量传递到iptables的链

net.bridge.bridge-nf-call-iptables=0(iptables不对bridge的数据进行处理)

vim  /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

sysctl --system

确保已加载了 br\_netfilter 模块

lsmod | grep br\_netfilter

要显示加载它，请调用 modprobe br\_netfilter

12.配置kubenetes的yum仓库（阿里yum源）

vim /etc/yum.repos.d/kubernetes.repo

\[kubernetes\] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86\_64/ enabled=1 gpgcheck=1 repo\_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg [https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg][https_mirrors.aliyun.com_kubernetes_yum_doc_rpm-package-key.gpg]

yum makecache (yum源生效)

13.安装kubelat、kubectl、kubeadm

yum -y install kubelet-1.18.0 kubeadm-1.18.0 kubectl-1.18.0 (kubectl可选)

rpm -aq kubelet kubectl kubeadm (查看安装软件)

systemctl enable kubelet (kubelet开机启动)

14.搭建keepalived+haproxy 高可用负载均衡集群

a.yum -y install pcre-devel bzip2-devel keepalived haproxy

b.修改haproxy配置文件

/etc/haproxy/haproxy.cfg

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /var/run/haproxy-admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        nbproc 1
    
    defaults
        log     global
        timeout connect 5000
        timeout client  10m
        timeout server  10m
    
    listen  admin_stats
        bind 0.0.0.0:10080
        mode http
        log 127.0.0.1 local0 err
        stats refresh 30s
        stats uri /status
        stats realm welcome login\ Haproxy
        stats auth admin:123456
        stats hide-version
        stats admin if TRUE
    
    listen kube-master
        bind 0.0.0.0:8443
        mode tcp
        option tcplog
        balance source
        server 192.168.9.104 192.168.9.104:6443 check inter 2000 fall 2 rise 2 weight 1
        server 192.168.9.105 192.168.9.105:6443 check inter 2000 fall 2 rise 2 weight 1

所有主节点文件一致

设置开机启动并重启：systemctl enable haproxy && systemctl restart haproxy

查看状态：systemctl status haproxy|grep Active

失败：journalctl -f 查看原因

c.修改keepalived配置文件

/etc/keepalived/keepalived.conf

master1

! Configuration File for keepalived
    
    global_defs {
        router_id lb-master-104
    }
    
    vrrp_script check-haproxy {
        script "killall -0 haproxy"
        interval 5
        weight -30
    }
    
    vrrp_instance VI-kube-master {
        state MASTER
        priority 120
        dont_track_primary
        interface enp0s25  #网卡
        virtual_router_id 68
        advert_int 3
        track_script {
            check-haproxy
        }
        virtual_ipaddress {
            192.168.9.111 ＃vip虚拟ip
        }
    }

master2.3.4....

! Configuration File for keepalived
    
    global_defs {
        router_id lb-backup-105
    }
    
    vrrp_script check-haproxy {
        script "killall -0 haproxy"
        interval 5
        weight -30
    }
    
    vrrp_instance VI-kube-master {
        state BACKUP
        priority 110
        dont_track_primary
        interface enp4s0 #网卡
        virtual_router_id 68
        advert_int 3
        track_script {
            check-haproxy
        }
        virtual_ipaddress {
            192.168.9.111　＃vip虚拟ip
        }
    }

启动keepalived

设置开机启动并重启 systemctl enable keepalived &&systemctl restart keepalived

查看状态：systemctl status keepalived|grep Active

失败：journalctl -f 查看原因

d.浏览器访问虚拟ip加端口

http:192.168.9.104(105):10080/status

帐户名密码：admin/123456

15:初始化master1

kubeadm init --kubernetes-version=v1.18.0 --control-plane-endpoint "192.168.9.111:8443" --upload-certs --image-repository registry.aliyuncs.com/google\_containers --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap

输出信息：

You can now join any number of the control-plane node running the following command on each as root:

kubeadm join 192.168.9.111:8443 --token vf8sku.klyq3gfwd4wl29ct \\

\--discovery-token-ca-cert-hash sha256:3e19607f26fe2af4c518f985be186c82c408a2988d352d38bc1392b90b07360c \\

\--control-plane --certificate-key 2f792f80628d539ec5a2c91f41f5dc20280dc982de5937de9cefc9675632f95a

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!

As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use

"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.9.111:8443 --token vf8sku.klyq3gfwd4wl29ct \\

\--discovery-token-ca-cert-hash sha256:3e19607f26fe2af4c518f985be186c82c408a2988d352d38bc1392b90b07360c

16:加入控制平面

kubeadm join 192.168.9.111:8443 --token vf8sku.klyq3gfwd4wl29ct \\

\--discovery-token-ca-cert-hash sha256:3e19607f26fe2af4c518f985be186c82c408a2988d352d38bc1392b90b07360c \\

\--control-plane --certificate-key 2f792f80628d539ec5a2c91f41f5dc20280dc982de5937de9cefc9675632f95a --ignore-preflight-errors=Swap

17.加入node节点

kubeadm join 192.168.9.111:8443 --token vf8sku.klyq3gfwd4wl29ct \\

\--discovery-token-ca-cert-hash sha256:3e19607f26fe2af4c518f985be186c82c408a2988d352d38bc1392b90b07360c --ignore-preflight-errors=Swap

18.使用kubectl工具

mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config

19.部署pod网络插件

此时master处于notredy,需要部署网络插件

此处使用cni的flannel

方法一 kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml kubectl get pods -n kube-system |grep flannel \#验证flannel网络插件是否部署成功（Running即为成功）

方法二

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml sed -i 's\#quay.io\#quay-mirror.qiniu.com\#g' kube-flannel.yml \#替换仓库地址 kubectl apply -f kube-flannel.yml

20.查看集群客户端和服务端程序版本信息

kubectl version --short=true

21.查看集群信息

kubectl cluster-info

22.节点故障，删除节点

kubectl drain <NODE-NAME> --delete-local-data --force --ignore-daemonsets

kubectl delete nodes node06

节点上执行

kubeadm reset

systemctl stop kubelet

systemctl stop docker

rm -rf /var/lib/cni/

rm -rf /var/lib/kubelet/\*

rm -rf /etc/cni/

ifconfig cni0 down

ifconfig flannel.1 down

ifconfig docker0 down

ip link delete cni0

ip link delete flannel.1

systemctl start docker

systemctl start kubelet

[https_mirrors.aliyun.com_docker-ce_linux_centos_docker-ce.repo]: https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[https_mirrors.aliyun.com_kubernetes_yum_doc_rpm-package-key.gpg]: https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg