使用kubeadm搭建生产环境的多master节点k8s高可用集群

红太狼 2023-09-27 18:20 128阅读 0赞

环境centos 7.9

**目录**

1.对安装 k8s 的节点进行初始化配置

2 通过 keepalived+nginx 实现 k8s apiserver 节点高可用

3、kubeadm 初始化 k8s 集群

4.扩容 k8s 控制节点，把 xuegod62 加入到 k8s 集群

5、扩容 k8s 控制节点，把 xuegod64 加入到 k8s 集群

6、扩容 k8s 集群-添加第一个工作节点

7、安装 kubernetes 网络组件-Calico

8、测试 k8s 集群的 DNS 解析和网络是否正常

9、etcd 配置成高可用状态

--------------------

k8s 实验环境网络规划：  
podSubnet（pod 网段） 10.244.0.0/16  
serviceSubnet（service 网段）: 10.96.0.0/12  
物理机网段：192.168.1.0/24

K8s 主机配置：  
操作系统：centos7.9  
配置： 4Gib 内存/4vCPU/60G 硬盘  
网络：机器相互可以通信

![b10e5e534a6d449f84ceedc6af1fe5a6.png][]

#### 1.对安装 k8s 的节点进行初始化配置 ####

初始化安装 k8s 集群需要的实验环境  
准备四台 Centos7.9 的 linux 机器。每台机器配置：4VCPU/4G 内存/60G 硬盘  
环境说明（centos7.9）：  
IP 主机名 角色 内存 cpu  
192.168.1.63 xuegod63 master 4G 4vCPU  
192.168.1.64 xuegod64 worker 4G 4vCPU  
192.168.1.62 xuegod62 worker 4G 4vCPU  
192.168.1.66 xuegod66 worker 4G 4vCPU

1、配置静态 IP：每台机器的网络模式要一致，能互相通信，机器网卡名字也要统一，机器要能联  
网。![1e8d46248b214363bbe91a32c8e6c123.png][]

2、永久关闭 selinux  
\[root@localhost ~\]\# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g'  
/etc/selinux/config

\#注意：修改 selinux 配置文件之后，重启机器，selinux 才能永久生效  
\[root@localhost~\]\#getenforce  
Disabled  
3、配置主机名  
在 192.168.1.63 上执行如下：

hostnamectl set-hostname xuegod63 && bash  
在 192.168.1.64 上执行如下：  
hostnamectl set-hostname xuegod64 && bash  
在 192.168.1.62 上执行如下：  
hostnamectl set-hostname xuegod62 && bash  
在 192.168.1.66 上执行如下：  
hostnamectl set-hostname xuegod66 && bash

4、配置 hosts 文件：  
修改每台机器的/etc/hosts 文件，在内容最后增加如下三行：  
192.168.1.63 xuegod63  
192.168.1.64 xuegod64  
192.168.1.62 xuegod62  
192.168.1.66 xuegod66

5、安装基础软件包  
\[root@xuegod63 ~\]\# yum install -y yum-utils device-mapper-persistent-data lvm2  
wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl  
curl-devel unzip sudo ntp libaio-devel vim ncurses-devel autoconf automake zlibdevel python-devel epel-release openssh-server socat conntrack ntpdate telnet ipvsadm  
\[root@xuegod64 ~\]\# yum install -y yum-utils device-mapper-persistent-data lvm2  
wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl  
curl-devel unzip sudo ntp libaio-devel vim ncurses-devel autoconf automake zlibdevel python-devel epel-release openssh-server socat conntrack ntpdate telnet ipvsadm  
\[root@xuegod62 ~\]\# yum install -y yum-utils device-mapper-persistent-data lvm2  
wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl  
curl-devel unzip sudo ntp libaio-devel vim ncurses-devel autoconf automake zlibdevel python-devel epel-release openssh-server socat conntrack ntpdate telnet ipvsadm  
\[root@xuegod66 ~\]\# yum install -y yum-utils device-mapper-persistent-data lvm2  
wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl  
curl-devel unzip sudo ntp libaio-devel vim ncurses-devel autoconf automake zlibdevel python-devel epel-release openssh-server socat conntrack ntpdate telnet ipvsadm

6、配置主机之间无密码登录  
1）配置 xuegod63 到其他机器免密登录  
\[root@xuegod63 ~\]\# ssh-keygen \#一路回车，不输入密码  
把本地的 ssh 公钥文件安装到远程主机对应的账户  
\[root@xuegod63 ~\]\# ssh-copy-id xuegod63  
\[root@xuegod63 ~\]\# ssh-copy-id xuegod64  
\[root@xuegod63 ~\]\# ssh-copy-id xuegod62  
\[root@xuegod63 ~\]\# ssh-copy-id xuegod66

2）配置 xuegod64 到其他机器免密登录  
\[root@xuegod64 ~\]\# ssh-keygen \#一路回车，不输入密码  
把本地的 ssh 公钥文件安装到远程主机对应的账户  
\[root@xuegod64 ~\]\# ssh-copy-id xuegod63  
\[root@xuegod64 ~\]\# ssh-copy-id xuegod64  
\[root@xuegod64 ~\]\# ssh-copy-id xuegod62  
\[root@xuegod64 ~\]\# ssh-copy-id xuegod66

3）配置 xuegod62 到其他机器免密登录  
\[root@xuegod62 ~\]\# ssh-keygen \#一路回车，不输入密码  
把本地的 ssh 公钥文件安装到远程主机对应的账户  
\[root@xuegod62 ~\]\# ssh-copy-id xuegod63  
\[root@xuegod62 ~\]\# ssh-copy-id xuegod64  
\[root@xuegod62 ~\]\# ssh-copy-id xuegod62  
\[root@xuegod62 ~\]\# ssh-copy-id xuegod66

4）配置 xuegod66 到其他机器免密登录  
\[root@xuegod66 ~\]\# ssh-keygen \#一路回车，不输入密码  
把本地的 ssh 公钥文件安装到远程主机对应的账户  
\[root@xuegod66 ~\]\# ssh-copy-id xuegod63  
\[root@xuegod66 ~\]\# ssh-copy-id xuegod64  
\[root@xuegod66 ~\]\# ssh-copy-id xuegod62  
\[root@xuegod66~\]\# ssh-copy-id xuegod66

7、关闭所有主机 firewalld 防火墙  
\[root@xuegod63 ~\]\# systemctl stop firewalld ; systemctl disable firewalld  
\[root@xuegod64 ~\]\# systemctl stop firewalld ; systemctl disable firewalld  
\[root@xuegod62 ~\]\# systemctl stop firewalld ; systemctl disable firewalld  
\[root@xuegod66 ~\]\# systemctl stop firewalld ; systemctl disable firewalld

8、关闭交换分区 swap  
\#临时关闭交换分区  
\[root@xuegod63 ~\]\# swapoff -a  
\[root@xuegod64 ~\]\# swapoff -a  
\[root@xuegod62 ~\]\# swapoff -a  
\[root@xuegod66 ~\]\# swapoff -a

永久关闭：注释 swap 挂载  
\[root@xuegod63 ~\]\# vim /etc/fstab \#给 swap 这行开头加一下注释\#

![3f719d00652b4fd8807c29b5b17df9e0.png][]  
\[root@xuegod64 ~\]\# vim /etc/fstab

![0c26cb6b7cbe4ff69f8dad9e5148c9b3.png][]  
\[root@xuegod62 ~\]\# vim /etc/fstab ![d731c3d36174465eb43fa608e219a1e0.png][]

\[root@xuegod66 ~\]\# vim /etc/fstab

![1148d68d401645dfab9824b11d1fcead.png][]

9、修改内核参数：  
\[root@xuegod63 ~\]\# modprobe br\_netfilter  
\[root@xuegod64 ~\]\# modprobe br\_netfilter  
\[root@xuegod62 ~\]\# modprobe br\_netfilter  
\[root@xuegod66 ~\]\# modprobe br\_netfilter  
\[root@xuegod63 ~\]\# cat > /etc/sysctl.d/k8s.conf <<EOF  
net.bridge.bridge-nf-call-ip6tables = 1  
net.bridge.bridge-nf-call-iptables = 1  
net.ipv4.ip\_forward = 1  
EOF  
\[root@xuegod63 ~\]\# sysctl -p /etc/sysctl.d/k8s.conf  
\[root@xuegod64 ~\]\# cat > /etc/sysctl.d/k8s.conf <<EOF  
net.bridge.bridge-nf-call-ip6tables = 1  
net.bridge.bridge-nf-call-iptables = 1  
net.ipv4.ip\_forward = 1  
EOF  
\[root@xuegod64 ~\]\# sysctl -p /etc/sysctl.d/k8s.conf  
\[root@xuegod62 ~\]\# cat > /etc/sysctl.d/k8s.conf <<EOF  
net.bridge.bridge-nf-call-ip6tables = 1  
net.bridge.bridge-nf-call-iptables = 1  
net.ipv4.ip\_forward = 1  
EOF  
\[root@xuegod62 ~\]\# sysctl -p /etc/sysctl.d/k8s.conf  
\[root@xuegod66 ~\]\# cat > /etc/sysctl.d/k8s.conf <<EOF  
net.bridge.bridge-nf-call-ip6tables = 1  
net.bridge.bridge-nf-call-iptables = 1  
net.ipv4.ip\_forward = 1  
EOF  
\[root@xuegod66 ~\]\# sysctl -p /etc/sysctl.d/k8s.conf

10、配置安装 docker 和 containerd 的需要的阿里云 yum 源  
\[root@xuegod63 ~\]\# yum-config-manager --add-repo  
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo  
\[root@xuegod64 ~\]\# yum-config-manager --add-repo  
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo  
\[root@xuegod62 ~\]\# yum-config-manager --add-repo

http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo  
\[root@xuegod66 ~\]\# yum-config-manager --add-repo  
http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

11、配置安装 k8s 命令行工具需要的阿里云的 yum 源  
配置阿里云 Kubernetes yum 源  
\[root@xuegod63 ~\]\# cat > /etc/yum.repos.d/kubernetes.repo <<EOF  
\[kubernetes\]  
name=Kubernetes  
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86\_64/  
enabled=1  
gpgcheck=0  
EOF

然后，再执行下面命令，使用在线 yum 源安装 kubeadm 和 kubelet  
将 xuegod63 上 Kubernetes 的 yum 源复制给 xuegod64、xuegod62、xuegod66  
\[root@xuegod63 ~\]\# scp /etc/yum.repos.d/kubernetes.repo  
xuegod64:/etc/yum.repos.d/  
\[root@xuegod63 ~\]\# scp /etc/yum.repos.d/kubernetes.repo  
xuegod62:/etc/yum.repos.d/  
\[root@xuegod63 ~\]\# scp /etc/yum.repos.d/kubernetes.repo  
xuegod66:/etc/yum.repos.d/

12、配置时间同步：  
\[root@xuegod63 ~\]\# yum install -y ntp ntpdate  
\[root@xuegod63 ~\]\# ntpdate cn.pool.ntp.org  
\#编写计划任务  
\[root@xuegod63 ~\]\# crontab -e  
\* \* \* \* \* /usr/sbin/ntpdate cn.pool.ntp.org  
\[root@xuegod64 ~\]\# yum install -y ntp ntpdate  
\[root@xuegod64 ~\]\# ntpdate cn.pool.ntp.org  
\#编写计划任务  
\[root@xuegod64 ~\]\# crontab -e  
\* \* \* \* \* /usr/sbin/ntpdate cn.pool.ntp.org

\[root@xuegod62~\]\# yum install -y ntp ntpdate  
\[root@xuegod62 ~\]\# ntpdate cn.pool.ntp.org  
\#编写计划任务  
\[root@xuegod62 ~\]\# crontab -e  
\* \* \* \* \* /usr/sbin/ntpdate cn.pool.ntp.org  
\[root@xuegod66~\]\# yum install -y ntp ntpdate

\[root@xuegod66 ~\]\# ntpdate cn.pool.ntp.org  
\#编写计划任务  
\[root@xuegod66 ~\]\# crontab -e  
\* \* \* \* \* /usr/sbin/ntpdate cn.pool.ntp.org

13、安装 containerd  
在 xuegod63 上安装 containerd  
\[root@xuegod63~\]\#yum install containerd.io-1.6.6 -y  
Containerd 版本要按照我这个版本，其他版本有问题。  
生成 containerd 的配置文件:  
\[root@xuegod63~\]\#mkdir -p /etc/containerd  
\[root@xuegod63 ~\]\#containerd config default > /etc/containerd/config.toml

修改配置文件，打开/etc/containerd/config.toml  
把 SystemdCgroup = false 修改成 SystemdCgroup = true  
把 sandbox\_image = "k8s.gcr.io/pause:3.6"修改成  
sandbox\_image="registry.aliyuncs.com/google\_containers/pause:3.7"  
找到 config\_path = ""，修改成如下目录：  
config\_path = "/etc/containerd/certs.d"

创建/etc/crictl.yaml 文件  
\[root@xuegod63 ~\]\#cat > /etc/crictl.yaml <<EOF  
runtime-endpoint: unix:///run/containerd/containerd.sock  
image-endpoint: unix:///run/containerd/containerd.sock  
timeout: 10  
debug: false  
EOF  
\[root@xuegod63 ~\]\#mkdir /etc/containerd/certs.d/docker.io/ -p  
\[root@xuegod63 ~\]\#vim /etc/containerd/certs.d/docker.io/hosts.toml  
\#写入如下内容：  
\[host."https://vh3bm52y.mirror.aliyuncs.com",host."https://registry.docker-cn.com"\]  
capabilities = \["pull","push"\]

启动 containerd、并设置开启自启动  
\[root@xuegod63 ~\]\#systemctl enable containerd --now

\#在 xuegod64 上安装 containerd  
\[root@xuegod64~\]\#yum install containerd.io-1.6.6 -y

生成 containerd 的配置文件:  
\[root@xuegod64~\]\#mkdir -p /etc/containerd  
\[root@xuegod64 ~\]\#containerd config default > /etc/containerd/config.toml  
修改配置文件，打开/etc/containerd/config.toml  
把 SystemdCgroup = false 修改成 SystemdCgroup = true  
把 sandbox\_image = "k8s.gcr.io/pause:3.6"修改成  
sandbox\_image="registry.aliyuncs.com/google\_containers/pause:3.7"  
找到 config\_path = ""，修改成如下目录：  
config\_path = "/etc/containerd/certs.d"  
创建/etc/crictl.yaml 文件  
\[root@xuegod64 ~\]\#cat > /etc/crictl.yaml <<EOF  
runtime-endpoint: unix:///run/containerd/containerd.sock  
image-endpoint: unix:///run/containerd/containerd.sock  
timeout: 10  
debug: false  
EOF  
\[root@xuegod64 ~\]\#mkdir /etc/containerd/certs.d/docker.io/ -p  
\[root@xuegod64 ~\]\#vim /etc/containerd/certs.d/docker.io/hosts.toml  
\#写入如下内容：  
\[host."https://vh3bm52y.mirror.aliyuncs.com",host."https://registry.docker-cn.com"\]  
capabilities = \["pull","push"\]  
启动 containerd、并设置开启自启动  
\[root@xuegod64 ~\]\#systemctl enable containerd --now  
\#在 xuegod62 上安装 containerd  
\[root@xuegod62~\]\#yum install containerd.io-1.6.6 -y  
生成 containerd 的配置文件:  
\[root@xuegod62~\]\#mkdir -p /etc/containerd  
\[root@xuegod62 ~\]\#containerd config default > /etc/containerd/config.toml  
修改配置文件，打开/etc/containerd/config.toml  
把 SystemdCgroup = false 修改成 SystemdCgroup = true  
把 sandbox\_image = "k8s.gcr.io/pause:3.6"修改成  
sandbox\_image="registry.aliyuncs.com/google\_containers/pause:3.7"  
找到 config\_path = ""，修改成如下目录：  
config\_path = "/etc/containerd/certs.d"

创建/etc/crictl.yaml 文件

\[root@xuegod62 ~\]\#cat > /etc/crictl.yaml <<EOF  
runtime-endpoint: unix:///run/containerd/containerd.sock  
image-endpoint: unix:///run/containerd/containerd.sock  
timeout: 10  
debug: false  
EOF

\[root@xuegod62 ~\]\#mkdir /etc/containerd/certs.d/docker.io/ -p  
\[root@xuegod62 ~\]\#vim /etc/containerd/certs.d/docker.io/hosts.toml  
\#写入如下内容：  
\[host."https://vh3bm52y.mirror.aliyuncs.com",host."https://registry.docker-cn.com"\]  
capabilities = \["pull","push"\]  
启动 containerd、并设置开启自启动  
\[root@xuegod62 ~\]\#systemctl enable containerd --now

在 xuegod66 上安装 containerd

\[root@xuegod66~\]\#yum install containerd.io-1.6.6 -y  
Containerd 版本要按照我这个版本，其他版本有问题。  
生成 containerd 的配置文件:  
\[root@xuegod66~\]\#mkdir -p /etc/containerd  
\[root@xuegod66 ~\]\#containerd config default > /etc/containerd/config.toml  
修改配置文件，打开/etc/containerd/config.toml  
把 SystemdCgroup = false 修改成 SystemdCgroup = true  
把 sandbox\_image = "k8s.gcr.io/pause:3.6"修改成  
sandbox\_image="registry.aliyuncs.com/google\_containers/pause:3.7"  
找到 config\_path = ""，修改成如下目录：  
config\_path = "/etc/containerd/certs.d"

创建/etc/crictl.yaml 文件  
\[root@xuegod66 ~\]\#cat > /etc/crictl.yaml <<EOF  
runtime-endpoint: unix:///run/containerd/containerd.sock  
image-endpoint: unix:///run/containerd/containerd.sock  
timeout: 10  
debug: false  
EOF  
\[root@xuegod66 ~\]\#mkdir /etc/containerd/certs.d/docker.io/ -p  
\[root@xuegod66 ~\]\#vim /etc/containerd/certs.d/docker.io/hosts.toml  
\#写入如下内容：

\[host."https://vh3bm52y.mirror.aliyuncs.com",host."https://registry.docker-cn.com"\]  
capabilities = \["pull","push"\]  
启动 containerd、并设置开启自启动  
\[root@xuegod66 ~\]\#systemctl enable containerd --now

14、安装 docker-ce  
K8s1.24 开始已经不支持 docker 了，但是还要把 docker 安装在 k8s 节点上，主要是为了用  
docker build 基于 dockerfile 做镜像，docker 跟 containerd 不冲突、  
\[root@xuegod63 ~\]\# yum install docker-ce-23.0.3 -y  
\[root@xuegod63 ~\]\# systemctl start docker && systemctl enable docker.service  
\[root@xuegod63 ~\]\# tee /etc/docker/daemon.json << 'EOF'  
\{  
"registry-mirrors":\["https://vh3bm52y.mirror.aliyuncs.com","https://registry.dockercn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hubmirror.c.163.com"\],  
"exec-opts": \["native.cgroupdriver=systemd"\]  
\}  
EOF  
\[root@xuegod63 ~\]\# systemctl restart docker  
\[root@xuegod64 ~\]\# yum install docker-ce-23.0.3 -y  
\[root@xuegod64 ~\]\# systemctl start docker && systemctl enable docker.service  
\[root@xuegod64 ~\]\# tee /etc/docker/daemon.json << 'EOF'  
\{  
"registry-mirrors":\["https://vh3bm52y.mirror.aliyuncs.com","https://registry.dockercn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hubmirror.c.163.com"\],  
"exec-opts": \["native.cgroupdriver=systemd"\]  
\}  
EOF  
\[root@xuegod62 ~\]\# systemctl restart docker  
\[root@xuegod62 ~\]\# yum install docker-ce-23.0.3 -y  
\[root@xuegod62 ~\]\# systemctl start docker && systemctl enable docker.service  
\[root@xuegod62 ~\]\# tee /etc/docker/daemon.json << 'EOF'  
\{  
"registry-mirrors":\["https://vh3bm52y.mirror.aliyuncs.com","https://registry.dockercn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hubmirror.c.163.com"\],"exec-opts": \["native.cgroupdriver=systemd"\]  
\}  
EOF

\[root@xuegod62 ~\]\# systemctl restart docker  
\[root@xuegod66 ~\]\# yum install docker-ce-23.0.3 -y  
\[root@xuegod66 ~\]\# systemctl start docker && systemctl enable docker.service  
\[root@xuegod66 ~\]\# tee /etc/docker/daemon.json << 'EOF'  
\{  
"registry-mirrors":\["https://vh3bm52y.mirror.aliyuncs.com","https://registry.dockercn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hubmirror.c.163.com"\],  
"exec-opts": \["native.cgroupdriver=systemd"\]  
\}  
EOF  
\[root@xuegod66 ~\]\# systemctl restart docker

15、安装初始化 k8s 需要的组件  
\[root@xuegod63 ~\]\# yum install -y kubelet-1.26.0 kubeadm-1.26.0 kubectl-1.26.0  
\[root@xuegod63 ~\]\# systemctl enable kubelet  
\[root@xuegod64 ~\]\# yum install -y kubelet-1.26.0 kubeadm-1.25.0 kubectl-1.26.0  
\[root@xuegod64~\]\# systemctl enable kubelet  
\[root@xuegod62 ~\]\# yum install -y kubelet-1.26.0 kubeadm-1.25.0 kubectl-1.26.0  
\[root@xuegod62~\]\# systemctl enable kubelet  
\[root@xuegod66 ~\]\# yum install -y kubelet-1.26.0 kubeadm-1.25.0 kubectl-1.26.0  
\[root@xuegod66~\]\# systemctl enable kubelet

#### 2 通过 keepalived+nginx 实现 k8s apiserver 节点高可用 ####

1、安装 nginx 和 keepalived  
在 xuegod63 和 xuegod64 上安装 keepalived 和 nginx，实现对 apiserver 负载均衡和反向代  
理。Xuegod63 是 keepalived 主节点，xuegod64 是 keepalived 备节点。  
\[root@xuegod63 ~\]\# yum install epel-release nginx keepalived nginx-mod-stream -y  
\[root@xuegod64 ~\]\# yum install epel-release nginx keepalived nginx-mod-stream -y  
2、修改 nginx 配置文件。主备一样  
\[root@xuegod63 ~\]\# vim /etc/nginx/nginx.conf  
user nginx;  
worker\_processes auto;

error\_log /var/log/nginx/error.log;  
pid /run/nginx.pid;  
include /usr/share/nginx/modules/\*.conf;  
events \{  
worker\_connections 1024;  
\}  
\# 四层负载均衡，为两台 Master apiserver 组件提供负载均衡  
stream \{  
log\_format main '$remote\_addr $upstream\_addr - \[$time\_local\] $status  
$upstream\_bytes\_sent';  
access\_log /var/log/nginx/k8s-access.log main;  
upstream k8s-apiserver \{  
server 192.168.1.63:6443 weight=5 max\_fails=3 fail\_timeout=30s;  
server 192.168.1.62:6443 weight=5 max\_fails=3 fail\_timeout=30s;  
server 192.168.1.64:6443 weight=5 max\_fails=3 fail\_timeout=30s;  
\}  
  
server \{  
listen 16443; \# 由于 nginx 与 master 节点复用，这个监听端口不能是 6443，否则会冲  
突  
proxy\_pass k8s-apiserver;  
\}  
\}  
http \{  
log\_format main '$remote\_addr - $remote\_user \[$time\_local\] "$request" '  
'$status $body\_bytes\_sent "$http\_referer" '  
'"$http\_user\_agent" "$http\_x\_forwarded\_for"';  
access\_log /var/log/nginx/access.log main;  
sendfile on;  
tcp\_nopush on;  
tcp\_nodelay on;  
keepalive\_timeout 65;  
types\_hash\_max\_size 2048;

include /etc/nginx/mime.types;  
default\_type application/octet-stream;  
server \{  
listen 80 default\_server;  
server\_name \_;  
location / \{  
\}  
\}  
\}  
\[root@xuegod64 ~\]\# vim /etc/nginx/nginx.conf  
user nginx;  
worker\_processes auto;  
error\_log /var/log/nginx/error.log;  
pid /run/nginx.pid;  
include /usr/share/nginx/modules/\*.conf;  
events \{  
worker\_connections 1024;  
\}  
\# 四层负载均衡，为两台 Master apiserver 组件提供负载均衡  
stream \{  
log\_format main '$remote\_addr $upstream\_addr - \[$time\_local\] $status  
$upstream\_bytes\_sent';  
access\_log /var/log/nginx/k8s-access.log main;  
upstream k8s-apiserver \{  
server 192.168.1.63:6443 weight=5 max\_fails=3 fail\_timeout=30s;  
server 192.168.1.62:6443 weight=5 max\_fails=3 fail\_timeout=30s;  
server 192.168.1.64:6443 weight=5 max\_fails=3 fail\_timeout=30s;  
\}  
  
server \{  
listen 16443; \# 由于 nginx 与 master 节点复用，这个监听端口不能是 6443，否则会冲  
突  
proxy\_pass k8s-apiserver;  
\}  
\}

http \{  
log\_format main '$remote\_addr - $remote\_user \[$time\_local\] "$request" '  
'$status $body\_bytes\_sent "$http\_referer" '  
'"$http\_user\_agent" "$http\_x\_forwarded\_for"';  
access\_log /var/log/nginx/access.log main;  
sendfile on;  
tcp\_nopush on;  
tcp\_nodelay on;  
keepalive\_timeout 65;  
types\_hash\_max\_size 2048;  
include /etc/nginx/mime.types;  
default\_type application/octet-stream;  
server \{  
listen 80 default\_server;  
server\_name \_;  
location / \{  
\}  
\}  
\}

3、修改 keepalive 配置文件，主备不一样，需要区分  
主 keepalived  
\[root@xuegod63 ~\]\# vim /etc/keepalived/keepalived.conf  
global\_defs \{  
notification\_email \{  
acassen@firewall.loc  
failover@firewall.loc  
sysadmin@firewall.loc  
\}  
notification\_email\_from Alexandre.Cassen@firewall.loc  
smtp\_server 127.0.0.1  
smtp\_connect\_timeout 30  
router\_id NGINX\_MASTER  
\}  
vrrp\_script check\_nginx \{  
script "/etc/keepalived/check\_nginx.sh"  
\}

vrrp\_instance VI\_1 \{  
state MASTER  
interface ens33 \# 修改为实际网卡名  
virtual\_router\_id 51 \# VRRP 路由 ID 实例，每个实例是唯一的  
priority 100 \# 优先级，备服务器设置 90  
advert\_int 1 \# 指定 VRRP 心跳包通告间隔时间，默认 1 秒  
authentication \{  
auth\_type PASS  
auth\_pass 1111  
\}  
\# 虚拟 IP  
virtual\_ipaddress \{  
192.168.1.199/24  
\}  
track\_script \{  
check\_nginx  
\}  
\}

\#vrrp\_script：指定检查 nginx 工作状态脚本（根据 nginx 状态判断是否故障转移）  
\#virtual\_ipaddress：虚拟 IP（VIP）  
\[root@xuegod63 ~\]\# vim /etc/keepalived/check\_nginx.sh  
\#!/bin/bash  
\#1、判断 Nginx 是否存活  
counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )  
if \[ $counter -eq 0 \]; then  
\#2、如果不存活则尝试启动 Nginx  
service nginx start  
sleep 2  
\#3、等待 2 秒后再次获取一次 Nginx 状态  
counter=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$" )  
\#4、再次进行判断，如 Nginx 还不存活则停止 Keepalived，让地址进行漂移  
if \[ $counter -eq 0 \]; then  
service keepalived stop  
fi  
fi  
\[root@xuegod63 ~\]\# chmod +x /etc/keepalived/check\_nginx.sh  
备 keepalive  
\[root@xuegod64 ~\]\# vim /etc/keepalived/keepalived.conf  
global\_defs \{  
notification\_email \{  
acassen@firewall.loc  
failover@firewall.loc  
sysadmin@firewall.loc  
\}  
notification\_email\_from Alexandre.Cassen@firewall.loc  
smtp\_server 127.0.0.1  
smtp\_connect\_timeout 30  
router\_id NGINX\_BACKUP  
\}  
vrrp\_script check\_nginx \{  
script "/etc/keepalived/check\_nginx.sh"  
\}  
vrrp\_instance VI\_1 \{  
state BACKUP  
interface ens33  
virtual\_router\_id 51 \# VRRP 路由 ID 实例，每个实例是唯一的  
priority 90  
advert\_int 1  
authentication \{  
auth\_type PASS  
auth\_pass 1111  
\}  
virtual\_ipaddress \{  
192.168.1.199/24  
\}  
track\_script \{  
check\_nginx  
\}  
\}  
\[root@xuegod64 ~\]\# chmod +x /etc/keepalived/check\_nginx.sh

4、启动服务：  
\[root@xuegod63 ~\]\# systemctl daemon-reload && systemctl start nginx  
\[root@xuegod63 ~\]\# systemctl start keepalived && systemctl enable nginx keepalived  
\[root@xuegod64 ~\]\# systemctl daemon-reload && systemctl start nginx  
\[root@xuegod64 ~\]\# systemctl start keepalived && systemctl enable nginx keepalived

5、测试 vip 是否绑定成功  
\[root@xuegod63 ~\]\# ip addr  
1: lo: <LOOPBACK,UP,LOWER\_UP> mtu 65536 qdisc noqueue state UNKNOWN group  
default qlen 1000  
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  
inet 127.0.0.1/8 scope host lo  
valid\_lft forever preferred\_lft forever  
inet6 ::1/128 scope host  
valid\_lft forever preferred\_lft forever  
2: ens33: <BROADCAST,MULTICAST,UP,LOWER\_UP> mtu 1500 qdisc pfifo\_fast state UP  
group default qlen 1000  
link/ether 00:0c:29:79:9e:36 brd ff:ff:ff:ff:ff:ff  
inet 192.168.1.63/24 brd 192.168.40.255 scope global noprefixroute ens33  
valid\_lft forever preferred\_lft forever  
inet 192.168.1.199/24 scope global secondary ens33  
valid\_lft forever preferred\_lft forever  
inet6 fe80::b6ef:8646:1cfc:3e0c/64 scope link noprefixroute  
valid\_lft forever preferred\_lft forever

6、测试 vip 能否漂移：  
停掉 xuegod63 上的 keepalived，Vip 会漂移到 xuegod64  
\[root@xuegod63 ~\]\# service keepalived stop  
\[root@xuegod64\]\# ip addr6、测试 vip 能否漂移：  
停掉 xuegod63 上的 keepalived，Vip 会漂移到 xuegod64  
\[root@xuegod63 ~\]\# service keepalived stop  
\[root@xuegod64\]\# ip addr

![d522491ca41f4f919d77ffbad6ae2768.png][]

\#启动 xuegod63 上的 nginx 和 keepalived，vip 又会漂移回来  
\[root@xuegod63 ~\]\# systemctl start nginx  
\[root@xuegod63 ~\]\# systemctl start keepalived  
\[root@xuegod63\]\# ip addr

![ca4c67c40a914cfc95da54c64f7f21d9.png][]

备注：  
nginx 配置文件参数解释：  
1、weight 指定了每个后端服务器的权重，用于调节请求的分配比例，例如上述配置中三个后端服务器  
的权重都为 5，则每个服务器会均衡地处理 1/3 的请求。  
2、max\_fails 指定了最大的失败次数，如果在 fail\_timeout 时间内连续失败了 max\_fails 次，则该  
后端服务器会被暂时认为是不可用的，不再向其分配请求。  
3、fail\_timeout 指定了服务器被认为是不可用的时间，即在该时间段内连续失败了 max\_fails 次，则  
该后端服务器会被暂时认为是不可用的。

#### 3、kubeadm 初始化 k8s 集群 ####

\#使用 kubeadm 初始化 k8s 集群  
\[root@xuegod63 ~\]\# kubeadm config print init-defaults > kubeadm.yaml  
根据我们自己的需求修改配置，比如修改 imageRepository 的值，kube-proxy 的模式为  
ipvs，需要注意的是由于我们使用的 containerd 作为运行时，所以在初始化节点的时候需要指定  
cgroupDriver 为 systemd  
kubeadm.yaml 配置文件如下：  
apiVersion: kubeadm.k8s.io/v1beta3  
bootstrapTokens:  
\- groups:  
\- system:bootstrappers:kubeadm:default-node-token  
token: abcdef.0123456789abcdef  
ttl: 24h0m0s  
usages:  
\- signing  
\- authentication  
kind: InitConfiguration![957a352d32574109a18df0f229fab734.png][]

nodeRegistration:  
criSocket: unix:///run/containerd/containerd.sock \#指定 containerd 容器运行时  
imagePullPolicy: IfNotPresent  
\#name: node \#前面加注释  
apiServer:  
timeoutForControlPlane: 4m0s  
apiVersion: kubeadm.k8s.io/v1beta3  
certificatesDir: /etc/kubernetes/pki  
clusterName: kubernetes  
controllerManager: \{\}  
dns: \{\}  
etcd:  
local:  
dataDir: /var/lib/etcd  
imageRepository: registry.cn-hangzhou.aliyuncs.com/google\_containers  
\#指定阿里云镜像仓库  
kind: ClusterConfiguration  
kubernetesVersion: 1.26.0  
\#新增加如下内容：  
controlPlaneEndpoint: 192.168.1.199:16443  
networking:  
dnsDomain: cluster.local  
podSubnet: 10.244.0.0/16 \#指定 pod 网段  
serviceSubnet: 10.96.0.0/12  
scheduler: \{\}  
\#追加如下内容  
\---  
apiVersion: kubeproxy.config.k8s.io/v1alpha1  
kind: KubeProxyConfiguration  
mode: ipvs  
\---  
apiVersion: kubelet.config.k8s.io/v1beta1  
kind: KubeletConfiguration  
cgroupDriver: systemd

\#基于 kubeadm.yaml 初始化 k8s 集群  
\[root@xuegod63 ~\]\# ctr -n=k8s.io images import k8s\_1.26.0.tar.gz  
\[root@xuegod62 ~\]\# ctr -n=k8s.io images import k8s\_1.26.0.tar.gz  
\[root@xuegod64 ~\]\# ctr -n=k8s.io images import k8s\_1.26.0.tar.gz  
\[root@xuegod66 ~\]\# ctr -n=k8s.io images import k8s\_1.26.0.tar.gz

\[root@xuegod63 ~\]\# kubeadm init --config=kubeadm.yaml --ignore-preflighterrors=SystemVerification

![d2656b52998b4d459040855f4a41a14e.png][]

\#配置 kubectl 的配置文件 config，相当于对 kubectl 进行授权，这样 kubectl 命令可以使用这个  
证书对 k8s 集群进行管理  
\[root@xuegod63 ~\]\# mkdir -p $HOME/.kube  
\[root@xuegod63 ~\]\# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config  
\[root@xuegod63 ~\]\# sudo chown $(id -u):$(id -g) $HOME/.kube/config  
\[root@xuegod63 ~\]\# kubectl get nodes  
![a81dfb8d1db44be4aee84fbbfa0f0c78.png][]

#### 4.扩容 k8s 控制节点，把 xuegod62 加入到 k8s 集群 ####

\#把 xuegod63 节点的证书拷贝到 xuegod62 上  
在 xuegod62 创建证书存放目录：  
\[root@xuegod62 ~\]\# cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p  
~/.kube/  
\#把 xuegod63 节点的证书拷贝到 xuegod62 上：  
scp /etc/kubernetes/pki/ca.crt xuegod62:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/ca.key xuegod62:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/sa.key xuegod62:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/sa.pub xuegod62:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/front-proxy-ca.crt xuegod62:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/front-proxy-ca.key xuegod62:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/etcd/ca.crt xuegod62:/etc/kubernetes/pki/etcd/  
scp /etc/kubernetes/pki/etcd/ca.key xuegod62:/etc/kubernetes/pki/etcd/  
在 xuegod63 上查看加入节点的命令：  
\[root@xuegod63 ~\]\# kubeadm token create --print-join-command

显示如下：  
kubeadm join 192.168.1.199:16443 --token zwzcks.u4jd8lj56wpckcwv \\  
\--discovery-token-ca-cert-hash  
sha256:1ba1b274090feecfef58eddc2a6f45590299c1d0624618f1f429b18a064cb728 \\  
在 xuegod62 上执行：  
\[root@xuegod62 ~\]\#kubeadm join 192.168.1.199:16443 --token  
zwzcks.u4jd8lj56wpckcwv \\  
\--discovery-token-ca-cert-hash  
sha256:1ba1b274090feecfef58eddc2a6f45590299c1d0624618f1f429b18a064cb728 \\  
\--control-plane --ignore-preflight-errors=SystemVerification

在 xuegod63 上查看集群状况：  
\[root@xuegod63 ~\]\# kubectl get nodes

![fd961e315f504d678ad8508df7034fb6.png][]

上面可以看到 xuegod62 已经加入到集群了

#### 5、扩容 k8s 控制节点，把 xuegod64 加入到 k8s 集群 ####

在 xuegod64 创建证书存放目录：  
\[root@xuegod64 ~\]\# cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p  
~/.kube/  
\#把 xuegod63 节点的证书拷贝到 xuegod64 上：  
scp /etc/kubernetes/pki/ca.crt xuegod64:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/ca.key xuegod64:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/sa.key xuegod64:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/sa.pub xuegod64:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/front-proxy-ca.crt xuegod64:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/front-proxy-ca.key xuegod64:/etc/kubernetes/pki/  
scp /etc/kubernetes/pki/etcd/ca.crt xuegod64:/etc/kubernetes/pki/etcd/  
scp /etc/kubernetes/pki/etcd/ca.key xuegod64:/etc/kubernetes/pki/etcd/  
在 xuegod63 上查看加入节点的命令：  
\[root@xuegod63 ~\]\# kubeadm token create --print-join-command  
显示如下：  
kubeadm join 192.168.1.199:16443 --token zwzcks.u4jd8lj56wpckcwv \\  
\--discovery-token-ca-cert-hash  
sha256:1ba1b274090feecfef58eddc2a6f45590299c1d0624618f1f429b18a064cb728 \\

**在 xuegod63 上执行：**

**\[root@xuegod63 ~\]\#kubeadm join 192.168.1.199:16443 --token  
zwzcks.u4jd8lj56wpckcwv \\  
\--discovery-token-ca-cert-hash  
sha256:1ba1b274090feecfef58eddc2a6f45590299c1d0624618f1f429b18a064cb728 \\  
\--control-plane --ignore-preflight-errors=SystemVerification**

**在 xuegod63 上查看集群状况：**

**\[root@xuegod63 ~\]\# kubectl get nodes**

![55c3ddc3b7c943d4ac417aa91cbc71e2.png][]

上面可以看到 xuegod64、xuegod62 已经加入到集群了

#### 6、扩容 k8s 集群-添加第一个工作节点 ####

在 xuegod63 上查看加入节点的命令：  
\[root@xuegod63 ~\]\# kubeadm token create --print-join-command  
显示如下：  
kubeadm join 192.168.1.199:16443 --token vulvta.9ns7da3saibv4pg1 --discoverytoken-ca-cert-hash  
sha256:72a0896e27521244850b8f1c3b600087292c2d10f2565adb56381f1f4ba7057a  
把 xuegod66 加入 k8s 集群：  
\[root@xuegod66~\]\# kubeadm join 192.168.1.199:16443 --token  
vulvta.9ns7da3saibv4pg1 --discovery-token-ca-cert-hash  
sha256:72a0896e27521244850b8f1c3b600087292c2d10f2565adb56381f1f4ba7057a --  
ignore-preflight-errors=SystemVerification

![30c606b2be82444f8d256cdbbd3c0841.png][]

\#看到上面说明 xuegod66 节点已经加入到集群了,充当工作节点  
\#在 xuegod63 上查看集群节点状况：  
\[root@xuegod63 ~\]\# kubectl get nodes

![d8218ae860be4e44886426b1c872085d.png][]

\#可以对 xuegod66 打个标签，显示 work  
\[root@xuegod63 ~\]\# kubectl label nodes xuegod66 noderole.kubernetes.io/work=work  
\[root@xuegod63 ~\]\# kubectl get nodes

![1fae734238f5499bb3edd3719fc4ecd0.png][]

#### 7、安装 kubernetes 网络组件-Calico ####

把安装 calico 需要的镜像 calico.tar.gz 传到 xuegod63、xuegod62、xuegod64 和 xuegod66  
节点，手动解压：  
\[root@xuegod63 ~\]\# ctr -n=k8s.io images import calico.tar.gz  
\[root@xuegod62 ~\]\# ctr -n=k8s.io images import calico.tar.gz  
\[root@xuegod64 ~\]\# ctr -n=k8s.io images import calico.tar.gz  
\[root@xuegod66 ~\]\# ctr -n=k8s.io images import calico.tar.gz  
上传 calico.yaml 到 xuegod63 上，使用 yaml 文件安装 calico 网络插件 。

修改 calico.yaml 文件：  
如果机器有多个网卡，需要在 calico 配置文件里指定可以联网的网卡，假如机器只有一个网卡，也  
要指定下，这样就直接找到可以用的网卡了。  
\- name: IP\_AUTODETECTION\_METHOD  
value: "interface=ens33"

![a4d2315c8e8e4237a32dad361697c077.png][]

\[root@xuegod63 ~\]\# kubectl apply -f calico.yaml  
\[root@xuegod63 ~\]\# kubectl get nodes

![646366b57b284471a9c456cddfd20cad.png][]

#### 8、测试 k8s 集群的 DNS 解析和网络是否正常 ####

\#把 busybox-1-28.tar.gz 上传到 xuegod66 节点，手动解压  
\[root@xuegod66 ~\]\# ctr images import busybox-1-28.tar.gz  
\[root@xuegod63 ~\]\# kubectl run busybox --image docker.io/library/busybox:1.28 --  
image-pull-policy=IfNotPresent --restart=Never --rm -it busybox -- sh  
/ \# ping www.baidu.com  
PING www.baidu.com (39.156.66.18): 56 data bytes  
64 bytes from 39.156.66.18: seq=0 ttl=127 time=39.3 ms  
\#通过上面可以看到能访问网络，说明 calico 网络插件已经被正常安装了  
/ \# nslookup kubernetes.default.svc.cluster.local  
Server: 10.96.0.10  
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local  
Name: kubernetes.default.svc.cluster.local  
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local  
看到上面内容，说明 k8s 的 coredns 服务正常  
/ \# exit \#退出 pod  
10.96.0.10 就是我们 coreDNS 的 clusterIP，说明 coreDNS 配置好了。  
解析内部 Service 的名称，是通过 coreDNS 去解析的。

#### 9、etcd 配置成高可用状态 ####

修改 xuegod63、xuegod62、xuegod64 上的 etcd.yaml 文件  
vim /etc/kubernetes/manifests/etcd.yaml

把  
\- --initial-cluster=xuegod63=https://192.168.1.63:2380  
变成如下：  
\- --initialcluster=xuegod63=https://192.168.1.63:2380,xuegod62=https://192.168.1.62:2380,xuegod  
64=https://192.168.1.64:2380

修改成功之后重启 kubelet：  
\[root@xuegod63 ~\]\# systemctl restart kubelet  
\[root@xuegod62 ~\]\# systemctl restart kubelet  
\[root@xuegod64 ~\]\# systemctl restart kubelet

测试 etcd 集群是否配置成功：  
\[root@xuegod63 ~\]\# docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes  
registry.cn-hangzhou.aliyuncs.com/google\_containers/etcd:3.5.4-0 etcdctl --cert  
/etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key --cacert  
/etc/kubernetes/pki/etcd/ca.crt member list

显示如下，说明 etcd 集群配置成功：  
1203cdd3ad75e761, started, xuegod63, https://192.168.1.63:2380,  
https://192.168.1.63:2379, false  
5c9f58513f7f9d01, started, xuegod62, https://192.168.1.62:2380,  
https://192.168.1.62:2379, false  
e4a737a7dcdd6fb5, started, xuegod63, https://192.168.1.64:2380,  
https://192.168.1.64:2379, false

\[root@xuegod63 ~\]\# docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes  
registry.cn-hangzhou.aliyuncs.com/google\_containers/etcd:3.5.4-0 etcdctl --cert  
/etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key --cacert  
/etc/kubernetes/pki/etcd/ca.crt --  
endpoints=https://192.168.1.63:2379,https://192.168.1.62:2379,https://192.168.1.64:2379  
endpoint health --cluster  
显示如下，说明 etcd 集群配置成功：  
https://192.168.1.62:2379 is healthy: successfully committed proposal: took =  
10.808798ms  
https://192.168.1.64:2379 is healthy: successfully committed proposal: took =  
11.179877ms  
https://192.168.1.63:2379 is healthy: successfully committed proposal: took =  
12.32604ms  
\[root@xuegod63 ~\]\# docker run --rm -it --net host -v /etc/kubernetes:/etc/kubernetes  
registry.cn-hangzhou.aliyuncs.com/google\_containers/etcd:3.5.4-0 etcdctl -w table --cert  
/etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key --cacert  
/etc/kubernetes/pki/etcd/ca.crt --

endpoints=https://192.168.1.63:2379,https://192.168.1.62:2379,https://192.168.1.64:2379  
endpoint status --cluster  
显示如下：![7c2cb6d52b9d4ae884f52b4511831f3c.png][]

完.

[b10e5e534a6d449f84ceedc6af1fe5a6.png]: https://img-blog.csdnimg.cn/b10e5e534a6d449f84ceedc6af1fe5a6.png
[1e8d46248b214363bbe91a32c8e6c123.png]: https://img-blog.csdnimg.cn/1e8d46248b214363bbe91a32c8e6c123.png
[3f719d00652b4fd8807c29b5b17df9e0.png]: https://img-blog.csdnimg.cn/3f719d00652b4fd8807c29b5b17df9e0.png
[0c26cb6b7cbe4ff69f8dad9e5148c9b3.png]: https://img-blog.csdnimg.cn/0c26cb6b7cbe4ff69f8dad9e5148c9b3.png
[d731c3d36174465eb43fa608e219a1e0.png]: https://img-blog.csdnimg.cn/d731c3d36174465eb43fa608e219a1e0.png
[1148d68d401645dfab9824b11d1fcead.png]: https://img-blog.csdnimg.cn/1148d68d401645dfab9824b11d1fcead.png
[d522491ca41f4f919d77ffbad6ae2768.png]: https://img-blog.csdnimg.cn/d522491ca41f4f919d77ffbad6ae2768.png
[ca4c67c40a914cfc95da54c64f7f21d9.png]: https://img-blog.csdnimg.cn/ca4c67c40a914cfc95da54c64f7f21d9.png
[957a352d32574109a18df0f229fab734.png]: https://img-blog.csdnimg.cn/957a352d32574109a18df0f229fab734.png
[d2656b52998b4d459040855f4a41a14e.png]: https://img-blog.csdnimg.cn/d2656b52998b4d459040855f4a41a14e.png
[a81dfb8d1db44be4aee84fbbfa0f0c78.png]: https://img-blog.csdnimg.cn/a81dfb8d1db44be4aee84fbbfa0f0c78.png
[fd961e315f504d678ad8508df7034fb6.png]: https://img-blog.csdnimg.cn/fd961e315f504d678ad8508df7034fb6.png
[55c3ddc3b7c943d4ac417aa91cbc71e2.png]: https://img-blog.csdnimg.cn/55c3ddc3b7c943d4ac417aa91cbc71e2.png
[30c606b2be82444f8d256cdbbd3c0841.png]: https://img-blog.csdnimg.cn/30c606b2be82444f8d256cdbbd3c0841.png
[d8218ae860be4e44886426b1c872085d.png]: https://img-blog.csdnimg.cn/d8218ae860be4e44886426b1c872085d.png
[1fae734238f5499bb3edd3719fc4ecd0.png]: https://img-blog.csdnimg.cn/1fae734238f5499bb3edd3719fc4ecd0.png
[a4d2315c8e8e4237a32dad361697c077.png]: https://img-blog.csdnimg.cn/a4d2315c8e8e4237a32dad361697c077.png
[646366b57b284471a9c456cddfd20cad.png]: https://img-blog.csdnimg.cn/646366b57b284471a9c456cddfd20cad.png
[7c2cb6d52b9d4ae884f52b4511831f3c.png]: https://img-blog.csdnimg.cn/7c2cb6d52b9d4ae884f52b4511831f3c.png