采用KubeSphere的kk，部署安装多节点服务的kubernetes-v1.18.6和kubesphere-v3.0.0的踩坑过程记录，及反思

分手后的思念是犯贱 2023-01-03 12:45 174阅读 0赞

# 前言 #

> KubeSphere® 是经 [CNCF 认证][CNCF]的 Kubernetes 主流开源发行版之一，在 Kubernetes 之上提供多种以容器为资源载体的业务功能模块，如多租户管理、集群运维、应用管理、DevOps、微服务治理等功能。

最近微服务，要部署到k8s，采用KubeSphere应用为中心的容器管理平台，于是捣鼓怎样去部署，第一次部署成功，好像不稳定，再次恢复四台服务器镜像，重新部署，其中遇到很多的问题及挫折，在此记录一下，以供大家参考。

# 准备服务器 #

1.  `master：172.16.7.12 centos7.5 + 8cpu + 16G（内存） + 20G（/） + 200G ( data）`
2.  `node5：172.16.7.15 centos7.5 + 8cpu + 16G（内存） + 20G（/） + 1T ( data）`
3.  `node3：172.16.7.16 centos7.5 + 8cpu + 16G（内存） + 20G（/） + 1T ( data）`
4.  `node4：172.16.7.17 centos7.5 + 8cpu + 16G（内存） + 20G（/） + 1T ( data）`

# 系统优化及安装准备软件 #

## 系统自动优化 ##

参考“[ansible安装部署CDH集群,与手动安装部署CDH集群，及CM配置和用户权限配置][ansible_CDH_CDH_CM]”，先进行对系统进行优化：

1.  `sh deploy_robot.sh init_ssh`
2.  `sh deploy_robot.sh init_sys`

关于deploy\_robot.sh的脚本在[https://github.com/fleapx/cdh-deploy-robot.git][https_github.com_fleapx_cdh-deploy-robot.git]

![20210105171304558.png][]

## ansible工具软件安装 ##

1.  `# 控制机器上安装ansible`
2.  `yum install -y ansible`

配置修改 /etc/ansible/hosts ，对需要管理的主机进行配置。

1.  `[all]`
2.  `172.16.7.12 master`
3.  `172.16.7.15 node5`
4.  `172.16.7.16 node3`
5.  `172.16.7.17 node4`

默认配置，需要修改编辑 /etc/ansible/ansible.cfg。

1.  `# uncomment this to disable SSH key host checking`
2.  `host_key_checking = False`

## 其他系统软件安装 ##

1.  `ansible all -m shell -a "wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo"`
2.  `ansible all -m shell -a "sed -i 's/^.*aliyuncs*/#&/g' /etc/yum.repos.d/CentOS-Base.repo"`
3.  `ansible all -m shell -a "wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo"`
4.  `ansible all -m shell -a "yum -y install ebtables socat ipset conntrack nfs-utils rpcbind"`
5.  `ansible all -m shell -a "yum install -y vim wget yum-utils device-mapper-persistent-data lvm2"`

## 集群时间校准 ##

1.  `ansible all -m shell -a "yum install chrony -y"`
2.  `ansible all -m shell -a "systemctl start chronyd"`
3.  `ansible all -m shell -a "sed -i -e '/^server/s/^/#/' -e '1a server ntp.aliyun.com iburst' /etc/chrony.conf"`
4.  `ansible all -m shell -a "systemctl restart chronyd"`
5.  `ansible all -m shell -a "timedatectl set-timezone Asia/Shanghai"`

## 系统其他优化 ##

1.  `ansible all -m shell -a "echo '* soft nofile 655360' >> /etc/security/limits.conf"`
2.  `ansible all -m shell -a "echo '* hard nofile 655360' >> /etc/security/limits.conf"`
3.  `ansible all -m shell -a "echo '* soft nproc 655360' >> /etc/security/limits.conf"`
4.  `ansible all -m shell -a "echo '* hard nproc 655360' >> /etc/security/limits.conf"`
5.  `ansible all -m shell -a "echo '* soft memlock unlimited' >> /etc/security/limits.conf"`
6.  `ansible all -m shell -a "echo '* hard memlock unlimited' >> /etc/security/limits.conf"`
7.  `ansible all -m shell -a "echo 'DefaultLimitNOFILE=1024000' >> /etc/systemd/system.conf"`
8.  `ansible all -m shell -a "echo 'DefaultLimitNPROC=1024000' >> /etc/systemd/system.conf"`

## 开放所有端口 ##

1.  `ansible all -m shell -a "iptables -P INPUT ACCEPT"`
2.  `ansible all -m shell -a "iptables -P FORWARD ACCEPT"`
3.  `ansible all -m shell -a "iptables -P OUTPUT ACCEPT"`
4.  `ansible all -m shell -a "iptables -F"`

## 安装docker ##

[CentOS 7系统下docker安装，及配置阿里云加速][CentOS 7_docker]，及所有配置请参加此文。

1.  `ansible all -m shell -a "yum remove docker docker-common docker-selinux docker-engine"`
2.  `ansible all -m shell -a "yum -y install docker-ce-19.03.8-3.el7"`

## 安装k8s的镜像源 ##

1.  `tee /etc/yum.repos.d/kubernetes.repo <<-'EOF'`
2.  `[kubernetes]`
3.  `name=Kubernetes`
4.  `baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64`
5.  `enabled=1`
6.  `gpgcheck=0`
7.  `repo_gpgcheck=0`
8.  `gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg`
9.  `https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg`
10. `EOF`

# 安装kubernetes-v1.18.6和kubesphere-v3.0 #

官方[https://kubesphere.com.cn/][https_kubesphere.com.cn]多节点安装地址：[https://kubesphere.com.cn/docs/quick-start/all-in-one-on-linux/][https_kubesphere.com.cn_docs_quick-start_all-in-one-on-linux]

![20210106001315902.gif][]

## 下载kk安装文件 ##

1.  `# 在国内先添加一个环境变量`
2.  `export KKZONE=cn`
3.  `curl -sfL https://get-kk.kubesphere.io | VERSION=v1.0.1 sh -`

然后，使kk成为全局可执行文件

mv ./kk /usr/local/bin

## 生成多节点集群的配置 ##

1.  `# 创建一个配置文件模版`
2.  `kk create config --with-kubernetes v1.18.6 --with-kubesphere v3.0.0 -f ./config-kubesphere.yaml`

## 修改配置文件 ##

修改配置文件config-kubesphere.yaml

1.  `apiVersion: kubekey.kubesphere.io/v1alpha1`
2.  `kind: Cluster`
3.  `metadata:`
4.  `name: sample`
5.  `spec:`
6.  `hosts:`
7.  `- {name: master, address: 172.16.7.12, internalAddress: 172.16.7.12, user: root, password: azdebug_it}`
8.  `- {name: node3, address: 172.16.7.16, internalAddress: 172.16.7.16, user: root, password: azdebug_it}`
9.  `- {name: node4, address: 172.16.7.17, internalAddress: 172.16.7.17, user: root, password: azdebug_it}`
10. `- {name: node5, address: 172.16.7.15, internalAddress: 172.16.7.15, user: root, password: azdebug_it}`
11. `roleGroups:`
12. `etcd:`
13. `- node3`
14. `master:`
15. `- master`
16. `worker:`
17. `- node4`
18. `- node3`
19. `- node5`
20. `controlPlaneEndpoint:`
21. `domain: lb.kubesphere.local`
22. `address: "172.16.7.12"`
23. `port: "6443"`
24. `kubernetes:`
25. `version: v1.18.6`
26. `imageRepo: kubesphere`
27. `clusterName: cluster.local`
28. `network:`
29. `plugin: calico`
30. `kubePodsCIDR: 10.233.64.0/18`
31. `kubeServiceCIDR: 10.233.0.0/18`
32. `registry:`
33. `registryMirrors: []`
34. `insecureRegistries: []`
35. `#privateRegistry: dockerhub.kubekey.local`
36. `addons: []`
37. 
38. 
39. `---`
40. `apiVersion: installer.kubesphere.io/v1alpha1`
41. `kind: ClusterConfiguration`
42. `metadata:`
43. `name: ks-installer`
44. `namespace: kubesphere-system`
45. `labels:`
46. `version: v3.0.0`
47. `spec:`
48. `local_registry: ""`
49. `persistence:`
50. `storageClass: ""`
51. `authentication:`
52. `jwtSecret: ""`
53. `etcd:`
54. `monitoring: true`
55. `endpointIps: 172.16.7.16`
56. `port: 2379`
57. `tlsEnable: true`
58. `common:`
59. `es:`
60. `elasticsearchDataVolumeSize: 20Gi`
61. `elasticsearchMasterVolumeSize: 4Gi`
62. `elkPrefix: logstash`
63. `logMaxAge: 7`
64. `mysqlVolumeSize: 20Gi`
65. `minioVolumeSize: 20Gi`
66. `etcdVolumeSize: 20Gi`
67. `openldapVolumeSize: 2Gi`
68. `redisVolumSize: 2Gi`
69. `console:`
70. `enableMultiLogin: true # enable/disable multi login`
71. `port: 30880`
72. `alerting:`
73. `enabled: true`
74. `auditing:`
75. `enabled: true`
76. `devops:`
77. `enabled: true`
78. `jenkinsMemoryLim: 5Gi`
79. `jenkinsMemoryReq: 1500Mi`
80. `jenkinsVolumeSize: 8Gi`
81. `jenkinsJavaOpts_Xms: 1024m`
82. `jenkinsJavaOpts_Xmx: 1024m`
83. `jenkinsJavaOpts_MaxRAM: 2g`
84. `events:`
85. `enabled: true`
86. `ruler:`
87. `enabled: true`
88. `replicas: 2`
89. `logging:`
90. `enabled: true`
91. `logsidecarReplicas: 2`
92. `metrics_server:`
93. `enabled: true`
94. `monitoring:`
95. `prometheusMemoryRequest: 400Mi`
96. `prometheusVolumeSize: 20Gi`
97. `multicluster:`
98. `clusterRole: none # host | member | none`
99. `networkpolicy:`
100. `enabled: true`
101. `notification:`
102. `enabled: true`
103. `openpitrix:`
104. `enabled: true`
105. `servicemesh:`
106. `enabled: true`

## 执行及安装kubernetes v1.18.6和kubesphere v3.0.0 ##

1.  `# 修改配置文件，添加上节点信息（节点名称，ip等）`
2.  `kk create cluster -f ./config-kubesphere.yaml`

## 正确执行结果 ##

1.  `[root@master ~]# sudo -E /bin/sh -c "/usr/local/bin/kubeadm init --config=/etc/kubernetes/kubeadm-config.yaml"`
2.  `W0105 22:45:15.009277 22248 utils.go:26] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]`
3.  `W0105 22:45:15.009521 22248 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]`
4.  `[init] Using Kubernetes version: v1.18.6`
5.  `[preflight] Running pre-flight checks`
6.  `error execution phase preflight: [preflight] Some fatal errors occurred:`
7.  `[ERROR ExternalEtcdVersion]: couldn't load external etcd's certificate and key pair /etc/ssl/etcd/ssl/node-node3.pem, /etc/ssl/etcd/ssl/node-node3-key.pem: open /etc/ssl/etcd/ssl/node-node3.pem: no such file or directory`
8.  `[ERROR ExternalEtcdClientCertificates]: /etc/ssl/etcd/ssl/node-node3.pem doesn't exist`
9.  `[ERROR ExternalEtcdClientCertificates]: /etc/ssl/etcd/ssl/node-node3-key.pem doesn't exist`
10. ``[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` ``
11. `To see the stack trace of this error execute with --v=5 or higher`
12. `[root@master ~]# cd /home/k`
13. `k8s-script/ kubesphere-all-v3.0.0-offline-linux-amd64.tar.gz`
14. `[root@master ~]# cd /home/k`
15. `k8s-script/ kubesphere-all-v3.0.0-offline-linux-amd64.tar.gz`
16. `[root@master ~]# cd /home/k8s-script/`
17. `[root@master k8s-script]# export KKZONE=cn`
18. `[root@master k8s-script]# ./kk create cluster -f ./k8s-config.yaml`
19. `+--------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+`
20. `| name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time |`
21. `+--------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+`
22. `| node5 | y | y | y | y | y | y | y | y | y | | | CST 22:55:35 |`
23. `| node4 | y | y | y | y | y | y | y | y | y | | | CST 22:55:35 |`
24. `| node3 | y | y | y | y | y | y | y | y | y | | | CST 22:55:35 |`
25. `| master | y | y | y | y | y | y | y | y | y | | | CST 22:55:35 |`
26. `+--------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+`
27. 
28. `This is a simple check of your environment.`
29. `Before installation, you should ensure that your machines meet all requirements specified at`
30. `https://github.com/kubesphere/kubekey#requirements-and-recommendations`
31. 
32. `Continue this installation? [yes/no]: yes`
33. `INFO[22:57:10 CST] Downloading Installation Files`
34. `INFO[22:57:10 CST] Downloading kubeadm ...`
35. `INFO[22:57:10 CST] Downloading kubelet ...`
36. `INFO[22:57:10 CST] Downloading kubectl ...`
37. `INFO[22:57:11 CST] Downloading helm ...`
38. `INFO[22:57:11 CST] Downloading kubecni ...`
39. `INFO[22:57:11 CST] Configurating operating system ...`
40. `[node5 172.16.7.15] MSG:`
41. `vm.swappiness = 1`
42. `net.ipv4.tcp_tw_reuse = 1`
43. `net.ipv4.tcp_tw_recycle = 1`
44. `net.ipv4.tcp_keepalive_time = 1200`
45. `net.ipv4.ip_local_port_range = 10000 65000`
46. `net.ipv4.tcp_max_syn_backlog = 8192`
47. `net.ipv4.tcp_max_tw_buckets = 5000`
48. `fs.file-max = 655350`
49. `net.ipv4.route.gc_timeout = 100`
50. `net.ipv4.tcp_syn_retries = 1`
51. `net.ipv4.tcp_synack_retries = 1`
52. `net.core.netdev_max_backlog = 16384`
53. `net.ipv4.tcp_max_orphans = 16384`
54. `net.ipv4.tcp_fin_timeout = 2`
55. `net.core.somaxconn = 32768`
56. `kernel.threads-max = 655360`
57. `kernel.pid_max = 655360`
58. `vm.max_map_count = 393210`
59. `net.ipv4.ip_forward = 1`
60. `net.bridge.bridge-nf-call-arptables = 1`
61. `net.bridge.bridge-nf-call-ip6tables = 1`
62. `net.bridge.bridge-nf-call-iptables = 1`
63. `net.ipv4.ip_local_reserved_ports = 30000-32767`
64. `[node4 172.16.7.17] MSG:`
65. `vm.swappiness = 1`
66. `net.ipv4.tcp_tw_reuse = 1`
67. `net.ipv4.tcp_tw_recycle = 1`
68. `net.ipv4.tcp_keepalive_time = 1200`
69. `net.ipv4.ip_local_port_range = 10000 65000`
70. `net.ipv4.tcp_max_syn_backlog = 8192`
71. `net.ipv4.tcp_max_tw_buckets = 5000`
72. `fs.file-max = 655350`
73. `net.ipv4.route.gc_timeout = 100`
74. `net.ipv4.tcp_syn_retries = 1`
75. `net.ipv4.tcp_synack_retries = 1`
76. `net.core.netdev_max_backlog = 16384`
77. `net.ipv4.tcp_max_orphans = 16384`
78. `net.ipv4.tcp_fin_timeout = 2`
79. `net.core.somaxconn = 32768`
80. `kernel.threads-max = 655360`
81. `kernel.pid_max = 655360`
82. `vm.max_map_count = 393210`
83. `net.ipv4.ip_forward = 1`
84. `net.bridge.bridge-nf-call-arptables = 1`
85. `net.bridge.bridge-nf-call-ip6tables = 1`
86. `net.bridge.bridge-nf-call-iptables = 1`
87. `net.ipv4.ip_local_reserved_ports = 30000-32767`
88. `[master 172.16.7.12] MSG:`
89. `vm.swappiness = 1`
90. `net.ipv4.tcp_tw_reuse = 1`
91. `net.ipv4.tcp_tw_recycle = 1`
92. `net.ipv4.tcp_keepalive_time = 1200`
93. `net.ipv4.ip_local_port_range = 10000 65000`
94. `net.ipv4.tcp_max_syn_backlog = 8192`
95. `net.ipv4.tcp_max_tw_buckets = 5000`
96. `fs.file-max = 655350`
97. `net.ipv4.route.gc_timeout = 100`
98. `net.ipv4.tcp_syn_retries = 1`
99. `net.ipv4.tcp_synack_retries = 1`
100. `net.core.netdev_max_backlog = 16384`
101. `net.ipv4.tcp_max_orphans = 16384`
102. `net.ipv4.tcp_fin_timeout = 2`
103. `net.core.somaxconn = 32768`
104. `kernel.threads-max = 655360`
105. `kernel.pid_max = 655360`
106. `vm.max_map_count = 393210`
107. `net.ipv4.ip_forward = 1`
108. `net.bridge.bridge-nf-call-arptables = 1`
109. `net.bridge.bridge-nf-call-ip6tables = 1`
110. `net.bridge.bridge-nf-call-iptables = 1`
111. `net.ipv4.ip_local_reserved_ports = 30000-32767`
112. `[node3 172.16.7.16] MSG:`
113. `vm.swappiness = 1`
114. `net.ipv4.tcp_tw_reuse = 1`
115. `net.ipv4.tcp_tw_recycle = 1`
116. `net.ipv4.tcp_keepalive_time = 1200`
117. `net.ipv4.ip_local_port_range = 10000 65000`
118. `net.ipv4.tcp_max_syn_backlog = 8192`
119. `net.ipv4.tcp_max_tw_buckets = 5000`
120. `fs.file-max = 655350`
121. `net.ipv4.route.gc_timeout = 100`
122. `net.ipv4.tcp_syn_retries = 1`
123. `net.ipv4.tcp_synack_retries = 1`
124. `net.core.netdev_max_backlog = 16384`
125. `net.ipv4.tcp_max_orphans = 16384`
126. `net.ipv4.tcp_fin_timeout = 2`
127. `net.core.somaxconn = 32768`
128. `kernel.threads-max = 655360`
129. `kernel.pid_max = 655360`
130. `vm.max_map_count = 393210`
131. `net.ipv4.ip_forward = 1`
132. `net.bridge.bridge-nf-call-arptables = 1`
133. `net.bridge.bridge-nf-call-ip6tables = 1`
134. `net.bridge.bridge-nf-call-iptables = 1`
135. `net.ipv4.ip_local_reserved_ports = 30000-32767`
136. `INFO[22:57:25 CST] Installing docker ...`
137. `INFO[22:57:35 CST] Start to download images on all nodes`
138. `[node5] Downloading image: kubesphere/pause:3.2`
139. `[master] Downloading image: kubesphere/pause:3.2`
140. `[node3] Downloading image: kubesphere/etcd:v3.3.12`
141. `[node4] Downloading image: kubesphere/pause:3.2`
142. `[node5] Downloading image: kubesphere/kube-proxy:v1.18.6`
143. `[node4] Downloading image: kubesphere/kube-proxy:v1.18.6`
144. `[node3] Downloading image: kubesphere/pause:3.2`
145. `[master] Downloading image: kubesphere/kube-apiserver:v1.18.6`
146. `[node5] Downloading image: coredns/coredns:1.6.9`
147. `[node4] Downloading image: coredns/coredns:1.6.9`
148. `[master] Downloading image: kubesphere/kube-controller-manager:v1.18.6`
149. `[node5] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12`
150. `[node3] Downloading image: kubesphere/kube-proxy:v1.18.6`
151. `[master] Downloading image: kubesphere/kube-scheduler:v1.18.6`
152. `[node5] Downloading image: calico/kube-controllers:v3.15.1`
153. `[node3] Downloading image: coredns/coredns:1.6.9`
154. `[node4] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12`
155. `[node3] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12`
156. `[master] Downloading image: kubesphere/kube-proxy:v1.18.6`
157. `[node5] Downloading image: calico/cni:v3.15.1`
158. `[node4] Downloading image: calico/kube-controllers:v3.15.1`
159. `[master] Downloading image: coredns/coredns:1.6.9`
160. `[node5] Downloading image: calico/node:v3.15.1`
161. `[node4] Downloading image: calico/cni:v3.15.1`
162. `[node3] Downloading image: calico/kube-controllers:v3.15.1`
163. `[node3] Downloading image: calico/cni:v3.15.1`
164. `[master] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12`
165. `[node5] Downloading image: calico/pod2daemon-flexvol:v3.15.1`
166. `[node4] Downloading image: calico/node:v3.15.1`
167. `[master] Downloading image: calico/kube-controllers:v3.15.1`
168. `[node3] Downloading image: calico/node:v3.15.1`
169. `[node4] Downloading image: calico/pod2daemon-flexvol:v3.15.1`
170. `[node3] Downloading image: calico/pod2daemon-flexvol:v3.15.1`
171. `[master] Downloading image: calico/cni:v3.15.1`
172. `[master] Downloading image: calico/node:v3.15.1`
173. `[master] Downloading image: calico/pod2daemon-flexvol:v3.15.1`
174. `INFO[23:01:26 CST] Generating etcd certs`
175. `INFO[23:01:32 CST] Synchronizing etcd certs`
176. `INFO[23:01:36 CST] Creating etcd service`
177. `INFO[23:01:49 CST] Starting etcd cluster`
178. `[node3 172.16.7.16] MSG:`
179. `Configuration file already exists`
180. `Waiting for etcd to start`
181. `INFO[23:01:58 CST] Refreshing etcd configuration`
182. `INFO[23:01:59 CST] Backup etcd data regularly`
183. `INFO[23:02:00 CST] Get cluster status`
184. `[master 172.16.7.12] MSG:`
185. `Cluster will be created.`
186. `INFO[23:02:01 CST] Installing kube binaries`
187. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubeadm to 172.16.7.16:/tmp/kubekey/kubeadm Done`
188. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubeadm to 172.16.7.15:/tmp/kubekey/kubeadm Done`
189. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubeadm to 172.16.7.17:/tmp/kubekey/kubeadm Done`
190. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubeadm to 172.16.7.12:/tmp/kubekey/kubeadm Done`
191. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubelet to 172.16.7.16:/tmp/kubekey/kubelet Done`
192. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubelet to 172.16.7.12:/tmp/kubekey/kubelet Done`
193. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubelet to 172.16.7.15:/tmp/kubekey/kubelet Done`
194. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubelet to 172.16.7.17:/tmp/kubekey/kubelet Done`
195. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubectl to 172.16.7.16:/tmp/kubekey/kubectl Done`
196. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubectl to 172.16.7.17:/tmp/kubekey/kubectl Done`
197. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubectl to 172.16.7.15:/tmp/kubekey/kubectl Done`
198. `Push /home/k8s-script/kubekey/v1.18.6/amd64/kubectl to 172.16.7.12:/tmp/kubekey/kubectl Done`
199. `Push /home/k8s-script/kubekey/v1.18.6/amd64/helm to 172.16.7.12:/tmp/kubekey/helm Done`
200. `Push /home/k8s-script/kubekey/v1.18.6/amd64/helm to 172.16.7.16:/tmp/kubekey/helm Done`
201. `Push /home/k8s-script/kubekey/v1.18.6/amd64/helm to 172.16.7.15:/tmp/kubekey/helm Done`
202. `Push /home/k8s-script/kubekey/v1.18.6/amd64/helm to 172.16.7.17:/tmp/kubekey/helm Done`
203. `Push /home/k8s-script/kubekey/v1.18.6/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 172.16.7.15:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done`
204. `Push /home/k8s-script/kubekey/v1.18.6/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 172.16.7.16:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done`
205. `Push /home/k8s-script/kubekey/v1.18.6/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 172.16.7.17:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done`
206. `Push /home/k8s-script/kubekey/v1.18.6/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 172.16.7.12:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done`
207. `INFO[23:02:50 CST] Initializing kubernetes cluster`
208. `[master 172.16.7.12] MSG:`
209. `W0105 23:02:51.587457 23847 utils.go:26] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]`
210. `W0105 23:02:51.587685 23847 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]`
211. `[init] Using Kubernetes version: v1.18.6`
212. `[preflight] Running pre-flight checks`
213. `[preflight] Pulling images required for setting up a Kubernetes cluster`
214. `[preflight] This might take a minute or two, depending on the speed of your internet connection`
215. `[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'`
216. `[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"`
217. `[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"`
218. `[kubelet-start] Starting the kubelet`
219. `[certs] Using certificateDir folder "/etc/kubernetes/pki"`
220. `[certs] Generating "ca" certificate and key`
221. `[certs] Generating "apiserver" certificate and key`
222. `[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local lb.kubesphere.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost lb.kubesphere.local master master.cluster.local node3 node3.cluster.local node4 node4.cluster.local node5 node5.cluster.local] and IPs [10.233.0.1 172.16.7.12 127.0.0.1 172.16.7.12 172.16.7.16 172.16.7.17 172.16.7.15 10.233.0.1]`
223. `[certs] Generating "apiserver-kubelet-client" certificate and key`
224. `[certs] Generating "front-proxy-ca" certificate and key`
225. `[certs] Generating "front-proxy-client" certificate and key`
226. `[certs] External etcd mode: Skipping etcd/ca certificate authority generation`
227. `[certs] External etcd mode: Skipping etcd/server certificate generation`
228. `[certs] External etcd mode: Skipping etcd/peer certificate generation`
229. `[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation`
230. `[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation`
231. `[certs] Generating "sa" key and public key`
232. `[kubeconfig] Using kubeconfig folder "/etc/kubernetes"`
233. `[kubeconfig] Writing "admin.conf" kubeconfig file`
234. `[kubeconfig] Writing "kubelet.conf" kubeconfig file`
235. `[kubeconfig] Writing "controller-manager.conf" kubeconfig file`
236. `[kubeconfig] Writing "scheduler.conf" kubeconfig file`
237. `[control-plane] Using manifest folder "/etc/kubernetes/manifests"`
238. `[control-plane] Creating static Pod manifest for "kube-apiserver"`
239. `W0105 23:03:00.466175 23847 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"`
240. `[control-plane] Creating static Pod manifest for "kube-controller-manager"`
241. `W0105 23:03:00.474746 23847 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"`
242. `[control-plane] Creating static Pod manifest for "kube-scheduler"`
243. `W0105 23:03:00.476002 23847 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"`
244. `[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s`
245. `[apiclient] All control plane components are healthy after 32.002873 seconds`
246. `[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace`
247. `[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster`
248. `[upload-certs] Skipping phase. Please see --upload-certs`
249. `[mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"`
250. `[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]`
251. `[bootstrap-token] Using token: 6zsarg.gxg5eijglkupq85j`
252. `[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles`
253. `[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes`
254. `[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials`
255. `[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token`
256. `[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster`
257. `[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace`
258. `[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key`
259. `[addons] Applied essential addon: CoreDNS`
260. `[addons] Applied essential addon: kube-proxy`
261. 
262. `Your Kubernetes control-plane has initialized successfully!`
263. 
264. `To start using your cluster, you need to run the following as a regular user:`
265. 
266. `mkdir -p $HOME/.kube`
267. `sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config`
268. `sudo chown $(id -u):$(id -g) $HOME/.kube/config`
269. 
270. `You should now deploy a pod network to the cluster.`
271. `Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:`
272. `https://kubernetes.io/docs/concepts/cluster-administration/addons/`
273. 
274. `You can now join any number of control-plane nodes by copying certificate authorities`
275. `and service account keys on each node and then running the following as root:`
276. 
277. `kubeadm join lb.kubesphere.local:6443 --token 6zsarg.gxg5eijglkupq85j \`
278. `--discovery-token-ca-cert-hash sha256:8e1405a3da9e80413ab9aec1952a8259490cb174dcc74ecb96c0c5eafa429fd9 \`
279. `--control-plane`
280. 
281. `Then you can join any number of worker nodes by running the following on each as root:`
282. 
283. `kubeadm join lb.kubesphere.local:6443 --token 6zsarg.gxg5eijglkupq85j \`
284. `--discovery-token-ca-cert-hash sha256:8e1405a3da9e80413ab9aec1952a8259490cb174dcc74ecb96c0c5eafa429fd9`
285. `[master 172.16.7.12] MSG:`
286. `service "kube-dns" deleted`
287. `[master 172.16.7.12] MSG:`
288. `service/coredns created`
289. `[master 172.16.7.12] MSG:`
290. `serviceaccount/nodelocaldns created`
291. `daemonset.apps/nodelocaldns created`
292. `[master 172.16.7.12] MSG:`
293. `configmap/nodelocaldns created`
294. `[master 172.16.7.12] MSG:`
295. `I0105 23:04:05.247536 26174 version.go:252] remote version is much newer: v1.20.1; falling back to: stable-1.18`
296. `W0105 23:04:06.468801 26174 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]`
297. `[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace`
298. `[upload-certs] Using certificate key:`
299. `13a993ef56fb292d7ecb9947a3095a0eca6c419dfa148569699c474a8d6c28df`
300. `[master 172.16.7.12] MSG:`
301. `secret/kubeadm-certs patched`
302. `[master 172.16.7.12] MSG:`
303. `secret/kubeadm-certs patched`
304. `[master 172.16.7.12] MSG:`
305. `secret/kubeadm-certs patched`
306. `[master 172.16.7.12] MSG:`
307. `W0105 23:04:08.563212 26292 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]`
308. `kubeadm join lb.kubesphere.local:6443 --token cfoibt.jdyzk3oc1aze53ri --discovery-token-ca-cert-hash sha256:8e1405a3da9e80413ab9aec1952a8259490cb174dcc74ecb96c0c5eafa429fd9`
309. `[master 172.16.7.12] MSG:`
310. `NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME`
311. `master NotReady master 39s v1.18.6 172.16.7.12 <none> CentOS Linux 7 (Core) 3.10.0-1160.11.1.el7.x86_64 docker://19.3.9`
312. `INFO[23:04:09 CST] Deploying network plugin ...`
313. `[master 172.16.7.12] MSG:`
314. `configmap/calico-config created`
315. `customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created`
316. `customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created`
317. `customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created`
318. `customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created`
319. `customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created`
320. `customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created`
321. `customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created`
322. `customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created`
323. `customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created`
324. `customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created`
325. `customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created`
326. `customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created`
327. `customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created`
328. `customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created`
329. `customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created`
330. `clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created`
331. `clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created`
332. `clusterrole.rbac.authorization.k8s.io/calico-node created`
333. `clusterrolebinding.rbac.authorization.k8s.io/calico-node created`
334. `daemonset.apps/calico-node created`
335. `serviceaccount/calico-node created`
336. `deployment.apps/calico-kube-controllers created`
337. `serviceaccount/calico-kube-controllers created`
338. `INFO[23:04:13 CST] Joining nodes to cluster`
339. `[node5 172.16.7.15] MSG:`
340. `W0105 23:04:14.035833 52825 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.`
341. `[preflight] Running pre-flight checks`
342. `[preflight] Reading configuration from the cluster...`
343. `[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'`
344. `W0105 23:04:20.030576 52825 utils.go:26] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]`
345. `[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace`
346. `[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"`
347. `[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"`
348. `[kubelet-start] Starting the kubelet`
349. `[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...`
350. 
351. `This node has joined the cluster:`
352. `* Certificate signing request was sent to apiserver and a response was received.`
353. `* The Kubelet was informed of the new secure connection details.`
354. 
355. `Run 'kubectl get nodes' on the control-plane to see this node join the cluster.`
356. `[node4 172.16.7.17] MSG:`
357. `W0105 23:04:14.432448 53936 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.`
358. `[preflight] Running pre-flight checks`
359. `[preflight] Reading configuration from the cluster...`
360. `[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'`
361. `W0105 23:04:19.870838 53936 utils.go:26] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]`
362. `[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace`
363. `[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"`
364. `[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"`
365. `[kubelet-start] Starting the kubelet`
366. `[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...`
367. 
368. `This node has joined the cluster:`
369. `* Certificate signing request was sent to apiserver and a response was received.`
370. `* The Kubelet was informed of the new secure connection details.`
371. 
372. `Run 'kubectl get nodes' on the control-plane to see this node join the cluster.`
373. `[node3 172.16.7.16] MSG:`
374. `W0105 23:04:14.376894 57091 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.`
375. `[preflight] Running pre-flight checks`
376. `[preflight] Reading configuration from the cluster...`
377. `[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'`
378. `W0105 23:04:20.949568 57091 utils.go:26] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]`
379. `[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace`
380. `[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"`
381. `[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"`
382. `[kubelet-start] Starting the kubelet`
383. `[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...`
384. 
385. `This node has joined the cluster:`
386. `* Certificate signing request was sent to apiserver and a response was received.`
387. `* The Kubelet was informed of the new secure connection details.`
388. 
389. `Run 'kubectl get nodes' on the control-plane to see this node join the cluster.`
390. `[node3 172.16.7.16] MSG:`
391. `node/node3 labeled`
392. `[node5 172.16.7.15] MSG:`
393. `node/node5 labeled`
394. `[node4 172.16.7.17] MSG:`
395. `node/node4 labeled`
396. `[master 172.16.7.12] MSG:`
397. `storageclass.storage.k8s.io/local created`
398. `serviceaccount/openebs-maya-operator created`
399. `clusterrole.rbac.authorization.k8s.io/openebs-maya-operator created`
400. `clusterrolebinding.rbac.authorization.k8s.io/openebs-maya-operator created`
401. `configmap/openebs-ndm-config created`
402. `daemonset.apps/openebs-ndm created`
403. `deployment.apps/openebs-ndm-operator created`
404. `deployment.apps/openebs-localpv-provisioner created`
405. `INFO[23:04:51 CST] Deploying KubeSphere ...`
406. `v3.0.0`
407. `[master 172.16.7.12] MSG:`
408. `namespace/kubesphere-system created`
409. `namespace/kubesphere-monitoring-system created`
410. `[master 172.16.7.12] MSG:`
411. `secret/kube-etcd-client-certs created`
412. `[master 172.16.7.12] MSG:`
413. `namespace/kubesphere-system unchanged`
414. `serviceaccount/ks-installer unchanged`
415. `customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged`
416. `clusterrole.rbac.authorization.k8s.io/ks-installer unchanged`
417. `clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged`
418. `deployment.apps/ks-installer unchanged`
419. `clusterconfiguration.installer.kubesphere.io/ks-installer created`
420. 
421. 
422. `INFO[23:10:23 CST] Installation is complete.`
423. 
424. `Please check the result using the command:`
425. 
426. `kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f`

# 遇到问题及解决方案 #

1.  `Push /home/k8s-script/kubekey/v1.18.6/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 172.16.8.35:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done`
2.  `Push /home/k8s-script/kubekey/v1.18.6/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 172.16.8.36:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done`
3.  `INFO[16:20:31 CST] Initializing kubernetes cluster`
4.  `[master 172.16.8.36] MSG:`
5.  `[preflight] Running pre-flight checks`
6.  `W0105 16:20:38.445541 19396 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory`
7.  `[reset] No etcd config found. Assuming external etcd`
8.  `[reset] Please, manually reset etcd to prevent further issues`
9.  `[reset] Stopping the kubelet service`
10. `[reset] Unmounting mounted directories in "/var/lib/kubelet"`
11. `W0105 16:20:38.453323 19396 cleanupnode.go:99] [reset] Failed to evaluate the "/var/lib/kubelet" directory. Skipping its unmount and cleanup: lstat /var/lib/kubelet: no such file or directory`
12. `[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]`
13. `[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]`
14. `[reset] Deleting contents of stateful directories: [/var/lib/dockershim /var/run/kubernetes /var/lib/cni]`
15. 
16. `The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d`
17. 
18. `The reset process does not reset or clean up iptables rules or IPVS tables.`
19. `If you wish to reset iptables, you must do so manually by using the "iptables" command.`
20. 
21. `If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)`
22. `to reset your system's IPVS tables.`
23. 
24. `The reset process does not clean your kubeconfig files and you must remove them manually.`
25. `Please, check the contents of the $HOME/.kube/config file.`
26. `[master 172.16.8.36] MSG:`
27. `[preflight] Running pre-flight checks`
28. `W0105 16:20:40.305612 19617 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory`
29. `[reset] No etcd config found. Assuming external etcd`
30. `[reset] Please, manually reset etcd to prevent further issues`
31. `[reset] Stopping the kubelet service`
32. `[reset] Unmounting mounted directories in "/var/lib/kubelet"`
33. `W0105 16:20:40.310273 19617 cleanupnode.go:99] [reset] Failed to evaluate the "/var/lib/kubelet" directory. Skipping its unmount and cleanup: lstat /var/lib/kubelet: no such file or directory`
34. `[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]`
35. `[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]`
36. `[reset] Deleting contents of stateful directories: [/var/lib/dockershim /var/run/kubernetes /var/lib/cni]`
37. 
38. `The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d`
39. 
40. `The reset process does not reset or clean up iptables rules or IPVS tables.`
41. `If you wish to reset iptables, you must do so manually by using the "iptables" command.`
42. 
43. `If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)`
44. `to reset your system's IPVS tables.`
45. 
46. `The reset process does not clean your kubeconfig files and you must remove them manually.`
47. `Please, check the contents of the $HOME/.kube/config file.`
48. `ERRO[16:20:41 CST] Failed to init kubernetes cluster: Failed to exec command: sudo -E /bin/sh -c "/usr/local/bin/kubeadm init --config=/etc/kubernetes/kubeadm-config.yaml"`
49. `W0105 16:20:40.826437 19657 utils.go:26] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10]`
50. `W0105 16:20:40.826682 19657 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]`
51. `[init] Using Kubernetes version: v1.18.6`
52. `[preflight] Running pre-flight checks`
53. `error execution phase preflight: [preflight] Some fatal errors occurred:`
54. `[ERROR ExternalEtcdVersion]: couldn't load external etcd's certificate and key pair /etc/ssl/etcd/ssl/node-node3.pem, /etc/ssl/etcd/ssl/node-node3-key.pem: open /etc/ssl/etcd/ssl/node-node3.pem: no such file or directory`
55. `[ERROR ExternalEtcdClientCertificates]: /etc/ssl/etcd/ssl/node-node3.pem doesn't exist`
56. `[ERROR ExternalEtcdClientCertificates]: /etc/ssl/etcd/ssl/node-node3-key.pem doesn't exist`
57. ``[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` ``
58. `To see the stack trace of this error execute with --v=5 or higher: Process exited with status 1 node=172.16.8.36`
59. `WARN[16:20:41 CST] Task failed ...`
60. `WARN[16:20:41 CST] error: interrupted by error`
61. `Error: Failed to init kubernetes cluster: interrupted by error`
62. `Usage:`
63. `kk create cluster [flags]`
64. 
65. `Flags:`
66. `-f, --filename string Path to a configuration file`
67. `-h, --help help for cluster`
68. `--skip-pull-images Skip pre pull images`
69. `--with-kubernetes string Specify a supported version of kubernetes`
70. `--with-kubesphere Deploy a specific version of kubesphere (default v3.0.0)`
71. `-y, --yes Skip pre-check of the installation`
72. 
73. `Global Flags:`
74. `--debug Print detailed information (default true)`
75. 
76. `Failed to init kubernetes cluster: interrupted by error`

在master节点执行

sudo -E /bin/sh -c "/usr/local/bin/kubeadm init --config=/etc/kubernetes/kubeadm-config.yaml"

提取关键错误

1.  `[ERROR ExternalEtcdVersion]: couldn't load external etcd's certificate and key pair /etc/ssl/etcd/ssl/node-node3.pem, /etc/ssl/etcd/ssl/node-node3-key.pem: open /etc/ssl/etcd/ssl/node-node3.pem: no such file or directory`
2.  `[ERROR ExternalEtcdClientCertificates]: /etc/ssl/etcd/ssl/node-node3.pem doesn't exist`
3.  `[ERROR ExternalEtcdClientCertificates]: /etc/ssl/etcd/ssl/node-node3-key.pem doesn't exist`

在master节点执行，再次查看

1.  `[root@master ~]# ls -lh /etc/ssl/etcd/ssl/`
2.  `总用量 32K`
3.  `-rw-r--r--. 1 root root 1.7K 1月 5 19:59 admin-node3-key.pem`
4.  `-rw-r--r--. 1 root root 1.4K 1月 5 19:59 admin-node3.pem`
5.  `-rw-r--r--. 1 root root 1.7K 1月 5 19:59 ca-key.pem`
6.  `-rw-r--r--. 1 root root 1.1K 1月 5 19:59 ca.pem`
7.  `-rw-r--r--. 1 root root 1.7K 1月 5 19:59 member-node3-key.pem`
8.  `-rw-r--r--. 1 root root 1.4K 1月 5 19:59 member-node3.pem`
9.  `-rw-r--r--. 1 root root 1.7K 1月 5 19:59 node-master-key.pem`
10. `-rw-r--r--. 1 root root 1.4K 1月 5 19:59 node-master.pem`

原来在/etc/ssl/etcd/ssl/中真不存在node-node3-key.pem、node-node3.pem，真么办？原来是我选择的是member模式。

解决方法

1.  `[root@master ~]# cp /etc/ssl/etcd/ssl/member-node3-key.pem /etc/ssl/etcd/ssl/node-node3-key.pem`
2.  `[root@master ~]# cp /etc/ssl/etcd/ssl/member-node3.pem /etc/ssl/etcd/ssl/node-node3.pem`

再次执行

1.  `export KKZONE=cn`
2.  `./kk create cluster -f ./k8s-config.yaml`

查看安装KubeSphere日志：

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

成功安装KubeSphere3.0

# 登陆kubesphere #

1.  `#ip:30880`
2.  `#用户名：admin`
3.  `#默认密码：P@88w0rd`

# ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FzNDU4OXNk_size_16_color_FFFFFF_t_70][] #

# 优秀参考 #

*  [https://www.cnblogs.com/elfcafe/p/13779619.html][https_www.cnblogs.com_elfcafe_p_13779619.html]
 *  [https://www.cnblogs.com/carriezhangyan/p/11551192.html][https_www.cnblogs.com_carriezhangyan_p_11551192.html]
 *  [https://www.cnblogs.com/xiao987334176/p/13267339.html][https_www.cnblogs.com_xiao987334176_p_13267339.html]
 *  [https://www.cnblogs.com/wenyang321/p/14086162.html][https_www.cnblogs.com_wenyang321_p_14086162.html]
 *  [https://www.cnblogs.com/technology178/p/13547342.html][https_www.cnblogs.com_technology178_p_13547342.html]
 *  [https://www.cnblogs.com/wuchangblog/p/14091717.html][https_www.cnblogs.com_wuchangblog_p_14091717.html]
 *  [https://www.cnblogs.com/it-peng/p/11393812.html][https_www.cnblogs.com_it-peng_p_11393812.html]

推荐这个博文：[https://blog.csdn.net/weixin\_43141746/article/details/110261158][https_blog.csdn.net_weixin_43141746_article_details_110261158]

[CNCF]: https://landscape.cncf.io/format=members&selected=kubesphere-member
[ansible_CDH_CDH_CM]: https://blog.csdn.net/as4589sd/article/details/108903591
[https_github.com_fleapx_cdh-deploy-robot.git]: https://github.com/fleapx/cdh-deploy-robot.git
[20210105171304558.png]: /images/20221119/3c1323f7ec1b484aade68fd97e8a5412.png
[CentOS 7_docker]: https://blog.csdn.net/as4589sd/article/details/109161443
[https_kubesphere.com.cn]: https://kubesphere.com.cn/
[https_kubesphere.com.cn_docs_quick-start_all-in-one-on-linux]: https://kubesphere.com.cn/docs/quick-start/all-in-one-on-linux/
[20210106001315902.gif]: /images/20221119/98d9985b385e4d969bf6e816f90a2be5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2FzNDU4OXNk_size_16_color_FFFFFF_t_70]: /images/20221119/7f41487e6c014b44a835b624e71ab4b4.png
[https_www.cnblogs.com_elfcafe_p_13779619.html]: https://www.cnblogs.com/elfcafe/p/13779619.html
[https_www.cnblogs.com_carriezhangyan_p_11551192.html]: https://www.cnblogs.com/carriezhangyan/p/11551192.html
[https_www.cnblogs.com_xiao987334176_p_13267339.html]: https://www.cnblogs.com/xiao987334176/p/13267339.html
[https_www.cnblogs.com_wenyang321_p_14086162.html]: https://www.cnblogs.com/wenyang321/p/14086162.html
[https_www.cnblogs.com_technology178_p_13547342.html]: https://www.cnblogs.com/technology178/p/13547342.html
[https_www.cnblogs.com_wuchangblog_p_14091717.html]: https://www.cnblogs.com/wuchangblog/p/14091717.html
[https_www.cnblogs.com_it-peng_p_11393812.html]: https://www.cnblogs.com/it-peng/p/11393812.html
[https_blog.csdn.net_weixin_43141746_article_details_110261158]: https://blog.csdn.net/weixin_43141746/article/details/110261158