二进制搭建Kubenetes集群

我不是女神ヾ 2022-10-14 10:57 116阅读 0赞

环境和Kubeadm方式搭建一样，使用两个节点来搭建。

注意yml文件的空格，有时直接复制空格会丢失造成启动报错。

# 1 初始化 #

# 关闭防火墙
    systemctl stop firewalld
    systemctl disable firewalld
    
    # 关闭 selinux
    sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
    setenforce 0 # 临时
    
    # 关闭 swap
    swapoff -a # 临时
    sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
    
    # 根据规划设置主机名 主节点m1 工作节点n1
    hostnamectl set-hostname <hostname>
    
    # 在 master 添加 hosts
    cat >> /etc/hosts << EOF
    master的ip m1
    node的ip n1
    EOF
    
    # 将桥接的 IPv4 流量传递到 iptables 的链
    cat > /etc/sysctl.d/k8s.conf << EOF
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    EOF
    sysctl --system # 生效
    
    # 时间同步
    yum install ntpdate -y
    ntpdate time.windows.com

# 2 部署Etcd #

### 1 准备 cfssl 证书生成工具 ###

cfssl 是一个开源的证书管理工具，使用 json 文件生成证书，相比 openssl 更方便使用。 找任意一台服务器操作，这里用 Master 节点。

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
    wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
    wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
    chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
    mv cfssl_linux-amd64 /usr/local/bin/cfssl
    mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
    mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

### 2 生成 Etcd 证书 ###

自签证书颁发机构（CA） 创建工作目录：

mkdir -p ~/TLS/{etcd,k8s} 
    cd TLS/etcd

自签 CA：

cat > ca-config.json<< EOF
    {
    "signing": {
    "default": {
    "expiry": "87600h"
    },
    "profiles": {
    "www": {
    "expiry": "87600h",
    "usages": [
    "signing",
    "key encipherment",
    "server auth",
    "client auth"
    ]
    }
    }
    }
    }
    EOF
    cat > ca-csr.json<< EOF
    {
    "CN": "etcd CA",
    "key": {
    "algo": "rsa",
    "size": 2048
    },
    "names": [
    {
    "C": "CN",
    "L": "Beijing",
    "ST": "Beijing"
    }
    ]
    }
    EOF

生成证书：

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
    ls *pem
    ca-key.pem ca.pem

使用自签 CA 签发 Etcd HTTPS 证书

创建证书申请文件：

# ip为两个节点的ip
    cat > server-csr.json<< EOF
    {
    "CN": "etcd",
    "hosts": [
    "m1节点ip",
    "n1节点ip"
    ],
    "key": {
    "algo": "rsa",
    "size": 2048
    },
    "names": [
    {
    "C": "CN",
    "L": "BeiJing",
    "ST": "BeiJing"
    }
    ]
    }
    EOF

注：上述文件 hosts 字段中 IP 为所有 etcd 节点的集群内部通信 IP，一个都不能少！为了 方便后期扩容可以多写几个预留的 IP。

生成证书：

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
    
    ls server*pem
    server-key.pem server.pem

# 3 搭建ETCD集群 #

Etcd 是一个分布式键值存储系统，Kubernetes 使用 Etcd 进行数据存储。

为了节省资源，在这两个节点上搭建一个ETCD集群。在1个节点上做如下操作，完成后按说明进行拷贝到另外一个节点上。

#下载安装包 上传到虚拟机上
    #下载地址：https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-
    linux-amd64.tar.gz
    
    # 创建工作目录 解压下载包
    mkdir /opt/etcd/{bin,cfg,ssl} –p
    tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
    mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

创建ETCD的配置文件

cat > /opt/etcd/cfg/etcd.conf << EOF
    #[Member]
    ETCD_NAME="etcd-1"
    ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
    ETCD_LISTEN_PEER_URLS="https://节点IP:2380"
    ETCD_LISTEN_CLIENT_URLS="https://节点IP:2379"
    #[Clustering]
    ETCD_INITIAL_ADVERTISE_PEER_URLS="https://节点IP:2380"
    ETCD_ADVERTISE_CLIENT_URLS="https://节点IP:2379"
    ETCD_INITIAL_CLUSTER="etcd-1=https://节点IP:2380,etcd-2=https://节点IP:2380"
    ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    ETCD_INITIAL_CLUSTER_STATE="new"
    EOF

配置说明

ETCD\_NAME：节点名称，集群中唯一 ETCD\_DATA\_DIR：数据目录 ETCD\_LISTEN\_PEER\_URLS：集群通信监听地址 ETCD\_LISTEN\_CLIENT\_URLS：客户端访问监听地址 ETCD\_INITIAL\_ADVERTISE\_PEER\_URLS：集群通告地址 ETCD\_ADVERTISE\_CLIENT\_URLS：客户端通告地址 ETCD\_INITIAL\_CLUSTER：集群节点地址 ETCD\_INITIAL\_CLUSTER\_TOKEN：集群 Token ETCD\_INITIAL\_CLUSTER\_STATE：加入集群的当前状态，new 是新集群，existing 表示加入 已有集群

systemd 管理 etcd

cat > /usr/lib/systemd/system/etcd.service << EOF
    [Unit]
    Description=Etcd Server
    After=network.target
    After=network-online.target
    Wants=network-online.target
    [Service]
    Type=notify
    EnvironmentFile=/opt/etcd/cfg/etcd.conf
    ExecStart=/opt/etcd/bin/etcd \
    --cert-file=/opt/etcd/ssl/server.pem \
    --key-file=/opt/etcd/ssl/server-key.pem \
    --peer-cert-file=/opt/etcd/ssl/server.pem \
    --peer-key-file=/opt/etcd/ssl/server-key.pem \
    --trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --logger=zap
    Restart=on-failure
    LimitNOFILE=65536
    [Install]
    WantedBy=multi-user.target
    EOF

拷贝刚才生成的证书

cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/

这个节点的配置已完成，将该节点的配置拷贝到另外一个节点。

scp -r /opt/etcd/ root@目标节点的IP:/opt/
    
    scp /usr/lib/systemd/system/etcd.service root@目标节点的IP:/usr/lib/systemd/system/

在另外一个节点上修改etcd的配置文件，主要是etcd\_name和节点IP

vi /opt/etcd/cfg/etcd.conf
    #[Member]
    ETCD_NAME="etcd-2" # 修改此处，节点 2 改为 etcd-2，节点 3 改为 etcd-3
    ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
    ETCD_LISTEN_PEER_URLS="https://节点IP:2380" # 修改此处为当前服务器 IP
    ETCD_LISTEN_CLIENT_URLS="https://节点IP:2379" # 修改此处为当前服务器 IP
    #[Clustering]
    ETCD_INITIAL_ADVERTISE_PEER_URLS="https://节点IP:2380" # 修改此处为当前
    服务器 IP
    ETCD_ADVERTISE_CLIENT_URLS="https://节点IP:2379" # 修改此处为当前服务器
    IP
    ETCD_INITIAL_CLUSTER="etcd-1=https://节点IP:2380,etcd2=https://另外一个节点IP:2380"
    ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    ETCD_INITIAL_CLUSTER_STATE="new

在每个节点上启动etcd

systemctl daemon-reload
    systemctl start etcd
    
    # 开机自启动设置
    systemctl enable etcd

启动后在一个节点查看状态

# 查看启动状态，如下表示成功
    [root@localhost etcd]# systemctl status etcd
    ● etcd.service - Etcd Server
       Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)
       Active: active (running) since 一 2021-05-24 11:34:10 CST; 19min ago
     Main PID: 2230 (etcd)
       CGroup: /system.slice/etcd.service
               └─2230 /opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-fil...
    
    5月 24 11:34:10 m1 etcd[2230]: {"level":"info","ts":"2021-05-24T11:34:10.261+0800","caller":"raft/raft.go:1302","msg":"90...tions"}
    5月 24 11:34:10 m1 etcd[2230]: {"level":"info","ts":"2021-05-24T11:34:10.261+0800","caller":"raft/raft.go:765","msg":"909... 1538"}
    5月 24 11:34:10 m1 etcd[2230]: {"level":"info","ts":"2021-05-24T11:34:10.261+0800","caller":"raft/node.go:325","msg":"raf... 1538"}
    5月 24 11:34:10 m1 etcd[2230]: {"level":"info","ts":"2021-05-24T11:34:10.265+0800","caller":"etcdserver/server.go:2036","msg":"p...
    5月 24 11:34:10 m1 systemd[1]: Started Etcd Server.
    5月 24 11:34:10 m1 etcd[2230]: {"level":"info","ts":"2021-05-24T11:34:10.266+0800","caller":"embed/serve.go:191","msg":"s...:2379"}
    5月 24 11:34:10 m1 etcd[2230]: {"level":"info","ts":"2021-05-24T11:34:10.274+0800","caller":"etcdserver/server.go:2527","...:"3.4"}
    5月 24 11:34:10 m1 etcd[2230]: {"level":"info","ts":"2021-05-24T11:34:10.275+0800","caller":"membership/cluster.go:558","...:"3.4"}
    5月 24 11:34:10 m1 etcd[2230]: {"level":"info","ts":"2021-05-24T11:34:10.275+0800","caller":"api/capability.go:76","msg":...:"3.4"}
    5月 24 11:34:10 m1 etcd[2230]: {"level":"info","ts":"2021-05-24T11:34:10.275+0800","caller":"etcdserver/server.go:2559","...:"3.4"}
    Hint: Some lines were ellipsized, use -l to show in full.

# 5 安装Docker #

下载地址：https://download.docker.com/linux/static/stable/x86\_64/docker19.03.9.tgz 以下在所有节点操作。这里采用二进制安装，用 yum 安装也一样。

解压二进制包
    tar zxvf docker-19.03.9.tgz
    mv docker/* /usr/bin

systemd 管理 docker

cat > /usr/lib/systemd/system/docker.service << EOF
    [Unit]
    Description=Docker Application Container Engine
    Documentation=https://docs.docker.com
    After=network-online.target firewalld.service
    Wants=network-online.target
    [Service]
    Type=notify
    ExecStart=/usr/bin/dockerd
    ExecReload=/bin/kill -s HUP $MAINPID
    LimitNOFILE=infinity
    LimitNPROC=infinity
    LimitCORE=infinity
    TimeoutStartSec=0
    Delegate=yes
    KillMode=process
    Restart=on-failure
    StartLimitBurst=3
    StartLimitInterval=60s
    [Install]
    WantedBy=multi-user.target
    EOF

配置阿里云加速

mkdir /etc/docker
    cat > /etc/docker/daemon.json << EOF
    {
    "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
    }
    EOF

启动并设置开机启动

systemctl daemon-reload
    systemctl start docker
    systemctl enable docker

# 6 部署Master Node #

## 1 生成 kube-apiserver 证书 ##

# 创建文件
    cat > ca-config.json<< EOF
    {
    "signing": {
    "default": {
    "expiry": "87600h"
    },
    "profiles": {
    "kubernetes": {
    "expiry": "87600h",
    "usages": [
    "signing",
    "key encipherment",
    "server auth",
    "client auth"
    ]
    }
    }
    }
    }
    EOF
    
    # 创建文件
    cat > ca-csr.json<< EOF
    {
    "CN": "kubernetes",
    "key": {
    "algo": "rsa",
    "size": 2048
    },
    "names": [
    {
    "C": "CN",
    "L": "Beijing",
    "ST": "Beijing",
    "O": "k8s",
    "OU": "System"
    }
    ]
    }
    EOF

基于以上两个json文件，生成证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
    ls *pem
    ca-key.pem ca.pem

使用自签 CA 签发 kube-apiserver HTTPS 证书

# 切换目录
    
    cd /root/TLS/k8s

创建证书申请文件

# IP 为集群NODE 所有IP
    cat > server-csr.json<< EOF
    {
    "CN": "kubernetes",
    "hosts": [
    "10.0.0.1",
    "127.0.0.1",
    "集群节点IP1",
    "集群节点IP2",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
    ],
    "key": {
    "algo": "rsa",
    "size": 2048
    },
    "names": [
    {
    "C": "CN",
    "L": "BeiJing",
    "ST": "BeiJing",
    "O": "k8s",
    "OU": "System"
    }
    ]
    }
    EOF

生成证书（需要指定上一步的证书ca-key.pem ca.pem的目录）

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -
    profile=kubernetes server-csr.json | cfssljson -bare server
    ls server*pem
    server-key.pem server.pem

## 2 部署 kube-apiserver ##

# 创建配置文件 etcd-servers 为etcd集群地址 
    cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
    KUBE_APISERVER_OPTS="--logtostderr=false \\
    --v=2 \\
    --log-dir=/opt/kubernetes/logs \
    --etcd-servers=https://192.168.204.140:2379,https://192.168.204.141:2379 \\
    --bind-address=本机IP \\
    --secure-port=6443 \\
    --advertise-address=本机IP\\
    --allow-privileged=true \\
    --service-cluster-ip-range=10.0.0.0/24 \\
    --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
    --authorization-mode=RBAC,Node \\
    --enable-bootstrap-token-auth=true \\
    --token-auth-file=/opt/kubernetes/cfg/token.csv \\
    --service-node-port-range=30000-32767 \\
    --kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
    --kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
    --tls-cert-file=/opt/kubernetes/ssl/server.pem \\
    --tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
    --client-ca-file=/opt/kubernetes/ssl/ca.pem \\
    --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
    --etcd-cafile=/opt/etcd/ssl/ca.pem \\
    --etcd-certfile=/opt/etcd/ssl/server.pem \\
    --etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
    --audit-log-maxage=30 \\
    --audit-log-maxbackup=3 \\
    --audit-log-maxsize=100 \\
    --audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

注：上面两个\\ \\ 第一个是转义符，第二个是换行符，使用转义符是为了使用 EOF 保留换 行符。 –logtostderr：启用日志 —v：日志等级 –log-dir：日志目录 –etcd-servers：etcd 集群地址 –bind-address：监听地址 –secure-port：https 安全端口 –advertise-address：集群通告地址 –allow-privileged：启用授权 –service-cluster-ip-range：Service 虚拟 IP 地址段 –enable-admission-plugins：准入控制模块 –authorization-mode：认证授权，启用 RBAC 授权和节点自管理 –enable-bootstrap-token-auth：启用 TLS bootstrap 机制 –token-auth-file：bootstrap token 文件 –service-node-port-range：Service nodeport 类型默认分配端口范围 –kubelet-client-xxx：apiserver 访问 kubelet 客户端证书 –tls-xxx-file：apiserver https 证书 –etcd-xxxfile：连接 Etcd 集群证书 –audit-log-xxx：审计日志

把刚才生成的证书拷贝到配置文件中的路径：

# ca证书 https证书
     cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

创建上述配置文件中 token 文件：

cat > /opt/kubernetes/cfg/token.csv << EOF
    c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:nodebootstrapper"
    EOF

systemd 管理 apiserver

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
    [Unit]
    Description=Kubernetes API Server
    Documentation=https://github.com/kubernetes/kubernetes
    [Service]
    EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
    ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

启动并设置开机启动

systemctl daemon-reload
    systemctl start kube-apiserver
    systemctl enable kube-apiserver

启动成功

[root@m1 ssl]# systemctl status kube-apiserver
    ● kube-apiserver.service - Kubernetes API Server
       Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
       Active: active (running) since 一 2021-05-24 16:27:57 CST; 6s ago
         Docs: https://github.com/kubernetes/kubernetes
     Main PID: 14501 (kube-apiserver)
        Tasks: 10
       Memory: 268.6M
       CGroup: /system.slice/kube-apiserver.service
               └─14501 /opt/kubernetes/bin/kube-apiserver --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --etcd-servers=htt...
    
    5月 24 16:27:57 m1 systemd[1]: Started Kubernetes API Server.
    5月 24 16:28:01 m1 kube-apiserver[14501]: E0524 16:28:01.077680   14501 controller.go:152] Unable to remove old endpoints ...orMsg:
    Hint: Some lines were ellipsized, use -l to show in full.

授权 kubelet-bootstrap 用户允许请求证书

kubectl create clusterrolebinding kubelet-bootstrap \
    --clusterrole=system:node-bootstrapper \
    --user=kubelet-bootstrap

## 3 部署署 kube-controller-manager ##

# 创建配置文件
    cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
    KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
    --v=2 \
    --log-dir=/opt/kubernetes/logs \
    --leader-elect=true \
    --master=127.0.0.1:8080 \
    --bind-address=127.0.0.1 \
    --allocate-node-cidrs=true \
    --cluster-cidr=10.244.0.0/16 \
    --service-cluster-ip-range=10.0.0.0/24 \
    --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
    --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
    --root-ca-file=/opt/kubernetes/ssl/ca.pem \
    --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
    --experimental-cluster-signing-duration=87600h0m0s"
    EOF

–master：通过本地非安全本地端口 8080 连接 apiserver。 –leader-elect：当该组件启动多个时，自动选举（HA） –cluster-signing-cert-file/–cluster-signing-key-file：自动为 kubelet 颁发证书 的 CA，与 apiserver 保持一致

systemd 管理 controller-manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
    [Unit]
    Description=Kubernetes Controller Manager
    Documentation=https://github.com/kubernetes/kubernetes
    [Service]
    EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
    ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

. 启动并设置开机启动

systemctl daemon-reload
    systemctl start kube-controller-manager
    systemctl enable kube-controller-manager

启动成功

[root@m1 ssl]# systemctl status kube-controller-manager
    ● kube-controller-manager.service - Kubernetes Controller Manager
       Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
       Active: active (running) since 一 2021-05-24 16:38:29 CST; 20s ago
         Docs: https://github.com/kubernetes/kubernetes
     Main PID: 14566 (kube-controller)
       CGroup: /system.slice/kube-controller-manager.service
               └─14566 /opt/kubernetes/bin/kube-controller-manager --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-...
    
    5月 24 16:38:29 m1 systemd[1]: Started Kubernetes Controller Manager.
    5月 24 16:38:45 m1 kube-controller-manager[14566]: E0524 16:38:45.645219   14566 core.go:229] failed to start cloud node li...vided
    Hint: Some lines were ellipsized, use -l to show in full.

## 4 部署 kube-scheduler ##

# 创建配置文件
    cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
    KUBE_SCHEDULER_OPTS="--logtostderr=false \
    --v=2 \
    --log-dir=/opt/kubernetes/logs \
    --leader-elect \
    --master=127.0.0.1:8080 \
    --bind-address=127.0.0.1"
    EOF

–master：通过本地非安全本地端口 8080 连接 apiserver。 –leader-elect：当该组件启动多个时，自动选举（HA）

systemd 管理 scheduler

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
    [Unit]
    Description=Kubernetes Scheduler
    Documentation=https://github.com/kubernetes/kubernetes
    [Service]
    EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
    ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target
    EOF

启动并设置开机启动

systemctl daemon-reload
    systemctl start kube-scheduler
    systemctl enable kube-scheduler

# 启动成功
    [root@m1 ssl]# systemctl status kube-scheduler
    ● kube-scheduler.service - Kubernetes Scheduler
       Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
       Active: active (running) since 一 2021-05-24 16:43:37 CST; 28s ago
         Docs: https://github.com/kubernetes/kubernetes
     Main PID: 14645 (kube-scheduler)
       CGroup: /system.slice/kube-scheduler.service
               └─14645 /opt/kubernetes/bin/kube-scheduler --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect --m...
    
    5月 24 16:43:37 m1 systemd[1]: Started Kubernetes Scheduler.
    5月 24 16:43:37 m1 kube-scheduler[14645]: I0524 16:43:37.897368   14645 registry.go:150] Registering EvenPodsSpread predic...nction
    5月 24 16:43:37 m1 kube-scheduler[14645]: I0524 16:43:37.897706   14645 registry.go:150] Registering EvenPodsSpread predic...nction
    Hint: Some lines were ellipsized, use -l to show in full.

## 5 查看各组件启动状态 ##

# 查看各组件启动状态，如下说明启动成功
    [root@m1 ssl]# kubectl get cs
    NAME                 STATUS    MESSAGE             ERROR
    controller-manager   Healthy   ok
    scheduler            Healthy   ok
    etcd-0               Healthy   {"health":"true"}
    etcd-1               Healthy   {"health":"true"}

至此 Master节点部署成功

# 7 部署WorkNode #

## 1 部署kubelet ##

# 在work node创建工作目录
    mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
    
    # 使用scp 在master节点上拷贝 kubelet kube-proxy 到work node的 /opt/kubernetes/bin
    cd ~/kubernetes/server/bin
    scp kubelet kube-proxy root@工作节点IP:/opt/kubernetes/bin/

# 创建配置文件
    cat > /opt/kubernetes/cfg/kubelet.conf << EOF
    KUBELET_OPTS="--logtostderr=false \\
    --v=2 \\
    --log-dir=/opt/kubernetes/logs \\
    --hostname-override=n1 \\
    --network-plugin=cni \\
    --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
    --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
    --config=/opt/kubernetes/cfg/kubelet-config.yml \\
    --cert-dir=/opt/kubernetes/ssl \\
    --pod-infra-container-image=lizhenliang/pause-amd64:3.0"
    EOF

注  hostname-override=n1 设置成当前节点hostname

cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
    kind: KubeletConfiguration
    apiVersion: kubelet.config.k8s.io/v1beta1
    address: 0.0.0.0
    port: 10250
    readOnlyPort: 10255
    cgroupDriver: cgroupfs
    clusterDNS:
    - 10.0.0.2
    clusterDomain: cluster.local
    failSwapOn: false
    authentication:
      anonymous:
        enabled: false
      webhook:
        cacheTTL: 2m0s
        enabled: true
      x509:
        clientCAFile: /opt/kubernetes/ssl/ca.pem
    authorization:
      mode: Webhook
      webhook:
        cacheAuthorizedTTL: 5m0s
        cacheUnauthorizedTTL: 30s
    evictionHard:
    imagefs.available: 15%
    memory.available: 100Mi
    nodefs.available: 10%
    nodefs.inodesFree: 5%
    maxOpenFiles: 1000000
    maxPods: 110
    EOF

#在master执行 把/opt/kubernetes/ssl下的ca证书拷贝到work节点的/opt/kubernetes/ssl/
    scp ca.pem root@work ip:/opt/kubernetes/ssl/

生成 bootstrap.kubeconfig 文件

#在master生成,执行下面的命令会得到文件 拷贝到work节点
    KUBE_APISERVER="https://master节点IP:6443" # apiserver IP:PORT
    TOKEN="c47ffb939f5ca36231d9e3121a252940" # 与 token.csv 里保持一致
    # 生成 kubelet bootstrap kubeconfig 配置文件
    kubectl config set-cluster kubernetes \
    --certificate-authority=/opt/kubernetes/ssl/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=bootstrap.kubeconfig
    kubectl config set-credentials "kubelet-bootstrap" \
    --token=${TOKEN} \
    --kubeconfig=bootstrap.kubeconfig
    kubectl config set-context default \
    --cluster=kubernetes \
    --user="kubelet-bootstrap" \
    --kubeconfig=bootstrap.kubeconfig
    kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
    
    
    #scp拷贝到配置文件路径：
     scp bootstrap.kubeconfig root@work节点IP:/opt/kubernetes/cfg/

# 启动脚本
    cat > /usr/lib/systemd/system/kubelet.service << EOF
    [Unit]
    Description=Kubernetes Kubelet
    After=docker.service
    [Service]
    EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
    ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
    Restart=on-failure
    LimitNOFILE=65536
    [Install]
    WantedBy=multi-user.target
    EOF

启动并设置开机启动

systemctl daemon-reload
    systemctl start kubelet
    systemctl enable kubelet

如下说明成功

[root@n1 cfg]# systemctl status kubelet
    ● kubelet.service - Kubernetes Kubelet
       Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
       Active: active (running) since 一 2021-05-24 18:55:28 CST; 43min ago
     Main PID: 13737 (kubelet)
        Tasks: 15
       Memory: 30.6M
       CGroup: /system.slice/kubelet.service
               └─13737 /opt/kubernetes/bin/kubelet --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --hostname-override=n1 --...
    
    5月 24 19:38:31 n1 kubelet[13737]: E0524 19:38:31.948836   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    5月 24 19:38:36 n1 kubelet[13737]: E0524 19:38:36.958487   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    5月 24 19:38:41 n1 kubelet[13737]: E0524 19:38:41.966524   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    5月 24 19:38:46 n1 kubelet[13737]: E0524 19:38:46.972973   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    5月 24 19:38:51 n1 kubelet[13737]: E0524 19:38:51.981558   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    5月 24 19:38:56 n1 kubelet[13737]: E0524 19:38:56.988564   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    5月 24 19:39:02 n1 kubelet[13737]: E0524 19:39:02.013727   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    5月 24 19:39:07 n1 kubelet[13737]: E0524 19:39:07.023013   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    5月 24 19:39:12 n1 kubelet[13737]: E0524 19:39:12.030665   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    5月 24 19:39:17 n1 kubelet[13737]: E0524 19:39:17.039752   13737 kubelet.go:2187] Container runtime network not ready: Ne...ialized
    Hint: Some lines were ellipsized, use -l to show in full.

批准 kubelet 证书申请并加入集群  在Master节点执行

# 查看 kubelet 证书请求
    kubectl get csr
    NAME AGE SIGNERNAME
    REQUESTOR CONDITION
    node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--K6M4G7bjhk8A 6m3s
    kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending
    # 批准申请
    kubectl certificate approve node-csr-uCEGPOIiDdlLODKts8J658HrFq9CZ--
    K6M4G7bjhk8A
    # 查看节点
    kubectl get node
    
    NAME   STATUS     ROLES    AGE   VERSION
    n1     NotReady   <none>   34m   v1.18.3

注：由于网络插件还没有部署，节点会没有准备就绪 NotReady

## 2 部署 kube-proxy ##

创建配置文件和配置参数文件，配置参数文件是yml文件，一定要注意空格

# 创建配置文件
    cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
    KUBE_PROXY_OPTS="--logtostderr=false \\
    --v=2 \\
    --log-dir=/opt/kubernetes/logs \\
    --config=/opt/kubernetes/cfg/kube-proxy-config.yml"
    EOF
    
    # 配置参数文件 hostnameOverride为本机hostname
    cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
    kind: KubeProxyConfiguration
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    bindAddress: 0.0.0.0
    metricsBindAddress: 0.0.0.0:10249
    clientConnection:
      kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
    hostnameOverride: n1
    clusterCIDR: 10.0.0.0/24
    EOF

生成 kube-proxy.kubeconfig 文件

该文件的生成需要在master节点执行，过程与搭建kubelet生成bootstrap.kubeconfig过程相差不多，需要用到搭建kubelet时候的ca.pem ca-key.pem  ca-config.json，将这三个文件拷贝到一个任意目录，进入这个目录，在这个目录做操作 详细过程如下：

# 创建证书请求文件
    cat > kube-proxy-csr.json<< EOF
    {
    "CN": "system:kube-proxy",
    "hosts": [],
    "key": {
    "algo": "rsa",
    "size": 2048
    },
    "names": [
    {
    "C": "CN",
    "L": "BeiJing",
    "ST": "BeiJing",
    "O": "k8s",
    "OU": "System"
    }
    ]
    }
    EOF

# 生成证书
    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

#查看证书
    ls kube-proxy*pem
    kube-proxy-key.pem kube-proxy.pem

在这个目录 生成 kube-proxy.kubeconfig

# 生成kube-proxy.kubeconfig  执行下列多个命令
    
    KUBE_APISERVER="https://master节点IP:6443"
    
    kubectl config set-cluster kubernetes \
    --certificate-authority=/opt/kubernetes/ssl/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=kube-proxy.kubeconfig
    
    kubectl config set-credentials kube-proxy \
    --client-certificate=./kube-proxy.pem \
    --client-key=./kube-proxy-key.pem \
    --embed-certs=true \
    --kubeconfig=kube-proxy.kubeconfig
    
    kubectl config set-context default \
    --cluster=kubernetes \
    --user=kube-proxy \
    --kubeconfig=kube-proxy.kubeconfig
    
    kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

生成 kube-proxy.kubeconfig完成，使用ls可以查看到该文件

使用scp 发送这个文件到work node的/opt/kubernetes/cfg/

scp kube-proxy.kubeconfig root@工作节点ip:/opt/kubernetes/cfg/

在工作节点执行：

创建启动文件

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
    [Unit]
    Description=Kubernetes Proxy
    After=network.target
    [Service]
    EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
    ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
    Restart=on-failure
    LimitNOFILE=65536
    [Install]
    WantedBy=multi-user.target
    EOF

启动

systemctl daemon-reload
    systemctl start kube-proxy
    systemctl enable kube-proxy

如下说明启动成功

[root@n1 cfg]# systemctl status kube-proxy
    ● kube-proxy.service - Kubernetes Proxy
       Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; disabled; vendor preset: disabled)
       Active: active (running) since 二 2021-05-25 00:27:24 CST; 6s ago

# 8 部署网络插件 #

在主节点上进行部署

# 没有部署网络插件时候查看node状态是NotReady
    [root@m1 ~]# kubectl get node
    NAME   STATUS     ROLES    AGE     VERSION
    n1     NotReady   <none>   7h13m   v1.18.3
    
    # 部署插件
    # 下载插件 上传到虚拟机
    https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
    
    # 解压二进制包并移动到默认工作目录
    mkdir /opt/cni/bin
    tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin
    
    # 部署 CNI 网络
    wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
    
    sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml
    
    kubectl apply -f kube-flannel.yml
    
    kubectl get pods -n kube-system
    
    # 执行完后有一定时间延迟 再查看状态 看到状态为ready说明成功
    kubectl get node

# 9 部署Nginx #

[root@m1 ~]# kubectl create deployment nginx --image=nginx
    deployment.apps/nginx created
    [root@m1 ~]# kubectl expose deployment nginx --port=80 --type=NodePort
    service/nginx exposed
    [root@m1 ~]# kubectl get pods,svc
    NAME                        READY   STATUS              RESTARTS   AGE
    pod/nginx-f89759699-96fdq   0/1     ContainerCreating   0          52s
    
    NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
    service/kubernetes   ClusterIP   10.0.0.1     <none>        443/TCP        12h
    service/nginx        NodePort    10.0.0.28    <none>        80:32586/TCP   11s

使用work Node的ip加映射的端口可以正常访问Nginx