Kubernetes上安装KubeSphere（亲测--实操完整版）

我会带着你远行 2024-02-05 23:54 7阅读 0赞

#### 文章目录 ####

*  0、准备工作
 *  1、安装Docker
 *  2、安装Kubernetes
 *   *  2.1、基本环境
     *  2.2、安装kubelet、kubeadm、kubectl
     *  2.3、初始化master节点
     *   *  2.3.1、初始化
         *  2.3.2、记录关键信息
         *  2.3.3、安装网络插件(calico与kube-flannel二选一)
         *   *  （1）、安装Calico网络插件（master节点）
             *  （2）、安装flannel网络插件（master节点）
     *  2.4、加入worker节点
     *   *  2.4.1、令牌过期,重新获取令牌
         *  2.4.2、从节点上运行命令kubectl
 *  3、安装KubeSphere前置环境
 *   *  3.1、nfs文件系统
     *   *  3.1.1、安装nfs-server
         *  3.1.2、配置nfs-client（选做）
         *  3.1.3、配置默认存储
     *  3.2、metrics-server
 *  4、安装KubeSphere
 *   *  4.1、下载核心文件
     *  4.2、修改cluster-configuration
     *  4.3、执行安装
     *  4.4、查看安装进度
 *  5、安装成功

## 0、准备工作 ##

安装步骤  
选择2核4G（master）、2核4G（node1）、2核4G（node2） 三台机器，操作系统CentOS7.9  
安装Docker  
安装Kubernetes  
安装KubeSphere前置环境  
安装KubeSphere

三台机子ip和hostname

192.168.162.31 k8s-master01  
192.168.162.41 k8s-node01  
192.168.162.42 k8s-node02

关闭防火墙

# 停止防火墙服务
    systemctl stop firewalld
    # 禁用防火墙服务
    systemctl disable firewalld
    # 查看防火墙状态
    systemctl status firewalld

## 1、安装Docker ##

sudo yum remove docker*
    sudo yum install -y yum-utils
    
    #配置docker的yum地址
    sudo yum-config-manager \
    --add-repo \
    http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
    
    
    #安装指定版本
    sudo yum install -y docker-ce-20.10.7 docker-ce-cli-20.10.7 containerd.io-1.4.6
    
    #	启动&开机启动docker
    systemctl enable docker --now
    
    # docker加速配置
    sudo mkdir -p /etc/docker
    sudo tee /etc/docker/daemon.json <<-'EOF'
    {
      "registry-mirrors": ["https://82m9ar63.mirror.aliyuncs.com"],
      "exec-opts": ["native.cgroupdriver=systemd"],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "100m"
      },
      "storage-driver": "overlay2"
    }
    EOF
    sudo systemctl daemon-reload
    sudo systemctl restart docker

#检查docker是否安装成功
    docker ps

## 2、安装Kubernetes ##

### 2.1、基本环境 ###

每个机器使用内网ip互通

每个机器配置自己的hostname，不能用localhost

#设置每个机器自己的hostname
    hostnamectl set-hostname xxx
    
    # 将 SELinux 设置为 permissive 模式（相当于将其禁用）
    sudo setenforce 0
    sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
    
    #关闭swap
    swapoff -a  
    sed -ri 's/.*swap.*/#&/' /etc/fstab
    
    #允许 iptables 检查桥接流量
    cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
    br_netfilter
    EOF
    
    cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    EOF
    sudo sysctl --system

### 2.2、安装kubelet、kubeadm、kubectl ###

#配置k8s的yum源地址
    cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=0
    repo_gpgcheck=0
    gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
    EOF
    
    
    #安装 kubelet，kubeadm，kubectl
    sudo yum install -y kubelet-1.20.9 kubeadm-1.20.9 kubectl-1.20.9
    
    #启动kubelet
    sudo systemctl enable --now kubelet
    
    #所有机器配置master域名
    echo "192.168.162.31  k8s-master01-31" >> /etc/hosts

#检查每个机子能联通master节点
    ping k8s-master01-31

### 2.3、初始化master节点 ###

#### 2.3.1、初始化 ####

kubeadm init \
    --apiserver-advertise-address=192.168.162.31 \
    --control-plane-endpoint=k8s-master01-31 \
    --image-repository registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images \
    --kubernetes-version v1.20.9 \
    --service-cidr=10.96.0.0/16 \
    --pod-network-cidr=10.244.0.0/16

耐心等待下载安装各种镜像…

#### 2.3.2、记录关键信息 ####

第一步安装完成后，记录master执行完成后的日志

Your Kubernetes control-plane has initialized successfully!
    
    To start using your cluster, you need to run the following as a regular user:
    
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    Alternatively, if you are the root user, you can run:
    
      export KUBECONFIG=/etc/kubernetes/admin.conf
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    You can now join any number of control-plane nodes by copying certificate authorities
    and service account keys on each node and then running the following as root:
    
      kubeadm join k8s-master01-31:6443 --token 8a22q8.fcw1b9dxpvuobyu0 \
        --discovery-token-ca-cert-hash sha256:f458760f66dfe9f76037b1f805f91855abbfbbade91178d5cb4e2f692911c2d0 \
        --control-plane 
    
    Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join k8s-master01-31:6443 --token 8a22q8.fcw1b9dxpvuobyu0 \
        --discovery-token-ca-cert-hash sha256:f458760f66dfe9f76037b1f805f91855abbfbbade91178d5cb4e2f692911c2d0

#### 2.3.3、安装网络插件(calico与kube-flannel二选一) ####

##### （1）、安装Calico网络插件（master节点） #####

curl https://docs.projectcalico.org/manifests/calico.yaml -O
    
    kubectl apply -f calico.yaml

如果上面链接下载不到calico，试一下下面的链接：

wget https://docs.projectcalico.org/v3.20/manifests/calico.yaml
    
    kubectl apply -f calico.yaml

##### （2）、安装flannel网络插件（master节点） #####

kube-flannel.yml文件内容如下

---
    apiVersion: policy/v1beta1
    kind: PodSecurityPolicy
    metadata:
      name: psp.flannel.unprivileged
      annotations:
        seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
        seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
        apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
        apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    spec:
      privileged: false
      volumes:
        - configMap
        - secret
        - emptyDir
        - hostPath
      allowedHostPaths:
        - pathPrefix: "/etc/cni/net.d"
        - pathPrefix: "/etc/kube-flannel"
        - pathPrefix: "/run/flannel"
      readOnlyRootFilesystem: false
      # Users and groups
      runAsUser:
        rule: RunAsAny
      supplementalGroups:
        rule: RunAsAny
      fsGroup:
        rule: RunAsAny
      # Privilege Escalation
      allowPrivilegeEscalation: false
      defaultAllowPrivilegeEscalation: false
      # Capabilities
      allowedCapabilities: ['NET_ADMIN']
      defaultAddCapabilities: []
      requiredDropCapabilities: []
      # Host namespaces
      hostPID: false
      hostIPC: false
      hostNetwork: true
      hostPorts:
      - min: 0
        max: 65535
      # SELinux
      seLinux:
        # SELinux is unused in CaaSP
        rule: 'RunAsAny'
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: flannel
    rules:
      - apiGroups: ['extensions']
        resources: ['podsecuritypolicies']
        verbs: ['use']
        resourceNames: ['psp.flannel.unprivileged']
      - apiGroups:
          - ""
        resources:
          - pods
        verbs:
          - get
      - apiGroups:
          - ""
        resources:
          - nodes
        verbs:
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - nodes/status
        verbs:
          - patch
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: flannel
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: flannel
    subjects:
    - kind: ServiceAccount
      name: flannel
      namespace: kube-system
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: flannel
      namespace: kube-system
    ---
    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: kube-flannel-cfg
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    data:
      cni-conf.json: |
        {
          "name": "cbr0",
          "cniVersion": "0.3.1",
          "plugins": [
            {
              "type": "flannel",
              "delegate": {
                "hairpinMode": true,
                "isDefaultGateway": true
              }
            },
            {
              "type": "portmap",
              "capabilities": {
                "portMappings": true
              }
            }
          ]
        }
      net-conf.json: |
        {
          "Network": "10.244.0.0/16",
          "Backend": {
            "Type": "vxlan"
          }
        }
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-amd64
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - amd64
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-amd64
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-amd64
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-arm64
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - arm64
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-arm64
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-arm64
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                 add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-arm
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - arm
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-arm
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-arm
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                 add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-ppc64le
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - ppc64le
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-ppc64le
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-ppc64le
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                 add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-s390x
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - s390x
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-s390x
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-s390x
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                 add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg

kubectl apply -f kube-flannel.yml

### 2.4、加入worker节点 ###

kubeadm join k8s-master01:6443 --token rjl8ez.rz5sc138nsgooop8 \
        --discovery-token-ca-cert-hash sha256:300c5d79b86954ea3f2bc7e34c54554f5ddbc18cbc2c09db37d549cdd5b86ecf

#### 2.4.1、令牌过期,重新获取令牌 ####

如果令牌过期了，重新 获取令牌。（在 master 服务器中 执行）

# 重新获取令牌
    kubeadm token create --print-join-command

#### 2.4.2、从节点上运行命令kubectl ####

Kubernetes的从节点上运行命令【kubectl】

将主节点中的【/etc/kubernetes/admin.conf】文件拷贝到从节点相同目录下，然后配置环境变量：

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

立即生效

source ~/.bash_profile

接着再运行kubectl命令就OK了

## 3、安装KubeSphere前置环境 ##

### 3.1、nfs文件系统 ###

#### 3.1.1、安装nfs-server ####

# 在每个机器。
    yum install -y nfs-utils
    
    # 在master 执行以下命令   以master节点作为nfs文件系统的服务器
    echo "/nfs/data/ *(insecure,rw,sync,no_root_squash)" > /etc/exports
    
    # 执行以下命令，启动 nfs 服务;创建共享目录
    mkdir -p /nfs/data
    
    # 在master执行
    systemctl enable rpcbind
    systemctl enable nfs-server
    systemctl start rpcbind
    systemctl start nfs-server
    
    # 使配置生效
    exportfs -r
    
    #检查配置是否生效
    exportfs

#### 3.1.2、配置nfs-client（选做） ####

# 在工作节点 执行以下命令   以工作节点作为nfs文件系统的客户端 下面ip修改为对应master节点的ip
    showmount -e 192.168.162.31
    
    mkdir -p /nfs/data
    
    mount -t nfs 192.168.162.31:/nfs/data /nfs/data

#### 3.1.3、配置默认存储 ####

配置动态供应的默认存储类（注意下面两处的ip修改为自己master节点的ip）sc.yaml

## 创建了一个存储类
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: nfs-storage
      annotations:
        storageclass.kubernetes.io/is-default-class: "true"
    provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
    parameters:
      archiveOnDelete: "true"  ## 删除pv的时候，pv的内容是否要备份
    
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nfs-client-provisioner
      labels:
        app: nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default
    spec:
      replicas: 1
      strategy:
        type: Recreate
      selector:
        matchLabels:
          app: nfs-client-provisioner
      template:
        metadata:
          labels:
            app: nfs-client-provisioner
        spec:
          serviceAccountName: nfs-client-provisioner
          containers:
            - name: nfs-client-provisioner
              image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2
              # resources:
              #    limits:
              #      cpu: 10m
              #    requests:
              #      cpu: 10m
              volumeMounts:
                - name: nfs-client-root
                  mountPath: /persistentvolumes
              env:
                - name: PROVISIONER_NAME
                  value: k8s-sigs.io/nfs-subdir-external-provisioner
                - name: NFS_SERVER
                  value: 192.168.162.31 ## 指定自己nfs服务器地址
                - name: NFS_PATH  
                  value: /nfs/data  ## nfs服务器共享的目录
          volumes:
            - name: nfs-client-root
              nfs:
                server: 192.168.162.31
                path: /nfs/data
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: nfs-client-provisioner-runner
    rules:
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["create", "update", "patch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: run-nfs-client-provisioner
    subjects:
      - kind: ServiceAccount
        name: nfs-client-provisioner
        # replace with namespace where provisioner is deployed
        namespace: default
    roleRef:
      kind: ClusterRole
      name: nfs-client-provisioner-runner
      apiGroup: rbac.authorization.k8s.io
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: leader-locking-nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default
    rules:
      - apiGroups: [""]
        resources: ["endpoints"]
        verbs: ["get", "list", "watch", "create", "update", "patch"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: leader-locking-nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: default
    subjects:
      - kind: ServiceAccount
        name: nfs-client-provisioner
        # replace with namespace where provisioner is deployed
        namespace: default
    roleRef:
      kind: Role
      name: leader-locking-nfs-client-provisioner
      apiGroup: rbac.authorization.k8s.io

#确认配置是否生效
    kubectl get sc

测试一下默认存储的使用

创建pvc申请存储书

apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: nginx-pvc
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          # 需要 200M的 PV
          storage: 200Mi
      # 上面 PV 写的什么 这里就写什么    
      storageClassName: nfs-storage

系统会自动创建对应的pv服务，并绑定pvc。

![请添加图片描述][7d4907c05b0341bc95de639ad13093ae.png]

### 3.2、metrics-server ###

集群指标监控组件 metrics.yaml

apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        k8s-app: metrics-server
      name: metrics-server
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        k8s-app: metrics-server
        rbac.authorization.k8s.io/aggregate-to-admin: "true"
        rbac.authorization.k8s.io/aggregate-to-edit: "true"
        rbac.authorization.k8s.io/aggregate-to-view: "true"
      name: system:aggregated-metrics-reader
    rules:
    - apiGroups:
      - metrics.k8s.io
      resources:
      - pods
      - nodes
      verbs:
      - get
      - list
      - watch
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      labels:
        k8s-app: metrics-server
      name: system:metrics-server
    rules:
    - apiGroups:
      - ""
      resources:
      - pods
      - nodes
      - nodes/stats
      - namespaces
      - configmaps
      verbs:
      - get
      - list
      - watch
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      labels:
        k8s-app: metrics-server
      name: metrics-server-auth-reader
      namespace: kube-system
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: extension-apiserver-authentication-reader
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        k8s-app: metrics-server
      name: metrics-server:system:auth-delegator
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:auth-delegator
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      labels:
        k8s-app: metrics-server
      name: system:metrics-server
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:metrics-server
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        k8s-app: metrics-server
      name: metrics-server
      namespace: kube-system
    spec:
      ports:
      - name: https
        port: 443
        protocol: TCP
        targetPort: https
      selector:
        k8s-app: metrics-server
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        k8s-app: metrics-server
      name: metrics-server
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          k8s-app: metrics-server
      strategy:
        rollingUpdate:
          maxUnavailable: 0
      template:
        metadata:
          labels:
            k8s-app: metrics-server
        spec:
          containers:
          - args:
            - --cert-dir=/tmp
            - --kubelet-insecure-tls
            - --secure-port=4443
            - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
            - --kubelet-use-node-status-port
            image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/metrics-server:v0.4.3
            imagePullPolicy: IfNotPresent
            livenessProbe:
              failureThreshold: 3
              httpGet:
                path: /livez
                port: https
                scheme: HTTPS
              periodSeconds: 10
            name: metrics-server
            ports:
            - containerPort: 4443
              name: https
              protocol: TCP
            readinessProbe:
              failureThreshold: 3
              httpGet:
                path: /readyz
                port: https
                scheme: HTTPS
              periodSeconds: 10
            securityContext:
              readOnlyRootFilesystem: true
              runAsNonRoot: true
              runAsUser: 1000
            volumeMounts:
            - mountPath: /tmp
              name: tmp-dir
          nodeSelector:
            kubernetes.io/os: linux
          priorityClassName: system-cluster-critical
          serviceAccountName: metrics-server
          volumes:
          - emptyDir: {}
            name: tmp-dir
    ---
    apiVersion: apiregistration.k8s.io/v1
    kind: APIService
    metadata:
      labels:
        k8s-app: metrics-server
      name: v1beta1.metrics.k8s.io
    spec:
      group: metrics.k8s.io
      groupPriorityMinimum: 100
      insecureSkipTLSVerify: true
      service:
        name: metrics-server
        namespace: kube-system
      version: v1beta1
      versionPriority: 100

安装集群监控服务

kubectl apply -f metrics.yaml

使用下面模块查看节点的资源占用情况：

kubectl top nodes

![请添加图片描述][3e5018e96acd43e7880fc273a1f827b8.png]

上面CPU单位中 1core = 1000m

查看所有运行pod的资源占用情况：

kubectl top pods -A

![在这里插入图片描述][51fea56571284a21a0112fd9514f4561.png]

## 4、安装KubeSphere ##

kubersphere官网参考地址： [https://kubesphere.com.cn/][https_kubesphere.com.cn]

https://kubesphere.io/zh/docs/v3.4/quick-start/minimal-kubesphere-on-k8s/

### 4.1、下载核心文件 ###

如果下载不到，请复制附录的内

wget https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/kubesphere-installer.yaml 
    
    wget https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/cluster-configuration.yaml

### 4.2、修改cluster-configuration ###

在 cluster-configuration.yaml中指定我们需要开启的功能

参照官网“启用可插拔组件”

https://kubesphere.io/zh/docs/v3.4/pluggable-components/

vim cluster-configuration.yaml

![在这里插入图片描述][f32d381d384447b1a1d759ea6b5f8134.png]

### 4.3、执行安装 ###

安装kubesphere

kubectl apply -f kubesphere-installer.yaml  ## 大概等待3分钟，完成pod的安装
    
    kubectl apply -f cluster-configuration.yaml

---
    apiVersion: installer.kubesphere.io/v1alpha1
    kind: ClusterConfiguration
    metadata:
      name: ks-installer
      namespace: kubesphere-system
      labels:
        version: v3.4.1
    spec:
      persistence:
        storageClass: ""        # If there is no default StorageClass in your cluster, you need to specify an existing StorageClass here.
      authentication:
        # adminPassword: ""     # Custom password of the admin user. If the parameter exists but the value is empty, a random password is generated. If the parameter does not exist, P@88w0rd is used.
        jwtSecret: ""           # Keep the jwtSecret consistent with the Host Cluster. Retrieve the jwtSecret by executing "kubectl -n kubesphere-system get cm kubesphere-config -o yaml | grep -v "apiVersion" | grep jwtSecret" on the Host Cluster.
      local_registry: ""        # Add your private registry address if it is needed.
    #  dev_tag: ""               # Add your kubesphere image tag you want to install, by default it's same as ks-installer release version.
      etcd:
        monitoring: true       # Enable or disable etcd monitoring dashboard installation. You have to create a Secret for etcd before you enable it.
        endpointIps: 192.168.162.31  # etcd cluster EndpointIps. It can be a bunch of IPs here.
        port: 2379              # etcd port.
        tlsEnable: true
      common:
        core:
          console:
            enableMultiLogin: true  # Enable or disable simultaneous logins. It allows different users to log in with the same account at the same time.
            port: 30880
            type: NodePort
    
        # apiserver:            # Enlarge the apiserver and controller manager's resource requests and limits for the large cluster
        #  resources: {}
        # controllerManager:
        #  resources: {}
        redis:
          enabled: true
          enableHA: true
          volumeSize: 2Gi # Redis PVC size.
        openldap:
          enabled: true
          volumeSize: 2Gi   # openldap PVC size.
        minio:
          volumeSize: 20Gi # Minio PVC size.
        monitoring:
          # type: external   # Whether to specify the external prometheus stack, and need to modify the endpoint at the next line.
          endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090 # Prometheus endpoint to get metrics data.
          GPUMonitoring:     # Enable or disable the GPU-related metrics. If you enable this switch but have no GPU resources, Kubesphere will set it to zero.
            enabled: false
        gpu:                 # Install GPUKinds. The default GPU kind is nvidia.com/gpu. Other GPU kinds can be added here according to your needs.
          kinds:
          - resourceName: "nvidia.com/gpu"
            resourceType: "GPU"
            default: true
        es:   # Storage backend for logging, events and auditing.
          # master:
          #   volumeSize: 4Gi  # The volume size of Elasticsearch master nodes.
          #   replicas: 1      # The total number of master nodes. Even numbers are not allowed.
          #   resources: {}
          # data:
          #   volumeSize: 20Gi  # The volume size of Elasticsearch data nodes.
          #   replicas: 1       # The total number of data nodes.
          #   resources: {}
          enabled: false
          logMaxAge: 7             # Log retention time in built-in Elasticsearch. It is 7 days by default.
          elkPrefix: logstash      # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log.
          basicAuth:
            enabled: false
            username: ""
            password: ""
          externalElasticsearchHost: ""
          externalElasticsearchPort: ""
        opensearch:   # Storage backend for logging, events and auditing.
          # master:
          #   volumeSize: 4Gi  # The volume size of Opensearch master nodes.
          #   replicas: 1      # The total number of master nodes. Even numbers are not allowed.
          #   resources: {}
          # data:
          #   volumeSize: 20Gi  # The volume size of Opensearch data nodes.
          #   replicas: 1       # The total number of data nodes.
          #   resources: {}
          enabled: true
          logMaxAge: 7             # Log retention time in built-in Opensearch. It is 7 days by default.
          opensearchPrefix: whizard      # The string making up index names. The index name will be formatted as ks-<opensearchPrefix>-logging.
          basicAuth:
            enabled: true
            username: "admin"
            password: "admin"
          externalOpensearchHost: ""
          externalOpensearchPort: ""
          dashboard:
            enabled: false
      alerting:                # (CPU: 0.1 Core, Memory: 100 MiB) It enables users to customize alerting policies to send messages to receivers in time with different time intervals and alerting levels to choose from.
        enabled: false         # Enable or disable the KubeSphere Alerting System.
        # thanosruler:
        #   replicas: 1
        #   resources: {}
      auditing:                # Provide a security-relevant chronological set of records，recording the sequence of activities happening on the platform, initiated by different tenants.
        enabled: false         # Enable or disable the KubeSphere Auditing Log System.
        # operator:
        #   resources: {}
        # webhook:
        #   resources: {}
      devops:                  # (CPU: 0.47 Core, Memory: 8.6 G) Provide an out-of-the-box CI/CD system based on Jenkins, and automated workflow tools including Source-to-Image & Binary-to-Image.
        enabled: false         # Enable or disable the KubeSphere DevOps System.
        jenkinsCpuReq: 0.5
        jenkinsCpuLim: 1
        jenkinsMemoryReq: 4Gi
        jenkinsMemoryLim: 4Gi  # Recommend keep same as requests.memory.
        jenkinsVolumeSize: 16Gi
      events:                  # Provide a graphical web console for Kubernetes Events exporting, filtering and alerting in multi-tenant Kubernetes clusters.
        enabled: false         # Enable or disable the KubeSphere Events System.
        # operator:
        #   resources: {}
        # exporter:
        #   resources: {}
        ruler:
          enabled: true
          replicas: 2
        #   resources: {}
      logging:                 # (CPU: 57 m, Memory: 2.76 G) Flexible logging functions are provided for log query, collection and management in a unified console. Additional log collectors can be added, such as Elasticsearch, Kafka and Fluentd.
        enabled: false         # Enable or disable the KubeSphere Logging System.
        logsidecar:
          enabled: true
          replicas: 2
          # resources: {}
      metrics_server:                    # (CPU: 56 m, Memory: 44.35 MiB) It enables HPA (Horizontal Pod Autoscaler).
        enabled: false                   # Enable or disable metrics-server.
      monitoring:
        storageClass: ""                 # If there is an independent StorageClass you need for Prometheus, you can specify it here. The default StorageClass is used by default.
        node_exporter:
          port: 9100
          # resources: {}
        # kube_rbac_proxy:
        #   resources: {}
        # kube_state_metrics:
        #   resources: {}
        # prometheus:
        #   replicas: 1  # Prometheus replicas are responsible for monitoring different segments of data source and providing high availability.
        #   volumeSize: 20Gi  # Prometheus PVC size.
        #   resources: {}
        #   operator:
        #     resources: {}
        # alertmanager:
        #   replicas: 1          # AlertManager Replicas.
        #   resources: {}
        # notification_manager:
        #   resources: {}
        #   operator:
        #     resources: {}
        #   proxy:
        #     resources: {}
        gpu:                           # GPU monitoring-related plug-in installation.
          nvidia_dcgm_exporter:        # Ensure that gpu resources on your hosts can be used normally, otherwise this plug-in will not work properly.
            enabled: false             # Check whether the labels on the GPU hosts contain "nvidia.com/gpu.present=true" to ensure that the DCGM pod is scheduled to these nodes.
            # resources: {}
      multicluster:
        clusterRole: none  # host | member | none  # You can install a solo cluster, or specify it as the Host or Member Cluster.
      network:
        networkpolicy: # Network policies allow network isolation within the same cluster, which means firewalls can be set up between certain instances (Pods).
          # Make sure that the CNI network plugin used by the cluster supports NetworkPolicy. There are a number of CNI network plugins that support NetworkPolicy, including Calico, Cilium, Kube-router, Romana and Weave Net.
          enabled: false # Enable or disable network policies.
        ippool: # Use Pod IP Pools to manage the Pod network address space. Pods to be created can be assigned IP addresses from a Pod IP Pool.
          type: calico # Specify "calico" for this field if Calico is used as your CNI plugin. "none" means that Pod IP Pools are disabled.
        topology: # Use Service Topology to view Service-to-Service communication based on Weave Scope.
          type: none # Specify "weave-scope" for this field to enable Service Topology. "none" means that Service Topology is disabled.
      openpitrix: # An App Store that is accessible to all platform tenants. You can use it to manage apps across their entire lifecycle.
        store:
          enabled: false # Enable or disable the KubeSphere App Store.
      servicemesh:         # (0.3 Core, 300 MiB) Provide fine-grained traffic management, observability and tracing, and visualized traffic topology.
        enabled: false     # Base component (pilot). Enable or disable KubeSphere Service Mesh (Istio-based).
        istio:  # Customizing the istio installation configuration, refer to https://istio.io/latest/docs/setup/additional-setup/customize-installation/
          components:
            ingressGateways:
            - name: istio-ingressgateway
              enabled: false
            cni:
              enabled: false
      edgeruntime:          # Add edge nodes to your cluster and deploy workloads on edge nodes.
        enabled: false
        kubeedge:        # kubeedge configurations
          enabled: false
          cloudCore:
            cloudHub:
              advertiseAddress: # At least a public IP address or an IP address which can be accessed by edge nodes must be provided.
                - ""            # Note that once KubeEdge is enabled, CloudCore will malfunction if the address is not provided.
            service:
              cloudhubNodePort: "30000"
              cloudhubQuicNodePort: "30001"
              cloudhubHttpsNodePort: "30002"
              cloudstreamNodePort: "30003"
              tunnelNodePort: "30004"
            # resources: {}
            # hostNetWork: false
          iptables-manager:
            enabled: true 
            mode: "external"
            # resources: {}
          # edgeService:
          #   resources: {}
      gatekeeper:        # Provide admission policy and rule management, A validating (mutating TBA) webhook that enforces CRD-based policies executed by Open Policy Agent.
        enabled: false   # Enable or disable Gatekeeper.
        # controller_manager:
        #   resources: {}
        # audit:
        #   resources: {}
      terminal:
        # image: 'alpine:3.15' # There must be an nsenter program in the image
        timeout: 600         # Container timeout, if set to 0, no timeout will be used. The unit is seconds

### 4.4、查看安装进度 ###

安装过程中可以通过下列命令查看检查安装日志：

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

![在这里插入图片描述][ce1d428a5ae146aea2febb5f516cdd74.png]

解决etcd监控证书找不到问题

kubectl -n kubesphere-monitoring-system create secret generic kube-etcd-client-certs  --from-file=etcd-client-ca.crt=/etc/kubernetes/pki/etcd/ca.crt  --from-file=etcd-client.crt=/etc/kubernetes/pki/apiserver-etcd-client.crt  --from-file=etcd-client.key=/etc/kubernetes/pki/apiserver-etcd-client.key

## 5、安装成功 ##

kubesphere操作界面：  
![在这里插入图片描述][c4c7447e9bac4233b151e3e87cdf7c0c.png]  
![在这里插入图片描述][8a6dc5c78b614d6db16ee40b6ae72e03.png]  
![在这里插入图片描述][b32aff8446de412f9c2c09fe0753a6e8.png]  
![在这里插入图片描述][ca366bd675be49c6965aa57a09811370.png]

[7d4907c05b0341bc95de639ad13093ae.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/30/8134de6004124e21902fe2468d6f19d8.png
[3e5018e96acd43e7880fc273a1f827b8.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/30/206442e63e9d4f16a4327322d7074038.png
[51fea56571284a21a0112fd9514f4561.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/30/b5785b374d0d4cafb62fb16b5ad16aa6.png
[https_kubesphere.com.cn]: https://kubesphere.com.cn/
[f32d381d384447b1a1d759ea6b5f8134.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/30/c0c37499116f43c688bfaa20a8ae4acd.png
[ce1d428a5ae146aea2febb5f516cdd74.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/30/9681dc50358143edb3b62aae09566988.png
[c4c7447e9bac4233b151e3e87cdf7c0c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/30/7850115e22d7458b9ab6eee7d9441a10.png
[8a6dc5c78b614d6db16ee40b6ae72e03.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/30/095bd927c632405aa96904e27becf5d3.png
[b32aff8446de412f9c2c09fe0753a6e8.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/30/2e2e46746b204f36b7094a43b239de83.png
[ca366bd675be49c6965aa57a09811370.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/01/30/485678da35c24913b038b4d0b21c7083.png