kubernetes1.9部署metrics-server0.3.1、dashboard、heapster

╰半橙微兮° 2022-02-21 08:55 219阅读 0赞

#### 简介 ####

K8S从1.8版本开始，CPU、内存等资源的metrics信息可以通过 Metrics API来获取，用户可以直接获取这些metrics信息（例如通过执行kubect top命令），HPA使用这些metics信息来实现动态伸缩。本文介绍K8S集群基于metric server的HPA。在开始之前我们需要了解一下Metrics API和Metrics Server。

###### Metrics API： ######

1、通过Metrics API我们可以获取到指定node或者pod的当前资源使用情况，API本身不存储任何信息，所以我们不可能通过API来获取资源的历史使用情况。  
2、Metrics API的获取路径位于：/apis/metrics.k8s.io/  
3、获取Metrics API的前提条件是metrics server要在K8S集群中成功部署  
4、更多的metrics资料请参考：[https://github.com/kubernetes/metrics][https_github.com_kubernetes_metrics]

###### Metrics server： ######

1、Metrics server是K8S集群资源使用情况的聚合器  
2、从1.8版本开始，Metrics [server默认可以通过kube-up.sh][server_kube-up.sh] 脚本以deployment的方式进行部署，也可以通过yaml文件的方式进行部署  
3、Metrics server收集所有node节点的metrics信息  
heapster 已经被废弃了，后续版本中会使用 metrics-server代替,不过kubenetes如果是1.10之前的版本，使用Metrics Server 还是要和heapster 配合的。k8s v1.10版本之前仍然要通过heapster获取指标数据，否则即使部署了metrics-server，kubectl top这个指令仍然连接heapster

##### 官网部署方法 #####

git clone https://github.com/kubernetes-incubator/metrics-server
    cd metrics-server
    kubectl create -f deploy/1.8+/
    kubectl -n kube-system get pods -l k8s-app=metrics-server

##### 我的实际部署步骤 #####

###### 下载文件 ######

[root@master01 1.8+]# ll
    total 28
    -rw-r--r-- 1 root root  384 Apr  9 14:03 aggregated-metrics-reader.yaml
    -rw-r--r-- 1 root root  308 Apr  9 14:03 auth-delegator.yaml
    -rw-r--r-- 1 root root  329 Apr  9 14:03 auth-reader.yaml
    -rw-r--r-- 1 root root  298 Apr  9 14:03 metrics-apiservice.yaml
    -rw-r--r-- 1 root root 1099 Apr 10 10:15 metrics-server-deployment.yaml
    -rw-r--r-- 1 root root  249 Apr  9 14:03 metrics-server-service.yaml
    -rw-r--r-- 1 root root  517 Apr 10 10:14 resource-reader.yaml

###### images ######

可以使用 [registry.cn-beijing.aliyuncs.com/minminmsn/metrics-server:v0.3.1][registry.cn-beijing.aliyuncs.com_minminmsn_metrics-server_v0.3.1] 感谢作者^^!  
Dockerfile文件地址：[https://github.com/minminmsn/k8s1.13/blob/master/metrics-server/Dockerfile][https_github.com_minminmsn_k8s1.13_blob_master_metrics-server_Dockerfile]

###### 修改metrics-server-deployment.yaml ######

---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: metrics-server
      namespace: kube-system
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: metrics-server
      namespace: kube-system
      labels:
        k8s-app: metrics-server
    spec:
      selector:
        matchLabels:
          k8s-app: metrics-server
      template:
        metadata:
          name: metrics-server
          labels:
            k8s-app: metrics-server
        spec:
          serviceAccountName: metrics-server
          volumes:
          # mount in tmp so we can safely use from-scratch images and/or read-only containers
          - name: tmp-dir
            emptyDir: { }
          containers:
          - name: metrics-server
            image: registry.cn-beijing.aliyuncs.com/minminmsn/metrics-server:v0.3.1  ## 修改镜像地址
            imagePullPolicy: Always
            command: ##新增
            - /metrics-server ##新增
            - --metric-resolution=30s ##新增
            - --kubelet-insecure-tls ##新增
            - --kubelet-preferred-address-types=InternalIP ##新增
            volumeMounts:
            - name: tmp-dir
              mountPath: /tmp

###### 修改 resource-reader.yaml ######

---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: system:metrics-server
    rules:
    - apiGroups:
      - ""
      resources:
      - pods
      - nodes
      - nodes/stats ##新增
      - namespaces  ##新增 
      verbs:
      - get
      - list
      - watch
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: system:metrics-server
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:metrics-server
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system

###### 部署 ######

kubectl create -f .

###### 报错 ######

I0109 05:55:43.708300       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
    Error: cluster doesn't provide requestheader-client-ca-file

解决方法 开启聚合层，Enable apiserver flags，修改kube-apiserver配置，重启服务

[root@elasticsearch01 cfg]# tail /k8s/kubernetes/cfg/kube-apiserver
    --etcd-cafile=/k8s/etcd/ssl/ca.pem \
    --etcd-certfile=/k8s/etcd/ssl/server.pem \
    --etcd-keyfile=/k8s/etcd/ssl/server-key.pem \
    --requestheader-client-ca-file=/k8s/kubernetes/ssl/ca.pem \
    --requestheader-allowed-names=aggregator \
    --requestheader-extra-headers-prefix=X-Remote-Extra- \
    --requestheader-group-headers=X-Remote-Group \
    --requestheader-username-headers=X-Remote-User \
    --proxy-client-cert-file=/k8s/kubernetes/ssl/kube-proxy.pem \
    --proxy-client-key-file=/k8s/kubernetes/ssl/kube-proxy-key.pem"

#### (重点)修改kube-apiservice和kube-controller-manager.service开启聚合层 ####

证书要准备好,我的证书（搭建k8s环境时候会生成证书，找到证书是难点。或者网上查找证书生成步骤、这里我不做多步骤啦）：

[root@master01 heapster]# ll /etc/kubernetes/ssl/*
    -rw-r--r-- 1 kube kube 1415 Apr 10 15:34 /etc/kubernetes/ssl/ca.pem  ##根证书
    -rw------- 1 kube kube 1679 Apr  4  2018 /etc/kubernetes/ssl/kubernetes-client-proxy-key.pem ##proxy的key文件
    -rw-r--r-- 1 kube kube 1562 Apr  4  2018 /etc/kubernetes/ssl/kubernetes-client-proxy.pem ##proxy的证书文件

注意证书的权限！！！ 我的是kube用户`chown kube:kube /etc/kubernetes/ssl`  
将这三个证书分发到其他node节点 `scp etc/kubernetes/ssl/* node01:/etc/kubernetes/ssl`

###### kube-apiservice文件： ######

新增以下几行

--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem    --requestheader-allowed-names=   --requestheader-extra-headers-prefix=X-Remote-Extra-        --requestheader-group-headers=X-Remote-Group    --requestheader-username-headers=X-Remote-User             --proxy-client-cert-file=/etc/kubernetes/ssl/kubernetes-client-proxy.pem   --proxy-client-key-file=/etc/kubernetes/ssl/kubernetes-client-proxy-key.pem  --runtime-config=api/all=true  --enable-aggregator-routing=true"

###### kube-controller-manager.service文件 ######

新增一行：`--horizontal-pod-autoscaler-use-rest-clients=true`

systemctl daemon-reload
     systemctl  restart kube-apiserver.service
     systemctl  restart kube-controller-manager

确认启动成功后重新部署etrics-server0.3.1

cd 1.8+/
    kubectl create -f .
    kubectl get pod,svc -n kube-system
    [root@master01 1.8+]# kubectl get pod -n kube-system 
    NAME                                      READY     STATUS    RESTARTS   AGE
    metrics-server-57596fd464-ms2zp           1/1       Running   0          39m

###### 部署heapster ######

kubenetes 1.10后废弃了，k8s v1.10版本之前仍然要通过heapster获取指标数据，否则即使部署了metrics-server，kubectl top这个指令仍然连接heapster

###### 创建heapster.yaml ######

[root@master01 heapster]# cat heapster.yaml 
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: heapster
      namespace: kube-system
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: heapster
    subjects:
      - kind: ServiceAccount
        name: heapster
        namespace: kube-system
    roleRef:
      kind: ClusterRole
      name: cluster-admin
      apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: heapster
      namespace: kube-system
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            task: monitoring
            k8s-app: heapster
        spec:
          serviceAccountName: heapster
          containers:
          - name: heapster
            image: harbor.reg/qijin_product/heapster-amd64:v1.4.3
            imagePullPolicy: IfNotPresent
            command:
            - /heapster
           # - --source=kubernetes:https://kubernetes.default
           # - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
            - --source=kubernetes:http://10.1.11.11:8080?inClusterConfig=false  ##api-server 机器IP和端口
            - --sink=influxdb:http://monitoring-influxdb:8086
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        task: monitoring
        # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
        # If you are NOT using this as an addon, you should comment out this line.
        kubernetes.io/cluster-service: 'true'
        kubernetes.io/name: Heapster
      name: heapster
      namespace: kube-system
    spec:
      ports:
      - port: 80
        targetPort: 8082
      selector:
        k8s-app: heapster

如果要持久化数据还要安装influxdb,yaml文件去链接:[https://pan.baidu.com/s/1WjCSZ9wQzJQ4Q0PCxXogZw][https_pan.baidu.com_s_1WjCSZ9wQzJQ4Q0PCxXogZw] 密码:6wtd下载

###### 获取v1beta1.metrics.k8s.io并验证 ######

[root@master01 1.8+]# kubectl get apiservice
    NAME                                    AGE
    v1.                                     1y
    v1.apps                                 1y
    v1.authentication.k8s.io                1y
    v1.authorization.k8s.io                 1y
    v1.autoscaling                          1y
    v1.batch                                1y
    v1.networking.k8s.io                    1y
    v1.rbac.authorization.k8s.io            1y
    v1.storage.k8s.io                       1y
    v1alpha1.admissionregistration.k8s.io   1h
    v1alpha1.rbac.authorization.k8s.io      1h
    v1alpha1.scheduling.k8s.io              1h
    v1alpha1.settings.k8s.io                1h
    v1alpha1.storage.k8s.io                 1h
    v1beta1.admissionregistration.k8s.io    1y
    v1beta1.apiextensions.k8s.io            1y
    v1beta1.apps                            1y
    v1beta1.authentication.k8s.io           1y
    v1beta1.authorization.k8s.io            1y
    v1beta1.batch                           1y
    v1beta1.certificates.k8s.io             1y
    v1beta1.events.k8s.io                   1y
    v1beta1.extensions                      1y
    v1beta1.metrics.k8s.io                  40m  ### 说明部署成功
    v1beta1.policy                          1y
    v1beta1.rbac.authorization.k8s.io       1y
    v1beta1.storage.k8s.io                  1y
    v1beta2.apps                            1y
    v2alpha1.batch                          1h
    v2beta1.autoscaling                     1y
    
    
    [root@master01 1.8+]# kubectl get apiservice v1beta1.metrics.k8s.io -o yaml 
    apiVersion: apiregistration.k8s.io/v1beta1
    kind: APIService
    metadata:
      creationTimestamp: 2019-04-10T08:40:19Z
      name: v1beta1.metrics.k8s.io
      resourceVersion: "41675073"
      selfLink: /apis/apiregistration.k8s.io/v1beta1/apiservices/v1beta1.metrics.k8s.io
      uid: 459dc419-5b6c-11e9-bac9-005056b678c1
    spec:
      caBundle: null
      group: metrics.k8s.io
      groupPriorityMinimum: 100
      insecureSkipTLSVerify: true
      service:
        name: metrics-server
        namespace: kube-system
      version: v1beta1
      versionPriority: 100
    status:
      conditions:
      - lastTransitionTime: 2019-04-10T08:40:24Z
        message: all checks passed
        reason: Passed
        status: "True"
        type: Available

###### 通过kubectl工具测试获取metrics数据 ######

yum -y install jq
    kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70]  
稍等一下查看pod的使用资源

[root@beta-k8s-master-1 ~]# kubectl top node
    NAME              CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
    beta-k8s-node-1   131m         1%        7413Mi          46%       
    beta-k8s-node-2   198m         2%        11421Mi         72%       
    beta-k8s-node-3   219m         2%        10914Mi         69%       
    
    [root@beta-k8s-master-1 ~]# kubectl top pod
    NAME                                                CPU(cores)   MEMORY(bytes)   
    frm-eureka-profile-a1-794995f789-fp5sd   18m          1731Mi

##### 部署dashboard #####

下载地址：  
链接:[https://pan.baidu.com/s/18PTR648nYQQWEXBWasLamg][https_pan.baidu.com_s_18PTR648nYQQWEXBWasLamg] 密码:gunr

###### 最后上成果图 ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70 2]

[https_github.com_kubernetes_metrics]: https://github.com/kubernetes/metrics
[server_kube-up.sh]: http://xn--serverkube-up-1m3ul00dkj4spyoaxlbw79m.sh
[registry.cn-beijing.aliyuncs.com_minminmsn_metrics-server_v0.3.1]: http://registry.cn-beijing.aliyuncs.com/minminmsn/metrics-server:v0.3.1
[https_github.com_minminmsn_k8s1.13_blob_master_metrics-server_Dockerfile]: https://github.com/minminmsn/k8s1.13/blob/master/metrics-server/Dockerfile
[https_pan.baidu.com_s_1WjCSZ9wQzJQ4Q0PCxXogZw]: https://pan.baidu.com/s/1WjCSZ9wQzJQ4Q0PCxXogZw
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70]: /images/20220221/689299adca424b128dfc9fb8f654f66c.png
[https_pan.baidu.com_s_18PTR648nYQQWEXBWasLamg]: https://pan.baidu.com/s/18PTR648nYQQWEXBWasLamg
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70 1]: /images/20220221/cb690c3afdd14cc3ad2a2c2f1beb2f96.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI1OTM0NDAx_size_16_color_FFFFFF_t_70 2]: /images/20220221/37a65be6370e4bb88bb6dcf8188b6597.png