31. API聚合层 + metrics-server

素颜马尾好姑娘i 2022-10-18 15:54 136阅读 0赞

知识结构梳理

*  HPA
    
     *  metrics api
 *  API aggregation
    
     *  metrics.k8s.io
        
         *  metrics-server
     *  custom.metrics.k8s.io
        
         *  prometheus adapter
     *  external.metrics.k8s.io
 *  安装扩展API服务器

> 链接资料  
> https://www.cnblogs.com/wjoyxt/p/10003159.html

### 31. API聚合层 + metrics-server ###

#### 一、metrics-server ####

##### 1. 获取metrics-server #####

github：https://github.com/kubernetes-sigs/metrics-server  
github获取k8s部署资源配置的参考清单，这里用的是V0.3.6版本，我们配置清单版本也选择这个版本（资源配置清单因具体情况还需一些小调整）  
https://github.com/kubernetes-sigs/metrics-server/tree/release-0.3/deploy/1.8%2B  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE5NjU1NDA1_size_16_color_FFFFFF_t_70]

拉取metrics-server镜像  
`docker pull mirrorgooglecontainers/metrics-server-amd64:v0.3.6`

##### 2. 配置清单（原始清单） #####

*  aggregated-metrics-reader.yaml

---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: system:aggregated-metrics-reader
      labels:
        rbac.authorization.k8s.io/aggregate-to-view: "true"
        rbac.authorization.k8s.io/aggregate-to-edit: "true"
        rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rules:
    - apiGroups: ["metrics.k8s.io"]
      resources: ["pods", "nodes"]
      verbs: ["get", "list", "watch"]

*  auth-delegator.yaml

---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
      name: metrics-server:system:auth-delegator
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:auth-delegator
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system

*  auth-reader.yaml

---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: RoleBinding
    metadata:
      name: metrics-server-auth-reader
      namespace: kube-system
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: extension-apiserver-authentication-reader
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system

*  resource-reader.yaml

---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: system:metrics-server
    rules:
    - apiGroups:
      - ""
      resources:
      - pods
      - nodes
      - nodes/stats
      - namespaces
      verbs:
      - get
      - list
      - watch
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: system:metrics-server
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: system:metrics-server
    subjects:
    - kind: ServiceAccount
      name: metrics-server
      namespace: kube-system

*  metrics-apiservice.yaml

---
    apiVersion: apiregistration.k8s.io/v1beta1
    kind: APIService
    metadata:
      name: v1beta1.metrics.k8s.io
    spec:
      service:
        name: metrics-server
        namespace: kube-system
      group: metrics.k8s.io
      version: v1beta1
      insecureSkipTLSVerify: true
      groupPriorityMinimum: 100
      versionPriority: 100

配置APIService资源，该资源是将apiserver的`metrics.k8s.io`转给metrics-server服务。HPA或其他组件访问apiserver的`metrics.k8s.io`接口时，apiserver会转访问metrics

*  metrics-server-service.yaml

---
    apiVersion: v1
    kind: Service
    metadata:
      name: metrics-server
      namespace: kube-system
      labels:
        kubernetes.io/name: "Metrics-server"
        kubernetes.io/cluster-service: "true"
    spec:
      selector:
        k8s-app: metrics-server
      ports:
      - port: 443
        protocol: TCP
        targetPort: 443

*  metrics-server-deployment.yaml

---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: metrics-server
      namespace: kube-system
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: metrics-server
      namespace: kube-system
      labels:
        k8s-app: metrics-server
    spec:
      selector:
        matchLabels:
          k8s-app: metrics-server
      template:
        metadata:
          name: metrics-server
          labels:
            k8s-app: metrics-server
        spec:
          serviceAccountName: metrics-server
          volumes:
          # mount in tmp so we can safely use from-scratch images and/or read-only containers
          - name: tmp-dir
            emptyDir: {
        }
          containers:
          - name: metrics-server
            image: harbor.hzwod.com/k8s/metrics-server-amd64:v0.3.6
            imagePullPolicy: Always
            volumeMounts:
            - name: tmp-dir
              mountPath: /tmp

具体deployment部署metrics-server容器时，需添加一下自定的配置，我们下文详细介绍

##### 3. metrics-server的关键配置说明 #####

我们可以通过下条命名查看metrics-server的帮助文档：  
`docker run --rm harbor.hzwod.com/k8s/metrics-server-amd64:v0.3.6 /metrics-server -h`  
我们主要关注这几个参数

*  `--kubelet-preferred-address-types` metrics默认使用hostname来通信的(默认值是\[Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP\]),若我们某些节点无法通过hostname来访问，那我们可以将该参数直接设置成InternalIP，让metrics使用ip访问各个节点
 *  **`--kubeconfig=xxxx.kubeconfig`** 若我们集群中kubelet使用了自定义的证书控制访问，那么metrics访问kubelet可能会被拒绝，通过这个参数可以给metrics提供一个kubeconfig文件，保存了用k8s同CA证书签发的client证书，用于和kubelet通信
 *  `--kubelet-insecure-tls` 不校验kubelets出示的服务证书CA，只建议测试用
 *  `--cert-dir` 指定本地的TLS证书目录，若设置了下面两个参数则这个参数会被忽略，否则metrics会在该目录下生成自签证书
 *  `--tls-cert-file`、`--tls-provate-key-file` 配置metrics提供HTTPS服务的证书

##### 4. 配置准备 #####

*  准备kubeconfig文件  
    参考部署kubelet时kubeconfig的制备过程，关键是配置文件中的证书和k8s集群证书源于同一个CA签发。这里这里直接使用部署kubelet时制备的k8s-node.kubeconfig文件
 *  将kubeconfig文件存入scerts资源  
    `kubectl -n kube-system create secret generic k8snode-kubeconfig-secrets --from-file=/xxx/k8s-node.kubeconfig`  
    `k8snode-kubeconfig-secrets`是secrets资源名称，下文就是通过这个名称挂载引用的
 *  deployment配置修改

...
          volumes:
          - name: k8snode-kubeconfig
            secret:
              secretName: k8snode-kubeconfig-secrets
          - name: tmp-dir
            emptyDir: {
        }
          containers:
          - name: metrics-server
            image: harbor.hzwod.com/k8s/metrics-server-amd64:v0.3.6
            imagePullPolicy: Always
            command:
              - /metrics-server
              - --kubelet-preferred-address-types=InternalIP
              - --kubeconfig=k8s-node.kubeconfig
              - --cert-dir=/tmp
            volumeMounts:
            - name: k8snode-kubeconfig
              mountPath: /kubeconfig
            - name: tmp-dir
              mountPath: /tmp

至此，我们的metrics就基本配置完成，我们将资源应用到k8s中；  
但此时，上面配置的APIServer可能并不能正常访问metrics，如下所示

~]# kubectl get apiservice
    v1beta1.metrics.k8s.io          kube-system/metrics-server    False (FailedDiscoveryCheck)
    ...

接下来我们要看看怎么修改kubenetes的启动参数来使用API聚合功能

#### 二、kube-apiserver API聚合层（还在编写中） ####

Aggregation Layer（聚合层）可以允许kube apiserver使用其他API扩展，

##### 1. kube-apiserver添加启动参数 #####

--requestheader-client-ca-file=/xxx/ca.pem
        --requestheader-allowed-names=aggregator
        --requestheader-extra-headers-prefix=X-Remote-Extra-
        --requestheader-group-headers=X-Remote-Group
        --requestheader-username-headers=X-Remote-User
        --proxy-client-cert-file=/xxx/proxy-client.pem
        --proxy-client-key-file=/xxx/proxy-client-key.pem

#### 三、异常 ####

启动后异常  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE5NjU1NDA1_size_16_color_FFFFFF_t_70 1]

异常二  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE5NjU1NDA1_size_16_color_FFFFFF_t_70 2]

通过metrics-server参数`--kubeconfig`配置上kubeconfig文件（内有证书信息）  
kubeconfig通过scerts类资源保存到etcd中，再挂载到metrics-server容器中使用

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE5NjU1NDA1_size_16_color_FFFFFF_t_70]: /images/20221005/936afd398003468a9a9a7da2074062bf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE5NjU1NDA1_size_16_color_FFFFFF_t_70 1]: /images/20221005/a0ebba1e997843e982118f804c84845d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE5NjU1NDA1_size_16_color_FFFFFF_t_70 2]: /images/20221005/4c93539ab5954c4496ef5d12f10c7c91.png