使用ssl_exporter监控K8S集群证书

清疚 2022-11-19 07:41 310阅读 0赞

使用kubeadm搭建的集群默认证书有效期是1年，续费证书其实是一件很快的事情。但是就怕出事了才发现，毕竟作为专业搬砖工程师，每天都很忙的。  
鉴于此，监控集群证书有效期是一件不得不做的事情。Prometheus作为云原生领域的王者，如果能用它来监控证书有效期并能及时告警，那就再好不过了。  
ssl\_exporter就是来做这个事情的。ssh\_exporter是一个Prometheus Exporter能提供多种针对 SSL 的检测手段，包括：https 证书生效/失效时间、文件证书生效/失效时间，OCSP 等相关指标。

下面就来监听集群证书的有效期。

apiVersion: v1
    kind: Service
    metadata:
      labels:
        name: ssl-exporter
      name: ssl-exporter
    spec:
      ports:
        - name: ssl-exporter
          protocol: TCP
          port: 9219
          targetPort: 9219
      selector:
        app: ssl-exporter
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: ssl-exporter
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: ssl-exporter
      template:
        metadata:
          name: ssl-exporter
          labels:
            app: ssl-exporter
        spec:
          initContainers:
            # Install kube ca cert as a root CA
            - name: ca
              image: alpine
              command:
                - sh
                - -c
                - |
                  set -e
                  apk add --update ca-certificates
                  cp /var/run/secrets/kubernetes.io/serviceaccount/ca.crt /usr/local/share/ca-certificates/kube-ca.crt
                  update-ca-certificates
                  cp /etc/ssl/certs/* /ssl-certs volumeMounts: - name: ssl-certs mountPath: /ssl-certs containers: - name: ssl-exporter image: ribbybibby/ssl-exporter:v0.6.0 ports: - name: tcp containerPort: 9219 volumeMounts: - name: ssl-certs mountPath: /etc/ssl/certs volumes: - name: ssl-certs emptyDir: {}

执行kubectl apply -f .安装即可。

待Pod正常运行，如下：

# kubectl get po -n monitoring -l app=ssl-exporter
    NAME                            READY   STATUS    RESTARTS   AGE
    ssl-exporter-7ff4759679-f4qbs   1/1     Running   0          21m

然后配置prometheus抓取规则。

> !! 由于我的Prometheus是通过Prometheus Operator部署的，所以通过additional的方式进行抓取。

首先创建一个文件prometheus-additional.yaml，其内容如下：

- job_name: ssl-exporter
      metrics_path: /probe
      static_configs:
      - targets:
        - kubernetes.default.svc:443
      relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: ssl-exporter.monitoring:9219

然后创建secret，命令如下：

kubectl delete secret additional-config -n monitoring
    kubectl -n monitoring create secret generic additional-config --from-file=prometheus-additional.yaml

然后修改prometheus-prometheus.yaml配置文件，新增如下内容：

additionalScrapeConfigs:
        name: additional-config 
        key: prometheus-additional.yaml

prometheus-prometheus.yaml的整体配置如下：

apiVersion: monitoring.coreos.com/v1
    kind: Prometheus
    metadata:
      labels:
        prometheus: k8s
      name: k8s
      namespace: monitoring
    spec:
      alerting:
        alertmanagers:
        - name: alertmanager-main
          namespace: monitoring
          port: web
      baseImage: quay.io/prometheus/prometheus
      nodeSelector:
        kubernetes.io/os: linux
      podMonitorNamespaceSelector: { }
      podMonitorSelector: { }
      replicas: 2
      resources:
        requests:
          memory: 400Mi
      ruleSelector:
        matchLabels:
          prometheus: k8s
          role: alert-rules
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 1000
      additionalScrapeConfigs:
        name: additional-config 
        key: prometheus-additional.yaml 
      serviceAccountName: prometheus-k8s
      serviceMonitorNamespaceSelector: { }
      serviceMonitorSelector: { }
      version: v2.11.0
      storage:
        volumeClaimTemplate:
          spec:
            storageClassName: managed-nfs-storage 
            resources:
              requests:
                storage: 10Gi

然后重新执行prometheus-prometheus.yaml文件，命令如下：  
![在这里插入图片描述][2021012716260130.png]  
然后通过(ssl\_cert\_not\_after-time())/3600/24即可看到证书还有多久失效。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3_size_16_color_FFFFFF_t_70]  
通过ssl\_tls\_connect\_success可以观测ssl链接是否正常。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3_size_16_color_FFFFFF_t_70 1]  
上面已经安装ssl\_exporter成功，并且能正常监控数据了，下面就配置一些告警规则，以便于运维能快速知道这个事情。

apiVersion: monitoring.coreos.com/v1
    kind: PrometheusRule
    metadata:
      name: monitoring-ssl-tls-rules
      namespace: monitoring
      labels:
        prometheus: k8s
        role: alert-rules
    spec:
      groups:
      - name: check_ssl_validity
        rules:
        - alert: "K8S集群证书在30天后过期"
          expr: (ssl_cert_not_after-time())/3600/24 <30
          for: 1h
          labels:
            severity: critical
          annotations:
            description: 'K8S集群的证书还有{ { printf "%.1f" $value }}天就过期了,请尽快更新证书'
            summary: "K8S集群证书证书过期警告"
      - name: ssl_connect_status
        rules:
        - alert: "K8S集群证书可用性异常"
          expr: ssl_tls_connect_success == 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "K8S集群证书连接异常"
            description: "K8S集群 { { $labels.instance }} 证书连接异常"

如下展示规则正常，在异常的时候就可以接收到告警了。  
![在这里插入图片描述][20210127162731663.png]

https://mp.weixin.qq.com/s/t0rGpblWTD1fyFPa\_Tlgqw

[2021012716260130.png]: /images/20221119/685ac65dd5d542b5998d5a9abffb161e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3_size_16_color_FFFFFF_t_70]: /images/20221119/2b06a9fdf87847248936bccc527e42ec.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQwOTA3OTc3_size_16_color_FFFFFF_t_70 1]: /images/20221119/786f6a4e363645fa97f5baffb59b8208.png
[20210127162731663.png]: /images/20221119/cacf56ea1d6a430e9e949c31e37cb6f8.png