[kubernetes]-两大grafana插件配合prometheus使用

朱雀 2022-11-13 11:22 200阅读 0赞

导语：想再grafana上监控k8s

### 提前部署node\_exporter ###

这个之前部署过了

### 提前部署kube-state-metrics ###

##### 创建kube-state-metrics.json #####

{
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      "metadata": {
        "name": "kube-state-metrics",
        "namespace": "kube-system"
      },
      "spec": {
        "selector": {
          "matchLabels": {
            "k8s-app": "kube-state-metrics",
            "grafanak8sapp": "true"
          }
        },
        "replicas": 1,
        "template": {
          "metadata": {
            "labels": {
              "k8s-app": "kube-state-metrics",
              "grafanak8sapp": "true"
            }
          },
          "spec": {
            "containers": [
              {
                "name": "kube-state-metrics",
                "image": "quay.io/coreos/kube-state-metrics:v1.1.0",
                "ports": [
                  {
                    "name": "http-metrics",
                    "containerPort": 8080
                  }
                ],
                "readinessProbe": {
                  "httpGet": {
                    "path": "/healthz",
                    "port": 8080
                  },
                  "initialDelaySeconds": 5,
                  "timeoutSeconds": 5
                }
              }
            ]
          }
        }
      }
    }

![3091aaddc706459761f89fa7286e7c19.png][]

修改prometheus的config

- job_name: 'kubernetes-pods'
        - job_name: 'kubernetes-kubelet'
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecure_skip_verify: true
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
          - role: node
          relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics
        - job_name: 'kubernetes-cadvisor'
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecure_skip_verify: true
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
          - role: node
          relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        - job_name: 'kubernetes-kube-state'
          kubernetes_sd_configs:
          - role: pod
          relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_pod_label_(.+)
          - source_labels: [__meta_kubernetes_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_pod_name]
            action: replace
            target_label: kubernetes_pod_name
          - source_labels: [__meta_kubernetes_pod_label_grafanak8sapp]
            regex: .*true.*
            action: keep
          - source_labels: ['__meta_kubernetes_pod_label_daemon', '__meta_kubernetes_pod_node_name']
            regex: 'node-exporter;(.*)'
            action: replace
            target_label: nodename

应用配置，重启prometheus的pod 使prometheus配置生效

![6b1204e11c9dd6d25e4a0084947e75b0.png][]

### 安装kubernetes-app ###

已经安装并enable 这个插件

安装包下载地址: https://grafana.com/api/plugins/grafana-kubernetes-app/versions/1.0.1/download

cd /data/k8s/grafana/plugins
    unzip grafana-kubernetes-app-31da28.zip

重启grafana的pod使插件生效 并enable 该插件

![d26dd1d86a1e6a304ebdb26ae3332ab5.png][]

在Data Sources添加一个prometheus的源

![ed23f895921bd67ce9cfa5b0467a0570.png][]

我这里因为已经设置过了白名单 所以填一下URL就可以了

![9411a3639dc58fc0b3fcbe508ecd5d89.png][]

保存后 点击Kubernetes-app的图标 添加cluster

![bb189c5a038b105602ef8a46c9ac1693.png][]

![c882d551e19b523581b21a1375dd6a92.png][]

![d97a6c34e4a9f148618a78ee6972a09f.png][]

按图填写如下信息

![750f2d15ffbe1104aed055d193f50913.png][]

apiserver 使用6443端口，以https形式提供服务。客户端访问apiserver需要认证客户端证书。该集群使用`kubeadm` 安装，会有一个`/etc/kubernetes/admin.conf` 文件，里面包含了客户端的证书和密码base64编码。复制 `client-certificate-data`和`client-key-data`的base64编码，分别执行 `echo "<base64 code>" | base64 -d` 就能还原成证书源文件。

通过以下命令获取相关信息

kubectl cluster-info

![eedfb346f7127953871905f05eb20e03.png][]

点击save 第一次成功添加会提示绿色的成功 我这里点快了重复添加了

![9ebe1b6e2b6d9230609f00f277ad1ca9.png][]

添加完成之后 在datasouces里可以看到它

![ce18ae412eadb04ffa7c91fe9a0b4dc1.png][]

我这里不知道是不是版本原因 也没报错 cluster界面一直是loading 就没继续下去了。没有自带的dashboard 不想自己去导入模版了。贴一下往网上的图

![c4e45cda1a5e3ac71589c9826ea09525.png][]

![3f2e22b1bb6eee1879388aa939d31ff0.png][]

![a1c3d33cce1adedb21e7ee10c95319b4.png][]

### 安装kubegraf ###

安装grafana插件并重启

# 进入pod安装
    grafana-cli plugins install devopsprodigy-kubegraf-app
    # 下载到插件文件夹解压也可
    # cd /data/k8s/grafana/plugins/
    # git clone https://github.com/devopsprodigy/kubegraf /var/lib/grafana/plugins/devopsprodigy-kubegraf-app # 这个后面是你plugins的路径

创建namespace及对应权限

kubectl create ns kubegraf
    kubectl apply -f https://raw.githubusercontent.com/devopsprodigy/kubegraf/master/kubernetes/serviceaccount.yaml
    kubectl apply -f https://raw.githubusercontent.com/devopsprodigy/kubegraf/master/kubernetes/clusterrole.yaml
    kubectl apply -f https://raw.githubusercontent.com/devopsprodigy/kubegraf/master/kubernetes/clusterrolebinding.yaml
    kubectl apply -f https://raw.githubusercontent.com/devopsprodigy/kubegraf/master/kubernetes/secret.yaml

创建private key and certificate

openssl genrsa -out ~/grafana-kubegraf.key 2048
    openssl req -new -key ~/grafana-kubegraf.key -out ~/grafana-kubegraf.csr -subj "/CN=grafana-kubegraf/O=monitoring"
    openssl x509 -req -in ~/grafana-kubegraf.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -out /etc/kubernetes/pki/grafana-kubegraf.crt -CAcreateserial

将证书拷贝到其余master

scp -r /etc/kubernetes/pki/grafana-kubegraf.crt prod-k8s-m002:/etc/kubernetes/pki/
    scp -r /etc/kubernetes/pki/grafana-kubegraf.crt prod-k8s-m003:/etc/kubernetes/pki/

![c2b9a2cbff0a921f7487ce86fd02c386.png][]

在grafana中enable 插件

![f7fdac287ea6bb697e232e93c5aa0205.png][]

添加cluster

![8e0026d0ea889956152e99d47c39d12f.png][]

填写对应信息及证书信息

![1878a81797b481446f20a4ef89415812.png][]

添加成功

![711c48fabe90ecd36368f4233b4b5e60.png][]

进入cluster界面

![a362ca738fe32d44634bcec1526a6a8c.png][]

查看对应各个dashboard

![cdc906bc9bd92f0260b253bc022a4219.png][]

![9968c6f345fd4ab89e7dd8fd31681c12.png][]

![90952ef506d6b39ea782b631ed3522a5.png][]

参考地址

https://grafana.com/grafana/plugins/grafana-kubernetes-app/

https://grafana.com/grafana/plugins/devopsprodigy-kubegraf-app/

[3091aaddc706459761f89fa7286e7c19.png]: /images/20221022/4005c1ae63d846388bdee69801a7b8da.png
[6b1204e11c9dd6d25e4a0084947e75b0.png]: /images/20221022/57b45c5d759c4aaaa7ed73b96af78f35.png
[d26dd1d86a1e6a304ebdb26ae3332ab5.png]: https://img-blog.csdnimg.cn/img_convert/d26dd1d86a1e6a304ebdb26ae3332ab5.png
[ed23f895921bd67ce9cfa5b0467a0570.png]: https://img-blog.csdnimg.cn/img_convert/ed23f895921bd67ce9cfa5b0467a0570.png
[9411a3639dc58fc0b3fcbe508ecd5d89.png]: https://img-blog.csdnimg.cn/img_convert/9411a3639dc58fc0b3fcbe508ecd5d89.png
[bb189c5a038b105602ef8a46c9ac1693.png]: https://img-blog.csdnimg.cn/img_convert/bb189c5a038b105602ef8a46c9ac1693.png
[c882d551e19b523581b21a1375dd6a92.png]: https://img-blog.csdnimg.cn/img_convert/c882d551e19b523581b21a1375dd6a92.png
[d97a6c34e4a9f148618a78ee6972a09f.png]: https://img-blog.csdnimg.cn/img_convert/d97a6c34e4a9f148618a78ee6972a09f.png
[750f2d15ffbe1104aed055d193f50913.png]: https://img-blog.csdnimg.cn/img_convert/750f2d15ffbe1104aed055d193f50913.png
[eedfb346f7127953871905f05eb20e03.png]: https://img-blog.csdnimg.cn/img_convert/eedfb346f7127953871905f05eb20e03.png
[9ebe1b6e2b6d9230609f00f277ad1ca9.png]: https://img-blog.csdnimg.cn/img_convert/9ebe1b6e2b6d9230609f00f277ad1ca9.png
[ce18ae412eadb04ffa7c91fe9a0b4dc1.png]: https://img-blog.csdnimg.cn/img_convert/ce18ae412eadb04ffa7c91fe9a0b4dc1.png
[c4e45cda1a5e3ac71589c9826ea09525.png]: https://img-blog.csdnimg.cn/img_convert/c4e45cda1a5e3ac71589c9826ea09525.png
[3f2e22b1bb6eee1879388aa939d31ff0.png]: https://img-blog.csdnimg.cn/img_convert/3f2e22b1bb6eee1879388aa939d31ff0.png
[a1c3d33cce1adedb21e7ee10c95319b4.png]: https://img-blog.csdnimg.cn/img_convert/a1c3d33cce1adedb21e7ee10c95319b4.png
[c2b9a2cbff0a921f7487ce86fd02c386.png]: https://img-blog.csdnimg.cn/img_convert/c2b9a2cbff0a921f7487ce86fd02c386.png
[f7fdac287ea6bb697e232e93c5aa0205.png]: https://img-blog.csdnimg.cn/img_convert/f7fdac287ea6bb697e232e93c5aa0205.png
[8e0026d0ea889956152e99d47c39d12f.png]: https://img-blog.csdnimg.cn/img_convert/8e0026d0ea889956152e99d47c39d12f.png
[1878a81797b481446f20a4ef89415812.png]: https://img-blog.csdnimg.cn/img_convert/1878a81797b481446f20a4ef89415812.png
[711c48fabe90ecd36368f4233b4b5e60.png]: https://img-blog.csdnimg.cn/img_convert/711c48fabe90ecd36368f4233b4b5e60.png
[a362ca738fe32d44634bcec1526a6a8c.png]: https://img-blog.csdnimg.cn/img_convert/a362ca738fe32d44634bcec1526a6a8c.png
[cdc906bc9bd92f0260b253bc022a4219.png]: https://img-blog.csdnimg.cn/img_convert/cdc906bc9bd92f0260b253bc022a4219.png
[9968c6f345fd4ab89e7dd8fd31681c12.png]: https://img-blog.csdnimg.cn/img_convert/9968c6f345fd4ab89e7dd8fd31681c12.png
[90952ef506d6b39ea782b631ed3522a5.png]: https://img-blog.csdnimg.cn/img_convert/90952ef506d6b39ea782b631ed3522a5.png