prometheus在k8s中的部署

Dear 丶 2022-10-15 08:57 360阅读 0赞

1.k8s的监控指标


























监控指标 具体实现 举例
Pod性能 cAdvisor 容器CPU，内存利用率
Node性能 node-exporter 节点CPU，内存利用率
K8S资源对象 kube-state-metrics Pod/Deployment/Service

2.创建namespace、sa账号，在k8s集群的master节点操作

#创建一个monitor-sa的名称空间

  1. kubectl create ns monitor-sa

#创建一个sa账号

  1. kubectl create serviceaccount monitor -n monitor-sa

#把sa账号monitor通过clusterrolebing绑定到clusterrole上

  1. kubectl create clusterrolebinding monitor-clusterrolebinding -n monitor-sa --clusterrole=cluster-admin  --serviceaccount=monitor-sa:monitor

3.创建数据目录

#在k8s集群的任何一个node节点操作，因为我的k8s集群只有一个node节点node1，所以我在node1上操作如下命令：

  1. mkdir /data/prometheus
  2. chmod 777 /data/prometheus

4. kube-state-metric的部署

prometheus通过 sa,clusterrolebinding来解决token、证书挂载问题
sa等配置： prometheus yaml中需要配置对应的saserviceAccountName

kube-state-metrics github项目地址

  1. .
  2. ├── cluster-role-binding.yaml
  3. ├── cluster-role.yaml
  4. ├── deployment.yaml
  5. ├── README.md
  6. ├── service-account.yaml
  7. └── service.yaml

service-account.yaml文件

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   labels:
  5.     app.kubernetes.io/name: kube-state-metrics
  6.     app.kubernetes.io/version: v1.8.0
  7.   name: kube-state-metrics
  8.   namespace: monitor-sa

cluster-role.yaml 文件

  1. apiVersion: rbac.authorization.k8s.io/v1
  2. kind: ClusterRole
  3. metadata:
  4.   labels:
  5.     app.kubernetes.io/name: kube-state-metrics
  6.     app.kubernetes.io/version: v1.8.0
  7.   name: kube-state-metrics
  8. rules:
  9. - apiGroups:
  10.   - ""
  11.   resources:
  12.   - configmaps
  13.   - secrets
  14.   - nodes
  15.   - pods
  16.   - services
  17.   - resourcequotas
  18.   - replicationcontrollers
  19.   - limitranges
  20.   - persistentvolumeclaims
  21.   - persistentvolumes
  22.   - namespaces
  23.   - endpoints
  24.   verbs:
  25.   - list
  26.   - watch
  27. - apiGroups:
  28.   - extensions
  29.   resources:
  30.   - daemonsets
  31.   - deployments
  32.   - replicasets
  33.   - ingresses
  34.   verbs:
  35.   - list
  36.   - watch
  37. - apiGroups:
  38.   - apps
  39.   resources:
  40.   - statefulsets
  41.   - daemonsets
  42.   - deployments
  43.   - replicasets
  44.   verbs:
  45.   - list
  46.   - watch
  47. - apiGroups:
  48.   - batch
  49.   resources:
  50.   - cronjobs
  51.   - jobs
  52.   verbs:
  53.   - list
  54.   - watch
  55. - apiGroups:
  56.   - autoscaling
  57.   resources:
  58.   - horizontalpodautoscalers
  59.   verbs:
  60.   - list
  61.   - watch
  62. - apiGroups:
  63.   - authentication.k8s.io
  64.   resources:
  65.   - tokenreviews
  66.   verbs:
  67.   - create
  68. - apiGroups:
  69.   - authorization.k8s.io
  70.   resources:
  71.   - subjectaccessreviews
  72.   verbs:
  73.   - create
  74. - apiGroups:
  75.   - policy
  76.   resources:
  77.   - poddisruptionbudgets
  78.   verbs:
  79.   - list
  80.   - watch
  81. - apiGroups:
  82.   - certificates.k8s.io
  83.   resources:
  84.   - certificatesigningrequests
  85.   verbs:
  86.   - list
  87.   - watch
  88. - apiGroups:
  89.   - storage.k8s.io
  90.   resources:
  91.   - storageclasses
  92.   - volumeattachments
  93.   verbs:
  94.   - list
  95.   - watch
  96. - apiGroups:
  97.   - admissionregistration.k8s.io
  98.   resources:
  99.   - mutatingwebhookconfigurations
  100.   - validatingwebhookconfigurations
  101.   verbs:
  102.   - list
  103.   - watch
  104. - apiGroups:
  105.   - networking.k8s.io
  106.   resources:
  107.   - networkpolicies
  108.   verbs:
  109.   - list
  110.   - watch

cluster-role-binding.yaml

  1. apiVersion: rbac.authorization.k8s.io/v1
  2. kind: ClusterRoleBinding
  3. metadata:
  4.   labels:
  5.     app.kubernetes.io/name: kube-state-metrics
  6.     app.kubernetes.io/version: v1.8.0
  7.   name: kube-state-metrics
  8. roleRef:
  9.   apiGroup: rbac.authorization.k8s.io
  10.   kind: ClusterRole
  11.   name: kube-state-metrics
  12. subjects:
  13. - kind: ServiceAccount
  14.   name: kube-state-metrics
  15.   namespace: monitor-sa

deployment.yaml 文件

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   labels:
  5.     app.kubernetes.io/name: kube-state-metrics
  6.     app.kubernetes.io/version: v1.8.0
  7.   name: kube-state-metrics
  8.   namespace: monitor-sa
  9. spec:
  10.   replicas: 1
  11.   selector:
  12.     matchLabels:
  13.       app.kubernetes.io/name: kube-state-metrics
  14.   template:
  15.     metadata:
  16.       labels:
  17.         app.kubernetes.io/name: kube-state-metrics
  18.         app.kubernetes.io/version: v1.8.0
  19.     spec:
  20.       containers:
  21.       - image: quay.io/coreos/kube-state-metrics:v1.8.0
  22.         livenessProbe:
  23.           httpGet:
  24.             path: /healthz
  25.             port: 8080
  26.           initialDelaySeconds: 5
  27.           timeoutSeconds: 5
  28.         name: kube-state-metrics
  29.         ports:
  30.         - containerPort: 8080
  31.           name: http-metrics
  32.         - containerPort: 8081
  33.           name: telemetry
  34.         readinessProbe:
  35.           httpGet:
  36.             path: /
  37.             port: 8081
  38.           initialDelaySeconds: 5
  39.           timeoutSeconds: 5
  40.       nodeSelector:
  41.         kubernetes.io/os: linux
  42.       serviceAccountName: kube-state-metrics

service.yaml文件

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   annotations:
  5.     prometheus.io/scrape: 'true'
  6.   name: kube-state-metrics
  7.   namespace: monitor-sa
  8.   labels:
  9.     app: kube-state-metrics
  10. spec:
  11.     ports:
  12.     - name: kube-state-metrics
  13.       port: 8080
  14.       protocol: TCP
  15.     - name: telemetry
  16.       port: 8081
  17.       protocol: TCP
  18.     selector:
  19.       app.kubernetes.io/name: kube-state-metrics

5.安装prometheus，以下步骤均在在k8s集群的master节点操作

1）创建一个configmap存储卷，用来存放prometheus配置信息

  1. kubectl get sa monitor  -n monitor-sa -o yaml
  2. apiVersion: v1
  3. kind: ServiceAccount
  4. metadata:
  5.   creationTimestamp: "2021-05-23T14:18:14Z"
  6.   name: monitor
  7.   namespace: monitor-sa
  8.   resourceVersion: "18761312"
  9.   selfLink: /api/v1/namespaces/monitor-sa/serviceaccounts/monitor
  10.   uid: 12ed67ab-dae8-4704-87b8-5a073a7047d2
  11. secrets:
  12. - name: monitor-token-p6wgp
  14. kubectl describe sa monitor  -n monitor-sa 
  15. Name:                monitor
  16. Namespace:           monitor-sa
  17. Labels:              <none>
  18. Annotations:         <none>
  19. Image pull secrets:  <none>
  20. Mountable secrets:   monitor-token-p6wgp
  21. Tokens:              monitor-token-p6wgp
  22. Events:              <none>
  25. kubectl describe secrets monitor-token-p6wgp -n monitor-sa 
  26. Name:         monitor-token-p6wgp
  27. Namespace:    monitor-sa
  28. Labels:       <none>
  29. Annotations:  kubernetes.io/service-account.name: monitor
  30.               kubernetes.io/service-account.uid: 12ed67ab-dae8-4704-87b8-5a073a7047d2
  32. Type:  kubernetes.io/service-account-token
  34. Data
  35. ====
  36. ca.crt:     1025 bytes
  37. namespace:  10 bytes
  38. token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IkY5eEJEUjZMRjNnejhxMVl6enJiYmVHX0RSOFYza1JfbVVpZmhCVXlucDQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1vbml0b3ItdG9rZW4tcDZ3Z3AiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibW9uaXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjEyZWQ2N2FiLWRhZTgtNDcwNC04N2I4LTVhMDczYTcwNDdkMiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yLXNhOm1vbml0b3IifQ.j2hWAze7aOZgVDg0j4NKOhoMUktu7XIJ56kU_RCRbt7RCaXYd_A4ijg7IJqVUHBitQKfx-_ZzNXcOqMZt5nCN5dtToKGWRK_Du0eqepKNcsfj9dzVvebaEbd-4t7LyhHvEdf5M1CviD0wnrw1O_9nXl1COpm9IojJB9I8tIzs9Y3fiMVd2oTUL3ctFKRSkwM4CTAEIm5SZN0QRgld7Ol8W7F-m8jjOh3c7MMm9FnnAn_NkQ57XSKJovMy_AdMA55gwZaufCYA225tubG9KS0eUyF70wgGvAKOMFn6yGpRZjHj26JcBDhoEZkwzFrBM4-blnGl9pMHXtPztAPlw-xQQ
  40. cat    >prometheus-cfg.yaml <<EOF --- kind: ConfigMap apiVersion: v1 metadata: labels: app: prometheus name: prometheus-config namespace: monitor-sa data: prometheus.yml: | global: scrape_interval: 15s scrape_timeout: 10s external_labels: monitor: 'AIUI-ceshi-k8s' evaluation_interval: 1m scrape_configs: - job_name: kubernetes-node kubernetes_sd_configs: - role: node tls_config: insecure_skip_verify: true bearer_token_file: /opt/k8s/k8s.token relabel_configs: - source_labels: [__address__] regex: '(.*):10250' replacement: '${1}:9100' target_label: __address__ action: replace - action: labelmap regex: __meta_kubernetes_node_label_(.+) - source_labels: [instance] regex: .*db002.* action: drop - job_name: 'kubernetes-apiservers' kubernetes_sd_configs: - role: endpoints scheme: https tls_config: insecure_skip_verify: true bearer_token_file: /opt/k8s/k8s.token relabel_configs: - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name] action: keep regex: default;kubernetes;https - job_name: 'kube-state-metrics' static_configs: - targets: ['kube-state-metrics:8080'] - job_name: 'kubernetes-node-cadvisor' kubernetes_sd_configs: - role: node scheme: https tls_config: insecure_skip_verify: true bearer_token_file: /opt/k8s/k8s.token relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) - target_label: __address__ replacement: 172.16.154.13:6443 - source_labels: [__meta_kubernetes_node_name] regex: (.+) target_label: __metrics_path__ replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor - job_name: 'kubernetes-service-endpoints' scrape_timeout: 10s kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (https?) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: kubernetes_name # - source_labels: [__meta_kubernetes_pod_container_port_number] # action: replace # target_label: container_port EOF
  42. cat >prometheus-deploy.yaml    <<EOF
  43. ---
  44. apiVersion: apps/v1
  45. kind: Deployment
  46. metadata:
  47.   name: prometheus-server
  48.   namespace: monitor-sa
  49.   labels:
  50.     app: prometheus
  51. spec:
  52.   replicas: 1
  53.   selector:
  54.     matchLabels:
  55.       app: prometheus
  56.       component: server
  57.   template:
  58.     metadata:
  59.       labels:
  60.         app: prometheus
  61.         component: server
  62.       annotations:
  63.         prometheus.io/scrape: 'false'
  64.     spec:
  65.       serviceAccountName: monitor
  66.       containers:
  67.       - name: prometheus
  68.         image: prom/prometheus:v2.2.1
  69.         imagePullPolicy: IfNotPresent
  70.         command:
  71.           - 'prometheus'
  72.           - '--config.file=/etc/prometheus/prometheus.yml'
  73.           - '--storage.tsdb.path=/prometheus'
  74.           - '--storage.tsdb.retention=720h'
  75.           - '--web.enable-lifecycle'
  76.         ports:
  77.         - containerPort: 9090
  78.           protocol: TCP
  79.         volumeMounts:
  80.         - mountPath: /etc/prometheus/prometheus.yml
  81.           name: prometheus-config
  82.           subPath: prometheus.yml
  83.         - mountPath: /prometheus/
  84.           name: prometheus-storage-volume
  85.         - mountPath: /opt/k8s/k8s.token
  86.           name: k8s-token
  87.           subPath: k8s.token
  88.       volumes:
  89.         - name: prometheus-config
  90.           configMap:
  91.             name: prometheus-config
  92.             items:
  93.               - key: prometheus.yml
  94.                 path: prometheus.yml
  95.                 mode: 0644
  96.         - name: prometheus-storage-volume
  97.           hostPath:
  98.            path: /data/prometheus
  99.            type: Directory
  100.         - name: k8s-token
  101.           hostPath:
  102.            path: /opt/k8s
  103.            type: Directory
  105. cat  > prometheus-svc.yaml  <<EOF --- apiVersion: v1 kind: Service metadata: annotations: prometheus.io/scrape: 'true' name: prometheus namespace: monitor-sa labels: app: prometheus spec: type: NodePort ports: - port: 9090 targetPort: 9090 nodePort: 30090 protocol: TCP selector: app: prometheus component: server EOF

在这里插入图片描述
在这里插入图片描述

发表评论

表情：
评论列表 （有 0 条评论，360人围观）

还没有评论，来说两句吧...

相关阅读