Prometheus GPU 监控

刺骨的言语ヽ痛彻心扉 2023-02-15 13:45 2阅读 0赞

### Prometheus GPU 监控 ###

*  1，Prometheus GPU 监控
 *  2，安装gpu-monitoring-tools
 *   *  2.1，设置\`dcgm-exporter\`开机启动
 *  3，Prometheus修改配置
 *  4，grafana
 *  5，使用监控面板\`9957\`可以切换节点
 *  6，Grafana设置
 *  7，使用\`12027\`
 *  8，使用GPU-Nodes-Metrics-Nvidia 12639

# 1，Prometheus GPU 监控 #

*  安装`DCGM`
 *  `datacenter-gpu-manager_1.7.2_amd64.deb`

# dcgmi --version
    
    dcgmi  version: 1.7.2

# 2，安装gpu-monitoring-tools #

# git clone https://github.com/NVIDIA/gpu-monitoring-tools.git
    # cd gpu-monitoring-tools/
    # make binary
    go build -o dcgm-exporter github.com/NVIDIA/gpu-monitoring-tools/pkg
    # make install
    go build -o dcgm-exporter github.com/NVIDIA/gpu-monitoring-tools/pkg
    install -m 557 dcgm-exporter /usr/bin/dcgm-exporter
    install -m 557 -D ./etc/dcgm-exporter/default-counters.csv /etc/dcgm-exporter/default-counters.csv
    install -m 557 -D ./etc/dcgm-exporter/dcp-metrics-included.csv /etc/dcgm-exporter/dcp-metrics-included.csv

*  运行`dcgm-exporter`

# which dcgm-exporter
    /usr/bin/dcgm-exporter
    # dcgm-exporter
    INFO[0000] Starting dcgm-exporter
    INFO[0000] DCGM successfully initialized!
    INFO[0000] Pipeline starting
    INFO[0000] Starting webserver

*  测试，可以看到监控数据

# curl 192.168.1.2:9400/metrics

## 2.1，设置`dcgm-exporter`开机启动 ##

*  `vim /lib/systemd/system/dcgm-exporter.service` 新建服务

[Unit]
    Description=dcgm-exporter service
    
    [Service]
    User=root
    ExecStart=/usr/bin/dcgm-exporter
    
    TimeoutStopSec=10
    Restart=on-failure
    RestartSec=5
    
    [Install]
    WantedBy=multi-user.target

# systemctl daemon-reload
    # systemctl enable dcgm-exporter.service
    # systemctl start dcgm-exporter.service
    # systemctl status dcgm-exporter.service

# 3，Prometheus修改配置 #

*  添加`dcgm-exporter`

# dcgm-exporter
      - job_name: 'gpu'
        static_configs:
        - targets: ['192.168.1.2:9400']

# cat prometheus.yml
      - job_name: 'prometheus'
    
        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.
    
        static_configs:
        - targets: ['localhost:9090']
    
    
        # node_exporter
      - job_name: 'node'
        static_configs:
        - targets: ['127.0.0.1:9100','192.168.1.2:9100']
    
        # dcgm-exporter
      - job_name: 'gpu'
        static_configs:
        - targets: ['192.168.1.2:9400']

*  重启`prometheus`

systemctl restart  prometheus.service

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70]

# 4，grafana #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 1]

# 5，使用监控面板`9957`可以切换节点 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 2]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 3]

# 6，Grafana设置 #

*  监控功率，`instance`为ip地址

DCGM_FI_DEV_POWER_USAGE{ instance="192.168.1.101:9400"}

*  显卡使用率

DCGM_FI_DEV_GPU_UTIL{ instance="192.168.1.101:9400"}

# 7，使用`12027` #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 4]

# dcgm-exporter
      - job_name: 'gpu-metrics'
        static_configs:
        - targets: ['127.0.0.1:9400','192.168.1.101:9400','192.168.1.102:9400']

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 5]

*  手动设置监控  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 6]
 *  查看显卡指标

curl http://127.0.0.1:9400/metrics

*  使用功率

DCGM_FI_DEV_POWER_USAGE{ instance="127.0.0.1:9400"}

*  内存使用

DCGM_FI_DEV_FB_USED{ instance="127.0.0.1:9400"}

*  总内存

DCGM_FI_DEV_FB_USED{ instance="127.0.0.1:9400"}+DCGM_FI_DEV_FB_FREE{ instance="127.0.0.1:9400"}

*  GPU使用率

DCGM_FI_DEV_GPU_UTIL{ instance="127.0.0.1:9400"}

*  GPU内存使用率

DCGM_FI_DEV_MEM_COPY_UTIL{ instance="192.168.0.114:9400"}

# 8，使用GPU-Nodes-Metrics-Nvidia 12639 #

参考：

1.  [Prometheus + Grafana 监控 NVIDIA GPU][Prometheus _ Grafana _ NVIDIA GPU]
2.  [DCGM 1.7.2 Downloads (December 2019)][DCGM 1.7.2 Downloads _December 2019]
3.  [GPU Nodes v2][]
4.  [NVIDIA/gpu-monitoring-tools][NVIDIA_gpu-monitoring-tools]
5.  [NVIDIA DCGM Exporter Dashboard][]
6.  [GPU Nodesby bkeyzers][]
7.  [Integrating with DCGM][]
8.  [安装dcgm][dcgm]
9.  [基于DCGM和Prometheus的GPU监控方案 dcgm r采集指标项以及含义][DCGM_Prometheus_GPU_ dcgm r]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/2020060814062749.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200608125829785.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/2020061614274660.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/2020061614281394.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200703103327354.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200703103355359.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200703103454803.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI=,size_16,color_FFFFFF,t_70
[Prometheus _ Grafana _ NVIDIA GPU]: https://www.yaoge123.com/blog/archives/2709
[DCGM 1.7.2 Downloads _December 2019]: https://developer.nvidia.com/dcgm
[GPU Nodes v2]: https://grafana.com/grafana/dashboards/11752
[NVIDIA_gpu-monitoring-tools]: https://github.com/NVIDIA/gpu-monitoring-tools
[NVIDIA DCGM Exporter Dashboard]: https://grafana.com/grafana/dashboards/12239
[GPU Nodesby bkeyzers]: https://grafana.com/grafana/dashboards/9957
[Integrating with DCGM]: https://s0docs0nvidia0com.icopy.site/datacenter/dcgm/1.6/dcgm-user-guide/integrating-with-dcgm.html
[dcgm]: https://blog.csdn.net/u010953692/article/details/103876660?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522159376093619724845001762%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fblog.%2522%257D&request_id=159376093619724845001762&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~blog~first_rank_v1~rank_blog_v1-2-103876660.pc_v1_rank_blog_v1&utm_term=DCGM
[DCGM_Prometheus_GPU_ dcgm r]: https://www.jianshu.com/p/dfc32a9db473