docker 访问 GPU NVIDIA Prometheus 监控

不念不忘少年蓝@ 2022-10-31 00:55 476阅读 0赞

### docker 访问 GPU NVIDIA ###

*  1.仓库配置
 *  2.安装
 *  3.验证
 *  4.Prometheus监控
 *  5.Prometheus修改配置文件
 *  6.grafana仪表板导入

# 1.仓库配置 #

# curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | sudo apt-key add -
    OK
    # distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
    
    # curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.list | sudo tee /etc/apt/sources.list.d/nvidia-container-runtime.list
    deb https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/$(ARCH) /
    #deb https://nvidia.github.io/libnvidia-container/experimental/ubuntu18.04/$(ARCH) /
    deb https://nvidia.github.io/nvidia-container-runtime/stable/ubuntu18.04/$(ARCH) /
    #deb https://nvidia.github.io/nvidia-container-runtime/experimental/ubuntu18.04/$(ARCH) /
    
    # sudo apt-get update

*  无法访问

# curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | sudo apt-key add -
    gpg: no valid OpenPGP data found.

*  浏览器下载 gpgkey

https://nvidia.github.io/nvidia-container-runtime/gpgkey

# sudo apt-key add gpgkey
    OK

curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | sudo apt-key add -

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)

curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.list | sudo tee /etc/apt/sources.list.d/nvidia-container-runtime.list

sudo apt-get update

# 2.安装 #

# apt-get install nvidia-container-runtime -y

# which nvidia-container-runtime-hook
    /usr/bin/nvidia-container-runtime-hook

systemctl restart docker.service

# 3.验证 #

# docker run -it --rm --gpus all ubuntu nvidia-smi

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70]

# 4.Prometheus监控 #

docker run -d --gpus all --rm -p 9400:9400 nvidia/dcgm-exporter:2.0.13-2.1.1-ubuntu18.04

# docker ps
    CONTAINER ID   IMAGE                                           COMMAND                  CREATED              STATUS              PORTS                    NAMES
    1b674d0565db   nvidia/dcgm-exporter:2.0.13-2.1.1-ubuntu18.04   "/usr/local/dcgm/dcg…"   About a minute ago   Up About a minute   0.0.0.0:9400->9400/tcp   vigilant_nightingale

# curl localhost:9400/metrics

# 5.Prometheus修改配置文件 #

*  `vim prometheus.yml`

scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'prometheus'
    
        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.
    
        static_configs:
        - targets: ['localhost:9090']
      # node
      - job_name: 'node'
        static_configs:
        - targets: ['localhost:9100','192.168.10.2:9100']
      # GPU
      - job_name: 'gpu'
        static_configs:
        - targets: ['192.168.10.2:9400']

systemctl restart  prometheus.service

# 6.grafana仪表板导入 #

*  使用 12639

--------------------

参考：

*  [GPU docker][]
 *  [nvidia-container-runtime][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTA5NTM2OTI_size_16_color_FFFFFF_t_70]: /images/20221024/9222cad32811410183b3b10a6d34947f.png
[GPU docker]: https://docs.docker.com/config/containers/resource_constraints/
[nvidia-container-runtime]: https://nvidia.github.io/nvidia-container-runtime/