Nova — 启动 GPU Passthrough 虚拟机（Nova Centric 模式）

﹏ヽ暗。殇╰゛Y 2023-10-01 21:34 6阅读 0赞

## 目录 ##

#### 文章目录 ####

*  目录
 *  NVIDIA Tesla K40
 *   *  环境
     *  硬件配置
     *  HostOS 配置
     *  OpenStack 配置
     *  VM 配置
     *  TS
 *  NVIDIA 2080Ti
 *   *  环境
     *  HostOS 配置
     *  OpenStack 配置
     *  VM 配置

## NVIDIA Tesla K40 ##

### 环境 ###

*  CentOS 7.9
 *  OpenStack Kolla/Train
 *  NVIDIA Tesla K40
 *  UEFI BIOS

### 硬件配置 ###

*  接入 6pin + 8pin 的电源线。

![在这里插入图片描述][09d2be97b87045a993ecab13822a0698.png]

![在这里插入图片描述][05deb4db4c6e4568a1c64f9eeced2240.png]

![在这里插入图片描述][87ad274827f9460c85c717a33dd46077.png]

### HostOS 配置 ###

1.  BIOS 开启 Intel VT-x、VT-d（Intel VT for Directed I/O）硬件辅助虚拟化功能。

# 确认开启了 CPU 的硬件辅助虚拟化功能。
    egrep -c '(vmx|svm)' /proc/cpuinfo

1.  Linux Kernel 开启 IOMMU 功能，使能 Intel VT-d。

# 在 GRUP CMD line 添加 intel_iommu=on 选项。
    $ vi /etc/default/grub
    ...
    GRUB_CMDLINE_LINUX="intel_iommu=on crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet"
    
    # 更新 GRUP 配置。
    # UEFI 启动模式
    grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
    # 普通启动模式
    $ grub2-mkconfig -o /boot/grub2/grub.cfg
    
    $ reboot
    
    # 检查
    $ cat /proc/cmdline | grep iommu
    BOOT_IMAGE=/vmlinuz-3.10.0-1160.66.1.el7.x86_64 root=/dev/mapper/centos-root ro intel_iommu=on crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet
    
    $ dmesg | grep -i "IOMMU enabled"
    [    0.000000] DMAR: IOMMU enabled

1.  查看 GPU 的 PCI 配置信息，可见其唯一的 BFD 为 10de:1023。

$ lspci -nn | grep NVIDIA
    01:00.0 3D controller [0302]: NVIDIA Corporation GK110BGL [Tesla K40m] [10de:1023] (rev a1)

1.  卸载 nouveau 驱动。

# 查看 GPU 的原生驱动。
    $ lspci -vv -s 01:00.0 | grep driver
    	Kernel driver in use: nouveau
    
    # 禁用 GPU 的默认驱动，为了保证设备不被宿主机使用。
    $ vi /etc/modprobe.d/blacklist.conf
    ...
    blacklist nouveau
    options nouveau modeset=0

1.  为 GPU 设备配置使用 VFIO（Virtual Function I/O）驱动。

# 加载 VFIO 驱动。
    $ vi /etc/modules-load.d/openstack-gpu.conf
    vfio_pci
    vfio
    vfio_iommu_type1
    pci_stub
    kvm
    kvm_intel
    
    # 配置 GPU 的 PCI 设备使用 VFIO 驱动。
    $ vi /etc/modprobe.d/vfio.conf
    options vfio-pci ids=10de:1023
    
    # 重建新的 Kernel initramfs 镜像文件。
    $ mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
    $ dracut /boot/initramfs-$(uname -r).img $(uname -r)
    
    $ reboot
    
    # 检查
    $ dmesg | grep -i vfio
    [ 4557.322686] VFIO - User Level meta-driver version: 0.3
    [ 4557.331414] vfio_pci: add [10de:1023[ffff:ffff]] class 0x000000/00000000
    
    $ lspci -vv -s 01:00.0 | grep driver
    	Kernel driver in use: vfio-pci

### OpenStack 配置 ###

1.  查看 GPU 的 PCI 配置的 BFD，其中 Verdor ID 为 10de、Product ID 为 1023。

$ lspci -nn | grep NVIDIA
    01:00.0 3D controller [0302]: NVIDIA Corporation GK110BGL [Tesla K40m] [10de:1023] (rev a1)

1.  配置 nova-api service，登记 GPU 的 PCI 信息。

$ vi /etc/kolla/nova-api/nova.conf
    ...
    [pci]
    alias = {
        "name":"nvk40", "vendor_id":"10de", "product_id":"1023", "device_type":"type-PCI"}
    
    $ docker restart nova_api

1.  配置 nova-scheduler service，启用 PciPassthroughFilter 调度过滤器。

$ vi /etc/kolla/nova-scheduler/nova.conf
    ...
    [filter_scheduler]
    enabled_filters = RetryFilter, AvailabilityZoneFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, PciPassthroughFilter
    available_filters = nova.scheduler.filters.all_filters
    
    $ docker restart nova_scheduler

1.  配置 nova-compute service，登记 GPU 的 PCI 信息。

$ vi /etc/kolla/nova-compute/nova.conf
    ...
    [pci]
    alias = {
        "name":"nvk40", "vendor_id":"10de", "product_id":"1023", "device_type":"type-PCI"}
    passthrough_whitelist = [{
         "vendor_id": "10de", "product_id": "1023" }]
    
    $ docker restart nova_compute

1.  创建 GPU VM flavor。

$ docker exec -it -u root kolla_toolbox bash
    $ source /var/lib/kolla/config_files/admin-openrc.sh
    
    # openstack flavor set FLAVOR-NAME --property pci_passthrough:alias=ALIAS:COUNT
    $ openstack flavor create --public --vcpus 2 --ram 2048 --disk 20 nvk40-2c2g --property pci_passthrough:alias='nvk40:1'

1.  隐藏 VM 的 Hypervisor ID，NIVIDIA GPU 的驱动程序会检测自己是否跑在 VM 里，如果是，则会出错。所以需要对 GPU 驱动程序隐藏 VM 的 Hypervisor ID。

# $ openstack image set [IMG-UUID] --property img_hide_hypervisor_id=true
    $ openstack image set centos7 --property img_hide_hypervisor_id=true

1.  启动 GPU VM。

### VM 配置 ###

1.  查看 Passthroughed 的 GPU 设备，以及检查 VM 的 Hypervisor ID 是否被隐藏。

$ yum install pciutils -y
    $ lspci -nn | grep NVIDIA
    00:05.0 3D controller [0302]: NVIDIA Corporation GK110BGL [Tesla K40m] [10de:1023] (rev a1)
    
    $ yum install cpuid -y
    $ cpuid | grep hypervisor_id
       hypervisor_id = "@  @      "
       hypervisor_id = "@  @      "

1.  安装 Kernel Headers 和 Developer 包。

$ yum update -y && yum upgrade -y
    
    $ uname -a ; rpm -qa kernel\* | sort
    kernel-3.10.0-1160.66.1.el7.x86_64
    kernel-3.10.0-1160.el7.x86_64
    kernel-tools-3.10.0-1160.66.1.el7.x86_64
    kernel-tools-libs-3.10.0-1160.66.1.el7.x86_64
    
    # 删除不必要的 Kernel
    $ rpm -e kernel-3.10.0-1160.el7.x86_64
    $ reboot
    
    $ uname -r
    3.10.0-1160.66.1.el7.x86_64
    
    $ yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r) -y
    
    $ uname -a ; rpm -qa kernel\* | sort
    kernel-3.10.0-1160.66.1.el7.x86_64
    kernel-devel-3.10.0-1160.66.1.el7.x86_64
    kernel-headers-3.10.0-1160.66.1.el7.x86_64
    kernel-tools-3.10.0-1160.66.1.el7.x86_64
    kernel-tools-libs-3.10.0-1160.66.1.el7.x86_64

1.  安装 NVIDIA Tesla Drivers（Download https://www.nvidia.com/download/index.aspx）。

# Linux 默认安装的是开源的 nouvea 显卡驱动，它与 nvidia 显卡驱动产生冲突。
    $ echo -e "blacklist nouveau\noptions nouveau modeset=0" > /etc/modprobe.d/blacklist.conf
    $ mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
    $ dracut /boot/initramfs-$(uname -r).img $(uname -r)
    $ reboot
    $ lsmod | grep nouveau
    # 没有输出
    
    $ yum install epel-release -y
    $ yum install gcc -y
    $ yum install dkms -y
    
    $ wget https://cn.download.nvidia.com/tesla/460.106.00/NVIDIA-Linux-x86_64-460.106.00.run
    $ sudo chmod +x NVIDIA-Linux-x86_64-460.106.00.run
    $ sudo /bin/bash ./NVIDIA-Linux-x86_64-460.106.00.run
    
    $ nvidia-smi

![在这里插入图片描述][05cf3c0915724195b81a8db10a7115df.png]

1.  安装 Docker CE。

$ sudo yum install -y https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.4.3-3.1.el7.x86_64.rpm
    $ sudo yum install docker-ce -y
    $ sudo systemctl --now enable docker
    $ sudo docker run --rm hello-world

1.  安装 CUDA Container Toolkit。

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
       && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
       
    $ sudo yum clean expire-cache
    $ sudo yum install -y nvidia-docker2
    $ sudo systemctl restart docker
    
    $ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

### TS ###

问题：

Unable to determine the device handle for GPU 0000:00:05.0: Unable to communicate with GPU because it is insufficiently powered.
    
    This may be because not all required external power cables are attached, or the attached cables are not seated properly.

## NVIDIA 2080Ti ##

### 环境 ###

*  CentOS 7.9
 *  OpenStack Train
 *  NVIDIA 2080Ti

### HostOS 配置 ###

1.  BIOS 开启 Intel VT-x、VT-d（Intel VT for Directed I/O）硬件辅助虚拟化功能，以及 Onboard VGA（图像显示卡）功能。

egrep -c '(vmx|svm)' /proc/cpuinfo

1.  Linux Kernel 开启 IOMMU 功能，使能 Intel VT-d。

$ vi /etc/default/grub
    ...
    GRUB_CMDLINE_LINUX="... intel_iommu=on"
    
    # MBR BIOS
    $ grub2-mkconfig -o /boot/grub2/grub.cfg
    # UEFI BIOS
    $ grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
    
    $ reboot
    
    $ dmesg | grep -i iommu
    ...
    [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.6.3.el7.x86_64 root=UUID=4e83b2b5-5ff1-4b1b-af0f-3f6a7f8275ea ro intel_iommu=on crashkernel=auto rhgb quiet
    [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-862.6.3.el7.x86_64 root=UUID=4e83b2b5-5ff1-4b1b-af0f-3f6a7f8275ea ro intel_iommu=on crashkernel=auto rhgb quiet
    [    0.000000] DMAR: IOMMU enabled
    [    0.257808] DMAR-IR: IOAPIC id 3 under DRHD base  0xfbffc000 IOMMU 0
    [    0.257810] DMAR-IR: IOAPIC id 1 under DRHD base  0xd7ffc000 IOMMU 1
    [    0.257812] DMAR-IR: IOAPIC id 2 under DRHD base  0xd7ffc000 IOMMU 1
    
    $ cat /proc/cmdline | grep iommu

1.  查看 GPU 设备的 PCIe 配置信息，可见一个 PCIe GPU 设备实际上由 4 个子设备（VGA、Audio、USB、Serial bus）组成。

$ yum install pciutils -y
    $ lspci -nn | grep NVIDIA
    ...
    06:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1e04] (rev a1)
    06:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f7] (rev a1)
    06:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad6] (rev a1)
    06:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad7] (rev a1)

1.  为 GPU 设备配置 VFIO（Virtual Function I/O）驱动，使得整个 PCIe GPU 设备的所有子设备都能够 Passthrough 到 VM。因为，同一个 PCIe 插槽上的若干个子设备，会被分配到同一个 IOMMU Group。进行 PCIe GPU Passthrough 时，需要将 IOMMU Group 中的所有设备都透传给同一个 VM，否则 nova-compute 会触发错误：Please ensure all devices within the iommu\_group are bound to their vfio bus driver.

# 禁用 GPU 的默认驱动，为了保证设备不被宿主机使用。
    $ vi /etc/modprobe.d/blacklist.conf
    ...
    blacklist nvidia         # for VGA
    blacklist snd_hda_intel  # for Audio
    blacklist xhci_hcd       # for USB
    blacklist nouveau
    blacklist nvidiafb
    blacklist vga16fb
    
    # 重建新的 initramfs 镜像文件。
    mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
    dracut /boot/initramfs-$(uname -r).img $(uname -r)
    
    # 加载 VFIO 驱动。
    $ vi /etc/modules-load.d/openstack-gpu.conf
    ...
    vfio_pci
    vfio
    vfio_iommu_type1
    pci_stub
    kvm
    kvm_intel
    
    # 配置 PCIe GPU 设备的 4 个子设备都使用 VFIO 驱动。
    $ vi /etc/modprobe.d/vfio.conf
    options vfio-pci ids=10de:1e04,10de:10f7,10de:1ad6,10de:1ad7
    
    # 写入到系统启动项。
    $ echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf
    
    $ reboot
    
    $ dmesg | grep -i vfio
    [    6.755346] VFIO - User Level meta-driver version: 0.3
    [    6.803197] vfio_pci: add [10de:1b06[ffff:ffff]] class 0x000000/00000000
    [    6.803306] vfio_pci: add [10de:10ef[ffff:ffff]] class 0x000000/00000000
    
    
    $ lspci -vv -s 06:00.0 | grep driver
    	Kernel driver in use: vfio-pci
    $ lspci -vv -s 06:00.1 | grep driver
    	Kernel driver in use: vfio-pci
    $ lspci -vv -s 06:00.2 | grep driver
    	Kernel driver in use: vfio-pci
    $ lspci -vv -s 06:00.3 | grep driver
    	Kernel driver in use: vfio-pci

### OpenStack 配置 ###

![在这里插入图片描述][4258b63bf4f34d37850c1a6da856d711.png]

1.  查看 PCIe 设备的 \[Verdor ID:Product ID\]。

$ lspci -v -s 06:00.0
    $ lspci -v -s 06:00.1
    $ lspci -v -s 06:00.2
    $ lspci -v -s 06:00.3

1.  配置 nova-scheduler service，启用 PciPassthroughFilter 调度过滤器。

[filter_scheduler]
    ...
    enabled_filters = ...,PciPassthroughFilter
    available_filters = nova.scheduler.filters.all_filters

1.  配置 nova-api service，登记 GPU 设备信息，需要根据 GPU 具体型号的实际情况来设置 device\_type。

[pci]
    # 2080
    alias = {
        "name":"nv2080vga","product_id":"1e04","vendor_id":"10de","device_type":"type-PCI"}
    alias = {
        "name":"nv2080aud","product_id":"10f7","vendor_id":"10de","device_type":"type-PCI"}
    alias = {
        "name":"nv2080usb","product_id":"1ad6","vendor_id":"10de","device_type":"type-PCI"}
    alias = {
        "name":"nv2080bus","product_id":"1ad7","vendor_id":"10de","device_type":"type-PCI"}
    
    ## T4 默认支持基于 SR-IOV 的 vGPU，所以默认的 device_type 为 PF。
    #alias = {"name":"nvT43D","product_id":"1eb8","vendor_id":"10de","device_type":"type-PF"}

1.  配置 nova-compute service。

[libvirt]
    ...
    cpu_mode = host-passthrough
    
    [pci]
    ...
    alias = {
        "name":"nv2080vga","product_id":"1e04","vendor_id":"10de","device_type":"type-PCI"}
    alias = {
        "name":"nv2080aud","product_id":"10f7","vendor_id":"10de","device_type":"type-PCI"}
    alias = {
        "name":"nv2080usb","product_id":"1ad6","vendor_id":"10de","device_type":"type-PCI"}
    alias = {
        "name":"nv2080bus","product_id":"1ad7","vendor_id":"10de","device_type":"type-PCI"}
    
    passthrough_whitelist = [{
         "vendor_id": "10de", "product_id": "1e04" },
    			             {
         "vendor_id": "10de", "product_id": "10f7" },
                             {
         "vendor_id": "10de", "product_id": "1ad6" },
                             {
         "vendor_id": "10de", "product_id": "1ad7" }]

1.  创建 GPU flavor。

openstack flavor create --public --ram 2048 --disk 20 --vcpus 2 m1.large
    
    # openstack flavor set FLAVOR-NAME --property pci_passthrough:alias=ALIAS:COUNT
    openstack flavor set m1.large --property pci_passthrough:alias='nv2080vga:1,nv2080aud:1,nv2080usb:1,nv2080bus:1'

1.  隐藏 VM 的 Hypervisor ID，NIVIDIA GPU 的驱动程序会检测自己是否跑在 VM 里，如果是，则会出错。所以需要对 GPU 驱动程序隐藏 VM 的 Hypervisor ID。

$ openstack image set [IMG-UUID] --property img_hide_hypervisor_id=true
    
    # VM 中执行
    $ cpuid | grep hypervisor_id
    hypervisor_id = "  @  @    "
    hypervisor_id = "  @  @    "

### VM 配置 ###

*  查看 GPU 信息。

lspci -k

*  GuestOS 配置。

$ rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
    $ rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
    $ yum install dkms gcc kernel-devel kernel-headers
    
    # 禁用系统默认安装的 nouveau 驱动
    $ echo -e "blacklist nouveau\noptions nouveau modeset=0" > /etc/modprobe.d/blacklist.conf
    
    # 重建新的 initramfs 镜像文件。
    mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak
    dracut /boot/initramfs-$(uname -r).img $(uname -r)
    
    reboot

*  安装 GPU 驱动程序。

$ sh NVIDIA-Linux-x86_64-450.80.02.run  --kernel-source-path=/usr/src/kernels/3.10.0-514.el7.x86_64  -k $(uname -r) --dkms -s -no-x-check -no-nouveau-check -no-opengl-files
    
    $ nvidia-smi
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  Tesla T4            Off  | 00000000:00:06.0 Off |                    0 |
    | N/A   73C    P0    29W /  70W |      0MiB / 15109MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
                                                                                   
    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

*  安装 CUDA 驱动程序。

yum install cuda cuda-drivers nvidia-driver-latest-dkms

*  配置 Docker 使用 GPU。

$ yum install docker-ce nvidia-docker2
    
    $ vim /etc/docker/daemon.json 
    ...
    {
        
        "runtimes": {
        
            "nvidia": {
        
                "path": "nvidia-container-runtime",
                "runtimeArgs": []
            }
        }
    }
    
    $ systemctl daemon-reload && systemctl restart docker

*  下载 CUDA 容器镜像。

$ docker pull nvidia/cuda:11.0-base
    $ nvidia-docker run -it {
        image_id} /bin/bash
    
    > nvidia-smi
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 455.45.01    Driver Version: 455.45.01    CUDA Version: 11.1     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  Tesla T4            Off  | 00000000:00:06.0 Off |                    0 |
    | N/A   56C    P0    19W /  70W |      0MiB / 15109MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+
                                                                                   
    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

[09d2be97b87045a993ecab13822a0698.png]: https://img-blog.csdnimg.cn/09d2be97b87045a993ecab13822a0698.png
[05deb4db4c6e4568a1c64f9eeced2240.png]: https://img-blog.csdnimg.cn/05deb4db4c6e4568a1c64f9eeced2240.png
[87ad274827f9460c85c717a33dd46077.png]: https://img-blog.csdnimg.cn/87ad274827f9460c85c717a33dd46077.png
[05cf3c0915724195b81a8db10a7115df.png]: https://img-blog.csdnimg.cn/05cf3c0915724195b81a8db10a7115df.png
[4258b63bf4f34d37850c1a6da856d711.png]: https://img-blog.csdnimg.cn/4258b63bf4f34d37850c1a6da856d711.png