- 1 Proxmox Configuration for GPU Passthrough
- 2 Debian 10 VM GPU Passthrough Configuration
- 3 Other Notes
Editors note: Last updated 12/26/2022
I’ve seen various parts of this documented on the internet, but I don’t think I’ve seen all the steps written down in one place, so in the interest of sharing and not banging my head next time I need to re-create my Plex VM: here’s how I was able to get my NVIDIA Quadro K620 GPU to work with my Plex VM running in Proxmox.
Here’s my setup:
- Proxmox (7.2-4 No-Subscription Repository) using the Linux 5.15.35-2-pve kernel
- Plex VM on Debian 10 using the Linux 4.19.0-20-amd64 kernel
- NVIDIA Quadro K620 GPU. Note: I’m only using this in headless mode for Plex transcoding.
The first part of the steps are based on https://3os.org/infrastructure/proxmox/gpu-passthrough/pgu-passthrough-to-vm/#proxmox-configuration-for-gpu-passthrough. However, installing the Debian driver didn’t work. Maybe because I’m using Debian 10 and perhaps the NVIDIA v470 driver has a bug? I think there are some repo differences between Ubuntu and Debian.
Proxmox Configuration for GPU Passthrough
Find GPU Bus Address and Device ID
Find the PCI address of the GPU Device:
lspci -nnv | grep VGA
0b:00.0 VGA compatible controller : Matrox Electronics Systems Ltd. G200eR2 [102b:0534] (prog-if 00 [VGA controller]) 42:00.0 VGA compatible controller : NVIDIA Corporation GM107GL [Quadro K620] [10de:13bb] (rev a2) (prog-if 00 [VGA controller]
What we are looking is the PCI address of the NVIDIA GPU device. In this case it’s
42:00.0 is only a part of of a group of PCI devices on the GPU.
We can list all the devices in the group
42:00 by using the following command:
lspci -s 42:00
The usual output will include VGA Device and Audio Device, which is what I have:
42:00.0 VGA compatible controller: NVIDIA Corporation GM107GL [Quadro K620] (rev a2) 42:00.1 Audio device: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] (rev a1)
Now we need to get the device ID’s of those devices. We can do this by using the following command:
lspci -s 42:00 -n
The output should look similar to this:
42:00.0 0300: 10de:13bb (rev a2) 42:00.1 0403: 10de:0fbc (rev a1)
What we are looking are the pairs, we will use those id to split the PCI Group to separate devices:
Edit the grub configuration file at
Find the line that starts with
GRUB_CMDLINE_LINUX_DEFAULT and updated it to look like this (Intel CPU example) — replace
vfio-pci.ids= with the ids for the GPU you want to passthrough:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifb:off video=vesa:off vfio-pci.ids=10de:13bb,10de:0fb vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu"
Note: if you have an AMD CPU, then use
amd_iommu=on instead of
Save the config changed and then update GRUB.
Next we need to add vfio modules to allow PCI passthrough. Edit the
/etc/modules file and add the following line to the end of the file:
# Modules required for PCI passthrough vfio vfio_iommu_type1 vfio_pci vfio_virqfd
Update configuration changes made in your /etc filesystem
update-initramfs -u -k all
Reboot and verify
Reboot Proxmox to apply the changes
Verify that IOMMU is enabled
dmesg | grep -e DMAR -e IOMMU
There should be a line that looks like
DMAR: IOMMU enabled. If there is no output, something is wrong.
Your Proxmox host should be ready to GPU passthrough!
Debian 10 VM GPU Passthrough Configuration
I’m using Debian 10 as my VM.
For best performance the VM should be configured the Machine type to
q35. This will allow the VM to utilize PCI-Express passthrough.
Add a PCI device to the VM (Add > PCI Device). Select the GPU Device, which you can find using it’s PCI address from before (42:00, in this example). Note: This list uses a different format for the PCI addresses id,
42:00.0 is listed as
- All Functions
…and then select Add. Note: Do not select Primary GPU.
Boot the VM. To test the GPU passthrough was successful, you can use the following command in the VM:
sudo lspci -nnv | grep VGA
The output should include the GPU:
00:01.0 VGA compatible controller : Device [1234:1111] (rev 02) (prog-if 00 [VGA controller]) 01:00.0 VGA compatible controller : NVIDIA Corporation GM107GL [Quadro K620] [10de:13bb] (rev a2) (prog-if 00 [VGA controller])
Download NVIDIA Driver
sudo apt-get install dkms linux-headers-$(uname -r)
Now we need to install the GPU Driver, and this is where things diverge from the instructions on 3os.org.
Go to https://www.nvidia.com/Download/index.aspx?lang=en-us and search for the driver you need:
Your search will come up with a file you can download to your Plex VM (right-click the download link to copy the download URL)
It doesn’t matter where you save it, since you can delete it once you’re done installing the driver.
Install NVIDIA Driver
You will need to
chmod +x the file so you can execute it. Then execute the file .
You might get a warning, which you can acknowledge. It didn’t seem to cause an issue for me.
I chose to install the 32-bit compatibility library. I’m not sure if it’s actually needed, but it didn’t seem to cause any problems.
And we’re done!
Verify Driver Installation
You should be able to verify that the system recognizes the GPU by running
If you’ve not done so already, you’ll want to enable hardware acceleration for Plex under Settings > Transcoder:
Select “Save Changes”.
To verify that Plex is actually offloading the transcoding, start playing a TV or movie from your Plex library (make sure that the Quality is any setting except Original). Then you can run nvidia-smi again and you’ll see that it lists Plex Transcoder under the process name (instead of “No running processes found”).
You can also install nvtop (
apt-get install nvtop) which is a nice way to view transcoding efforts over time:
Debian vs Ubuntu
There’s a lot of guides and questions where Ubuntu is the OS. I’m not sure why, but none of those work for Debian when it comes to NVIDIA drivers. The NVIDIA Driver Installation Quickstart Guide (see below list of links) lists Ubuntu as supported, but not Debian. I’m not sure if that’s just because there’s no Debian package manager support, or if Debian is technically not supported by NVIDIA at all.
Removing the driver and upgrading
You can upgrade another .run file on top of an existing NVIDIA driver installation (as far as I can tell). You’ll need to run:
…to uninstall the existing driver. You’ll need to reboot the VM and then you can install the new driver.
Other links that probably aren’t useful, but which I have stumbled on as part of my research. Sometimes these are things that don’t work (or don’t work for what I was wanting to do). Regardless, they are worth mentioning:
- NVIDIA Driver Installation Quickstart Guide: https://docs.nvidia.com/datacenter/tesla/tesla-installation-notes/index.html
- Using FFmpeg with NVIDIA GPU Hardware Acceleration: https://docs.nvidia.com/video-technologies/video-codec-sdk/ffmpeg-with-nvidia-gpu/
- NVIDIA FFmpeg Transcoding Guide: https://developer.nvidia.com/blog/nvidia-ffmpeg-transcoding-guide/
- Plex in LXC on Proxmox, with Nvidia GPU passthrough: https://blog.twan.ovh/homelab/plex-lxc-proxmox-gpu-passthrough/
- Proxmox | GPU Passthrough to Ubuntu VM (Plex): https://lab.evishome.com/proxmox-gpu-passthrough-to-plex/
- How do I Set Up a NVIDIA Mobile GPU with Headless Debian 10?: https://www.reddit.com/r/debian/comments/jd2wvh/how_do_i_set_up_a_nvidia_mobile_gpu_with_headless/
- How to Install Nvidia Drivers on Debian: https://phoenixnap.com/kb/nvidia-drivers-debian
- How to install Nvidia drivers on Debian 11 (Bullseye) guide???: https://forums.debian.net/viewtopic.php?t=150825
How do you typeset NVIDIA?
I thought it was supposed to be nVidia, but wikipedia says Nvidia with a footnote saying it’s “Officially written as NVIDIA.” Or maybe it’s supposed to be nVIDIA?2