Plex Proxmox VM with NVIDIA GPU passthrough

Contents

Editors note: Last updated 12/26/2022

I’ve seen various parts of this documented on the internet, but I don’t think I’ve seen all the steps written down in one place, so in the interest of sharing and not banging my head next time I need to re-create my Plex VM: here’s how I was able to get my NVIDIA Quadro K620 GPU to work with my Plex VM running in Proxmox.

Here’s my setup:

  • Proxmox (7.2-4 No-Subscription Repository) using the Linux 5.15.35-2-pve kernel
  • Plex VM on Debian 10 using the Linux 4.19.0-20-amd64 kernel
  • NVIDIA Quadro K620 GPU. Note: I’m only using this in headless mode for Plex transcoding.

The first part of the steps are based on https://3os.org/infrastructure/proxmox/gpu-passthrough/pgu-passthrough-to-vm/#proxmox-configuration-for-gpu-passthrough. However, installing the Debian driver didn’t work. Maybe because I’m using Debian 10 and perhaps the NVIDIA v470 driver has a bug? I think there are some repo differences between Ubuntu and Debian.

Proxmox Configuration for GPU Passthrough

Find GPU Bus Address and Device ID

Find the PCI address of the GPU Device:

lspci -nnv | grep VGA

We get:

0b:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. G200eR2 [102b:0534] (prog-if 00 [VGA controller])
42:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107GL [Quadro K620] [10de:13bb] (rev a2) (prog-if 00 [VGA controller]

What we are looking is the PCI address of the NVIDIA GPU device. In this case it’s 42:00.0.
42:00.0 is only a part of of a group of PCI devices on the GPU.
We can list all the devices in the group 42:00 by using the following command:

lspci -s 42:00

The usual output will include VGA Device and Audio Device, which is what I have:

42:00.0 VGA compatible controller: NVIDIA Corporation GM107GL [Quadro K620] (rev a2)
42:00.1 Audio device: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] (rev a1)

Now we need to get the device ID’s of those devices. We can do this by using the following command:

lspci -s 42:00 -n

The output should look similar to this:

42:00.0 0300: 10de:13bb (rev a2)
42:00.1 0403: 10de:0fbc (rev a1)

What we are looking are the pairs, we will use those id to split the PCI Group to separate devices: 10de:13bb,10de:0fbc

Edit grub

Edit the grub configuration file at /etc/default/grub

Find the line that starts with GRUB_CMDLINE_LINUX_DEFAULT and updated it to look like this (Intel CPU example) — replace vfio-pci.ids= with the ids for the GPU you want to passthrough:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction video=efifb:off video=vesa:off vfio-pci.ids=10de:13bb,10de:0fb vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu"

Note: if you have an AMD CPU, then use amd_iommu=on instead of intel_iommu=on.

Save the config changed and then update GRUB.

update-grub

Update modules

Next we need to add vfio modules to allow PCI passthrough. Edit the /etc/modules file and add the following line to the end of the file:

# Modules required for PCI passthrough
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Update configuration changes made in your /etc filesystem

update-initramfs -u -k all

Reboot and verify

Reboot Proxmox to apply the changes

Verify that IOMMU is enabled

dmesg | grep -e DMAR -e IOMMU

There should be a line that looks like DMAR: IOMMU enabled. If there is no output, something is wrong.

Your Proxmox host should be ready to GPU passthrough!

Debian 10 VM GPU Passthrough Configuration

VM Setup

I’m using Debian 10 as my VM.

For best performance the VM should be configured the Machine type to q35. This will allow the VM to utilize PCI-Express passthrough.

Add a PCI device to the VM (Add > PCI Device). Select the GPU Device, which you can find using it’s PCI address from before (42:00, in this example). Note: This list uses a different format for the PCI addresses id, 42:00.0 is listed as 0000:42:00.0

Select:

  • All Functions
  • ROM-Bar
  • PCI-Epress

…and then select Add. Note: Do not select Primary GPU.

Boot the VM. To test the GPU passthrough was successful, you can use the following command in the VM: 

 sudo lspci -nnv | grep VGA

The output should include the GPU:

00:01.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02) (prog-if 00 [VGA controller])
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107GL [Quadro K620] [10de:13bb] (rev a2) (prog-if 00 [VGA controller])

Download NVIDIA Driver

Before we install the NVIDIA driver, we need to install the linux header files and dkms so when the linux kernel is updated the NVIDIA driver will automatically recompile

sudo apt-get install dkms linux-headers-$(uname -r)

Now we need to install the GPU Driver, and this is where things diverge from the instructions on 3os.org.

Go to https://www.nvidia.com/Download/index.aspx?lang=en-us and search for the driver you need:

Your search will come up with a file you can download to your Plex VM (right-click the download link to copy the download URL)

I used the following file: https://www.nvidia.com/content/DriverDownload-March2009/confirmation.php?url=/XFree86/Linux-x86_64/515.48.07/NVIDIA-Linux-x86_64-515.48.07.run&lang=us&type=TITAN

It doesn’t matter where you save it, since you can delete it once you’re done installing the driver.

Install NVIDIA Driver

You will need to chmod +x the file so you can execute it. Then execute the file with the DKMS flag (e.g. sudo sh ./NVIDIA-Linux-x86_64-515.48.07.run --dkms).

Building the kernel modules

You might get a warning, which you can acknowledge. It didn’t seem to cause an issue for me.

Warning about guessing X library path. Select OK to dismiss. We’re using this headless, so don’t care about x.org stuff.

I chose to install the 32-bit compatibility library. I’m not sure if it’s actually needed, but it didn’t seem to cause any problems.

Install NVIDIA’s 32-bit compatibility libraries? Select Yes
Asking to run nvidia-xconfig utility. Select No. We’re using this headless, so don’t care about x.org stuff.

And we’re done!

NVIDIA driver installation complete!

Verify Driver Installation

You should be able to verify that the system recognizes the GPU by running nvidia-smi

Output of nvidia-smi when no GPU processes are running

Configure Plex

If you’ve not done so already, you’ll want to enable hardware acceleration for Plex under Settings > Transcoder:

Select “Save Changes”.

To verify that Plex is actually offloading the transcoding, start playing a TV or movie from your Plex library (make sure that the Quality is any setting except Original). Then you can run nvidia-smi again and you’ll see that it lists Plex Transcoder under the process name (instead of “No running processes found”).

Output of nvidia-smi when Plex is transcoding

You can also install nvtop (apt-get install nvtop) which is a nice way to view transcoding efforts over time:

Using nvtop

Other Notes

Debian vs Ubuntu

There’s a lot of guides and questions where Ubuntu is the OS. I’m not sure why, but none of those work for Debian when it comes to NVIDIA drivers. The NVIDIA Driver Installation Quickstart Guide (see below list of links) lists Ubuntu as supported, but not Debian. I’m not sure if that’s just because there’s no Debian package manager support, or if Debian is technically not supported by NVIDIA at all.

Removing the driver and upgrading

You can upgrade another .run file on top of an existing NVIDIA driver installation (as far as I can tell). You’ll need to run:

/usr/bin/nvidia-uninstall

…to uninstall the existing driver. You’ll need to reboot the VM and then you can install the new driver.

Somewhat related links

Other links that probably aren’t useful, but which I have stumbled on as part of my research. Sometimes these are things that don’t work (or don’t work for what I was wanting to do). Regardless, they are worth mentioning:

How do you typeset NVIDIA?

I thought it was supposed to be nVidia, but wikipedia says Nvidia with a footnote saying it’s “Officially written as NVIDIA.” Or maybe it’s supposed to be nVIDIA?

11
Tags: