NVIDIA GPUs in Proxmox Containers

Tutorial
, , , ,
1

Getting an NVIDIA graphics card to pass through into an unprivileged Proxmox LXC container can be tricky due to missing device nodes, varying cgroup numbers, and unprivileged user namespace permissions.

This guide summarizes the final working solution, including a fix for a subtle boot-race condition that causes the GPU to randomly stop working inside the container even when everything looks correctly configured and working at first.

Step 1: Install the Driver on the Host

Install the official NVIDIA driver on your Proxmox host (download the latest production version from the NVIDIA Unix Drivers page). Make the installer executable with chmod +x and run it via ./.

Step 2: Install User-Space Utilities in the Container

To use nvidia-smi and talk to the host’s GPU driver without conflicting with its kernel modules, you must install the user-space portion inside your LXC container:

  1. Download the exact same NVIDIA driver version matching your host from the NVIDIA Unix Drivers page.
  2. Make the installer executable:
   chmod +x NVIDIA-Linux-x86_64-*.run
  1. Run the installer using the --no-kernel-module flag:
   ./NVIDIA-Linux-x86_64-*.run --no-kernel-module

This ensures it only installs nvidia-smi and user-space libraries, leaving the kernel module management entirely to the Proxmox host.

Step 3: Check Your Host Cgroup Numbers

Before editing your container, run this command on your Proxmox host to verify the major numbers for your specific devices:

ls -l /dev/nvidia* /dev/dri/* 2>/dev/null
cat /proc/devices | grep -i nvidia

Look at the number before the comma (e.g., 226, 507, 195). Ensure the major numbers in the next step match what your host outputs.

Note: these numbers are not always static. nvidia-uvm in particular is dynamically assigned by the kernel and can drift after a driver update or module reload (e.g. 507 today, 508 next time). If your container suddenly loses GPU access after a driver upgrade, re-check this output before anything else.

Step 4: Configure the Container (/etc/pve/lxc/100.conf)

Open your container configuration file on the host (nano /etc/pve/lxc/100.conf) and add the required cgroup device allowances along with the explicit bind mounts.

Using the optional, create=file, and mode=0666 flags ensures that:

  1. The container won’t crash on boot if the host driver isn’t fully initialized yet.
  2. Unprivileged containers can bypass permission locks by making the files globally readable/writable (0666) inside the container.

Add these lines to your container configuration file (replacing 100 with your actual container ID):

# Cgroup permissions (adjust major numbers if yours differ)
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 507:* rwm
lxc.cgroup2.devices.allow: c 195:* rwm
# Mount entries with mode=0666 override permissions inside unprivileged containers
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file,mode=0666
lxc.mount.entry: /dev/dri/renderD129 dev/dri/renderD129 none bind,optional,create=file,mode=0666

Step 5: Restart the Container

Apply the changes by performing a full stop and start cycle from the Proxmox host:

pct stop 100
pct start 100

Step 6: Verify Inside the Container

Jump into your container to confirm everything is communicating properly:

pct enter 100
nvidia-smi

At this point the GPU should work, but there’s a gotcha most guides miss.

Step 7: Fix the Boot-Time Race Condition (important!)

Here’s the trap: the NVIDIA kernel module is not loaded automatically at boot. It loads lazily, the first time something on the host touches the GPU (e.g. running nvidia-smi manually). Until that happens, /dev/nvidia* doesn’t exist on the host at all.

Because your lxc.mount.entry lines use optional, LXC silently skips the bind mounts if the device nodes aren’t there yet, no error, no crash, the container just boots without GPU access. If Proxmox autostarts your CT (onboot: 1) before anything has triggered the driver to load, the container gets stuck without the GPU for its entire runtime, even though the host will show a perfectly healthy nvidia-smi moments later. Only a full pct stop / pct start after the driver is loaded will fix it, a host reboot alone reproduces the same race every time.

The fix is to force the driver to load, and stay loaded, before Proxmox starts any guests.

7a. Install and enable nvidia-persistenced on the host

This keeps the GPU driver “warm” continuously, so the module never unloads and device nodes never disappear during normal operation.

If nvidia-persistenced isn’t already present (check with which nvidia-persistenced), the systemd unit usually isn’t installed by the .run installer either. Create it manually:

cat > /etc/systemd/system/nvidia-persistenced.service << 'EOF'
[Unit]
Description=NVIDIA Persistence Daemon
Wants=syslog.target
Before=pve-guests.service

[Service]
Type=forking
PIDFile=/var/run/nvidia-persistenced/nvidia-persistenced.pid
Restart=always
ExecStart=/usr/bin/nvidia-persistenced --verbose
ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced

[Install]
WantedBy=multi-user.target
EOF

Don’t try to install nvidia-persistenced via apt if your driver was installed via the official .run installer, Debian’s repo package (e.g. 550.163.01) will conflict with your actual driver version and fail with unsatisfiable dependency errors. If the binary is missing, re-run your driver installer with ./NVIDIA-Linux-x86_64-*.run --no-kernel-module on the host to repair the userspace tools without touching the working kernel module.

7b. Add a boot-order guard as a second layer of protection

Even with persistenced running, it’s worth guaranteeing the driver is confirmed live before Proxmox starts any guests, this protects against edge cases like a driver crash mid-uptime unloading the module.

cat > /etc/systemd/system/nvidia-boot.service << 'EOF'
[Unit]
Description=Load NVIDIA driver and create device nodes before LXC guests start
After=nvidia-persistenced.service
Before=pve-guests.service
DefaultDependencies=no

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

7c. Make Proxmox’s guest-startup service wait on both

Proxmox’s autostart path is pve-guests.service. Hook into it with a drop-in:

mkdir -p /etc/systemd/system/pve-guests.service.d
cat > /etc/systemd/system/pve-guests.service.d/nvidia.conf << 'EOF'
[Unit]
After=nvidia-persistenced.service nvidia-boot.service
Wants=nvidia-persistenced.service nvidia-boot.service
EOF

7d. Enable everything and verify

systemctl daemon-reload
systemctl enable --now nvidia-persistenced.service
systemctl enable nvidia-boot.service
systemctl list-dependencies pve-guests.service

Confirm both nvidia-persistenced.service and nvidia-boot.service show up as dependencies of pve-guests.service.

Step 8: Confirm with a Full Host Reboot

This is the real test, a plain pct restart will “work” even without the fix, since the driver is usually already loaded by then. Only a cold host reboot exercises the actual race condition:

reboot

After the host comes back up:

journalctl -u nvidia-boot.service -u pve-guests.service --boot
pct enter 100
nvidia-smi

If nvidia-smi works inside the container on the very first boot, no manual pct stop/start needed, the race condition is fixed.

Congratulations! You should now have a GPU passthrough setup that survives host reboots and driver reloads without manual intervention. Enjoy.