hosts/MacPro31/README.md

@@ -82,6 +82,32 @@ docker run --rm --device=nvidia.com/gpu=all nvidia/cuda:12.9.1-base-ubuntu24.04
 - Docker socket is local-only (no TCP listener, unlike the Pi). Users need the
   `docker` group; the registry already grants it.
 
+### "Driver Not Loaded" from the CDI generator
+
+`nvidia-container-toolkit-cdi-generator.service` fails with
+`failed to initialize NVML: Driver Not Loaded` whenever the `nvidia` kernel
+module is not loaded in the **running** kernel. After a kernel bump that is
+unavoidable — the rebuilt module cannot load until reboot — so the unit is
+guarded with `ConditionPathExists=/proc/driver/nvidia/version` and skips
+instead of failing. Without that guard it also takes `docker.service`
+(`requiredBy`) with it and makes `nixos-rebuild switch` exit non-zero.
+
+**Reboot after a rebuild that touches the driver or the kernel.** The toolkit's
+udev rule restarts the generator when the GPU device appears, so the CDI specs
+are written on the next boot. To check the state:
+
+```sh
+lsmod | grep nvidia            # nvidia, nvidia_modeset, nvidia_drm, nvidia_uvm
+cat /proc/driver/nvidia/version
+nvidia-smi
+systemctl status nvidia-container-toolkit-cdi-generator.service
+ls /var/run/cdi                # the generated spec
+```
+
+If the module is genuinely absent after a reboot, check `dmesg | grep -i
+nvidia` (build/version mismatch, or nouveau still bound — the module blacklists
+it, so that should not happen).
+
 ## Claude Code — not installed here
 
 The dual Harpertown Xeons are **x86-64-v1** (SSE4.1, but no SSE4.2/POPCNT) and

hosts/MacPro31/nvidia.nix

@@ -28,6 +28,18 @@
     open = false;
   };
 
+  # The NVIDIA module only puts these in boot.kernelModules when
+  # services.xserver.enable is true, which is false on this Wayland-only host --
+  # so load them explicitly rather than relying on udev modalias autoloading.
+  # nvidia_uvm (needed by CUDA) is deliberately absent: the module's modprobe
+  # softdep pulls it in after the GPU device exists, which is the supported
+  # ordering.
+  boot.kernelModules = [
+    "nvidia"
+    "nvidia_modeset"
+    "nvidia_drm"
+  ];
+
   # wlroots refuses the proprietary NVIDIA driver unless told to proceed. The
   # greeter's compositor (cage) has no such check; only Sway needs the flag,
   # which the module bakes into the wrapper the session's .desktop file runs.
@@ -40,4 +52,15 @@
   # with `docker run --device=nvidia.com/gpu=all ...`. The deprecated
   # virtualisation.docker.enableNvidia runtime wrapper is deliberately not used.
   hardware.nvidia-container-toolkit.enable = true;
+
+  # The generator needs a loaded kernel module: without one it aborts with
+  # "failed to initialize NVML: Driver Not Loaded". That is guaranteed after a
+  # kernel bump, where the rebuilt module cannot load until reboot -- and since
+  # the unit is requiredBy docker.service and wantedBy multi-user.target, the
+  # failure takes Docker down and makes `nixos-rebuild switch` exit non-zero.
+  # Skip the run instead when no driver is loaded; the toolkit's udev rule
+  # restarts the unit as soon as the nvidia device appears, so the CDI specs are
+  # still generated on the next boot.
+  systemd.services.nvidia-container-toolkit-cdi-generator.unitConfig.ConditionPathExists =
+    "/proc/driver/nvidia/version";
 }