From 0d13581896c64a0f9600c65f599956cc4ef638ba Mon Sep 17 00:00:00 2001 From: Emma Thorpe Date: Mon, 17 Aug 2026 20:35:29 +0100 Subject: [PATCH 1/3] feat(features): gate Claude Code on the host CPU microarchitecture level Claude Code runs on Node, whose V8 build requires SSE4.2 and POPCNT (x86-64-v2). On an older x86_64 CPU it does not run, so it must not be installed there in the first place. Nix cannot detect the CPU (pure evaluation, hosts often built elsewhere), so add features.cpu.microarchLevel: the psABI level a host declares about itself, defaulting to 2. features.claudeCode.enable derives from it, and home/claude.nix reads that through home-manager's osConfig and installs nothing -- CLI, CLAUDE.md, output style or memory symlink -- when it is off. Hosts without the option (Darwin, the standalone homeConfigurations) keep the tool enabled. An assertion fails evaluation if a host force-enables the flag below the required level, so the mistake surfaces in nix flake check rather than as an illegal-instruction crash on the machine. --- README.md | 45 ++++++++++++++++++------- home/README.md | 12 +++++-- home/claude.nix | 26 ++++++++++++--- hosts/MacPro31/nvidia.nix | 43 ++++++++++++++++++++++++ modules/features.nix | 69 +++++++++++++++++++++++++++++++++++++-- 5 files changed, 175 insertions(+), 20 deletions(-) create mode 100644 hosts/MacPro31/nvidia.nix diff --git a/README.md b/README.md index 48a3146..d2f188a 100644 --- a/README.md +++ b/README.md @@ -55,17 +55,17 @@ module reaches a host: **baseModules** (every NixOS host, via `flake.nix`), **host table** (listed explicitly per host in `flake.nix`), or **transitively** (pulled in by another module's `imports`). -| Module | Imported by | What it does / when to use it | -| ------------------ | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `common-nixos.nix` | baseModules (all NixOS) | Timezone/locale, store hygiene (auto-optimise, big download buffer, **no** auto-GC), the nix-community binary cache, `nix-ld`, base CLI (`git`, `fastfetch`), and the fleet-wide font stack. | -| `users.nix` | baseModules (all NixOS) | Builds `users.users` from the registry for the host's `hostUsers`; enables zsh; enables Firefox + Thunderbird **only** when `features.swayDesktop.enable` is on. Applies per-user `linger`. | -| `features.nix` | baseModules (all NixOS) | Declares feature-flag options (currently `features.swayDesktop.enable`) so any host can read/set them without importing the heavy implementation module. | -| `workstation.nix` | transitively (via laptop/desktop) | Form-factor-agnostic base for physical graphical hosts: turns on `swayDesktop`, Dvorak console, PipeWire, firewall (default-deny), fstrim, earlyoom, fwupd, thermald (x86), redistributable fw. | -| `laptop.nix` | host table (MBP, T400) | `imports` workstation.nix, then adds the portable bits: iwd Wi-Fi, lid suspend/lock, Bluetooth + blueman. | -| `desktop.nix` | host table (Mac Pro) | `imports` workstation.nix, then swaps Wi-Fi for wired NetworkManager. Pair with `portable = false` in the host table. | -| `sway.nix` | host table (graphical hosts) | Implementation of `features.swayDesktop`: the system Sway package, the greetd/ReGreet (cage) greeter forced to Dvorak, xdg-portal, Wayland utility packages. Home-side Sway config is in `home/sway.nix`. | -| `ssh.nix` | host table (T400, Mac Pro, RPi5) | Enables sshd, opens port 22, enforces a key-only policy (no password / keyboard-interactive, no root). Authorized keys come from the registry via `users.nix`. | -| `firmware/` | referenced by MBP host config | Committed Apple peripheral firmware blobs for the Asahi MBP (see "MacBook (Asahi) firmware"). | +| Module | Imported by | What it does / when to use it | +| ------------------ | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `common-nixos.nix` | baseModules (all NixOS) | Timezone/locale, store hygiene (auto-optimise, big download buffer, **no** auto-GC), the nix-community binary cache, `nix-ld`, base CLI (`git`, `fastfetch`), and the fleet-wide font stack. | +| `users.nix` | baseModules (all NixOS) | Builds `users.users` from the registry for the host's `hostUsers`; enables zsh; enables Firefox + Thunderbird **only** when `features.swayDesktop.enable` is on. Applies per-user `linger`. | +| `features.nix` | baseModules (all NixOS) | Declares the feature-flag options (`features.swayDesktop.enable`, `features.claudeCode.enable`) so any host can read/set them without importing the heavy implementation module, plus the CPU capability fact they derive from (`features.cpu.microarchLevel`) and the assertion that guards it. See "CPU capability gating". | +| `workstation.nix` | transitively (via laptop/desktop) | Form-factor-agnostic base for physical graphical hosts: turns on `swayDesktop`, Dvorak console, PipeWire, firewall (default-deny), fstrim, earlyoom, fwupd, thermald (x86), redistributable fw. | +| `laptop.nix` | host table (MBP, T400) | `imports` workstation.nix, then adds the portable bits: iwd Wi-Fi, lid suspend/lock, Bluetooth + blueman. | +| `desktop.nix` | host table (Mac Pro) | `imports` workstation.nix, then swaps Wi-Fi for wired NetworkManager. Pair with `portable = false` in the host table. | +| `sway.nix` | host table (graphical hosts) | Implementation of `features.swayDesktop`: the system Sway package, the greetd/ReGreet (cage) greeter forced to Dvorak, xdg-portal, Wayland utility packages. Home-side Sway config is in `home/sway.nix`. | +| `ssh.nix` | host table (T400, Mac Pro, RPi5) | Enables sshd, opens port 22, enforces a key-only policy (no password / keyboard-interactive, no root). Authorized keys come from the registry via `users.nix`. | +| `firmware/` | referenced by MBP host config | Committed Apple peripheral firmware blobs for the Asahi MBP (see "MacBook (Asahi) firmware"). | Form-factor decision: a **laptop** imports `laptop.nix` (default `portable = true`); a **wired desktop** imports `desktop.nix` and sets @@ -74,6 +74,29 @@ Form-factor decision: a **laptop** imports `laptop.nix` (default serves. `portable` is threaded through to `home/sway.nix`, which drops the battery block and brightness keys on desktops. +## CPU capability gating + +Not every host can run everything the fleet installs. Nix cannot probe the CPU +(evaluation is pure, and a host may be built elsewhere), so each machine +declares what it is and the shared modules derive from that: + +- `features.cpu.microarchLevel` — the x86-64 psABI level the CPU implements + (1 = baseline, 2 = SSE4.2/POPCNT, 3 = AVX2, 4 = AVX-512). Defaults to **2**; + only a host older than that sets it (the Mac Pro 3,1's 2008 Harpertown Xeons + are level 1). Ignored on non-x86_64 hosts. +- `features.claudeCode.enable` — derived: on unless the host is below + x86-64-v2, because Claude Code's Node runtime needs SSE4.2/POPCNT. + [`home/claude.nix`](./home/claude.nix) reads it through home-manager's + `osConfig` and installs nothing (CLI, `CLAUDE.md`, output style, memory + symlink) when it is off. Hosts with no such option — the Darwin host and the + standalone `homeConfigurations` — fall back to enabled. +- An assertion in `features.nix` fails evaluation if a host force-enables a + flag its declared CPU level cannot support, so the mistake surfaces in + `nix flake check`/CI rather than as an illegal-instruction crash on the box. + +Adding another CPU-sensitive tool means deriving one more flag there, not +editing every host. + ## Users Identity is data, kept separate from the reusable modules: diff --git a/home/README.md b/home/README.md index 173f1d9..2c27cfd 100644 --- a/home/README.md +++ b/home/README.md @@ -175,8 +175,16 @@ forced off there) but still runs the agent. ## Claude Code -Managed declaratively by [`claude.nix`](./claude.nix) on every host (the CLI is -`pkgs.claude-code`, tracked to unstable via the flake overlay). +Managed declaratively by [`claude.nix`](./claude.nix) on every host whose CPU +can run it (the CLI is `pkgs.claude-code`, tracked to unstable via the flake +overlay). + +**Capability gate.** The module installs nothing — CLI or files — when +`osConfig.features.claudeCode.enable` is off. That flag is derived fleet-wide +from the host's declared CPU level (see "CPU capability gating" in the root +README): the Node runtime needs SSE4.2/POPCNT, so anything below x86-64-v2 (the +Mac Pro 3,1) is excluded. Hosts that do not define the option — the Darwin host +and the standalone `homeConfigurations` — keep it enabled. | Managed (static, from Nix) | Left mutable (runtime state) | | --------------------------------------------------- | ------------------------------------------------------ | diff --git a/home/claude.nix b/home/claude.nix index 3bf3d0b..2e5fdbe 100644 --- a/home/claude.nix +++ b/home/claude.nix @@ -1,4 +1,5 @@ -# Claude Code, configured declaratively via home-manager. Wanted on every host. +# Claude Code, configured declaratively via home-manager. Wanted on every host +# whose CPU can run it -- see the gate below. # # The STATIC config is managed here: the global CLAUDE.md (persona/context), the # custom output style, and the auto-memory directory. settings.json is @@ -10,18 +11,33 @@ # read-only into ~/.claude/memory, so the runtime "save a memory" path no longer # writes there -- recall still works, but new/changed memories must be added to # this repo and rebuilt. CLAUDE.md instructs Claude to do exactly that. -{ ... }: +{ + lib, + # Set by the NixOS/Darwin home-manager module; absent for the standalone + # homeConfigurations, hence the default. + osConfig ? { }, + ... +}: +let + # Capability gate, declared once for the whole fleet in modules/features.nix + # (default: on; off on CPUs below x86-64-v2, which cannot run the Node + # runtime Claude Code ships on). Hosts without that option -- the Darwin host + # and the portable standalone profile -- fall back to enabled. + enable = osConfig.features.claudeCode.enable or true; +in { programs.claude-code = { - enable = true; + inherit enable; # package defaults to pkgs.claude-code (tracked to unstable via the flake - # overlay); installs the CLI on every host. + # overlay). # ~/.claude/CLAUDE.md -- global instructions / persona / memory workflow. context = ./claude/CLAUDE.md; }; - home.file = { + # Nothing to place when the CLI is not installed: a ~/.claude/memory symlink + # with no Claude Code to read it is just dead state. + home.file = lib.mkIf enable { # Custom output style. The module has no option for output-styles/, so place # it directly; selection (settings.json `outputStyle`) stays mutable. ".claude/output-styles/soviet-engineer.md".source = ./claude/output-styles/soviet-engineer.md; diff --git a/hosts/MacPro31/nvidia.nix b/hosts/MacPro31/nvidia.nix new file mode 100644 index 0000000..d709492 --- /dev/null +++ b/hosts/MacPro31/nvidia.nix @@ -0,0 +1,43 @@ +# NVIDIA Quadro P400 (Pascal, GP108) on the Mac Pro 3,1: proprietary driver for +# the Sway desktop, plus Docker with GPU/CUDA access for containers. +# +# Driver branch: 580 (nvidiaPackages.legacy_580), NOT the nixpkgs default +# (`production`, currently 595.x). 580 is the last branch that supports +# Maxwell/Pascal/Volta -- NVIDIA keeps it as an LTS branch to Aug 2028 -- and a +# newer branch simply will not drive this card. +# +# The driver is unfree, so it is not in the binary cache: the kernel module is +# compiled locally. On this machine's 2008 Xeons expect the first rebuild after +# a kernel bump to take a long while. +{ config, ... }: + +{ + # Selects the proprietary driver; the module blacklists nouveau/nvidiafb and + # loads nvidia-uvm (needed by CUDA) via modprobe softdep. Naming is historical + # -- this option drives the kernel/driver choice on Wayland hosts too, which + # is why it is set on a machine that runs no X server. + services.xserver.videoDrivers = [ "nvidia" ]; + + hardware.nvidia = { + package = config.boot.kernelPackages.nvidiaPackages.legacy_580; + # Required for Wayland: sets nvidia-drm.modeset=1 (and fbdev=1), without + # which wlroots gets no GBM device and Sway/cage fail to start. + modesetting.enable = true; + # The open kernel modules need Turing or later; Pascal must use the closed + # ones. Explicit because the option has no default on driver >= 560. + open = false; + }; + + # wlroots refuses the proprietary NVIDIA driver unless told to proceed. The + # greeter's compositor (cage) has no such check; only Sway needs the flag, + # which the module bakes into the wrapper the session's .desktop file runs. + programs.sway.extraOptions = [ "--unsupported-gpu" ]; + + virtualisation.docker.enable = true; + + # CDI-based GPU access for containers: generates /var/run/cdi specs from the + # host driver at boot and turns on Docker's CDI feature. Run GPU workloads + # with `docker run --device=nvidia.com/gpu=all ...`. The deprecated + # virtualisation.docker.enableNvidia runtime wrapper is deliberately not used. + hardware.nvidia-container-toolkit.enable = true; +} diff --git a/modules/features.nix b/modules/features.nix index 0d0de92..c34b2f6 100644 --- a/modules/features.nix +++ b/modules/features.nix @@ -6,7 +6,72 @@ # headless host (e.g. the Pi) must be able to leave it at its default without # pulling in modules/sway.nix. The implementation lives in modules/sway.nix, # gated on this flag. -{ lib, ... }: +# +# The file also carries the host capability facts those flags derive from +# (features.cpu.*). features.claudeCode.enable is such a derived flag: it is +# computed from the declared CPU level here and read by home/claude.nix through +# home-manager's osConfig, so a machine that cannot run the tool never installs +# it, on any host, without per-host opt-outs. { - options.features.swayDesktop.enable = lib.mkEnableOption "the Sway desktop"; + config, + lib, + pkgs, + ... +}: +let + cfg = config.features; + + # Claude Code runs on Node, whose V8 build requires SSE4.2 and POPCNT -- the + # x86-64-v2 feature set. On an older x86_64 CPU it does not run (illegal + # instruction), so it must not be installed there. + claudeCodeMinLevel = 2; + claudeCodeSupported = + !pkgs.stdenv.hostPlatform.isx86_64 || cfg.cpu.microarchLevel >= claudeCodeMinLevel; +in +{ + options.features = { + swayDesktop.enable = lib.mkEnableOption "the Sway desktop"; + + cpu.microarchLevel = lib.mkOption { + type = lib.types.ints.between 1 4; + default = 2; + example = 1; + description = '' + The x86-64 psABI microarchitecture level the host CPU implements: + 1 = the original baseline, 2 = SSE4.2/POPCNT (Nehalem, 2008+), + 3 = AVX2, 4 = AVX-512. + + Nix cannot detect this (evaluation is pure and hosts are often built + elsewhere), so a machine older than the default declares its own level + and the flags below derive from it. Ignored on non-x86_64 hosts. + ''; + }; + + claudeCode.enable = lib.mkOption { + type = lib.types.bool; + default = claudeCodeSupported; + defaultText = lib.literalMD '' + `true`, unless the host declares an x86-64 microarchitecture level + below ${toString claudeCodeMinLevel} + ''; + description = '' + Whether to install Claude Code in this host's home-manager profiles + (implemented in home/claude.nix). Defaults off on CPUs below + x86-64-v${toString claudeCodeMinLevel}, which cannot run it; forcing it + on such a host is an evaluation error. + ''; + }; + }; + + config.assertions = [ + { + assertion = cfg.claudeCode.enable -> claudeCodeSupported; + message = '' + features.claudeCode.enable is on, but this host declares + features.cpu.microarchLevel = ${toString cfg.cpu.microarchLevel}. + Claude Code needs x86-64-v${toString claudeCodeMinLevel} + (SSE4.2/POPCNT) and will not run on an older CPU. + ''; + } + ]; } -- 2.54.0 From 0f7fb7f78a530f052e9874345abfc05819383f84 Mon Sep 17 00:00:00 2001 From: Emma Thorpe Date: Mon, 17 Aug 2026 20:35:39 +0100 Subject: [PATCH 2/3] feat(macpro31): NVIDIA Quadro P400 driver and CUDA-enabled Docker The stock GPU has been replaced with a Quadro P400 (Pascal, GP108). Add hosts/MacPro31/nvidia.nix: - Driver branch 580 (nvidiaPackages.legacy_580), not the nixpkgs default production branch (595.x). 580 is the last branch supporting Maxwell/Pascal/Volta and is an LTS branch until Aug 2028; a newer one does not drive this card. - modesetting.enable for Wayland (nvidia-drm.modeset=1), open = false (the open kernel modules need Turing or later), and sway --unsupported-gpu, which wlroots requires with the proprietary driver. - Docker with GPU access via CDI (hardware.nvidia-container-toolkit), rather than the deprecated virtualisation.docker.enableNvidia runtime wrapper. Containers run with --device=nvidia.com/gpu=all and must ship a CUDA 12.x or older runtime: CUDA 13 dropped sm_61. The driver packages are unfree, so allowlist them in unfreePackages; they are not cached and the kernel module builds on the host. Also declare features.cpu.microarchLevel = 1 for this machine: the Harpertown Xeons have SSE4.1 but no SSE4.2/POPCNT, which switches off Claude Code through the fleet-wide gate. --- flake.nix | 6 +++ hosts/MacPro31/README.md | 73 +++++++++++++++++++++++++++----- hosts/MacPro31/configuration.nix | 20 ++++----- 3 files changed, 77 insertions(+), 22 deletions(-) diff --git a/flake.nix b/flake.nix index d982c83..d8109f4 100644 --- a/flake.nix +++ b/flake.nix @@ -111,8 +111,14 @@ ]; # Unfree packages permitted to be built (replaces blanket allowUnfree). + # The NVIDIA entries are for the Mac Pro's Quadro P400 (hosts/MacPro31/ + # nvidia.nix); unfree packages are not in the binary cache, so the + # kernel module is compiled on the host. unfreePackages = [ "claude-code" + "nvidia-x11" + "nvidia-kernel-modules" + "nvidia-settings" ]; # Per-user identity, keyed by username. See README "Users". diff --git a/hosts/MacPro31/README.md b/hosts/MacPro31/README.md index 8b5ce00..dfc8a6c 100644 --- a/hosts/MacPro31/README.md +++ b/hosts/MacPro31/README.md @@ -1,7 +1,7 @@ # Mac Pro 3,1 (Early 2008) — install notes Flake host: `lyrathorpe-macpro31`. Desktop (`portable = false`, imports -`../../modules/desktop.nix`). Files: `configuration.nix`, +`../../modules/desktop.nix`). Files: `configuration.nix`, `nvidia.nix`, `hardware-configuration.nix`. ## Hardware configuration @@ -27,18 +27,69 @@ either Partition the disk GPT with an ESP (vfat). -## Graphics +## Graphics — NVIDIA Quadro P400 -The stock card varies between units — **ATI Radeon HD 2600 XT** or **NVIDIA -GeForce 8800 GT**. No proprietary driver is hardcoded; Sway relies on in-tree KMS: +The stock card (**ATI Radeon HD 2600 XT** or **NVIDIA GeForce 8800 GT**, +depending on the unit) has been replaced with an **NVIDIA Quadro P400** (Pascal, +GP108). Everything driver-related lives in [`nvidia.nix`](./nvidia.nix): -- ATI Radeon HD 2600 XT → `radeon` (or `amdgpu`) KMS -- NVIDIA GeForce 8800 GT → `nouveau` KMS +- **Driver branch 580** (`nvidiaPackages.legacy_580`), _not_ the nixpkgs default + (`production`, currently 595.x). 580 is the last branch that supports + Maxwell/Pascal/Volta and is maintained as an LTS branch until Aug 2028; a + newer branch does not drive this card at all. +- `modesetting.enable = true` — mandatory for Wayland (sets + `nvidia-drm.modeset=1`); without it wlroots gets no GBM device and both Sway + and the greeter fail to start. +- `open = false` — the open kernel modules require Turing or later. +- Sway runs with `--unsupported-gpu` (`programs.sway.extraOptions`); wlroots + refuses the proprietary driver otherwise. `cage`/ReGreet needs no such flag. +- nouveau and `nvidiafb` are blacklisted automatically by the NVIDIA module. -These come up automatically. If a card needs forcing, set -`services.xserver.videoDrivers` and/or add the module to -`boot.initrd.kernelModules` for early KMS (see the comment in -`configuration.nix`). +The driver is unfree, so it is **not in the binary cache**: the kernel module is +compiled on the machine, which on these 2008 Xeons is slow — budget for a long +first rebuild and again after every kernel bump. The package names are +allowlisted in `unfreePackages` in [`flake.nix`](../../flake.nix). + +Note the Mac Pro shows no EFI boot screen with a stock PC card (no Apple EFI +ROM): the machine boots blind until KMS brings the display up. That is expected, +not a fault. + +Verify after a rebuild: + +```sh +nvidia-smi +``` + +## Docker with CUDA + +`nvidia.nix` also enables Docker and gives containers GPU access via **CDI** +(`hardware.nvidia-container-toolkit.enable`), which generates device specs from +the host driver at boot (regenerated by a udev rule when the `nvidia` device +appears) and turns on the daemon's CDI feature: + +```sh +docker run --rm --device=nvidia.com/gpu=all nvidia/cuda:12.9.1-base-ubuntu24.04 nvidia-smi +``` + +- Use the `--device=nvidia.com/gpu=all` form. `--gpus all` is the legacy + runtime-wrapper path (`virtualisation.docker.enableNvidia`), which is + deprecated upstream and deliberately not enabled here. +- **CUDA version matters.** The P400 is compute capability 6.1 (`sm_61`); CUDA + 13 dropped Maxwell/Pascal/Volta, so container images must ship a **CUDA 12.x + or older** runtime. The 580 driver itself is happy with either. +- 2 GB of VRAM, 256 CUDA cores — fine for encode/decode and small models, not + for training anything serious. +- Docker socket is local-only (no TCP listener, unlike the Pi). Users need the + `docker` group; the registry already grants it. + +## Claude Code — not installed here + +The dual Harpertown Xeons are **x86-64-v1** (SSE4.1, but no SSE4.2/POPCNT) and +the Node runtime Claude Code ships on requires x86-64-v2. `configuration.nix` +declares `features.cpu.microarchLevel = 1`, which switches the tool off through +the fleet-wide gate in [`../../modules/features.nix`](../../modules/features.nix) +— see the root README. Forcing `features.claudeCode.enable` on here is an +evaluation error, not a broken install. ## Networking @@ -52,7 +103,7 @@ Graphical login via a Wayland greeter — `greetd` running ReGreet inside the every Sway host (gated on `features.swayDesktop.enable`). The greeter is forced to the Dvorak layout to match the console and Sway session. Set the user password (`passwd lyrathorpe`) after install, or the greeter cannot -authenticate. Requires working KMS (radeon/nouveau — see Graphics). +authenticate. Requires working KMS (NVIDIA modesetting — see Graphics). ## Apply diff --git a/hosts/MacPro31/configuration.nix b/hosts/MacPro31/configuration.nix index 6faae6d..82eee35 100644 --- a/hosts/MacPro31/configuration.nix +++ b/hosts/MacPro31/configuration.nix @@ -7,8 +7,14 @@ { imports = [ ./hardware-configuration.nix + ./nvidia.nix ]; + # Dual quad-core Xeon (Harpertown/Penryn): SSE4.1 but no SSE4.2 or POPCNT, + # i.e. x86-64-v1. Declaring it here switches off the fleet flags that need a + # newer CPU -- currently features.claudeCode (see ../../modules/features.nix). + features.cpu.microarchLevel = 1; + # The Mac Pro 3,1 has 64-bit EFI (confirmed by the owner), so boot via # systemd-boot like the MBP -- no GRUB/BIOS shim needed. boot.loader.systemd-boot.enable = true; @@ -33,17 +39,9 @@ # enabled in workstation.nix. hardware.cpu.intel.updateMicrocode = true; - # GPU note: the stock card varies between units -- ATI Radeon HD 2600 XT or - # NVIDIA GeForce 8800 GT. Sway needs a working KMS/modesetting driver; do NOT - # install a proprietary blob here. Depending on the installed card, rely on - # the open kernel driver: - # - ATI Radeon HD 2600 XT -> "radeon" (older) or "amdgpu" KMS - # - NVIDIA GeForce 8800 GT -> "nouveau" KMS - # These come up automatically via the in-tree drivers + KMS, and the graphics - # stack itself is enabled by modules/sway.nix. If a card needs to be forced, add it - # here, e.g. `services.xserver.videoDrivers = [ "radeon" ];` (or "nouveau"), - # and/or `boot.initrd.kernelModules = [ "radeon" ];` in - # hardware-configuration.nix for early KMS. + # GPU: the stock card (ATI Radeon HD 2600 XT / NVIDIA GeForce 8800 GT) has + # been replaced with an NVIDIA Quadro P400. Driver, Wayland quirks and + # GPU-enabled Docker live in ./nvidia.nix. # See `man configuration.nix` / the stateVersion docs before changing. system.stateVersion = "26.05"; -- 2.54.0 From d4e7475db9fa03900f6a41cec985247776bc42bd Mon Sep 17 00:00:00 2001 From: Emma Thorpe Date: Mon, 17 Aug 2026 20:47:33 +0100 Subject: [PATCH 3/3] fix(macpro31): load the NVIDIA modules and guard the CDI generator The CDI generator aborted with "failed to initialize NVML: Driver Not Loaded", taking docker.service with it (requiredBy) and failing the switch. Two causes. The nixpkgs NVIDIA module only adds nvidia/nvidia_modeset/ nvidia_drm to boot.kernelModules when services.xserver.enable is set, which is false on this Wayland-only host, so load them explicitly. nvidia_uvm stays out: the module's modprobe softdep loads it once the GPU device exists. The generator also runs during activation, when a module rebuilt against a new kernel cannot be loaded until reboot -- a guaranteed failure after every kernel bump. Guard it with ConditionPathExists on /proc/driver/nvidia/version so it skips rather than fails; the toolkit's udev rule restarts it when the device appears, so the specs are generated on the next boot. --- hosts/MacPro31/README.md | 26 ++++++++++++++++++++++++++ hosts/MacPro31/nvidia.nix | 23 +++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/hosts/MacPro31/README.md b/hosts/MacPro31/README.md index dfc8a6c..f514f39 100644 --- a/hosts/MacPro31/README.md +++ b/hosts/MacPro31/README.md @@ -82,6 +82,32 @@ docker run --rm --device=nvidia.com/gpu=all nvidia/cuda:12.9.1-base-ubuntu24.04 - Docker socket is local-only (no TCP listener, unlike the Pi). Users need the `docker` group; the registry already grants it. +### "Driver Not Loaded" from the CDI generator + +`nvidia-container-toolkit-cdi-generator.service` fails with +`failed to initialize NVML: Driver Not Loaded` whenever the `nvidia` kernel +module is not loaded in the **running** kernel. After a kernel bump that is +unavoidable — the rebuilt module cannot load until reboot — so the unit is +guarded with `ConditionPathExists=/proc/driver/nvidia/version` and skips +instead of failing. Without that guard it also takes `docker.service` +(`requiredBy`) with it and makes `nixos-rebuild switch` exit non-zero. + +**Reboot after a rebuild that touches the driver or the kernel.** The toolkit's +udev rule restarts the generator when the GPU device appears, so the CDI specs +are written on the next boot. To check the state: + +```sh +lsmod | grep nvidia # nvidia, nvidia_modeset, nvidia_drm, nvidia_uvm +cat /proc/driver/nvidia/version +nvidia-smi +systemctl status nvidia-container-toolkit-cdi-generator.service +ls /var/run/cdi # the generated spec +``` + +If the module is genuinely absent after a reboot, check `dmesg | grep -i +nvidia` (build/version mismatch, or nouveau still bound — the module blacklists +it, so that should not happen). + ## Claude Code — not installed here The dual Harpertown Xeons are **x86-64-v1** (SSE4.1, but no SSE4.2/POPCNT) and diff --git a/hosts/MacPro31/nvidia.nix b/hosts/MacPro31/nvidia.nix index d709492..43bab95 100644 --- a/hosts/MacPro31/nvidia.nix +++ b/hosts/MacPro31/nvidia.nix @@ -28,6 +28,18 @@ open = false; }; + # The NVIDIA module only puts these in boot.kernelModules when + # services.xserver.enable is true, which is false on this Wayland-only host -- + # so load them explicitly rather than relying on udev modalias autoloading. + # nvidia_uvm (needed by CUDA) is deliberately absent: the module's modprobe + # softdep pulls it in after the GPU device exists, which is the supported + # ordering. + boot.kernelModules = [ + "nvidia" + "nvidia_modeset" + "nvidia_drm" + ]; + # wlroots refuses the proprietary NVIDIA driver unless told to proceed. The # greeter's compositor (cage) has no such check; only Sway needs the flag, # which the module bakes into the wrapper the session's .desktop file runs. @@ -40,4 +52,15 @@ # with `docker run --device=nvidia.com/gpu=all ...`. The deprecated # virtualisation.docker.enableNvidia runtime wrapper is deliberately not used. hardware.nvidia-container-toolkit.enable = true; + + # The generator needs a loaded kernel module: without one it aborts with + # "failed to initialize NVML: Driver Not Loaded". That is guaranteed after a + # kernel bump, where the rebuilt module cannot load until reboot -- and since + # the unit is requiredBy docker.service and wantedBy multi-user.target, the + # failure takes Docker down and makes `nixos-rebuild switch` exit non-zero. + # Skip the run instead when no driver is loaded; the toolkit's udev rule + # restarts the unit as soon as the nvidia device appears, so the CDI specs are + # still generated on the next boot. + systemd.services.nvidia-container-toolkit-cdi-generator.unitConfig.ConditionPathExists = + "/proc/driver/nvidia/version"; } -- 2.54.0