Merge pull request 'feat(macpro31): NVIDIA P400 with CUDA Docker, and a fleet-wide CPU capability gate' (#94) from feat/macpro31-nvidia-cuda into main
CI / flake (push) Successful in 4m44s

Reviewed-on: #94
This commit was merged in pull request #94.
This commit is contained in:
2026-08-17 20:59:21 +01:00
8 changed files with 301 additions and 42 deletions
+34 -11
View File
@@ -55,17 +55,17 @@ module reaches a host: **baseModules** (every NixOS host, via `flake.nix`),
**host table** (listed explicitly per host in `flake.nix`), or **transitively**
(pulled in by another module's `imports`).
| Module | Imported by | What it does / when to use it |
| ------------------ | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `common-nixos.nix` | baseModules (all NixOS) | Timezone/locale, store hygiene (auto-optimise, big download buffer, **no** auto-GC), the nix-community binary cache, `nix-ld`, base CLI (`git`, `fastfetch`), and the fleet-wide font stack. |
| `users.nix` | baseModules (all NixOS) | Builds `users.users` from the registry for the host's `hostUsers`; enables zsh; enables Firefox + Thunderbird **only** when `features.swayDesktop.enable` is on. Applies per-user `linger`. |
| `features.nix` | baseModules (all NixOS) | Declares feature-flag options (currently `features.swayDesktop.enable`) so any host can read/set them without importing the heavy implementation module. |
| `workstation.nix` | transitively (via laptop/desktop) | Form-factor-agnostic base for physical graphical hosts: turns on `swayDesktop`, Dvorak console, PipeWire, firewall (default-deny), fstrim, earlyoom, fwupd, thermald (x86), redistributable fw. |
| `laptop.nix` | host table (MBP, T400) | `imports` workstation.nix, then adds the portable bits: iwd Wi-Fi, lid suspend/lock, Bluetooth + blueman. |
| `desktop.nix` | host table (Mac Pro) | `imports` workstation.nix, then swaps Wi-Fi for wired NetworkManager. Pair with `portable = false` in the host table. |
| `sway.nix` | host table (graphical hosts) | Implementation of `features.swayDesktop`: the system Sway package, the greetd/ReGreet (cage) greeter forced to Dvorak, xdg-portal, Wayland utility packages. Home-side Sway config is in `home/sway.nix`. |
| `ssh.nix` | host table (T400, Mac Pro, RPi5) | Enables sshd, opens port 22, enforces a key-only policy (no password / keyboard-interactive, no root). Authorized keys come from the registry via `users.nix`. |
| `firmware/` | referenced by MBP host config | Committed Apple peripheral firmware blobs for the Asahi MBP (see "MacBook (Asahi) firmware"). |
| Module | Imported by | What it does / when to use it |
| ------------------ | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `common-nixos.nix` | baseModules (all NixOS) | Timezone/locale, store hygiene (auto-optimise, big download buffer, **no** auto-GC), the nix-community binary cache, `nix-ld`, base CLI (`git`, `fastfetch`), and the fleet-wide font stack. |
| `users.nix` | baseModules (all NixOS) | Builds `users.users` from the registry for the host's `hostUsers`; enables zsh; enables Firefox + Thunderbird **only** when `features.swayDesktop.enable` is on. Applies per-user `linger`. |
| `features.nix` | baseModules (all NixOS) | Declares the feature-flag options (`features.swayDesktop.enable`, `features.claudeCode.enable`) so any host can read/set them without importing the heavy implementation module, plus the CPU capability fact they derive from (`features.cpu.microarchLevel`) and the assertion that guards it. See "CPU capability gating". |
| `workstation.nix` | transitively (via laptop/desktop) | Form-factor-agnostic base for physical graphical hosts: turns on `swayDesktop`, Dvorak console, PipeWire, firewall (default-deny), fstrim, earlyoom, fwupd, thermald (x86), redistributable fw. |
| `laptop.nix` | host table (MBP, T400) | `imports` workstation.nix, then adds the portable bits: iwd Wi-Fi, lid suspend/lock, Bluetooth + blueman. |
| `desktop.nix` | host table (Mac Pro) | `imports` workstation.nix, then swaps Wi-Fi for wired NetworkManager. Pair with `portable = false` in the host table. |
| `sway.nix` | host table (graphical hosts) | Implementation of `features.swayDesktop`: the system Sway package, the greetd/ReGreet (cage) greeter forced to Dvorak, xdg-portal, Wayland utility packages. Home-side Sway config is in `home/sway.nix`. |
| `ssh.nix` | host table (T400, Mac Pro, RPi5) | Enables sshd, opens port 22, enforces a key-only policy (no password / keyboard-interactive, no root). Authorized keys come from the registry via `users.nix`. |
| `firmware/` | referenced by MBP host config | Committed Apple peripheral firmware blobs for the Asahi MBP (see "MacBook (Asahi) firmware"). |
Form-factor decision: a **laptop** imports `laptop.nix` (default
`portable = true`); a **wired desktop** imports `desktop.nix` and sets
@@ -74,6 +74,29 @@ Form-factor decision: a **laptop** imports `laptop.nix` (default
serves. `portable` is threaded through to `home/sway.nix`, which drops the
battery block and brightness keys on desktops.
## CPU capability gating
Not every host can run everything the fleet installs. Nix cannot probe the CPU
(evaluation is pure, and a host may be built elsewhere), so each machine
declares what it is and the shared modules derive from that:
- `features.cpu.microarchLevel` — the x86-64 psABI level the CPU implements
(1 = baseline, 2 = SSE4.2/POPCNT, 3 = AVX2, 4 = AVX-512). Defaults to **2**;
only a host older than that sets it (the Mac Pro 3,1's 2008 Harpertown Xeons
are level 1). Ignored on non-x86_64 hosts.
- `features.claudeCode.enable` — derived: on unless the host is below
x86-64-v2, because Claude Code's Node runtime needs SSE4.2/POPCNT.
[`home/claude.nix`](./home/claude.nix) reads it through home-manager's
`osConfig` and installs nothing (CLI, `CLAUDE.md`, output style, memory
symlink) when it is off. Hosts with no such option — the Darwin host and the
standalone `homeConfigurations` — fall back to enabled.
- An assertion in `features.nix` fails evaluation if a host force-enables a
flag its declared CPU level cannot support, so the mistake surfaces in
`nix flake check`/CI rather than as an illegal-instruction crash on the box.
Adding another CPU-sensitive tool means deriving one more flag there, not
editing every host.
## Users
Identity is data, kept separate from the reusable modules:
+6
View File
@@ -111,8 +111,14 @@
];
# Unfree packages permitted to be built (replaces blanket allowUnfree).
# The NVIDIA entries are for the Mac Pro's Quadro P400 (hosts/MacPro31/
# nvidia.nix); unfree packages are not in the binary cache, so the
# kernel module is compiled on the host.
unfreePackages = [
"claude-code"
"nvidia-x11"
"nvidia-kernel-modules"
"nvidia-settings"
];
# Per-user identity, keyed by username. See README "Users".
+10 -2
View File
@@ -175,8 +175,16 @@ forced off there) but still runs the agent.
## Claude Code
Managed declaratively by [`claude.nix`](./claude.nix) on every host (the CLI is
`pkgs.claude-code`, tracked to unstable via the flake overlay).
Managed declaratively by [`claude.nix`](./claude.nix) on every host whose CPU
can run it (the CLI is `pkgs.claude-code`, tracked to unstable via the flake
overlay).
**Capability gate.** The module installs nothing — CLI or files — when
`osConfig.features.claudeCode.enable` is off. That flag is derived fleet-wide
from the host's declared CPU level (see "CPU capability gating" in the root
README): the Node runtime needs SSE4.2/POPCNT, so anything below x86-64-v2 (the
Mac Pro 3,1) is excluded. Hosts that do not define the option — the Darwin host
and the standalone `homeConfigurations` — keep it enabled.
| Managed (static, from Nix) | Left mutable (runtime state) |
| --------------------------------------------------- | ------------------------------------------------------ |
+21 -5
View File
@@ -1,4 +1,5 @@
# Claude Code, configured declaratively via home-manager. Wanted on every host.
# Claude Code, configured declaratively via home-manager. Wanted on every host
# whose CPU can run it -- see the gate below.
#
# The STATIC config is managed here: the global CLAUDE.md (persona/context), the
# custom output style, and the auto-memory directory. settings.json is
@@ -10,18 +11,33 @@
# read-only into ~/.claude/memory, so the runtime "save a memory" path no longer
# writes there -- recall still works, but new/changed memories must be added to
# this repo and rebuilt. CLAUDE.md instructs Claude to do exactly that.
{ ... }:
{
lib,
# Set by the NixOS/Darwin home-manager module; absent for the standalone
# homeConfigurations, hence the default.
osConfig ? { },
...
}:
let
# Capability gate, declared once for the whole fleet in modules/features.nix
# (default: on; off on CPUs below x86-64-v2, which cannot run the Node
# runtime Claude Code ships on). Hosts without that option -- the Darwin host
# and the portable standalone profile -- fall back to enabled.
enable = osConfig.features.claudeCode.enable or true;
in
{
programs.claude-code = {
enable = true;
inherit enable;
# package defaults to pkgs.claude-code (tracked to unstable via the flake
# overlay); installs the CLI on every host.
# overlay).
# ~/.claude/CLAUDE.md -- global instructions / persona / memory workflow.
context = ./claude/CLAUDE.md;
};
home.file = {
# Nothing to place when the CLI is not installed: a ~/.claude/memory symlink
# with no Claude Code to read it is just dead state.
home.file = lib.mkIf enable {
# Custom output style. The module has no option for output-styles/, so place
# it directly; selection (settings.json `outputStyle`) stays mutable.
".claude/output-styles/soviet-engineer.md".source = ./claude/output-styles/soviet-engineer.md;
+88 -11
View File
@@ -1,7 +1,7 @@
# Mac Pro 3,1 (Early 2008) — install notes
Flake host: `lyrathorpe-macpro31`. Desktop (`portable = false`, imports
`../../modules/desktop.nix`). Files: `configuration.nix`,
`../../modules/desktop.nix`). Files: `configuration.nix`, `nvidia.nix`,
`hardware-configuration.nix`.
## Hardware configuration
@@ -27,18 +27,95 @@ either
Partition the disk GPT with an ESP (vfat).
## Graphics
## Graphics — NVIDIA Quadro P400
The stock card varies between units — **ATI Radeon HD 2600 XT** or **NVIDIA
GeForce 8800 GT**. No proprietary driver is hardcoded; Sway relies on in-tree KMS:
The stock card (**ATI Radeon HD 2600 XT** or **NVIDIA GeForce 8800 GT**,
depending on the unit) has been replaced with an **NVIDIA Quadro P400** (Pascal,
GP108). Everything driver-related lives in [`nvidia.nix`](./nvidia.nix):
- ATI Radeon HD 2600 XT → `radeon` (or `amdgpu`) KMS
- NVIDIA GeForce 8800 GT → `nouveau` KMS
- **Driver branch 580** (`nvidiaPackages.legacy_580`), _not_ the nixpkgs default
(`production`, currently 595.x). 580 is the last branch that supports
Maxwell/Pascal/Volta and is maintained as an LTS branch until Aug 2028; a
newer branch does not drive this card at all.
- `modesetting.enable = true` — mandatory for Wayland (sets
`nvidia-drm.modeset=1`); without it wlroots gets no GBM device and both Sway
and the greeter fail to start.
- `open = false` — the open kernel modules require Turing or later.
- Sway runs with `--unsupported-gpu` (`programs.sway.extraOptions`); wlroots
refuses the proprietary driver otherwise. `cage`/ReGreet needs no such flag.
- nouveau and `nvidiafb` are blacklisted automatically by the NVIDIA module.
These come up automatically. If a card needs forcing, set
`services.xserver.videoDrivers` and/or add the module to
`boot.initrd.kernelModules` for early KMS (see the comment in
`configuration.nix`).
The driver is unfree, so it is **not in the binary cache**: the kernel module is
compiled on the machine, which on these 2008 Xeons is slow — budget for a long
first rebuild and again after every kernel bump. The package names are
allowlisted in `unfreePackages` in [`flake.nix`](../../flake.nix).
Note the Mac Pro shows no EFI boot screen with a stock PC card (no Apple EFI
ROM): the machine boots blind until KMS brings the display up. That is expected,
not a fault.
Verify after a rebuild:
```sh
nvidia-smi
```
## Docker with CUDA
`nvidia.nix` also enables Docker and gives containers GPU access via **CDI**
(`hardware.nvidia-container-toolkit.enable`), which generates device specs from
the host driver at boot (regenerated by a udev rule when the `nvidia` device
appears) and turns on the daemon's CDI feature:
```sh
docker run --rm --device=nvidia.com/gpu=all nvidia/cuda:12.9.1-base-ubuntu24.04 nvidia-smi
```
- Use the `--device=nvidia.com/gpu=all` form. `--gpus all` is the legacy
runtime-wrapper path (`virtualisation.docker.enableNvidia`), which is
deprecated upstream and deliberately not enabled here.
- **CUDA version matters.** The P400 is compute capability 6.1 (`sm_61`); CUDA
13 dropped Maxwell/Pascal/Volta, so container images must ship a **CUDA 12.x
or older** runtime. The 580 driver itself is happy with either.
- 2 GB of VRAM, 256 CUDA cores — fine for encode/decode and small models, not
for training anything serious.
- Docker socket is local-only (no TCP listener, unlike the Pi). Users need the
`docker` group; the registry already grants it.
### "Driver Not Loaded" from the CDI generator
`nvidia-container-toolkit-cdi-generator.service` fails with
`failed to initialize NVML: Driver Not Loaded` whenever the `nvidia` kernel
module is not loaded in the **running** kernel. After a kernel bump that is
unavoidable — the rebuilt module cannot load until reboot — so the unit is
guarded with `ConditionPathExists=/proc/driver/nvidia/version` and skips
instead of failing. Without that guard it also takes `docker.service`
(`requiredBy`) with it and makes `nixos-rebuild switch` exit non-zero.
**Reboot after a rebuild that touches the driver or the kernel.** The toolkit's
udev rule restarts the generator when the GPU device appears, so the CDI specs
are written on the next boot. To check the state:
```sh
lsmod | grep nvidia # nvidia, nvidia_modeset, nvidia_drm, nvidia_uvm
cat /proc/driver/nvidia/version
nvidia-smi
systemctl status nvidia-container-toolkit-cdi-generator.service
ls /var/run/cdi # the generated spec
```
If the module is genuinely absent after a reboot, check `dmesg | grep -i
nvidia` (build/version mismatch, or nouveau still bound — the module blacklists
it, so that should not happen).
## Claude Code — not installed here
The dual Harpertown Xeons are **x86-64-v1** (SSE4.1, but no SSE4.2/POPCNT) and
the Node runtime Claude Code ships on requires x86-64-v2. `configuration.nix`
declares `features.cpu.microarchLevel = 1`, which switches the tool off through
the fleet-wide gate in [`../../modules/features.nix`](../../modules/features.nix)
— see the root README. Forcing `features.claudeCode.enable` on here is an
evaluation error, not a broken install.
## Networking
@@ -52,7 +129,7 @@ Graphical login via a Wayland greeter — `greetd` running ReGreet inside the
every Sway host (gated on `features.swayDesktop.enable`). The greeter is forced
to the Dvorak layout to match the console and Sway session. Set the user
password (`passwd lyrathorpe`) after install, or the greeter cannot
authenticate. Requires working KMS (radeon/nouveau — see Graphics).
authenticate. Requires working KMS (NVIDIA modesetting — see Graphics).
## Apply
+9 -11
View File
@@ -7,8 +7,14 @@
{
imports = [
./hardware-configuration.nix
./nvidia.nix
];
# Dual quad-core Xeon (Harpertown/Penryn): SSE4.1 but no SSE4.2 or POPCNT,
# i.e. x86-64-v1. Declaring it here switches off the fleet flags that need a
# newer CPU -- currently features.claudeCode (see ../../modules/features.nix).
features.cpu.microarchLevel = 1;
# The Mac Pro 3,1 has 64-bit EFI (confirmed by the owner), so boot via
# systemd-boot like the MBP -- no GRUB/BIOS shim needed.
boot.loader.systemd-boot.enable = true;
@@ -33,17 +39,9 @@
# enabled in workstation.nix.
hardware.cpu.intel.updateMicrocode = true;
# GPU note: the stock card varies between units -- ATI Radeon HD 2600 XT or
# NVIDIA GeForce 8800 GT. Sway needs a working KMS/modesetting driver; do NOT
# install a proprietary blob here. Depending on the installed card, rely on
# the open kernel driver:
# - ATI Radeon HD 2600 XT -> "radeon" (older) or "amdgpu" KMS
# - NVIDIA GeForce 8800 GT -> "nouveau" KMS
# These come up automatically via the in-tree drivers + KMS, and the graphics
# stack itself is enabled by modules/sway.nix. If a card needs to be forced, add it
# here, e.g. `services.xserver.videoDrivers = [ "radeon" ];` (or "nouveau"),
# and/or `boot.initrd.kernelModules = [ "radeon" ];` in
# hardware-configuration.nix for early KMS.
# GPU: the stock card (ATI Radeon HD 2600 XT / NVIDIA GeForce 8800 GT) has
# been replaced with an NVIDIA Quadro P400. Driver, Wayland quirks and
# GPU-enabled Docker live in ./nvidia.nix.
# See `man configuration.nix` / the stateVersion docs before changing.
system.stateVersion = "26.05";
+66
View File
@@ -0,0 +1,66 @@
# NVIDIA Quadro P400 (Pascal, GP108) on the Mac Pro 3,1: proprietary driver for
# the Sway desktop, plus Docker with GPU/CUDA access for containers.
#
# Driver branch: 580 (nvidiaPackages.legacy_580), NOT the nixpkgs default
# (`production`, currently 595.x). 580 is the last branch that supports
# Maxwell/Pascal/Volta -- NVIDIA keeps it as an LTS branch to Aug 2028 -- and a
# newer branch simply will not drive this card.
#
# The driver is unfree, so it is not in the binary cache: the kernel module is
# compiled locally. On this machine's 2008 Xeons expect the first rebuild after
# a kernel bump to take a long while.
{ config, ... }:
{
# Selects the proprietary driver; the module blacklists nouveau/nvidiafb and
# loads nvidia-uvm (needed by CUDA) via modprobe softdep. Naming is historical
# -- this option drives the kernel/driver choice on Wayland hosts too, which
# is why it is set on a machine that runs no X server.
services.xserver.videoDrivers = [ "nvidia" ];
hardware.nvidia = {
package = config.boot.kernelPackages.nvidiaPackages.legacy_580;
# Required for Wayland: sets nvidia-drm.modeset=1 (and fbdev=1), without
# which wlroots gets no GBM device and Sway/cage fail to start.
modesetting.enable = true;
# The open kernel modules need Turing or later; Pascal must use the closed
# ones. Explicit because the option has no default on driver >= 560.
open = false;
};
# The NVIDIA module only puts these in boot.kernelModules when
# services.xserver.enable is true, which is false on this Wayland-only host --
# so load them explicitly rather than relying on udev modalias autoloading.
# nvidia_uvm (needed by CUDA) is deliberately absent: the module's modprobe
# softdep pulls it in after the GPU device exists, which is the supported
# ordering.
boot.kernelModules = [
"nvidia"
"nvidia_modeset"
"nvidia_drm"
];
# wlroots refuses the proprietary NVIDIA driver unless told to proceed. The
# greeter's compositor (cage) has no such check; only Sway needs the flag,
# which the module bakes into the wrapper the session's .desktop file runs.
programs.sway.extraOptions = [ "--unsupported-gpu" ];
virtualisation.docker.enable = true;
# CDI-based GPU access for containers: generates /var/run/cdi specs from the
# host driver at boot and turns on Docker's CDI feature. Run GPU workloads
# with `docker run --device=nvidia.com/gpu=all ...`. The deprecated
# virtualisation.docker.enableNvidia runtime wrapper is deliberately not used.
hardware.nvidia-container-toolkit.enable = true;
# The generator needs a loaded kernel module: without one it aborts with
# "failed to initialize NVML: Driver Not Loaded". That is guaranteed after a
# kernel bump, where the rebuilt module cannot load until reboot -- and since
# the unit is requiredBy docker.service and wantedBy multi-user.target, the
# failure takes Docker down and makes `nixos-rebuild switch` exit non-zero.
# Skip the run instead when no driver is loaded; the toolkit's udev rule
# restarts the unit as soon as the nvidia device appears, so the CDI specs are
# still generated on the next boot.
systemd.services.nvidia-container-toolkit-cdi-generator.unitConfig.ConditionPathExists =
"/proc/driver/nvidia/version";
}
+67 -2
View File
@@ -6,7 +6,72 @@
# headless host (e.g. the Pi) must be able to leave it at its default without
# pulling in modules/sway.nix. The implementation lives in modules/sway.nix,
# gated on this flag.
{ lib, ... }:
#
# The file also carries the host capability facts those flags derive from
# (features.cpu.*). features.claudeCode.enable is such a derived flag: it is
# computed from the declared CPU level here and read by home/claude.nix through
# home-manager's osConfig, so a machine that cannot run the tool never installs
# it, on any host, without per-host opt-outs.
{
options.features.swayDesktop.enable = lib.mkEnableOption "the Sway desktop";
config,
lib,
pkgs,
...
}:
let
cfg = config.features;
# Claude Code runs on Node, whose V8 build requires SSE4.2 and POPCNT -- the
# x86-64-v2 feature set. On an older x86_64 CPU it does not run (illegal
# instruction), so it must not be installed there.
claudeCodeMinLevel = 2;
claudeCodeSupported =
!pkgs.stdenv.hostPlatform.isx86_64 || cfg.cpu.microarchLevel >= claudeCodeMinLevel;
in
{
options.features = {
swayDesktop.enable = lib.mkEnableOption "the Sway desktop";
cpu.microarchLevel = lib.mkOption {
type = lib.types.ints.between 1 4;
default = 2;
example = 1;
description = ''
The x86-64 psABI microarchitecture level the host CPU implements:
1 = the original baseline, 2 = SSE4.2/POPCNT (Nehalem, 2008+),
3 = AVX2, 4 = AVX-512.
Nix cannot detect this (evaluation is pure and hosts are often built
elsewhere), so a machine older than the default declares its own level
and the flags below derive from it. Ignored on non-x86_64 hosts.
'';
};
claudeCode.enable = lib.mkOption {
type = lib.types.bool;
default = claudeCodeSupported;
defaultText = lib.literalMD ''
`true`, unless the host declares an x86-64 microarchitecture level
below ${toString claudeCodeMinLevel}
'';
description = ''
Whether to install Claude Code in this host's home-manager profiles
(implemented in home/claude.nix). Defaults off on CPUs below
x86-64-v${toString claudeCodeMinLevel}, which cannot run it; forcing it
on such a host is an evaluation error.
'';
};
};
config.assertions = [
{
assertion = cfg.claudeCode.enable -> claudeCodeSupported;
message = ''
features.claudeCode.enable is on, but this host declares
features.cpu.microarchLevel = ${toString cfg.cpu.microarchLevel}.
Claude Code needs x86-64-v${toString claudeCodeMinLevel}
(SSE4.2/POPCNT) and will not run on an older CPU.
'';
}
];
}