diff --git a/README.md b/README.md index 48a3146..d2f188a 100644 --- a/README.md +++ b/README.md @@ -55,17 +55,17 @@ module reaches a host: **baseModules** (every NixOS host, via `flake.nix`), **host table** (listed explicitly per host in `flake.nix`), or **transitively** (pulled in by another module's `imports`). -| Module | Imported by | What it does / when to use it | -| ------------------ | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `common-nixos.nix` | baseModules (all NixOS) | Timezone/locale, store hygiene (auto-optimise, big download buffer, **no** auto-GC), the nix-community binary cache, `nix-ld`, base CLI (`git`, `fastfetch`), and the fleet-wide font stack. | -| `users.nix` | baseModules (all NixOS) | Builds `users.users` from the registry for the host's `hostUsers`; enables zsh; enables Firefox + Thunderbird **only** when `features.swayDesktop.enable` is on. Applies per-user `linger`. | -| `features.nix` | baseModules (all NixOS) | Declares feature-flag options (currently `features.swayDesktop.enable`) so any host can read/set them without importing the heavy implementation module. | -| `workstation.nix` | transitively (via laptop/desktop) | Form-factor-agnostic base for physical graphical hosts: turns on `swayDesktop`, Dvorak console, PipeWire, firewall (default-deny), fstrim, earlyoom, fwupd, thermald (x86), redistributable fw. | -| `laptop.nix` | host table (MBP, T400) | `imports` workstation.nix, then adds the portable bits: iwd Wi-Fi, lid suspend/lock, Bluetooth + blueman. | -| `desktop.nix` | host table (Mac Pro) | `imports` workstation.nix, then swaps Wi-Fi for wired NetworkManager. Pair with `portable = false` in the host table. | -| `sway.nix` | host table (graphical hosts) | Implementation of `features.swayDesktop`: the system Sway package, the greetd/ReGreet (cage) greeter forced to Dvorak, xdg-portal, Wayland utility packages. Home-side Sway config is in `home/sway.nix`. | -| `ssh.nix` | host table (T400, Mac Pro, RPi5) | Enables sshd, opens port 22, enforces a key-only policy (no password / keyboard-interactive, no root). Authorized keys come from the registry via `users.nix`. | -| `firmware/` | referenced by MBP host config | Committed Apple peripheral firmware blobs for the Asahi MBP (see "MacBook (Asahi) firmware"). | +| Module | Imported by | What it does / when to use it | +| ------------------ | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `common-nixos.nix` | baseModules (all NixOS) | Timezone/locale, store hygiene (auto-optimise, big download buffer, **no** auto-GC), the nix-community binary cache, `nix-ld`, base CLI (`git`, `fastfetch`), and the fleet-wide font stack. | +| `users.nix` | baseModules (all NixOS) | Builds `users.users` from the registry for the host's `hostUsers`; enables zsh; enables Firefox + Thunderbird **only** when `features.swayDesktop.enable` is on. Applies per-user `linger`. | +| `features.nix` | baseModules (all NixOS) | Declares the feature-flag options (`features.swayDesktop.enable`, `features.claudeCode.enable`) so any host can read/set them without importing the heavy implementation module, plus the CPU capability fact they derive from (`features.cpu.microarchLevel`) and the assertion that guards it. See "CPU capability gating". | +| `workstation.nix` | transitively (via laptop/desktop) | Form-factor-agnostic base for physical graphical hosts: turns on `swayDesktop`, Dvorak console, PipeWire, firewall (default-deny), fstrim, earlyoom, fwupd, thermald (x86), redistributable fw. | +| `laptop.nix` | host table (MBP, T400) | `imports` workstation.nix, then adds the portable bits: iwd Wi-Fi, lid suspend/lock, Bluetooth + blueman. | +| `desktop.nix` | host table (Mac Pro) | `imports` workstation.nix, then swaps Wi-Fi for wired NetworkManager. Pair with `portable = false` in the host table. | +| `sway.nix` | host table (graphical hosts) | Implementation of `features.swayDesktop`: the system Sway package, the greetd/ReGreet (cage) greeter forced to Dvorak, xdg-portal, Wayland utility packages. Home-side Sway config is in `home/sway.nix`. | +| `ssh.nix` | host table (T400, Mac Pro, RPi5) | Enables sshd, opens port 22, enforces a key-only policy (no password / keyboard-interactive, no root). Authorized keys come from the registry via `users.nix`. | +| `firmware/` | referenced by MBP host config | Committed Apple peripheral firmware blobs for the Asahi MBP (see "MacBook (Asahi) firmware"). | Form-factor decision: a **laptop** imports `laptop.nix` (default `portable = true`); a **wired desktop** imports `desktop.nix` and sets @@ -74,6 +74,29 @@ Form-factor decision: a **laptop** imports `laptop.nix` (default serves. `portable` is threaded through to `home/sway.nix`, which drops the battery block and brightness keys on desktops. +## CPU capability gating + +Not every host can run everything the fleet installs. Nix cannot probe the CPU +(evaluation is pure, and a host may be built elsewhere), so each machine +declares what it is and the shared modules derive from that: + +- `features.cpu.microarchLevel` — the x86-64 psABI level the CPU implements + (1 = baseline, 2 = SSE4.2/POPCNT, 3 = AVX2, 4 = AVX-512). Defaults to **2**; + only a host older than that sets it (the Mac Pro 3,1's 2008 Harpertown Xeons + are level 1). Ignored on non-x86_64 hosts. +- `features.claudeCode.enable` — derived: on unless the host is below + x86-64-v2, because Claude Code's Node runtime needs SSE4.2/POPCNT. + [`home/claude.nix`](./home/claude.nix) reads it through home-manager's + `osConfig` and installs nothing (CLI, `CLAUDE.md`, output style, memory + symlink) when it is off. Hosts with no such option — the Darwin host and the + standalone `homeConfigurations` — fall back to enabled. +- An assertion in `features.nix` fails evaluation if a host force-enables a + flag its declared CPU level cannot support, so the mistake surfaces in + `nix flake check`/CI rather than as an illegal-instruction crash on the box. + +Adding another CPU-sensitive tool means deriving one more flag there, not +editing every host. + ## Users Identity is data, kept separate from the reusable modules: diff --git a/flake.nix b/flake.nix index d982c83..d8109f4 100644 --- a/flake.nix +++ b/flake.nix @@ -111,8 +111,14 @@ ]; # Unfree packages permitted to be built (replaces blanket allowUnfree). + # The NVIDIA entries are for the Mac Pro's Quadro P400 (hosts/MacPro31/ + # nvidia.nix); unfree packages are not in the binary cache, so the + # kernel module is compiled on the host. unfreePackages = [ "claude-code" + "nvidia-x11" + "nvidia-kernel-modules" + "nvidia-settings" ]; # Per-user identity, keyed by username. See README "Users". diff --git a/home/README.md b/home/README.md index 173f1d9..2c27cfd 100644 --- a/home/README.md +++ b/home/README.md @@ -175,8 +175,16 @@ forced off there) but still runs the agent. ## Claude Code -Managed declaratively by [`claude.nix`](./claude.nix) on every host (the CLI is -`pkgs.claude-code`, tracked to unstable via the flake overlay). +Managed declaratively by [`claude.nix`](./claude.nix) on every host whose CPU +can run it (the CLI is `pkgs.claude-code`, tracked to unstable via the flake +overlay). + +**Capability gate.** The module installs nothing — CLI or files — when +`osConfig.features.claudeCode.enable` is off. That flag is derived fleet-wide +from the host's declared CPU level (see "CPU capability gating" in the root +README): the Node runtime needs SSE4.2/POPCNT, so anything below x86-64-v2 (the +Mac Pro 3,1) is excluded. Hosts that do not define the option — the Darwin host +and the standalone `homeConfigurations` — keep it enabled. | Managed (static, from Nix) | Left mutable (runtime state) | | --------------------------------------------------- | ------------------------------------------------------ | diff --git a/home/claude.nix b/home/claude.nix index 3bf3d0b..2e5fdbe 100644 --- a/home/claude.nix +++ b/home/claude.nix @@ -1,4 +1,5 @@ -# Claude Code, configured declaratively via home-manager. Wanted on every host. +# Claude Code, configured declaratively via home-manager. Wanted on every host +# whose CPU can run it -- see the gate below. # # The STATIC config is managed here: the global CLAUDE.md (persona/context), the # custom output style, and the auto-memory directory. settings.json is @@ -10,18 +11,33 @@ # read-only into ~/.claude/memory, so the runtime "save a memory" path no longer # writes there -- recall still works, but new/changed memories must be added to # this repo and rebuilt. CLAUDE.md instructs Claude to do exactly that. -{ ... }: +{ + lib, + # Set by the NixOS/Darwin home-manager module; absent for the standalone + # homeConfigurations, hence the default. + osConfig ? { }, + ... +}: +let + # Capability gate, declared once for the whole fleet in modules/features.nix + # (default: on; off on CPUs below x86-64-v2, which cannot run the Node + # runtime Claude Code ships on). Hosts without that option -- the Darwin host + # and the portable standalone profile -- fall back to enabled. + enable = osConfig.features.claudeCode.enable or true; +in { programs.claude-code = { - enable = true; + inherit enable; # package defaults to pkgs.claude-code (tracked to unstable via the flake - # overlay); installs the CLI on every host. + # overlay). # ~/.claude/CLAUDE.md -- global instructions / persona / memory workflow. context = ./claude/CLAUDE.md; }; - home.file = { + # Nothing to place when the CLI is not installed: a ~/.claude/memory symlink + # with no Claude Code to read it is just dead state. + home.file = lib.mkIf enable { # Custom output style. The module has no option for output-styles/, so place # it directly; selection (settings.json `outputStyle`) stays mutable. ".claude/output-styles/soviet-engineer.md".source = ./claude/output-styles/soviet-engineer.md; diff --git a/hosts/MacPro31/README.md b/hosts/MacPro31/README.md index 8b5ce00..f514f39 100644 --- a/hosts/MacPro31/README.md +++ b/hosts/MacPro31/README.md @@ -1,7 +1,7 @@ # Mac Pro 3,1 (Early 2008) — install notes Flake host: `lyrathorpe-macpro31`. Desktop (`portable = false`, imports -`../../modules/desktop.nix`). Files: `configuration.nix`, +`../../modules/desktop.nix`). Files: `configuration.nix`, `nvidia.nix`, `hardware-configuration.nix`. ## Hardware configuration @@ -27,18 +27,95 @@ either Partition the disk GPT with an ESP (vfat). -## Graphics +## Graphics — NVIDIA Quadro P400 -The stock card varies between units — **ATI Radeon HD 2600 XT** or **NVIDIA -GeForce 8800 GT**. No proprietary driver is hardcoded; Sway relies on in-tree KMS: +The stock card (**ATI Radeon HD 2600 XT** or **NVIDIA GeForce 8800 GT**, +depending on the unit) has been replaced with an **NVIDIA Quadro P400** (Pascal, +GP108). Everything driver-related lives in [`nvidia.nix`](./nvidia.nix): -- ATI Radeon HD 2600 XT → `radeon` (or `amdgpu`) KMS -- NVIDIA GeForce 8800 GT → `nouveau` KMS +- **Driver branch 580** (`nvidiaPackages.legacy_580`), _not_ the nixpkgs default + (`production`, currently 595.x). 580 is the last branch that supports + Maxwell/Pascal/Volta and is maintained as an LTS branch until Aug 2028; a + newer branch does not drive this card at all. +- `modesetting.enable = true` — mandatory for Wayland (sets + `nvidia-drm.modeset=1`); without it wlroots gets no GBM device and both Sway + and the greeter fail to start. +- `open = false` — the open kernel modules require Turing or later. +- Sway runs with `--unsupported-gpu` (`programs.sway.extraOptions`); wlroots + refuses the proprietary driver otherwise. `cage`/ReGreet needs no such flag. +- nouveau and `nvidiafb` are blacklisted automatically by the NVIDIA module. -These come up automatically. If a card needs forcing, set -`services.xserver.videoDrivers` and/or add the module to -`boot.initrd.kernelModules` for early KMS (see the comment in -`configuration.nix`). +The driver is unfree, so it is **not in the binary cache**: the kernel module is +compiled on the machine, which on these 2008 Xeons is slow — budget for a long +first rebuild and again after every kernel bump. The package names are +allowlisted in `unfreePackages` in [`flake.nix`](../../flake.nix). + +Note the Mac Pro shows no EFI boot screen with a stock PC card (no Apple EFI +ROM): the machine boots blind until KMS brings the display up. That is expected, +not a fault. + +Verify after a rebuild: + +```sh +nvidia-smi +``` + +## Docker with CUDA + +`nvidia.nix` also enables Docker and gives containers GPU access via **CDI** +(`hardware.nvidia-container-toolkit.enable`), which generates device specs from +the host driver at boot (regenerated by a udev rule when the `nvidia` device +appears) and turns on the daemon's CDI feature: + +```sh +docker run --rm --device=nvidia.com/gpu=all nvidia/cuda:12.9.1-base-ubuntu24.04 nvidia-smi +``` + +- Use the `--device=nvidia.com/gpu=all` form. `--gpus all` is the legacy + runtime-wrapper path (`virtualisation.docker.enableNvidia`), which is + deprecated upstream and deliberately not enabled here. +- **CUDA version matters.** The P400 is compute capability 6.1 (`sm_61`); CUDA + 13 dropped Maxwell/Pascal/Volta, so container images must ship a **CUDA 12.x + or older** runtime. The 580 driver itself is happy with either. +- 2 GB of VRAM, 256 CUDA cores — fine for encode/decode and small models, not + for training anything serious. +- Docker socket is local-only (no TCP listener, unlike the Pi). Users need the + `docker` group; the registry already grants it. + +### "Driver Not Loaded" from the CDI generator + +`nvidia-container-toolkit-cdi-generator.service` fails with +`failed to initialize NVML: Driver Not Loaded` whenever the `nvidia` kernel +module is not loaded in the **running** kernel. After a kernel bump that is +unavoidable — the rebuilt module cannot load until reboot — so the unit is +guarded with `ConditionPathExists=/proc/driver/nvidia/version` and skips +instead of failing. Without that guard it also takes `docker.service` +(`requiredBy`) with it and makes `nixos-rebuild switch` exit non-zero. + +**Reboot after a rebuild that touches the driver or the kernel.** The toolkit's +udev rule restarts the generator when the GPU device appears, so the CDI specs +are written on the next boot. To check the state: + +```sh +lsmod | grep nvidia # nvidia, nvidia_modeset, nvidia_drm, nvidia_uvm +cat /proc/driver/nvidia/version +nvidia-smi +systemctl status nvidia-container-toolkit-cdi-generator.service +ls /var/run/cdi # the generated spec +``` + +If the module is genuinely absent after a reboot, check `dmesg | grep -i +nvidia` (build/version mismatch, or nouveau still bound — the module blacklists +it, so that should not happen). + +## Claude Code — not installed here + +The dual Harpertown Xeons are **x86-64-v1** (SSE4.1, but no SSE4.2/POPCNT) and +the Node runtime Claude Code ships on requires x86-64-v2. `configuration.nix` +declares `features.cpu.microarchLevel = 1`, which switches the tool off through +the fleet-wide gate in [`../../modules/features.nix`](../../modules/features.nix) +— see the root README. Forcing `features.claudeCode.enable` on here is an +evaluation error, not a broken install. ## Networking @@ -52,7 +129,7 @@ Graphical login via a Wayland greeter — `greetd` running ReGreet inside the every Sway host (gated on `features.swayDesktop.enable`). The greeter is forced to the Dvorak layout to match the console and Sway session. Set the user password (`passwd lyrathorpe`) after install, or the greeter cannot -authenticate. Requires working KMS (radeon/nouveau — see Graphics). +authenticate. Requires working KMS (NVIDIA modesetting — see Graphics). ## Apply diff --git a/hosts/MacPro31/configuration.nix b/hosts/MacPro31/configuration.nix index 6faae6d..82eee35 100644 --- a/hosts/MacPro31/configuration.nix +++ b/hosts/MacPro31/configuration.nix @@ -7,8 +7,14 @@ { imports = [ ./hardware-configuration.nix + ./nvidia.nix ]; + # Dual quad-core Xeon (Harpertown/Penryn): SSE4.1 but no SSE4.2 or POPCNT, + # i.e. x86-64-v1. Declaring it here switches off the fleet flags that need a + # newer CPU -- currently features.claudeCode (see ../../modules/features.nix). + features.cpu.microarchLevel = 1; + # The Mac Pro 3,1 has 64-bit EFI (confirmed by the owner), so boot via # systemd-boot like the MBP -- no GRUB/BIOS shim needed. boot.loader.systemd-boot.enable = true; @@ -33,17 +39,9 @@ # enabled in workstation.nix. hardware.cpu.intel.updateMicrocode = true; - # GPU note: the stock card varies between units -- ATI Radeon HD 2600 XT or - # NVIDIA GeForce 8800 GT. Sway needs a working KMS/modesetting driver; do NOT - # install a proprietary blob here. Depending on the installed card, rely on - # the open kernel driver: - # - ATI Radeon HD 2600 XT -> "radeon" (older) or "amdgpu" KMS - # - NVIDIA GeForce 8800 GT -> "nouveau" KMS - # These come up automatically via the in-tree drivers + KMS, and the graphics - # stack itself is enabled by modules/sway.nix. If a card needs to be forced, add it - # here, e.g. `services.xserver.videoDrivers = [ "radeon" ];` (or "nouveau"), - # and/or `boot.initrd.kernelModules = [ "radeon" ];` in - # hardware-configuration.nix for early KMS. + # GPU: the stock card (ATI Radeon HD 2600 XT / NVIDIA GeForce 8800 GT) has + # been replaced with an NVIDIA Quadro P400. Driver, Wayland quirks and + # GPU-enabled Docker live in ./nvidia.nix. # See `man configuration.nix` / the stateVersion docs before changing. system.stateVersion = "26.05"; diff --git a/hosts/MacPro31/nvidia.nix b/hosts/MacPro31/nvidia.nix new file mode 100644 index 0000000..43bab95 --- /dev/null +++ b/hosts/MacPro31/nvidia.nix @@ -0,0 +1,66 @@ +# NVIDIA Quadro P400 (Pascal, GP108) on the Mac Pro 3,1: proprietary driver for +# the Sway desktop, plus Docker with GPU/CUDA access for containers. +# +# Driver branch: 580 (nvidiaPackages.legacy_580), NOT the nixpkgs default +# (`production`, currently 595.x). 580 is the last branch that supports +# Maxwell/Pascal/Volta -- NVIDIA keeps it as an LTS branch to Aug 2028 -- and a +# newer branch simply will not drive this card. +# +# The driver is unfree, so it is not in the binary cache: the kernel module is +# compiled locally. On this machine's 2008 Xeons expect the first rebuild after +# a kernel bump to take a long while. +{ config, ... }: + +{ + # Selects the proprietary driver; the module blacklists nouveau/nvidiafb and + # loads nvidia-uvm (needed by CUDA) via modprobe softdep. Naming is historical + # -- this option drives the kernel/driver choice on Wayland hosts too, which + # is why it is set on a machine that runs no X server. + services.xserver.videoDrivers = [ "nvidia" ]; + + hardware.nvidia = { + package = config.boot.kernelPackages.nvidiaPackages.legacy_580; + # Required for Wayland: sets nvidia-drm.modeset=1 (and fbdev=1), without + # which wlroots gets no GBM device and Sway/cage fail to start. + modesetting.enable = true; + # The open kernel modules need Turing or later; Pascal must use the closed + # ones. Explicit because the option has no default on driver >= 560. + open = false; + }; + + # The NVIDIA module only puts these in boot.kernelModules when + # services.xserver.enable is true, which is false on this Wayland-only host -- + # so load them explicitly rather than relying on udev modalias autoloading. + # nvidia_uvm (needed by CUDA) is deliberately absent: the module's modprobe + # softdep pulls it in after the GPU device exists, which is the supported + # ordering. + boot.kernelModules = [ + "nvidia" + "nvidia_modeset" + "nvidia_drm" + ]; + + # wlroots refuses the proprietary NVIDIA driver unless told to proceed. The + # greeter's compositor (cage) has no such check; only Sway needs the flag, + # which the module bakes into the wrapper the session's .desktop file runs. + programs.sway.extraOptions = [ "--unsupported-gpu" ]; + + virtualisation.docker.enable = true; + + # CDI-based GPU access for containers: generates /var/run/cdi specs from the + # host driver at boot and turns on Docker's CDI feature. Run GPU workloads + # with `docker run --device=nvidia.com/gpu=all ...`. The deprecated + # virtualisation.docker.enableNvidia runtime wrapper is deliberately not used. + hardware.nvidia-container-toolkit.enable = true; + + # The generator needs a loaded kernel module: without one it aborts with + # "failed to initialize NVML: Driver Not Loaded". That is guaranteed after a + # kernel bump, where the rebuilt module cannot load until reboot -- and since + # the unit is requiredBy docker.service and wantedBy multi-user.target, the + # failure takes Docker down and makes `nixos-rebuild switch` exit non-zero. + # Skip the run instead when no driver is loaded; the toolkit's udev rule + # restarts the unit as soon as the nvidia device appears, so the CDI specs are + # still generated on the next boot. + systemd.services.nvidia-container-toolkit-cdi-generator.unitConfig.ConditionPathExists = + "/proc/driver/nvidia/version"; +} diff --git a/modules/features.nix b/modules/features.nix index 0d0de92..c34b2f6 100644 --- a/modules/features.nix +++ b/modules/features.nix @@ -6,7 +6,72 @@ # headless host (e.g. the Pi) must be able to leave it at its default without # pulling in modules/sway.nix. The implementation lives in modules/sway.nix, # gated on this flag. -{ lib, ... }: +# +# The file also carries the host capability facts those flags derive from +# (features.cpu.*). features.claudeCode.enable is such a derived flag: it is +# computed from the declared CPU level here and read by home/claude.nix through +# home-manager's osConfig, so a machine that cannot run the tool never installs +# it, on any host, without per-host opt-outs. { - options.features.swayDesktop.enable = lib.mkEnableOption "the Sway desktop"; + config, + lib, + pkgs, + ... +}: +let + cfg = config.features; + + # Claude Code runs on Node, whose V8 build requires SSE4.2 and POPCNT -- the + # x86-64-v2 feature set. On an older x86_64 CPU it does not run (illegal + # instruction), so it must not be installed there. + claudeCodeMinLevel = 2; + claudeCodeSupported = + !pkgs.stdenv.hostPlatform.isx86_64 || cfg.cpu.microarchLevel >= claudeCodeMinLevel; +in +{ + options.features = { + swayDesktop.enable = lib.mkEnableOption "the Sway desktop"; + + cpu.microarchLevel = lib.mkOption { + type = lib.types.ints.between 1 4; + default = 2; + example = 1; + description = '' + The x86-64 psABI microarchitecture level the host CPU implements: + 1 = the original baseline, 2 = SSE4.2/POPCNT (Nehalem, 2008+), + 3 = AVX2, 4 = AVX-512. + + Nix cannot detect this (evaluation is pure and hosts are often built + elsewhere), so a machine older than the default declares its own level + and the flags below derive from it. Ignored on non-x86_64 hosts. + ''; + }; + + claudeCode.enable = lib.mkOption { + type = lib.types.bool; + default = claudeCodeSupported; + defaultText = lib.literalMD '' + `true`, unless the host declares an x86-64 microarchitecture level + below ${toString claudeCodeMinLevel} + ''; + description = '' + Whether to install Claude Code in this host's home-manager profiles + (implemented in home/claude.nix). Defaults off on CPUs below + x86-64-v${toString claudeCodeMinLevel}, which cannot run it; forcing it + on such a host is an evaluation error. + ''; + }; + }; + + config.assertions = [ + { + assertion = cfg.claudeCode.enable -> claudeCodeSupported; + message = '' + features.claudeCode.enable is on, but this host declares + features.cpu.microarchLevel = ${toString cfg.cpu.microarchLevel}. + Claude Code needs x86-64-v${toString claudeCodeMinLevel} + (SSE4.2/POPCNT) and will not run on an older CPU. + ''; + } + ]; }