Emma Thorpe
d4e7475db9
CI / flake (push) Skipped
CI / flake (pull_request) Successful in 4m12s
The CDI generator aborted with "failed to initialize NVML: Driver Not Loaded", taking docker.service with it (requiredBy) and failing the switch. Two causes. The nixpkgs NVIDIA module only adds nvidia/nvidia_modeset/ nvidia_drm to boot.kernelModules when services.xserver.enable is set, which is false on this Wayland-only host, so load them explicitly. nvidia_uvm stays out: the module's modprobe softdep loads it once the GPU device exists. The generator also runs during activation, when a module rebuilt against a new kernel cannot be loaded until reboot -- a guaranteed failure after every kernel bump. Guard it with ConditionPathExists on /proc/driver/nvidia/version so it skips rather than fails; the toolkit's udev rule restarts it when the device appears, so the specs are generated on the next boot.
139 lines
5.9 KiB
Markdown
139 lines
5.9 KiB
Markdown
# Mac Pro 3,1 (Early 2008) — install notes
|
|
|
|
Flake host: `lyrathorpe-macpro31`. Desktop (`portable = false`, imports
|
|
`../../modules/desktop.nix`). Files: `configuration.nix`, `nvidia.nix`,
|
|
`hardware-configuration.nix`.
|
|
|
|
## Hardware configuration
|
|
|
|
`hardware-configuration.nix` here is the real config generated by
|
|
`nixos-generate-config` on the machine. Root is an **LVM** logical volume
|
|
(`/dev/mapper/MacPro-Root`, ext4); the ESP (vfat) and swap are referenced by
|
|
UUID. The initrd carries `dm-snapshot` for the LVM root. Regenerate and commit
|
|
if the disk layout changes.
|
|
|
|
## Bootloader
|
|
|
|
The Mac Pro 3,1 has **64-bit EFI**, so it uses **systemd-boot** (no GRUB/CSM
|
|
shim). `canTouchEfiVariables = false` because Apple's firmware does not reliably
|
|
accept `efibootmgr` NVRAM writes.
|
|
|
|
Apple-EFI quirk: if the firmware boot picker does not show NixOS after install,
|
|
either
|
|
|
|
- uncomment `boot.loader.efi.efiInstallAsRemovable = true;` in
|
|
`configuration.nix` (installs the fallback `\EFI\BOOT\BOOTX64.EFI`), and/or
|
|
- "bless" the ESP from macOS.
|
|
|
|
Partition the disk GPT with an ESP (vfat).
|
|
|
|
## Graphics — NVIDIA Quadro P400
|
|
|
|
The stock card (**ATI Radeon HD 2600 XT** or **NVIDIA GeForce 8800 GT**,
|
|
depending on the unit) has been replaced with an **NVIDIA Quadro P400** (Pascal,
|
|
GP108). Everything driver-related lives in [`nvidia.nix`](./nvidia.nix):
|
|
|
|
- **Driver branch 580** (`nvidiaPackages.legacy_580`), _not_ the nixpkgs default
|
|
(`production`, currently 595.x). 580 is the last branch that supports
|
|
Maxwell/Pascal/Volta and is maintained as an LTS branch until Aug 2028; a
|
|
newer branch does not drive this card at all.
|
|
- `modesetting.enable = true` — mandatory for Wayland (sets
|
|
`nvidia-drm.modeset=1`); without it wlroots gets no GBM device and both Sway
|
|
and the greeter fail to start.
|
|
- `open = false` — the open kernel modules require Turing or later.
|
|
- Sway runs with `--unsupported-gpu` (`programs.sway.extraOptions`); wlroots
|
|
refuses the proprietary driver otherwise. `cage`/ReGreet needs no such flag.
|
|
- nouveau and `nvidiafb` are blacklisted automatically by the NVIDIA module.
|
|
|
|
The driver is unfree, so it is **not in the binary cache**: the kernel module is
|
|
compiled on the machine, which on these 2008 Xeons is slow — budget for a long
|
|
first rebuild and again after every kernel bump. The package names are
|
|
allowlisted in `unfreePackages` in [`flake.nix`](../../flake.nix).
|
|
|
|
Note the Mac Pro shows no EFI boot screen with a stock PC card (no Apple EFI
|
|
ROM): the machine boots blind until KMS brings the display up. That is expected,
|
|
not a fault.
|
|
|
|
Verify after a rebuild:
|
|
|
|
```sh
|
|
nvidia-smi
|
|
```
|
|
|
|
## Docker with CUDA
|
|
|
|
`nvidia.nix` also enables Docker and gives containers GPU access via **CDI**
|
|
(`hardware.nvidia-container-toolkit.enable`), which generates device specs from
|
|
the host driver at boot (regenerated by a udev rule when the `nvidia` device
|
|
appears) and turns on the daemon's CDI feature:
|
|
|
|
```sh
|
|
docker run --rm --device=nvidia.com/gpu=all nvidia/cuda:12.9.1-base-ubuntu24.04 nvidia-smi
|
|
```
|
|
|
|
- Use the `--device=nvidia.com/gpu=all` form. `--gpus all` is the legacy
|
|
runtime-wrapper path (`virtualisation.docker.enableNvidia`), which is
|
|
deprecated upstream and deliberately not enabled here.
|
|
- **CUDA version matters.** The P400 is compute capability 6.1 (`sm_61`); CUDA
|
|
13 dropped Maxwell/Pascal/Volta, so container images must ship a **CUDA 12.x
|
|
or older** runtime. The 580 driver itself is happy with either.
|
|
- 2 GB of VRAM, 256 CUDA cores — fine for encode/decode and small models, not
|
|
for training anything serious.
|
|
- Docker socket is local-only (no TCP listener, unlike the Pi). Users need the
|
|
`docker` group; the registry already grants it.
|
|
|
|
### "Driver Not Loaded" from the CDI generator
|
|
|
|
`nvidia-container-toolkit-cdi-generator.service` fails with
|
|
`failed to initialize NVML: Driver Not Loaded` whenever the `nvidia` kernel
|
|
module is not loaded in the **running** kernel. After a kernel bump that is
|
|
unavoidable — the rebuilt module cannot load until reboot — so the unit is
|
|
guarded with `ConditionPathExists=/proc/driver/nvidia/version` and skips
|
|
instead of failing. Without that guard it also takes `docker.service`
|
|
(`requiredBy`) with it and makes `nixos-rebuild switch` exit non-zero.
|
|
|
|
**Reboot after a rebuild that touches the driver or the kernel.** The toolkit's
|
|
udev rule restarts the generator when the GPU device appears, so the CDI specs
|
|
are written on the next boot. To check the state:
|
|
|
|
```sh
|
|
lsmod | grep nvidia # nvidia, nvidia_modeset, nvidia_drm, nvidia_uvm
|
|
cat /proc/driver/nvidia/version
|
|
nvidia-smi
|
|
systemctl status nvidia-container-toolkit-cdi-generator.service
|
|
ls /var/run/cdi # the generated spec
|
|
```
|
|
|
|
If the module is genuinely absent after a reboot, check `dmesg | grep -i
|
|
nvidia` (build/version mismatch, or nouveau still bound — the module blacklists
|
|
it, so that should not happen).
|
|
|
|
## Claude Code — not installed here
|
|
|
|
The dual Harpertown Xeons are **x86-64-v1** (SSE4.1, but no SSE4.2/POPCNT) and
|
|
the Node runtime Claude Code ships on requires x86-64-v2. `configuration.nix`
|
|
declares `features.cpu.microarchLevel = 1`, which switches the tool off through
|
|
the fleet-wide gate in [`../../modules/features.nix`](../../modules/features.nix)
|
|
— see the root README. Forcing `features.claudeCode.enable` on here is an
|
|
evaluation error, not a broken install.
|
|
|
|
## Networking
|
|
|
|
Wired Ethernet via NetworkManager (from `desktop.nix`) — the Mac Pro has two
|
|
gigabit ports.
|
|
|
|
## Login
|
|
|
|
Graphical login via a Wayland greeter — `greetd` running ReGreet inside the
|
|
`cage` kiosk compositor — configured centrally in `../../modules/sway.nix` for
|
|
every Sway host (gated on `features.swayDesktop.enable`). The greeter is forced
|
|
to the Dvorak layout to match the console and Sway session. Set the user
|
|
password (`passwd lyrathorpe`) after install, or the greeter cannot
|
|
authenticate. Requires working KMS (NVIDIA modesetting — see Graphics).
|
|
|
|
## Apply
|
|
|
|
```sh
|
|
sudo nixos-rebuild switch --flake .#lyrathorpe-macpro31
|
|
```
|