perf: size the encoder pool to the CPUs the container may use #3

2026-08-21T16:56:59+01:00

lyrathorpe commented

2026-08-21 16:56:59 +01:00

Measurements

libmp3lame reports Threading capabilities: none. One encode uses one core, so concurrency is entirely a matter of how many ffmpeg processes run at once.
41.9× realtime per core at V0, measured on an EPYC 7763 (Zen 3, 3.5 GHz boost). A 5600X core boosts to 4.6 GHz, so 55–60× is the expectation there.
A 4-minute FLAC costs about 4 s of CPU to encode; the embedded-art probe that preceded it costs about 30 ms.

The defect

Pool width came from os.cpu_count(), which reports the host's cores and ignores a container's cpus: allowance. On a 12-thread host limited to 4 CPUs that is threefold oversubscription: no extra throughput, just context switches and a less responsive NAS.

The default now reads the cgroup v2 quota, falling back to process CPU affinity and then to the host count. Verified in a container on a 4-CPU host:

--cpus=2   -> pass starting with 2 concurrent encoders
no limit   -> pass starting with 4 concurrent encoders

Each pass logs the number it chose, so tuning MUSIC_MIRROR_JOBS is not guesswork.

The tidy-up

The old code ran an ffprobe per file to decide whether to attach external cover art. That question only matters when a cover file sits beside the track — with none, -map 0:v:0? already carries embedded art if present. So it now probes only in that case, and caches the per-directory cover lookup.

Honest scale: ~30 ms against ~4 s of encoding, under 1%. This is tidiness, not a speed-up, and the commit message says so.

Verification

21 tests pass. Container built and run under a CPU limit and without one.

## Measurements - `libmp3lame` reports `Threading capabilities: none`. One encode uses one core, so concurrency is entirely a matter of how many ffmpeg processes run at once. - **41.9× realtime** per core at V0, measured on an EPYC 7763 (Zen 3, 3.5 GHz boost). A 5600X core boosts to 4.6 GHz, so 55–60× is the expectation there. - A 4-minute FLAC costs about 4 s of CPU to encode; the embedded-art probe that preceded it costs about 30 ms. ## The defect Pool width came from `os.cpu_count()`, which reports the **host's** cores and ignores a container's `cpus:` allowance. On a 12-thread host limited to 4 CPUs that is threefold oversubscription: no extra throughput, just context switches and a less responsive NAS. The default now reads the cgroup v2 quota, falling back to process CPU affinity and then to the host count. Verified in a container on a 4-CPU host: ``` --cpus=2 -> pass starting with 2 concurrent encoders no limit -> pass starting with 4 concurrent encoders ``` Each pass logs the number it chose, so tuning `MUSIC_MIRROR_JOBS` is not guesswork. ## The tidy-up The old code ran an `ffprobe` per file to decide whether to attach external cover art. That question only matters when a cover file sits beside the track — with none, `-map 0:v:0?` already carries embedded art if present. So it now probes only in that case, and caches the per-directory cover lookup. Honest scale: ~30 ms against ~4 s of encoding, under 1%. This is tidiness, not a speed-up, and the commit message says so. ## Verification 21 tests pass. Container built and run under a CPU limit and without one.

lyrathorpe added 1 commit 2026-08-21 16:56:59 +01:00

perf: size the encoder pool to the CPUs the container may use

Build and publish container / build (pull_request) Successful in 11m13s

Details

2b831daa1a

libmp3lame is single-threaded, so concurrency is one ffmpeg process per file
and the pool width is the whole of it. The width came from os.cpu_count(),
which reports the host's cores and ignores a container's cpus: allowance -- on
a 12-thread host limited to 4 CPUs that is threefold oversubscription, which
costs context switches and NAS responsiveness for no throughput.

The default now reads the cgroup v2 quota, falling back to process CPU
affinity and then to the host count. Each pass logs the number it chose.

Also stops probing every file for embedded cover art. The probe only changes
the command when a cover file sits beside the track, so ask only then; with no
cover file, -map 0:v:0? already carries embedded art if there is any. Worth
roughly 30 ms per track against about 4 s of encoding, so this is tidiness
rather than a speed-up. The per-directory cover lookup is cached alongside.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

lyrathorpe merged commit 9984bedd02 into main

2026-08-24 11:35:04 +01:00

lyrathorpe deleted branch perf/encode-concurrency

2026-08-24 11:35:06 +01:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lyrathorpe/music-mirror#3