feat: mirror a lossless library to MP3 for iPod sync #1

Merged

lyrathorpe merged 6 commits from feat/initial-implementation into main 2026-08-21 15:30:59 +01:00

Conversation 0 Commits 6 Files Changed 11

+1346 -1

lyrathorpe commented 2026-08-21 14:45:03 +01:00

Owner

Copy Link

Copy Source

What

A path-for-path MP3 mirror of a lossless library. FLAC in, MP3 out, same relative layout, tags and cover art carried across. Already-MP3 sources are copied rather than re-encoded. Mirror files whose source has gone are deleted, along with any directory they emptied. The source library is never written to — mounted read-only in the compose file, and never opened for writing in code.

Situation	Action
No mirror file	encode
Source modified since the mirror	re-encode (a Lidarr quality upgrade)
Mirror up to date	skip
Source is already MP3	copy verbatim
Source gone	delete, prune empty dirs

Design notes

• No database. Freshness is mtime: an encode is stamped with its source's mtime, so a file is stale exactly when the two differ. Lidarr owns the library; a second tool with its own index would only fall out of step with it. This is also why it is not beets.
• Atomic writes. Encode to a temp file, rename into place. An interrupted run cannot leave a truncated MP3 that the next run treats as finished.
• A lock file in the mirror root stops two passes overlapping.
• Refuses a mirror inside the source tree, which would otherwise recurse.
• --subdir never prunes: a partial pass cannot distinguish an orphan from a file outside its scope.

Shipping

• Python package with a music-mirror console script, no runtime dependencies beyond ffmpeg.
• Dockerfile plus compose.yaml as a TrueNAS Scale Custom App: source dataset read-only, mirror dataset writable, MUSIC_MIRROR_INTERVAL=6h.
• CI runs the tests inside the image, against the ffmpeg that ships, on every pull request, and on merge publishes multi-arch (amd64 + arm64) to this Gitea's registry using the PACKAGES_SECRET repository secret. Versioning follows the same conventional-commit scheme as legacy-email-proxy, including writing the released version back into pyproject.toml so the packaging metadata cannot drift behind the tag.

Behaviour under Lidarr

Lidarr does this	The mirror does this
Replaces a file with a better rip	Re-encodes in place; same path in, same path out, so no duplicate
Upgrades MP3 to FLAC	Both map to the same .mp3 mirror path, so the old one is overwritten
Renames a track, album or artist folder	Old path pruned, new path encoded — correct, though it re-encodes rather than moving
Deletes an album or artist	Every orphaned mirror file goes, and the directories they emptied with them

Three defects were found and fixed while writing those tests, each verified to
fail against the previous code:

• Pruning probed the source tree for a mirror file's original name, lowercase
  extensions only. A .FLAC source was never found, so its mirror file was
  deleted as an orphan and rebuilt on the next pass, for ever. Pruning now works
  from the set of paths the pass actually accounted for.
• Two sources could claim one mirror path — 01 Song.flac beside a leftover
  01 Song.mp3. Both encoded to the same destination and each pass found the
  loser stale. The better format now wins, ties break on path.
• The source mtime was read after encoding rather than before, so a file still
  being written when the pass reached it could be stamped current while holding
  truncated audio.

Why there is no flake

The deployment target is a container. A flake here would sit on no path between
the source and the NAS, and nix flake check would test against nixpkgs' ffmpeg
while the artefact ships Debian's — precisely the layer these tests exercise. It
was removed in favour of running the suite inside the image. The sibling
legacy-email-proxy keeps its flake because there the flake is the deployment
mechanism.

Verification

• 21 tests, all real ffmpeg round-trips: layout, tag survival, MP3 output, skip-when-current, re-encode-on-change, orphan pruning, empty-dir removal, --no-prune, MP3 passthrough, --dry-run, --subdir, external cover art embedding, both refusal paths, and the Lidarr lifecycle cases below.
• The suite also runs in the multi-stage Docker test stage: 21 passed. The published runtime stage carries neither the tests nor pytest, verified by inspecting the image.
• Container built and run locally against a sample library: correct output path, Cyrillic tags intact.
• The release step was extracted from the workflow and run against a scratch repository: it commits and tags when the version changes, and skips the commit but still tags when pyproject.toml already carries it.
• Not tested against the real library — the first run there should be --dry-run.

Follow-ups, not in this PR

• Lidarr imports are picked up on the next scheduled pass rather than instantly. A webhook trigger is the obvious next step if six hours feels slow.
• PACKAGES_SECRET must exist as a repository secret before the first merge, or the login step fails.

## What A path-for-path MP3 mirror of a lossless library. FLAC in, MP3 out, same relative layout, tags and cover art carried across. Already-MP3 sources are copied rather than re-encoded. Mirror files whose source has gone are deleted, along with any directory they emptied. The source library is never written to — mounted read-only in the compose file, and never opened for writing in code. | Situation | Action | | -------------------------------- | ------------------------------------ | | No mirror file | encode | | Source modified since the mirror | re-encode (a Lidarr quality upgrade) | | Mirror up to date | skip | | Source is already MP3 | copy verbatim | | Source gone | delete, prune empty dirs | ## Design notes - **No database.** Freshness is mtime: an encode is stamped with its source's mtime, so a file is stale exactly when the two differ. Lidarr owns the library; a second tool with its own index would only fall out of step with it. This is also why it is not beets. - **Atomic writes.** Encode to a temp file, rename into place. An interrupted run cannot leave a truncated MP3 that the next run treats as finished. - **A lock file** in the mirror root stops two passes overlapping. - **Refuses a mirror inside the source tree**, which would otherwise recurse. - `--subdir` never prunes: a partial pass cannot distinguish an orphan from a file outside its scope. ## Shipping - Python package with a `music-mirror` console script, no runtime dependencies beyond ffmpeg. - `Dockerfile` plus `compose.yaml` as a TrueNAS Scale Custom App: source dataset read-only, mirror dataset writable, `MUSIC_MIRROR_INTERVAL=6h`. - CI runs the tests **inside the image**, against the ffmpeg that ships, on every pull request, and on merge publishes multi-arch (amd64 + arm64) to this Gitea's registry using the `PACKAGES_SECRET` repository secret. Versioning follows the same conventional-commit scheme as `legacy-email-proxy`, including writing the released version back into `pyproject.toml` so the packaging metadata cannot drift behind the tag. ## Behaviour under Lidarr | Lidarr does this | The mirror does this | | --------------------------------------- | -------------------------------------------------------------------------------- | | Replaces a file with a better rip | Re-encodes in place; same path in, same path out, so no duplicate | | Upgrades MP3 to FLAC | Both map to the same `.mp3` mirror path, so the old one is overwritten | | Renames a track, album or artist folder | Old path pruned, new path encoded — correct, though it re-encodes rather than moving | | Deletes an album or artist | Every orphaned mirror file goes, and the directories they emptied with them | Three defects were found and fixed while writing those tests, each verified to fail against the previous code: - Pruning probed the source tree for a mirror file's original name, lowercase extensions only. A `.FLAC` source was never found, so its mirror file was deleted as an orphan and rebuilt on the next pass, for ever. Pruning now works from the set of paths the pass actually accounted for. - Two sources could claim one mirror path — `01 Song.flac` beside a leftover `01 Song.mp3`. Both encoded to the same destination and each pass found the loser stale. The better format now wins, ties break on path. - The source mtime was read after encoding rather than before, so a file still being written when the pass reached it could be stamped current while holding truncated audio. ## Why there is no flake The deployment target is a container. A flake here would sit on no path between the source and the NAS, and `nix flake check` would test against nixpkgs' ffmpeg while the artefact ships Debian's — precisely the layer these tests exercise. It was removed in favour of running the suite inside the image. The sibling `legacy-email-proxy` keeps its flake because there the flake *is* the deployment mechanism. ## Verification - 21 tests, all real ffmpeg round-trips: layout, tag survival, MP3 output, skip-when-current, re-encode-on-change, orphan pruning, empty-dir removal, `--no-prune`, MP3 passthrough, `--dry-run`, `--subdir`, external cover art embedding, both refusal paths, and the Lidarr lifecycle cases below. - The suite also runs in the multi-stage Docker `test` stage: 21 passed. The published `runtime` stage carries neither the tests nor pytest, verified by inspecting the image. - Container built and run locally against a sample library: correct output path, Cyrillic tags intact. - The release step was extracted from the workflow and run against a scratch repository: it commits and tags when the version changes, and skips the commit but still tags when `pyproject.toml` already carries it. - **Not tested against the real library** — the first run there should be `--dry-run`. ## Follow-ups, not in this PR - Lidarr imports are picked up on the next scheduled pass rather than instantly. A webhook trigger is the obvious next step if six hours feels slow. - `PACKAGES_SECRET` must exist as a repository secret before the first merge, or the login step fails.

lyrathorpe added 2 commits 2026-08-21 14:45:03 +01:00

feat: mirror a lossless library to MP3 for iPod sync ... 177761bb75

Apple's Music app cannot read FLAC, so getting a lossless library onto an iPod
requires a converted copy somewhere. This keeps that copy beside the library on
a NAS rather than on a laptop, and keeps it current unattended.

Walks a source tree and reproduces it path for path as MP3: tags and cover art
carried across, already-MP3 sources copied rather than re-encoded, and mirror
files whose source has gone deleted along with any directories they emptied.
The source library is never written to.

Freshness is tracked by modification time -- an encoded file is stamped with
its source's mtime, so a file is stale exactly when the two differ. That keeps
runs idempotent without a database that could fall out of step with whatever
owns the library, which here is Lidarr.

Encodes go to a temporary file and are renamed into place, so an interrupted
run cannot leave a truncated MP3 that the next run mistakes for finished work.
A lock file in the mirror root prevents overlapping passes.

Ships as a Python package with a console script, a container image with a
TrueNAS Scale compose file, and a Nix flake providing the package, an overlay
and a dev shell. The test suite runs real ffmpeg encodes rather than mocks --
the failures worth catching are in what ffmpeg does with tags, cover art and
container formats.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

ci: publish the container image to the Gitea registry ... 82c9da6d62

Adopts the release scheme from legacy-email-proxy so both repositories behave
the same way: the version is derived from conventional commits since the last
v* tag, the image is pushed under the full version, the truncated major.minor
and major forms, and latest, and non-release builds are published as
sha-<short>. Multi-arch (amd64 for the NAS, arm64 so the same image runs on a
Pi). Authentication uses the PACKAGES_SECRET repository secret.

The release step also writes the computed version into pyproject.toml and
commits it as chore(release) before tagging, so the packaging metadata cannot
drift behind the release. It skips the commit when the file already carries
that version, which would otherwise fail the job after the image had been
pushed.

compose.yaml and the README now reference the published image instead of
instructing the NAS to build one locally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

lyrathorpe added 1 commit 2026-08-21 14:59:47 +01:00

fix: keep the mirror stable across Lidarr upgrades and renames ... 7a57e03c7b

Three defects in how the mirror tracked its source, all of which show up on a
library that something else reorganises.

Pruning probed the source tree for a mirror file's original name, trying each
known extension in turn. A source saved as .FLAC was never found, so its mirror
file was deleted as an orphan and re-encoded on the next pass, for ever.
Pruning now works from the set of paths the pass actually accounted for, which
cannot disagree with the walk over letter case or extension coverage.

Two sources could also claim one mirror path -- 01 Song.flac beside a leftover
01 Song.mp3, which is what an interrupted upgrade leaves behind. Both encoded
to the same destination, whichever finished last won the race, and every later
pass found the other one stale. The best-quality source now wins, ties break on
path, and the loser is logged.

The source mtime was read after encoding rather than before. A file still being
written when the pass reached it would be stamped with its final mtime while
holding truncated audio, and would never be revisited.

Adds regression tests for all three, plus the format-upgrade, album-rename and
whole-library-deletion cases, each verified to fail before the change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

lyrathorpe added 1 commit 2026-08-21 15:06:08 +01:00

build: drop the Nix packaging and test inside the container ... cbcbf30228

The deployment target is a container on TrueNAS Scale, so the flake sat on no
path between the source and the NAS. It was carried over from a sibling project
where the flake is the deployment mechanism; here it only added a second build
path and a second dependency pin.

Worse, it tested the wrong thing: `nix flake check` ran the suite against
nixpkgs' ffmpeg while the shipped artefact contains Debian's, and encoder and
muxer behaviour is exactly what these tests cover.

The Dockerfile gains a `test` stage that installs pytest and runs the suite
against the image's own ffmpeg; a failing test fails the build. CI runs
`docker build --target test` in place of the host-based Python setup, and the
push build states `target: runtime` so the published image is the lean stage
rather than the last one in the file. The runtime image carries neither the
tests nor pytest.

The release step now calls python3 rather than python, since setup-python is
no longer in the job to provide the alias.

Removes package.nix, flake.nix and flake.lock. A dev shell is a `nix shell`
away for anyone who wants one, and the README says so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

lyrathorpe added 1 commit 2026-08-21 15:21:14 +01:00

build: base the image on Alpine to drop 387 MB of graphics stack ... 81dce801ce

Debian's ffmpeg package depends on libavdevice, which can capture from and
render to X11, Wayland, SDL and OpenGL. That pulls in Mesa, and Mesa pulls in
LLVM for its software rasteriser -- 127 MB of it -- plus Z3 at 27 MB and a
speech synthesiser at 28 MB, in an image whose only job is to encode MP3s on a
headless NAS.

Alpine's ffmpeg brings none of that. The runtime image goes from 576 MB to
189 MB. Small X11 and Wayland client libraries remain, but they are kilobytes
rather than megabytes.

The test suite passes unchanged inside the new image, which is the point of
running it there: the encoder under test is now a musl build from a different
distribution, and the tests cover exactly the tag, cover art and container
behaviour that could have differed. Verified separately that a CBR 256 encode
comes out at the expected bitrate with non-ASCII tags intact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

lyrathorpe added 1 commit 2026-08-21 15:25:47 +01:00

build: publish amd64 only ... 6df53d7c42

The NAS is the only host this container runs on. Building linux/arm64 as well
meant emulating it under QEMU on every release for a consumer that does not
exist, so both the second platform and the QEMU setup step are dropped.

Nothing in the image is architecture-specific; restoring arm64 is a one-line
change if it ever gains a home on the Pi.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

lyrathorpe merged commit d4ccff3b75 into main 2026-08-21 15:30:59 +01:00

lyrathorpe deleted branch feat/initial-implementation 2026-08-21 15:31:00 +01:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

lyrathorpe

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lyrathorpe/music-mirror#1