lyrathorpeandEmma Thorpe
d4ccff3b75
## What A path-for-path MP3 mirror of a lossless library. FLAC in, MP3 out, same relative layout, tags and cover art carried across. Already-MP3 sources are copied rather than re-encoded. Mirror files whose source has gone are deleted, along with any directory they emptied. The source library is never written to — mounted read-only in the compose file, and never opened for writing in code. | Situation | Action | | -------------------------------- | ------------------------------------ | | No mirror file | encode | | Source modified since the mirror | re-encode (a Lidarr quality upgrade) | | Mirror up to date | skip | | Source is already MP3 | copy verbatim | | Source gone | delete, prune empty dirs | ## Design notes - **No database.** Freshness is mtime: an encode is stamped with its source's mtime, so a file is stale exactly when the two differ. Lidarr owns the library; a second tool with its own index would only fall out of step with it. This is also why it is not beets. - **Atomic writes.** Encode to a temp file, rename into place. An interrupted run cannot leave a truncated MP3 that the next run treats as finished. - **A lock file** in the mirror root stops two passes overlapping. - **Refuses a mirror inside the source tree**, which would otherwise recurse. - `--subdir` never prunes: a partial pass cannot distinguish an orphan from a file outside its scope. ## Shipping - Python package with a `music-mirror` console script, no runtime dependencies beyond ffmpeg. - `Dockerfile` plus `compose.yaml` as a TrueNAS Scale Custom App: source dataset read-only, mirror dataset writable, `MUSIC_MIRROR_INTERVAL=6h`. - CI runs the tests **inside the image**, against the ffmpeg that ships, on every pull request, and on merge publishes multi-arch (amd64 + arm64) to this Gitea's registry using the `PACKAGES_SECRET` repository secret. Versioning follows the same conventional-commit scheme as `legacy-email-proxy`, including writing the released version back into `pyproject.toml` so the packaging metadata cannot drift behind the tag. ## Behaviour under Lidarr | Lidarr does this | The mirror does this | | --------------------------------------- | -------------------------------------------------------------------------------- | | Replaces a file with a better rip | Re-encodes in place; same path in, same path out, so no duplicate | | Upgrades MP3 to FLAC | Both map to the same `.mp3` mirror path, so the old one is overwritten | | Renames a track, album or artist folder | Old path pruned, new path encoded — correct, though it re-encodes rather than moving | | Deletes an album or artist | Every orphaned mirror file goes, and the directories they emptied with them | Three defects were found and fixed while writing those tests, each verified to fail against the previous code: - Pruning probed the source tree for a mirror file's original name, lowercase extensions only. A `.FLAC` source was never found, so its mirror file was deleted as an orphan and rebuilt on the next pass, for ever. Pruning now works from the set of paths the pass actually accounted for. - Two sources could claim one mirror path — `01 Song.flac` beside a leftover `01 Song.mp3`. Both encoded to the same destination and each pass found the loser stale. The better format now wins, ties break on path. - The source mtime was read after encoding rather than before, so a file still being written when the pass reached it could be stamped current while holding truncated audio. ## Why there is no flake The deployment target is a container. A flake here would sit on no path between the source and the NAS, and `nix flake check` would test against nixpkgs' ffmpeg while the artefact ships Debian's — precisely the layer these tests exercise. It was removed in favour of running the suite inside the image. The sibling `legacy-email-proxy` keeps its flake because there the flake *is* the deployment mechanism. ## Verification - 21 tests, all real ffmpeg round-trips: layout, tag survival, MP3 output, skip-when-current, re-encode-on-change, orphan pruning, empty-dir removal, `--no-prune`, MP3 passthrough, `--dry-run`, `--subdir`, external cover art embedding, both refusal paths, and the Lidarr lifecycle cases below. - The suite also runs in the multi-stage Docker `test` stage: 21 passed. The published `runtime` stage carries neither the tests nor pytest, verified by inspecting the image. - Container built and run locally against a sample library: correct output path, Cyrillic tags intact. - The release step was extracted from the workflow and run against a scratch repository: it commits and tags when the version changes, and skips the commit but still tags when `pyproject.toml` already carries it. - **Not tested against the real library** — the first run there should be `--dry-run`. ## Follow-ups, not in this PR - Lidarr imports are picked up on the next scheduled pass rather than instantly. A webhook trigger is the obvious next step if six hours feels slow. - `PACKAGES_SECRET` must exist as a repository secret before the first merge, or the login step fails. --------- Co-authored-by: Emma Thorpe <emma.thorpe@citrix.com> Reviewed-on: #1
7.3 KiB
music-mirror
Maintain a lossy MP3 mirror of a lossless music library.
Walks a source library and reproduces it, path for path, as MP3 in a separate tree. Tags and cover art are carried across; sources that are already MP3 are copied rather than re-encoded; mirror files whose source has been deleted are removed. The source library is never written to — it is mounted read-only in the supplied compose file, and nothing in the code opens it for writing.
The intended use is an iPod. Apple's Music app cannot read FLAC at all, so a converted copy has to exist somewhere; this keeps that copy next to the library on a NAS instead of on a laptop, and keeps it current without a human remembering to do anything.
How it decides what to do
| Situation | Action |
|---|---|
| No mirror file | encode |
| Source modified since the mirror | re-encode (a Lidarr quality upgrade) |
| Mirror up to date | skip |
| Source is already MP3 | copy verbatim |
| Source gone | delete the mirror file, prune empty dirs |
Freshness is modification time: an encoded file is stamped with its source's mtime, so a file is stale exactly when the two differ. There is no database to fall out of step with the library, which matters when something else — Lidarr, in this case — is the thing that owns and reorganises it.
What that means for Lidarr
| Lidarr does this | The mirror does this |
|---|---|
| Replaces a file with a better rip | Re-encodes in place. Same path in, same path out, so no duplicate |
| Upgrades MP3 to FLAC | Both map to the same .mp3 mirror path, so the old one is overwritten |
| Renames a track, album or artist folder | Old path pruned, new path encoded. Correct, but it re-encodes rather than moving |
| Deletes an album or artist | Every orphaned mirror file is deleted and the emptied directories go too |
Pruning is driven by what the pass actually found, not by guessing source
filenames from mirror ones: a .FLAC source would not be found by a search for
.flac, and the mirror file would be deleted and rebuilt on alternate passes
for ever.
If two sources want the same mirror path — a 01 Song.flac next to a leftover
01 Song.mp3, which is what an interrupted upgrade leaves — the better format
wins, ties break on path, and the loser is logged. Without that rule both
encode to the same destination and every pass finds one of them stale.
The mtime is read before encoding rather than after. A file still being written when the pass reaches it would otherwise be stamped with its final mtime while holding truncated audio, and never be revisited.
Encodes are written to a temporary file and renamed into place, so an interrupted run cannot leave a truncated MP3 that the next run mistakes for finished work. A lock file in the mirror root stops two passes overlapping.
Usage
music-mirror --source /music --mirror /music-mp3 # one pass
music-mirror --source /music --mirror /music-mp3 --interval 6h # keep running
music-mirror --source /music --mirror /music-mp3 --dry-run # report only
music-mirror --source /music --mirror /music-mp3 --subdir "Artist/Album"
| Option | Environment variable | Default | Meaning |
|---|---|---|---|
--source |
MUSIC_MIRROR_SOURCE |
— | Root of the lossless library, read-only |
--mirror |
MUSIC_MIRROR_MIRROR |
— | Root of the MP3 mirror |
--quality |
MUSIC_MIRROR_QUALITY |
V0 |
LAME VBR level V0–V9, or kbps e.g. 256 |
--jobs |
MUSIC_MIRROR_JOBS |
CPU count | Concurrent encodes |
--interval |
MUSIC_MIRROR_INTERVAL |
unset | Repeat forever, e.g. 45m, 6h, 1d |
--subdir |
— | unset | Limit the pass to one directory; skips pruning |
--no-prune |
— | off | Keep mirror files whose source has gone |
--dry-run |
— | off | Report what would change, write nothing |
--subdir never prunes: a partial pass cannot tell an orphan from a file
outside its own scope.
Requires ffmpeg and ffprobe on PATH. The container image provides both.
Running it on TrueNAS Scale
compose.yaml is a Custom App definition. Adjust the two host paths and the
user: to match your pool, then add it as a custom app. The image is published
to this Gitea's registry on every release:
code.emmathe.dev/lyrathorpe/music-mirror:latest
Tags are latest, the full version, and the truncated major.minor and
major forms; builds that are not releases are published as sha-<short>.
Point the mirror at its own dataset rather than a directory inside the music dataset — it is derived data, so it wants its own snapshot policy, its own quota, and its own SMB share. The tool refuses to run with a mirror inside the source tree.
New Lidarr imports are picked up on the next pass. With MUSIC_MIRROR_INTERVAL
at 6h that is the worst case; run --subdir by hand if you want an album
immediately.
Tests
docker build --target test . # what CI runs
pytest # needs ffmpeg and pytest on PATH
The suite runs real ffmpeg encodes rather than mocking them. The interesting
failures are in what ffmpeg actually does with tags, cover art and container
formats, and a mock cannot fail that way — which is also why CI runs the tests
inside the image, against the ffmpeg that ships, rather than against whatever
the build runner provides. The published image is the runtime stage and
carries neither the tests nor pytest.
Run them directly instead if you prefer; they skip when ffmpeg is absent. On a Nix machine:
nix shell nixpkgs#python3Packages.pytest nixpkgs#ffmpeg -c pytest
Getting the result onto an iPod
The mirror is just a directory of MP3s, so any client will do:
- macOS. Add the mirror's SMB share to the Music app with Copy files to
Music Media folder and Keep Media folder organised both off. The Mac
then stores a library database and nothing else. Keep the share mounted at a
stable path — if it is missing when Music opens, every track shows
!. - Linux. Rhythmbox links
libgpodand handles iPod sync. An iPod Video (5th generation) predates the models whose database has to be signed, so no firmware-hash trickery is needed.
Neither client transcodes at sync time; they copy finished MP3s.
Two device-side details worth knowing: the iPod reads cover art from the file's
tags and ignores folder.jpg, which is why art is embedded here; and volume
levelling on the device uses iTunes' Soundcheck tag, not ReplayGain, so
ReplayGain tags in the source are not carried over as such.