The option table listed it, but compose.yaml is what actually gets copied into
a TrueNAS Custom App, so an option absent from there is an option nobody finds.
Quoted deliberately: an unquoted yes or true is a YAML 1.1 boolean, and compose
rejects a boolean as an environment value outright.
tools/sync-to-ipod.sh does the whole transfer to a Rockbox device, so the only
manual part left is the disk-mode button sequence.
The guards are the substance rather than decoration. rsync --delete is being
aimed at a whole filesystem, so the destination must exist, be its own mount
point, and be a FAT filesystem; the mirror must be non-empty and must not be
the destination. Emptying the wrong directory is not a mistake that announces
itself.
It also excludes /.rockbox, the scrobbler logs and the usual filesystem
metadata directories. The mirror does not contain them, so a sync to the card
root would otherwise have deleted the Rockbox installation -- which the first
draft of this script would have done.
The unmount is why this is a script at all. FAT32 has no journal, the device is
reached through the Apple firmware's disk mode because Rockbox's own mass
storage is unreliable on an iFlash, and an interrupted write is corruption that
needs fsck.vfat from another machine.
tools/submit_scrobbles.py sends the Rockbox scrobbler log to Last.fm and sets
it aside. Rockbox writes it in AUDIOSCROBBLER 1.1: tab-separated, one line per
track, rated L for listened or S for skipped, and only the listened ones are a
play. It runs before the copy, because the plays already happened and a failed
transfer is no reason to lose them as well.
Two things there differ from every other Last.fm call in these projects.
Scrobbling is a write method, so it needs the API secret and a session key
obtained once through the browser rather than the read-only key. And a target
with no real-time clock gets /.scrobbler-timeless.log with every timestamp set
to zero; those are counted and reported but never sent, since submitting them
would mean inventing when they happened.
Signature generation sorts parameter names by the ASCII table rather than
numerically, so artist[10] precedes artist[1]. Sorting them the obvious way
produces an invalid signature and no other symptom, so there is a test for it.
The log is renamed rather than deleted once accepted, so that if Last.fm
quietly dropped something the evidence is still on the device.
Two changes for playing the mirror on a Rockbox iPod, where the device is FAT32
and Rockbox reads a plain directory tree rather than a database.
--fat32-safe names mirror files acceptably: the reserved characters and control
characters become underscores, trailing dots and spaces are stripped because
FAT eats them silently and the name then round-trips as a different one, and a
component left empty becomes an underscore. Names differing only in case are
detected as collisions, since two files here are one file there and the second
would silently overwrite the first. "Kick Out the Epic Motherf**ker" is a real
example from a real library, and without this it simply never arrives.
Off by default. It renames files, and that should be a decision rather than a
surprise on somebody's next pass.
Turning it on does not re-encode anything. Every track whose name held a
reserved character changes path, and encoding those again would be hours of
work producing files that already exist byte for byte, so the run moves them
instead and logs each one. Prune then finds nothing left behind.
Album art is now also copied into the mirror as cover.jpg beside the tracks.
Rockbox searches the filesystem for art -- cover.jpg, folder.jpg and the rest,
in the track's directory or its parent -- and that search never looks at the
picture embedded in the tag, so a mirror that only embeds art displays none of
it on the device. Embedding continues for the Apple firmware; both are now
satisfied. A cover whose tracks have all been pruned is removed too, or its
directory would never look empty and never go.
Adds tools/check_fat32.py, which reports unacceptable paths before a copy
rather than during one: rsync reports them too, but scattered through fifty
thousand files where they are easy to lose. It exits non-zero so it can gate a
script.
The README documents the rsync invocation, including why --modify-window=2 is
required against FAT and why Rhythmbox must be kept out of the transfer --
rb_ipod_helpers_is_ipod() reads access-protocols from media-player-info and
returns true on the USB id alone, without looking at the filesystem, so
removing iPod_Control changes nothing.
Runs take fifteen to eighteen minutes, and the log shows why: the image is
built twice, in full.
The test stage is built by the runner's docker daemon. The runtime stage was
then built by docker/build-push-action, which runs under a buildx builder that
setup-buildx-action creates in its own container with its own cache. The two
share nothing, so the second build spent seventy-six seconds booting buildkit
and then installed ffmpeg and the package all over again -- around a hundred
and ten seconds for the apk and another hundred for pip, neither of which
produced anything the first build had not already made. The comment above the
test step claimed those layers were shared, which is what made this look
reasonable.
buildx earns that overhead when producing several architectures. This produces
linux/amd64 only, by an explicit decision recorded in the workflow, so it earns
nothing here. Use plain docker build against the same daemon that ran the
tests, and push with docker push. The runtime stage is a strict prefix of the
test stage, so every layer is a cache hit: measured at 1.3 seconds locally.
Identical to the change made in music-curator, whose workflow this one was
copied from.
The umask handling added for group access cleared only the group bits and left
owner and other to the environment. A container whose umask carries 0400 then
produces mirror directories of mode 0300: writable and enterable, unreadable to
the very run that created them, and unreadable to anything serving the share.
Clear the owner read and execute bits from the umask as well. The `other` bits
stay where the environment puts them, because whether the mirror is
world-readable is a real policy question; being able to read a directory the
process itself just created is not.
Files were never exposed to this: mkstemp sets 0600 outright and copy2 takes
the source file's mode, both ignoring the umask.
Copies of already-MP3 sources were written straight to their destination while
encodes went via a temporary file and a rename. A copy interrupted by a full
disk, a killed container or an I/O error therefore left a truncated MP3 in the
mirror -- and because shutil.copy2 reproduces the source's mtime along with its
bytes, staleness detection would read that fragment as up to date and never
replace it. The damage is silent and permanent until someone plays the track.
Give copy the same temporary-file-and-rename path encode already uses, so the
destination either has the whole file or has nothing.
Explain why the group bits are set explicitly rather than left to the umask,
what is deliberately not touched, and that an existing mirror is repaired in
place rather than re-encoded.
The mirror is written by one account and read by another -- an SMB share, or
whatever else serves it -- but nothing here produced a group-readable file.
Encodes go through `tempfile.mkstemp`, which creates 0600 regardless of the
umask and keeps that mode through the rename into place, so every encoded
track landed unreadable. Copies of existing MP3s inherit the mode of a source
file in a library this tool does not own, which may be no better.
Add the group-read bit explicitly: to the temporary file before it is renamed,
so a mirror file is never visible without it, and to a copy once it has landed.
Directories are handled by clearing the group bits from the process umask
rather than chmod'ing each one, since a file the group cannot reach is no more
useful than one it cannot read. Only the group bits are touched; the world bits
and ownership stay with the umask as before.
Mirror files written before this are repaired on the next pass. Their mtimes
are correct, so no other part of the pass would revisit them, and topping up
the mode costs a stat rather than a re-encode.
libmp3lame is single-threaded, so concurrency is one ffmpeg process per file
and the pool width is the whole of it. The width came from os.cpu_count(),
which reports the host's cores and ignores a container's cpus: allowance -- on
a 12-thread host limited to 4 CPUs that is threefold oversubscription, which
costs context switches and NAS responsiveness for no throughput.
The default now reads the cgroup v2 quota, falling back to process CPU
affinity and then to the host count. Each pass logs the number it chose.
Also stops probing every file for embedded cover art. The probe only changes
the command when a cover file sits beside the track, so ask only then; with no
cover file, -map 0:v:0? already carries embedded art if there is any. Worth
roughly 30 ms per track against about 4 s of encoding, so this is tidiness
rather than a speed-up. The per-directory cover lookup is cached alongside.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The login step referenced PACKAGES_SECRET while the secret configured on this
repository -- and on its sibling -- is PACKAGES_TOKEN. The expression resolved
to an empty string and docker/login-action failed with "Password required",
so the first release published nothing and created no tag.
The push trigger only fires on image-affecting paths, which a workflow-only
change is not, so merging this fix alone would not produce a release.
workflow_dispatch can now cut one, restricted to main so a manual run on
another branch cannot tag a commit that is not on the default branch.
Verified by running the version step across every event and ref combination:
push and workflow_dispatch on main release, everything else does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What
A path-for-path MP3 mirror of a lossless library. FLAC in, MP3 out, same relative layout, tags and cover art carried across. Already-MP3 sources are copied rather than re-encoded. Mirror files whose source has gone are deleted, along with any directory they emptied. The source library is never written to — mounted read-only in the compose file, and never opened for writing in code.
| Situation | Action |
| -------------------------------- | ------------------------------------ |
| No mirror file | encode |
| Source modified since the mirror | re-encode (a Lidarr quality upgrade) |
| Mirror up to date | skip |
| Source is already MP3 | copy verbatim |
| Source gone | delete, prune empty dirs |
## Design notes
- **No database.** Freshness is mtime: an encode is stamped with its source's mtime, so a file is stale exactly when the two differ. Lidarr owns the library; a second tool with its own index would only fall out of step with it. This is also why it is not beets.
- **Atomic writes.** Encode to a temp file, rename into place. An interrupted run cannot leave a truncated MP3 that the next run treats as finished.
- **A lock file** in the mirror root stops two passes overlapping.
- **Refuses a mirror inside the source tree**, which would otherwise recurse.
- `--subdir` never prunes: a partial pass cannot distinguish an orphan from a file outside its scope.
## Shipping
- Python package with a `music-mirror` console script, no runtime dependencies beyond ffmpeg.
- `Dockerfile` plus `compose.yaml` as a TrueNAS Scale Custom App: source dataset read-only, mirror dataset writable, `MUSIC_MIRROR_INTERVAL=6h`.
- CI runs the tests **inside the image**, against the ffmpeg that ships, on every pull request, and on merge publishes multi-arch (amd64 + arm64) to this Gitea's registry using the `PACKAGES_SECRET` repository secret. Versioning follows the same conventional-commit scheme as `legacy-email-proxy`, including writing the released version back into `pyproject.toml` so the packaging metadata cannot drift behind the tag.
## Behaviour under Lidarr
| Lidarr does this | The mirror does this |
| --------------------------------------- | -------------------------------------------------------------------------------- |
| Replaces a file with a better rip | Re-encodes in place; same path in, same path out, so no duplicate |
| Upgrades MP3 to FLAC | Both map to the same `.mp3` mirror path, so the old one is overwritten |
| Renames a track, album or artist folder | Old path pruned, new path encoded — correct, though it re-encodes rather than moving |
| Deletes an album or artist | Every orphaned mirror file goes, and the directories they emptied with them |
Three defects were found and fixed while writing those tests, each verified to
fail against the previous code:
- Pruning probed the source tree for a mirror file's original name, lowercase
extensions only. A `.FLAC` source was never found, so its mirror file was
deleted as an orphan and rebuilt on the next pass, for ever. Pruning now works
from the set of paths the pass actually accounted for.
- Two sources could claim one mirror path — `01 Song.flac` beside a leftover
`01 Song.mp3`. Both encoded to the same destination and each pass found the
loser stale. The better format now wins, ties break on path.
- The source mtime was read after encoding rather than before, so a file still
being written when the pass reached it could be stamped current while holding
truncated audio.
## Why there is no flake
The deployment target is a container. A flake here would sit on no path between
the source and the NAS, and `nix flake check` would test against nixpkgs' ffmpeg
while the artefact ships Debian's — precisely the layer these tests exercise. It
was removed in favour of running the suite inside the image. The sibling
`legacy-email-proxy` keeps its flake because there the flake *is* the deployment
mechanism.
## Verification
- 21 tests, all real ffmpeg round-trips: layout, tag survival, MP3 output, skip-when-current, re-encode-on-change, orphan pruning, empty-dir removal, `--no-prune`, MP3 passthrough, `--dry-run`, `--subdir`, external cover art embedding, both refusal paths, and the Lidarr lifecycle cases below.
- The suite also runs in the multi-stage Docker `test` stage: 21 passed. The published `runtime` stage carries neither the tests nor pytest, verified by inspecting the image.
- Container built and run locally against a sample library: correct output path, Cyrillic tags intact.
- The release step was extracted from the workflow and run against a scratch repository: it commits and tags when the version changes, and skips the commit but still tags when `pyproject.toml` already carries it.
- **Not tested against the real library** — the first run there should be `--dry-run`.
## Follow-ups, not in this PR
- Lidarr imports are picked up on the next scheduled pass rather than instantly. A webhook trigger is the obvious next step if six hours feels slow.
- `PACKAGES_SECRET` must exist as a repository secret before the first merge, or the login step fails.
---------
Co-authored-by: Emma Thorpe <emma.thorpe@citrix.com>
Reviewed-on: #1
lyrathorpe
lyrathorpe