Emma Thorpe
d5f67c6de5
Build and publish container / build (pull_request) Successful in 2m16s
Rockbox's MAX_PATH is 260, defined in firmware/include/fs_defines.h and used to size the directory entry buffer in dir.h. It bounds the path as the device sees it, so the directory the mirror is copied into spends part of the same budget; --device-prefix accounts for that and defaults to /Music. Over-budget paths are shortened from the deepest component outward. The track name carries the least navigational value and the artist directory the most, so the filename is cut first and the artist only if nothing else will serve. A shortened component keeps its extension and gains four hex digits of the original name: two long titles sharing a prefix cut to the same string otherwise, and a silent collision between two tracks is a worse outcome than an ugly filename. The result is stable. The same source always yields the same shortened name, so one pass does not rename what the last one wrote -- an unstable scheme would churn the whole mirror every six hours. A path too deeply nested to fit without reducing every component to nonsense is left alone and reported rather than mangled. Migration now tries more than one previous naming, because there is more than one. A mirror already running with --fat32-safe holds sanitised but unshortened paths, and matching only the original unsanitised name would have re-encoded every one of them instead of moving it. The checker gains the same two options, since it was measuring the mirror-relative path against a limit that applies to the device-absolute one, and so under-reported by the length of the destination directory.
312 lines
16 KiB
Markdown
312 lines
16 KiB
Markdown
# music-mirror
|
||
|
||
Maintain a lossy MP3 mirror of a lossless music library.
|
||
|
||
Walks a source library and reproduces it, path for path, as MP3 in a separate
|
||
tree. Tags and cover art are carried across; sources that are already MP3 are
|
||
copied rather than re-encoded; mirror files whose source has been deleted are
|
||
removed. **The source library is never written to** — it is mounted read-only
|
||
in the supplied compose file, and nothing in the code opens it for writing.
|
||
|
||
The intended use is an iPod. Apple's Music app cannot read FLAC at all, so a
|
||
converted copy has to exist somewhere; this keeps that copy next to the
|
||
library on a NAS instead of on a laptop, and keeps it current without a human
|
||
remembering to do anything.
|
||
|
||
## How it decides what to do
|
||
|
||
| Situation | Action |
|
||
| -------------------------------- | ---------------------------------------- |
|
||
| No mirror file | encode |
|
||
| Source modified since the mirror | re-encode (a Lidarr quality upgrade) |
|
||
| Mirror up to date | skip |
|
||
| Source is already MP3 | copy verbatim |
|
||
| Source gone | delete the mirror file, prune empty dirs |
|
||
|
||
Freshness is modification time: an encoded file is stamped with its source's
|
||
mtime, so a file is stale exactly when the two differ. There is no database to
|
||
fall out of step with the library, which matters when something else — Lidarr,
|
||
in this case — is the thing that owns and reorganises it.
|
||
|
||
### What that means for Lidarr
|
||
|
||
| Lidarr does this | The mirror does this |
|
||
| --------------------------------------- | -------------------------------------------------------------------------------- |
|
||
| Replaces a file with a better rip | Re-encodes in place. Same path in, same path out, so no duplicate |
|
||
| Upgrades MP3 to FLAC | Both map to the same `.mp3` mirror path, so the old one is overwritten |
|
||
| Renames a track, album or artist folder | Old path pruned, new path encoded. Correct, but it re-encodes rather than moving |
|
||
| Deletes an album or artist | Every orphaned mirror file is deleted and the emptied directories go too |
|
||
|
||
Pruning is driven by what the pass actually found, not by guessing source
|
||
filenames from mirror ones: a `.FLAC` source would not be found by a search for
|
||
`.flac`, and the mirror file would be deleted and rebuilt on alternate passes
|
||
for ever.
|
||
|
||
If two sources want the same mirror path — a `01 Song.flac` next to a leftover
|
||
`01 Song.mp3`, which is what an interrupted upgrade leaves — the better format
|
||
wins, ties break on path, and the loser is logged. Without that rule both
|
||
encode to the same destination and every pass finds one of them stale.
|
||
|
||
The mtime is read _before_ encoding rather than after. A file still being
|
||
written when the pass reaches it would otherwise be stamped with its final
|
||
mtime while holding truncated audio, and never be revisited.
|
||
|
||
Both encodes and copies are written to a temporary file and renamed into place,
|
||
so an interrupted run cannot leave a truncated MP3 that the next run mistakes
|
||
for finished work. Copies need it as much as encodes do: the mtime comes across
|
||
with the bytes, so a half-written copy would look current for ever. A lock file
|
||
in the mirror root stops two passes overlapping.
|
||
|
||
### Permissions
|
||
|
||
Everything written into the mirror is made group-readable, and its directories
|
||
group-traversable, so the mirror can be read back by whatever serves it. Neither
|
||
writer does that unaided: the temporary file an encode renames into place is
|
||
created `0600` regardless of the umask, and a straight copy of an existing MP3
|
||
inherits the mode of a source file in a library this tool does not own.
|
||
|
||
Directories are handled by clearing the owner and group read/execute bits from
|
||
the process umask, once, at startup. Owner as well as group, because a umask
|
||
carrying `0400` produces directories of mode `0300` — writable and enterable,
|
||
unreadable to the very run that created them. The `other` bits are left where
|
||
the umask puts them: whether the mirror is world-readable is a genuine policy
|
||
question, and so is its ownership.
|
||
|
||
Mirror files written before this existed are topped up on the next pass. Their
|
||
mtimes are correct, so nothing else would revisit them — and they are not
|
||
re-encoded, only chmod'ed.
|
||
|
||
## Usage
|
||
|
||
```sh
|
||
music-mirror --source /music --mirror /music-mp3 # one pass
|
||
music-mirror --source /music --mirror /music-mp3 --interval 6h # keep running
|
||
music-mirror --source /music --mirror /music-mp3 --dry-run # report only
|
||
music-mirror --source /music --mirror /music-mp3 --subdir "Artist/Album"
|
||
```
|
||
|
||
| Option | Environment variable | Default | Meaning |
|
||
| ------------ | ----------------------- | --------- | ---------------------------------------------- |
|
||
| `--source` | `MUSIC_MIRROR_SOURCE` | — | Root of the lossless library, read-only |
|
||
| `--mirror` | `MUSIC_MIRROR_MIRROR` | — | Root of the MP3 mirror |
|
||
| `--quality` | `MUSIC_MIRROR_QUALITY` | `V0` | LAME VBR level `V0`–`V9`, or kbps e.g. `256` |
|
||
| `--jobs` | `MUSIC_MIRROR_JOBS` | CPU count | Concurrent encodes |
|
||
| `--interval` | `MUSIC_MIRROR_INTERVAL` | unset | Repeat forever, e.g. `45m`, `6h`, `1d` |
|
||
| `--subdir` | — | unset | Limit the pass to one directory; skips pruning |
|
||
| `--fat32-safe` | `MUSIC_MIRROR_FAT32_SAFE` | off | Name files so a FAT32 device accepts them |
|
||
| `--no-prune` | — | off | Keep mirror files whose source has gone |
|
||
| `--dry-run` | — | off | Report what would change, write nothing |
|
||
|
||
`--subdir` never prunes: a partial pass cannot tell an orphan from a file
|
||
outside its own scope.
|
||
|
||
### Concurrency
|
||
|
||
LAME is single-threaded — ffmpeg reports `Threading capabilities: none` for
|
||
`libmp3lame` — so throughput comes entirely from running several encoders at
|
||
once, one process per file. `--jobs` defaults to the CPUs the process may
|
||
actually use, which inside a container means the `cpus:` allowance rather than
|
||
the host's core count. Each pass logs the number it settled on.
|
||
|
||
As a rough guide, a Zen 3 core encodes about 40–60× realtime at V0 depending on
|
||
clock, so six cores clear roughly 250 hours of audio per hour of wall clock.
|
||
The first full pass is the expensive one; after that only new and changed files
|
||
are touched. Lower `MUSIC_MIRROR_JOBS` if you would rather the NAS stayed
|
||
responsive than finished sooner.
|
||
|
||
Requires `ffmpeg` and `ffprobe` on `PATH`. The container image provides both.
|
||
|
||
## Running it on TrueNAS Scale
|
||
|
||
`compose.yaml` is a Custom App definition. Adjust the two host paths and the
|
||
`user:` to match your pool, then add it as a custom app. The image is published
|
||
to this Gitea's registry on every release:
|
||
|
||
```
|
||
code.emmathe.dev/lyrathorpe/music-mirror:latest
|
||
```
|
||
|
||
Tags are `latest`, the full version, and the truncated `major.minor` and
|
||
`major` forms; builds that are not releases are published as `sha-<short>`.
|
||
|
||
Point the mirror at its own dataset rather than a directory inside the music
|
||
dataset — it is derived data, so it wants its own snapshot policy, its own
|
||
quota, and its own SMB share. The tool refuses to run with a mirror inside the
|
||
source tree.
|
||
|
||
New Lidarr imports are picked up on the next pass. With `MUSIC_MIRROR_INTERVAL`
|
||
at `6h` that is the worst case; run `--subdir` by hand if you want an album
|
||
immediately.
|
||
|
||
## Tools
|
||
|
||
Host-side scripts under `tools/`, not part of the container image.
|
||
|
||
`sync-to-ipod.sh` does a whole transfer: submits the scrobbler log, checks the
|
||
mirror, rsyncs, syncs and unmounts.
|
||
|
||
```sh
|
||
tools/sync-to-ipod.sh /mnt/tank/media/music-mp3 /media/IPOD/Music
|
||
tools/sync-to-ipod.sh -n /mnt/tank/media/music-mp3 /media/IPOD/Music # dry run
|
||
```
|
||
|
||
It refuses to start unless the destination is a mounted FAT filesystem that is
|
||
its own mount point, because `--delete` aimed at the wrong directory empties it
|
||
and does not announce itself. It also excludes `/.rockbox`, the scrobbler logs
|
||
and the various filesystem metadata directories from deletion — the mirror does
|
||
not contain them, and without the exclusion a sync to the card root would
|
||
remove the Rockbox install.
|
||
|
||
The unmount is the point of doing this in a script. FAT32 has no journal and
|
||
the device is reached through disk mode, so an interrupted write is corruption
|
||
that needs `fsck.vfat` from another machine.
|
||
|
||
`submit_scrobbles.py` sends the Rockbox scrobbler log to Last.fm and sets it
|
||
aside. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
|
||
tab-separated line per track rated `L` for listened or `S` for skipped; only
|
||
the listened ones are sent. It runs **before** the copy, since the plays
|
||
already happened and a failed transfer is no reason to lose them.
|
||
|
||
Two things are unlike every other Last.fm call in these projects. Scrobbling is
|
||
a *write* method, so it needs `LASTFM_API_SECRET` and a session key obtained
|
||
once through the browser, not just the read-only key. And on a target with no
|
||
real-time clock Rockbox writes `/.scrobbler-timeless.log` with every timestamp
|
||
set to zero; those are counted and reported but never submitted, because
|
||
scrobbling them would mean inventing when they happened.
|
||
|
||
The log is renamed rather than deleted once accepted. If Last.fm quietly
|
||
dropped something, the evidence is still on the device.
|
||
|
||
`check_fat32.py` reports paths a FAT32 device will not accept — reserved
|
||
characters, trailing dots and spaces, over-long components and paths, and names
|
||
colliding case-insensitively. Run it against the mirror **before** an rsync:
|
||
rsync reports the failures too, but scattered through fifty thousand files where
|
||
they are easy to miss. Exits non-zero when it finds anything, so it can gate a
|
||
script.
|
||
|
||
## Tests
|
||
|
||
```sh
|
||
docker build --target test . # what CI runs
|
||
pytest # needs ffmpeg and pytest on PATH
|
||
```
|
||
|
||
The suite runs real ffmpeg encodes rather than mocking them. The interesting
|
||
failures are in what ffmpeg actually does with tags, cover art and container
|
||
formats, and a mock cannot fail that way — which is also why CI runs the tests
|
||
_inside the image_, against the ffmpeg that ships, rather than against whatever
|
||
the build runner provides. The published image is the `runtime` stage and
|
||
carries neither the tests nor pytest.
|
||
|
||
Run them directly instead if you prefer; they skip when ffmpeg is absent. On a
|
||
Nix machine:
|
||
|
||
```sh
|
||
nix shell nixpkgs#python3Packages.pytest nixpkgs#ffmpeg -c pytest
|
||
```
|
||
|
||
## FAT32 and Rockbox
|
||
|
||
`--fat32-safe` names mirror files so a FAT32 device will accept them. Off by
|
||
default, because turning it on renames files and that should be a decision
|
||
rather than a surprise.
|
||
|
||
What it handles, per path component:
|
||
|
||
| Problem | Treatment |
|
||
| ------------------------------- | ------------------------------ |
|
||
| `< > : " \ \| ? *` and control characters | replaced with `_` |
|
||
| trailing dots and spaces | stripped — FAT eats them silently, so the name round-trips as a different name |
|
||
| a component left empty | becomes `_` |
|
||
| names differing only in case | detected and reported; one wins, as with any other collision |
|
||
|
||
`Dada Life - Kick Out the Epic Motherf**ker` is a real example from a real
|
||
library. Without this it simply never arrives on the device.
|
||
|
||
**Turning it on does not re-encode anything.** Every track whose name held a
|
||
reserved character changes path, and re-encoding those would be hours of work
|
||
producing files that already exist byte for byte. The run moves them instead,
|
||
and says so. Prune then finds nothing to remove because nothing was left
|
||
behind.
|
||
|
||
Renames are counted apart from encodes in the pass summary, and `--dry-run`
|
||
reports `would rename` rather than `would encode` — the difference between the
|
||
two is a minute against an afternoon, so a preview that conflated them would be
|
||
worse than no preview. A dry run also does not list the pre-rename files as
|
||
orphans: nothing was moved, so they are still there, but they are what a real
|
||
run would move rather than what it would delete.
|
||
|
||
### Path length
|
||
|
||
Rockbox's `MAX_PATH` is 260, from `firmware/include/fs_defines.h`, and it bounds
|
||
the path *as the device sees it*. The directory the mirror is copied into comes
|
||
out of the same budget, so `--device-prefix` (default `/Music`) is subtracted
|
||
from `--max-path` to get what a mirror-relative path may spend.
|
||
|
||
Over-budget paths are shortened from the **deepest component outward**: the
|
||
track name carries the least navigational value and the artist directory the
|
||
most, so the filename goes first and the artist is touched only if nothing else
|
||
will do. A shortened component keeps its extension and gains four hex digits of
|
||
the original name — two long names sharing a prefix would otherwise cut to the
|
||
same string, and a silent collision between two tracks is worse than an ugly
|
||
filename.
|
||
|
||
The result is stable: the same source always produces the same shortened name,
|
||
so a pass does not rename what the previous pass wrote. A path too deeply
|
||
nested to fit without reducing every component to nonsense is left alone and
|
||
reported instead.
|
||
|
||
### Album art
|
||
|
||
Rockbox looks for cover art **on the filesystem** — `cover.jpg`, `folder.jpg`
|
||
and friends beside the track or in its parent — and that search never touches
|
||
the picture embedded in the tag. So a JPEG cover found beside the source is now
|
||
copied into the mirror as `cover.jpg`, in addition to being embedded. The iPod
|
||
firmware reads the embedded one; Rockbox reads the file. Both are satisfied.
|
||
|
||
A cover left behind in a directory whose tracks have all gone is pruned, or the
|
||
directory would never look empty and never be removed.
|
||
|
||
## Getting the result onto an iPod
|
||
|
||
The mirror is just a directory of MP3s, so any client will do:
|
||
|
||
- **macOS.** Add the mirror's SMB share to the Music app with _Copy files to
|
||
Music Media folder_ and _Keep Media folder organised_ both **off**. The Mac
|
||
then stores a library database and nothing else. Keep the share mounted at a
|
||
stable path — if it is missing when Music opens, every track shows `!`.
|
||
- **Linux.** Rhythmbox links `libgpod` and handles iPod sync. An iPod Video
|
||
(5th generation) predates the models whose database has to be signed, so no
|
||
firmware-hash trickery is needed.
|
||
- **Rockbox.** No database to write at all — it reads a plain directory tree
|
||
and builds its own index from tags. Copy the mirror across with rsync:
|
||
|
||
```sh
|
||
python3 tools/check_fat32.py /mnt/tank/media/music-mp3 # before, not during
|
||
rsync -rtv --delete --modify-window=2 \
|
||
/mnt/tank/media/music-mp3/ /media/IPOD/Music/
|
||
```
|
||
|
||
`--modify-window=2` because FAT stores modification times to two-second
|
||
resolution; without it rsync re-copies the entire library on every run. `-rt`
|
||
rather than `-a` because owners, groups and permissions mean nothing on FAT
|
||
and asking for them only produces errors.
|
||
|
||
Do not route this through Rhythmbox. Its `rb_ipod_helpers_is_ipod()` reads
|
||
`access-protocols` from media-player-info first and returns true without
|
||
looking at the filesystem at all, so an iPod in disk mode is identified by its
|
||
USB id and managed as an iPod — writing a database Rockbox does not want.
|
||
Deleting `iPod_Control` does not change this. Either use rsync, or untick
|
||
Preferences → Plugins → Portable Players - iPod.
|
||
|
||
Faster still for the first bulk copy: take the card out of the iFlash adapter
|
||
and use a card reader. Fifty thousand files over USB 2.0 through an iPod is a
|
||
long evening, and it avoids Rockbox's USB stack entirely.
|
||
|
||
Neither client transcodes at sync time; they copy finished MP3s.
|
||
|
||
Two device-side details worth knowing: the iPod reads cover art from the file's
|
||
tags and ignores `folder.jpg`, which is why art is embedded here; and volume
|
||
levelling on the device uses iTunes' Soundcheck tag, not ReplayGain, so
|
||
ReplayGain tags in the source are not carried over as such.
|