Emma Thorpe
284069d9eb
--dry-run described a FAT32 rename as an encode. process() skipped the rename whenever dry_run was set, then found no file at the target and fell through to the encode path, so a preview of enabling --fat32-safe announced a full re-encode of every track whose name held a reserved character. The real run moves those files in a moment. A preview that inverts the cost of the thing being previewed is worse than no preview at all. Prune compounded it. With nothing renamed, the pre-sanitisation files are still on disk, and they were reported as orphans due for deletion -- so the same dry run claimed the library would be re-encoded and the originals thrown away, neither of which is true. Renames are now their own outcome: reported as "would rename" in a dry run, counted separately from encodes in the pass summary, and excluded from the orphan list when a dry run leaves them in place.
292 lines
15 KiB
Markdown
292 lines
15 KiB
Markdown
# music-mirror
|
||
|
||
Maintain a lossy MP3 mirror of a lossless music library.
|
||
|
||
Walks a source library and reproduces it, path for path, as MP3 in a separate
|
||
tree. Tags and cover art are carried across; sources that are already MP3 are
|
||
copied rather than re-encoded; mirror files whose source has been deleted are
|
||
removed. **The source library is never written to** — it is mounted read-only
|
||
in the supplied compose file, and nothing in the code opens it for writing.
|
||
|
||
The intended use is an iPod. Apple's Music app cannot read FLAC at all, so a
|
||
converted copy has to exist somewhere; this keeps that copy next to the
|
||
library on a NAS instead of on a laptop, and keeps it current without a human
|
||
remembering to do anything.
|
||
|
||
## How it decides what to do
|
||
|
||
| Situation | Action |
|
||
| -------------------------------- | ---------------------------------------- |
|
||
| No mirror file | encode |
|
||
| Source modified since the mirror | re-encode (a Lidarr quality upgrade) |
|
||
| Mirror up to date | skip |
|
||
| Source is already MP3 | copy verbatim |
|
||
| Source gone | delete the mirror file, prune empty dirs |
|
||
|
||
Freshness is modification time: an encoded file is stamped with its source's
|
||
mtime, so a file is stale exactly when the two differ. There is no database to
|
||
fall out of step with the library, which matters when something else — Lidarr,
|
||
in this case — is the thing that owns and reorganises it.
|
||
|
||
### What that means for Lidarr
|
||
|
||
| Lidarr does this | The mirror does this |
|
||
| --------------------------------------- | -------------------------------------------------------------------------------- |
|
||
| Replaces a file with a better rip | Re-encodes in place. Same path in, same path out, so no duplicate |
|
||
| Upgrades MP3 to FLAC | Both map to the same `.mp3` mirror path, so the old one is overwritten |
|
||
| Renames a track, album or artist folder | Old path pruned, new path encoded. Correct, but it re-encodes rather than moving |
|
||
| Deletes an album or artist | Every orphaned mirror file is deleted and the emptied directories go too |
|
||
|
||
Pruning is driven by what the pass actually found, not by guessing source
|
||
filenames from mirror ones: a `.FLAC` source would not be found by a search for
|
||
`.flac`, and the mirror file would be deleted and rebuilt on alternate passes
|
||
for ever.
|
||
|
||
If two sources want the same mirror path — a `01 Song.flac` next to a leftover
|
||
`01 Song.mp3`, which is what an interrupted upgrade leaves — the better format
|
||
wins, ties break on path, and the loser is logged. Without that rule both
|
||
encode to the same destination and every pass finds one of them stale.
|
||
|
||
The mtime is read _before_ encoding rather than after. A file still being
|
||
written when the pass reaches it would otherwise be stamped with its final
|
||
mtime while holding truncated audio, and never be revisited.
|
||
|
||
Both encodes and copies are written to a temporary file and renamed into place,
|
||
so an interrupted run cannot leave a truncated MP3 that the next run mistakes
|
||
for finished work. Copies need it as much as encodes do: the mtime comes across
|
||
with the bytes, so a half-written copy would look current for ever. A lock file
|
||
in the mirror root stops two passes overlapping.
|
||
|
||
### Permissions
|
||
|
||
Everything written into the mirror is made group-readable, and its directories
|
||
group-traversable, so the mirror can be read back by whatever serves it. Neither
|
||
writer does that unaided: the temporary file an encode renames into place is
|
||
created `0600` regardless of the umask, and a straight copy of an existing MP3
|
||
inherits the mode of a source file in a library this tool does not own.
|
||
|
||
Directories are handled by clearing the owner and group read/execute bits from
|
||
the process umask, once, at startup. Owner as well as group, because a umask
|
||
carrying `0400` produces directories of mode `0300` — writable and enterable,
|
||
unreadable to the very run that created them. The `other` bits are left where
|
||
the umask puts them: whether the mirror is world-readable is a genuine policy
|
||
question, and so is its ownership.
|
||
|
||
Mirror files written before this existed are topped up on the next pass. Their
|
||
mtimes are correct, so nothing else would revisit them — and they are not
|
||
re-encoded, only chmod'ed.
|
||
|
||
## Usage
|
||
|
||
```sh
|
||
music-mirror --source /music --mirror /music-mp3 # one pass
|
||
music-mirror --source /music --mirror /music-mp3 --interval 6h # keep running
|
||
music-mirror --source /music --mirror /music-mp3 --dry-run # report only
|
||
music-mirror --source /music --mirror /music-mp3 --subdir "Artist/Album"
|
||
```
|
||
|
||
| Option | Environment variable | Default | Meaning |
|
||
| ------------ | ----------------------- | --------- | ---------------------------------------------- |
|
||
| `--source` | `MUSIC_MIRROR_SOURCE` | — | Root of the lossless library, read-only |
|
||
| `--mirror` | `MUSIC_MIRROR_MIRROR` | — | Root of the MP3 mirror |
|
||
| `--quality` | `MUSIC_MIRROR_QUALITY` | `V0` | LAME VBR level `V0`–`V9`, or kbps e.g. `256` |
|
||
| `--jobs` | `MUSIC_MIRROR_JOBS` | CPU count | Concurrent encodes |
|
||
| `--interval` | `MUSIC_MIRROR_INTERVAL` | unset | Repeat forever, e.g. `45m`, `6h`, `1d` |
|
||
| `--subdir` | — | unset | Limit the pass to one directory; skips pruning |
|
||
| `--fat32-safe` | `MUSIC_MIRROR_FAT32_SAFE` | off | Name files so a FAT32 device accepts them |
|
||
| `--no-prune` | — | off | Keep mirror files whose source has gone |
|
||
| `--dry-run` | — | off | Report what would change, write nothing |
|
||
|
||
`--subdir` never prunes: a partial pass cannot tell an orphan from a file
|
||
outside its own scope.
|
||
|
||
### Concurrency
|
||
|
||
LAME is single-threaded — ffmpeg reports `Threading capabilities: none` for
|
||
`libmp3lame` — so throughput comes entirely from running several encoders at
|
||
once, one process per file. `--jobs` defaults to the CPUs the process may
|
||
actually use, which inside a container means the `cpus:` allowance rather than
|
||
the host's core count. Each pass logs the number it settled on.
|
||
|
||
As a rough guide, a Zen 3 core encodes about 40–60× realtime at V0 depending on
|
||
clock, so six cores clear roughly 250 hours of audio per hour of wall clock.
|
||
The first full pass is the expensive one; after that only new and changed files
|
||
are touched. Lower `MUSIC_MIRROR_JOBS` if you would rather the NAS stayed
|
||
responsive than finished sooner.
|
||
|
||
Requires `ffmpeg` and `ffprobe` on `PATH`. The container image provides both.
|
||
|
||
## Running it on TrueNAS Scale
|
||
|
||
`compose.yaml` is a Custom App definition. Adjust the two host paths and the
|
||
`user:` to match your pool, then add it as a custom app. The image is published
|
||
to this Gitea's registry on every release:
|
||
|
||
```
|
||
code.emmathe.dev/lyrathorpe/music-mirror:latest
|
||
```
|
||
|
||
Tags are `latest`, the full version, and the truncated `major.minor` and
|
||
`major` forms; builds that are not releases are published as `sha-<short>`.
|
||
|
||
Point the mirror at its own dataset rather than a directory inside the music
|
||
dataset — it is derived data, so it wants its own snapshot policy, its own
|
||
quota, and its own SMB share. The tool refuses to run with a mirror inside the
|
||
source tree.
|
||
|
||
New Lidarr imports are picked up on the next pass. With `MUSIC_MIRROR_INTERVAL`
|
||
at `6h` that is the worst case; run `--subdir` by hand if you want an album
|
||
immediately.
|
||
|
||
## Tools
|
||
|
||
Host-side scripts under `tools/`, not part of the container image.
|
||
|
||
`sync-to-ipod.sh` does a whole transfer: submits the scrobbler log, checks the
|
||
mirror, rsyncs, syncs and unmounts.
|
||
|
||
```sh
|
||
tools/sync-to-ipod.sh /mnt/tank/media/music-mp3 /media/IPOD/Music
|
||
tools/sync-to-ipod.sh -n /mnt/tank/media/music-mp3 /media/IPOD/Music # dry run
|
||
```
|
||
|
||
It refuses to start unless the destination is a mounted FAT filesystem that is
|
||
its own mount point, because `--delete` aimed at the wrong directory empties it
|
||
and does not announce itself. It also excludes `/.rockbox`, the scrobbler logs
|
||
and the various filesystem metadata directories from deletion — the mirror does
|
||
not contain them, and without the exclusion a sync to the card root would
|
||
remove the Rockbox install.
|
||
|
||
The unmount is the point of doing this in a script. FAT32 has no journal and
|
||
the device is reached through disk mode, so an interrupted write is corruption
|
||
that needs `fsck.vfat` from another machine.
|
||
|
||
`submit_scrobbles.py` sends the Rockbox scrobbler log to Last.fm and sets it
|
||
aside. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
|
||
tab-separated line per track rated `L` for listened or `S` for skipped; only
|
||
the listened ones are sent. It runs **before** the copy, since the plays
|
||
already happened and a failed transfer is no reason to lose them.
|
||
|
||
Two things are unlike every other Last.fm call in these projects. Scrobbling is
|
||
a *write* method, so it needs `LASTFM_API_SECRET` and a session key obtained
|
||
once through the browser, not just the read-only key. And on a target with no
|
||
real-time clock Rockbox writes `/.scrobbler-timeless.log` with every timestamp
|
||
set to zero; those are counted and reported but never submitted, because
|
||
scrobbling them would mean inventing when they happened.
|
||
|
||
The log is renamed rather than deleted once accepted. If Last.fm quietly
|
||
dropped something, the evidence is still on the device.
|
||
|
||
`check_fat32.py` reports paths a FAT32 device will not accept — reserved
|
||
characters, trailing dots and spaces, over-long components and paths, and names
|
||
colliding case-insensitively. Run it against the mirror **before** an rsync:
|
||
rsync reports the failures too, but scattered through fifty thousand files where
|
||
they are easy to miss. Exits non-zero when it finds anything, so it can gate a
|
||
script.
|
||
|
||
## Tests
|
||
|
||
```sh
|
||
docker build --target test . # what CI runs
|
||
pytest # needs ffmpeg and pytest on PATH
|
||
```
|
||
|
||
The suite runs real ffmpeg encodes rather than mocking them. The interesting
|
||
failures are in what ffmpeg actually does with tags, cover art and container
|
||
formats, and a mock cannot fail that way — which is also why CI runs the tests
|
||
_inside the image_, against the ffmpeg that ships, rather than against whatever
|
||
the build runner provides. The published image is the `runtime` stage and
|
||
carries neither the tests nor pytest.
|
||
|
||
Run them directly instead if you prefer; they skip when ffmpeg is absent. On a
|
||
Nix machine:
|
||
|
||
```sh
|
||
nix shell nixpkgs#python3Packages.pytest nixpkgs#ffmpeg -c pytest
|
||
```
|
||
|
||
## FAT32 and Rockbox
|
||
|
||
`--fat32-safe` names mirror files so a FAT32 device will accept them. Off by
|
||
default, because turning it on renames files and that should be a decision
|
||
rather than a surprise.
|
||
|
||
What it handles, per path component:
|
||
|
||
| Problem | Treatment |
|
||
| ------------------------------- | ------------------------------ |
|
||
| `< > : " \ \| ? *` and control characters | replaced with `_` |
|
||
| trailing dots and spaces | stripped — FAT eats them silently, so the name round-trips as a different name |
|
||
| a component left empty | becomes `_` |
|
||
| names differing only in case | detected and reported; one wins, as with any other collision |
|
||
|
||
`Dada Life - Kick Out the Epic Motherf**ker` is a real example from a real
|
||
library. Without this it simply never arrives on the device.
|
||
|
||
**Turning it on does not re-encode anything.** Every track whose name held a
|
||
reserved character changes path, and re-encoding those would be hours of work
|
||
producing files that already exist byte for byte. The run moves them instead,
|
||
and says so. Prune then finds nothing to remove because nothing was left
|
||
behind.
|
||
|
||
Renames are counted apart from encodes in the pass summary, and `--dry-run`
|
||
reports `would rename` rather than `would encode` — the difference between the
|
||
two is a minute against an afternoon, so a preview that conflated them would be
|
||
worse than no preview. A dry run also does not list the pre-rename files as
|
||
orphans: nothing was moved, so they are still there, but they are what a real
|
||
run would move rather than what it would delete.
|
||
|
||
### Album art
|
||
|
||
Rockbox looks for cover art **on the filesystem** — `cover.jpg`, `folder.jpg`
|
||
and friends beside the track or in its parent — and that search never touches
|
||
the picture embedded in the tag. So a JPEG cover found beside the source is now
|
||
copied into the mirror as `cover.jpg`, in addition to being embedded. The iPod
|
||
firmware reads the embedded one; Rockbox reads the file. Both are satisfied.
|
||
|
||
A cover left behind in a directory whose tracks have all gone is pruned, or the
|
||
directory would never look empty and never be removed.
|
||
|
||
## Getting the result onto an iPod
|
||
|
||
The mirror is just a directory of MP3s, so any client will do:
|
||
|
||
- **macOS.** Add the mirror's SMB share to the Music app with _Copy files to
|
||
Music Media folder_ and _Keep Media folder organised_ both **off**. The Mac
|
||
then stores a library database and nothing else. Keep the share mounted at a
|
||
stable path — if it is missing when Music opens, every track shows `!`.
|
||
- **Linux.** Rhythmbox links `libgpod` and handles iPod sync. An iPod Video
|
||
(5th generation) predates the models whose database has to be signed, so no
|
||
firmware-hash trickery is needed.
|
||
- **Rockbox.** No database to write at all — it reads a plain directory tree
|
||
and builds its own index from tags. Copy the mirror across with rsync:
|
||
|
||
```sh
|
||
python3 tools/check_fat32.py /mnt/tank/media/music-mp3 # before, not during
|
||
rsync -rtv --delete --modify-window=2 \
|
||
/mnt/tank/media/music-mp3/ /media/IPOD/Music/
|
||
```
|
||
|
||
`--modify-window=2` because FAT stores modification times to two-second
|
||
resolution; without it rsync re-copies the entire library on every run. `-rt`
|
||
rather than `-a` because owners, groups and permissions mean nothing on FAT
|
||
and asking for them only produces errors.
|
||
|
||
Do not route this through Rhythmbox. Its `rb_ipod_helpers_is_ipod()` reads
|
||
`access-protocols` from media-player-info first and returns true without
|
||
looking at the filesystem at all, so an iPod in disk mode is identified by its
|
||
USB id and managed as an iPod — writing a database Rockbox does not want.
|
||
Deleting `iPod_Control` does not change this. Either use rsync, or untick
|
||
Preferences → Plugins → Portable Players - iPod.
|
||
|
||
Faster still for the first bulk copy: take the card out of the iFlash adapter
|
||
and use a card reader. Fifty thousand files over USB 2.0 through an iPod is a
|
||
long evening, and it avoids Rockbox's USB stack entirely.
|
||
|
||
Neither client transcodes at sync time; they copy finished MP3s.
|
||
|
||
Two device-side details worth knowing: the iPod reads cover art from the file's
|
||
tags and ignores `folder.jpg`, which is why art is embedded here; and volume
|
||
levelling on the device uses iTunes' Soundcheck tag, not ReplayGain, so
|
||
ReplayGain tags in the source are not carried over as such.
|