music-mirror/README.md

# music-mirror

Maintain a lossy MP3 mirror of a lossless music library.

Walks a source library and reproduces it, path for path, as MP3 in a separate
tree. Tags and cover art are carried across; sources that are already MP3 are
copied rather than re-encoded; mirror files whose source has been deleted are
removed. **The source library is never written to** — it is mounted read-only
in the supplied compose file, and nothing in the code opens it for writing.

The intended use is an iPod. Apple's Music app cannot read FLAC at all, so a
converted copy has to exist somewhere; this keeps that copy next to the
library on a NAS instead of on a laptop, and keeps it current without a human
remembering to do anything.

## How it decides what to do

| Situation                        | Action                                   |
| -------------------------------- | ---------------------------------------- |
| No mirror file                   | encode                                   |
| Source modified since the mirror | re-encode (a Lidarr quality upgrade)     |
| Mirror up to date                | skip                                     |
| Source is already MP3            | copy verbatim                            |
| Source gone                      | delete the mirror file, prune empty dirs |
| An album gained or lost a track  | re-measure its ReplayGain, tags only     |

Freshness is modification time: an encoded file is stamped with its source's
mtime, so a file is stale exactly when the two differ. There is no database to
fall out of step with the library, which matters when something else — Lidarr,
in this case — is the thing that owns and reorganises it.

### What that means for Lidarr

| Lidarr does this                        | The mirror does this                                                             |
| --------------------------------------- | -------------------------------------------------------------------------------- |
| Replaces a file with a better rip       | Re-encodes in place. Same path in, same path out, so no duplicate                |
| Upgrades MP3 to FLAC                    | Both map to the same `.mp3` mirror path, so the old one is overwritten           |
| Renames a track, album or artist folder | Old path pruned, new path encoded. Correct, but it re-encodes rather than moving |
| Deletes an album or artist              | Every orphaned mirror file is deleted and the emptied directories go too         |

Pruning is driven by what the pass actually found, not by guessing source
filenames from mirror ones: a `.FLAC` source would not be found by a search for
`.flac`, and the mirror file would be deleted and rebuilt on alternate passes
for ever.

If two sources want the same mirror path — a `01 Song.flac` next to a leftover
`01 Song.mp3`, which is what an interrupted upgrade leaves — the better format
wins, ties break on path, and the loser is logged. Without that rule both
encode to the same destination and every pass finds one of them stale.

The mtime is read _before_ encoding rather than after. A file still being
written when the pass reaches it would otherwise be stamped with its final
mtime while holding truncated audio, and never be revisited.

Both encodes and copies are written to a temporary file and renamed into place,
so an interrupted run cannot leave a truncated MP3 that the next run mistakes
for finished work. Copies need it as much as encodes do: the mtime comes across
with the bytes, so a half-written copy would look current for ever. A lock file
in the mirror root stops two passes overlapping.

### Permissions

Everything written into the mirror is made group-readable, and its directories
group-traversable, so the mirror can be read back by whatever serves it. Neither
writer does that unaided: the temporary file an encode renames into place is
created `0600` regardless of the umask, and a straight copy of an existing MP3
inherits the mode of a source file in a library this tool does not own.

Directories are handled by clearing the owner and group read/execute bits from
the process umask, once, at startup. Owner as well as group, because a umask
carrying `0400` produces directories of mode `0300` — writable and enterable,
unreadable to the very run that created them. The `other` bits are left where
the umask puts them: whether the mirror is world-readable is a genuine policy
question, and so is its ownership.

Mirror files written before this existed are topped up on the next pass. Their
mtimes are correct, so nothing else would revisit them — and they are not
re-encoded, only chmod'ed.

## Usage

```sh
music-mirror --source /music --mirror /music-mp3            # one pass
music-mirror --source /music --mirror /music-mp3 --interval 6h   # keep running
music-mirror --source /music --mirror /music-mp3 --dry-run  # report only
music-mirror --source /music --mirror /music-mp3 --subdir "Artist/Album"
```

| Option       | Environment variable    | Default   | Meaning                                        |
| ------------ | ----------------------- | --------- | ---------------------------------------------- |
| `--source`   | `MUSIC_MIRROR_SOURCE`   | —         | Root of the lossless library, read-only        |
| `--mirror`   | `MUSIC_MIRROR_MIRROR`   | —         | Root of the MP3 mirror                         |
| `--quality`  | `MUSIC_MIRROR_QUALITY`  | `V0`      | LAME VBR level `V0`–`V9`, or kbps e.g. `256`   |
| `--jobs`     | `MUSIC_MIRROR_JOBS`     | CPU count | Concurrent encodes                             |
| `--interval` | `MUSIC_MIRROR_INTERVAL` | unset     | Repeat forever, e.g. `45m`, `6h`, `1d`         |
| `--subdir`   | —                       | unset     | Limit the pass to one directory; skips pruning |
| `--fat32-safe` | `MUSIC_MIRROR_FAT32_SAFE` | off     | Name files so a FAT32 device accepts them      |
| `--no-replaygain` | `MUSIC_MIRROR_REPLAYGAIN` | on   | Write ReplayGain tags; set the variable to `0` to skip |
| `--no-prune` | —                       | off       | Keep mirror files whose source has gone        |
| `--dry-run`  | —                       | off       | Report what would change, write nothing        |

`--subdir` never prunes: a partial pass cannot tell an orphan from a file
outside its own scope.

### Concurrency

LAME is single-threaded — ffmpeg reports `Threading capabilities: none` for
`libmp3lame` — so throughput comes entirely from running several encoders at
once, one process per file. `--jobs` defaults to the CPUs the process may
actually use, which inside a container means the `cpus:` allowance rather than
the host's core count. Each pass logs the number it settled on.

As a rough guide, a Zen 3 core encodes about 40–60× realtime at V0 depending on
clock, so six cores clear roughly 250 hours of audio per hour of wall clock.
The first full pass is the expensive one; after that only new and changed files
are touched. Lower `MUSIC_MIRROR_JOBS` if you would rather the NAS stayed
responsive than finished sooner.

Requires `ffmpeg` and `ffprobe` on `PATH`, and `rsgain` for volume levelling.
The container image provides all three. A missing `rsgain` is reported once per
pass and costs the ReplayGain tags; everything else still runs.

## Running it on TrueNAS Scale

`compose.yaml` is a Custom App definition. Adjust the two host paths and the
`user:` to match your pool, then add it as a custom app. The image is published
to this Gitea's registry on every release:

```
code.emmathe.dev/lyrathorpe/music-mirror:latest
```

Tags are `latest`, the full version, and the truncated `major.minor` and
`major` forms; builds that are not releases are published as `sha-<short>`.

Point the mirror at its own dataset rather than a directory inside the music
dataset — it is derived data, so it wants its own snapshot policy, its own
quota, and its own SMB share. The tool refuses to run with a mirror inside the
source tree.

New Lidarr imports are picked up on the next pass. With `MUSIC_MIRROR_INTERVAL`
at `6h` that is the worst case; run `--subdir` by hand if you want an album
immediately.

## Tools

Host-side scripts under `tools/`, not part of the container image.

`sync-to-ipod.sh` does a whole transfer: submits the scrobbler log, checks the
mirror, rsyncs, syncs and unmounts.

```sh
tools/sync-to-ipod.sh /mnt/tank/media/music-mp3 /media/IPOD/Music
tools/sync-to-ipod.sh -n /mnt/tank/media/music-mp3 /media/IPOD/Music   # dry run
```

The destination is where the artist folders should end up — normally a
subdirectory such as `/media/IPOD/Music`, not the card root. A subdirectory is
the better target: `--delete` is confined to it, and the device path budget is
derived from it rather than configured, so the two cannot disagree.

It refuses to start unless the destination is on a mounted FAT filesystem. That
check is also what catches an unmounted device — `/media/IPOD/Music` then
resolves to the host's own root filesystem, and this refuses to empty that.
`--help` says all of it. It also excludes `/.rockbox`, the scrobbler logs
and the various filesystem metadata directories from deletion — the mirror does
not contain them, and without the exclusion a sync to the card root would
remove the Rockbox install.

Progress is a single line that rewrites itself:

```
[ 24%] 12,345/49,600  3.2 GiB/13.1 GiB  4.4 MiB/s  ETA 38m12s  King Gizzard / Petro…
```

The estimate comes from rsync's `%l`, which gives each file's size as it
completes. Bytes done over time elapsed is the same arithmetic rsync would do,
and needs nothing it does not already print. The rate is measured over a
trailing thirty seconds rather than the whole run, so it follows a device that
slows down instead of averaging the slowdown away — and it is suppressed
entirely for the first two seconds, where the window is microseconds wide and
would report gigabytes per second.

rsync says nothing at all while it builds its file list, which on fifty
thousand files is minutes of apparent hang, and its own `progress2` percentage
is computed against a list it has not finished discovering. So the script
renders its own.

**The percentage and the estimate are opt-in, via `-P`.** They need a total,
the total needs a counting pass, and that pass walks and compares both trees in
full exactly as the transfer does. Measured on a real card: read from the
source at 35 MB/s and write to the card at 21 MB/s, yet the sync crawled —
because the traversal, not the data, was the cost, and it was being paid twice.
Without `-P` the line still shows the running count, the rate and the album in
flight; only the two figures that needed the second walk are missing.

Piped to a log it prints a plain line every thirty seconds instead, with no
carriage returns, and a summary at the end either way.

### The Rockbox database

Point `MUSIC_MIRROR_DATABASE_TOOL` at Rockbox's host-side builder and the sync
rebuilds the database itself, so it never has to happen on the device.

```sh
git clone --depth 1 https://github.com/Rockbox/rockbox.git
cd rockbox && mkdir build-db && cd build-db
../tools/configure --target=ipodvideo --type=d && make -j$(nproc)
```

It needs a native compiler and SDL2 development headers, not the ARM
cross-toolchain, and `tools/configure` detects `__aarch64__` correctly. On a
distribution without `/usr/bin/perl` or `gcc-ar` — NixOS, say — patch the
shebangs in `tools/*.pl` and pass `AR=ar`.

Building it here rather than on the device is not merely faster. The on-device
commit sorts the whole index in whatever memory `core_alloc_maximum()` can
scrape together; on a fifty-thousand-track library it runs for hours or aborts
outright.

The scan runs against a scratch root — a real `.rockbox` beside a symlink
standing in for wherever the music lands on the device — so the paths recorded
are the ones Rockbox will look up, while the **bytes are read from the mirror
rather than over USB**. Only the dozen `.tcd` files cross to the card.

Cost, measured: the parser makes about 49 reads and 43 seeks per file, probing
the head for ID3v2 and the tail for ID3v1. On a local disk that is 2,000 files
in half a second. Over SMB, readahead absorbs most of the reads but the opens
and the head/tail split are real round trips, so a first full scan is minutes
rather than seconds. It is a one-time cost: the builder is incremental, and the
scratch root is kept between runs, so a later pass over unchanged files does no
metadata reads at all.

If minutes is still too many, run the builder where the mirror is local — on
the NAS — and copy the `.tcd` files across. There is nothing to parallelise:
the tool is single-threaded, and two instances cannot produce one database.

### If the sync is interrupted

No partially copied track is ever left under a name Rockbox would play. rsync
writes to a hidden temporary file and only renames it into place once the file
is complete, and *"by default, rsync will delete any partially transferred file
if the transfer is interrupted"*. `--partial` is deliberately not used, and
there is a test asserting it never will be.

After an unclean kill or a power cut a hidden `.track.mp3.XXXXXX` can survive.
It is not playable, it is not in the source, and the next run's `--delete`
removes it.

Interrupting does not, by itself, endanger the filesystem. The kernel flushes
dirty pages within `dirty_expire_centisecs` — thirty seconds by default — and
`umount` always syncs before it returns. Losing data needs you to interrupt,
*and* pull the card inside that window, *and* skip the unmount.

The script still traps `INT` and `TERM` and flushes and unmounts on the way
out, exiting 130. Not because a Ctrl-C is dangerous, but because it removes the
manual step and makes the exit deterministic — you get the same "safe to
disconnect" either way, rather than having to remember which path you took.
Both paths call the same function, so they cannot drift apart.

What genuinely does lose data is pulling the cable or the card without
unmounting at all, interrupted or not. FAT32 has no journal. Wait for the
unmount line.

### Making it faster over a network mount

The transfer is metadata-bound, not throughput-bound: 49,600 files means 49,600
round trips, and the counting pass doubles that. In rough order of what it is
worth doing:

| Lever | Why |
| ----- | --- |
| Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. |
| Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. |
| Counting is off by default | The percentage costs a second full traversal of both trees. On a FAT card of fifty thousand files that is slower than the transfer. `-P` asks for it. |
| `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. |

**NFS instead of SMB** is worth trying but is not the big win it looks like.
Its attribute caching defaults are far more generous than SMB's — `acregmax` of
sixty seconds against `actimeo=1` — which is precisely the gap that
`actimeo=60` closes on the mount you already have. Bulk read throughput between
the two is much of a muchness on a gigabit link. Try the mount option first; it
is one line and needs no change on the NAS.

And if the destination is the iPod rather than a card reader, none of this
matters much: the source can feed data faster than USB 2.0 through an iPod will
take it either way.

The unmount is the point of doing this in a script. FAT32 has no journal and
the device is reached through disk mode, so an interrupted write is corruption
that needs `fsck.vfat` from another machine.

`submit_scrobbles.py` sends what was played to Last.fm and sets the logs aside.

It reads **Rockbox's own `playback.log`**, which core Rockbox writes whenever
"play log" is enabled, with no plugin running. Each line is
`timestamp:elapsed_ms:length_ms:path` — a path and nothing else, which is why
the on-device scrobbler plugin exists at all: reading tags back off the player
is slow. Off the mirror it is free, so `--mirror` lets the conversion happen
here and the plugin never has to be run. A play counts as listened at half the
track's length, the same fraction the plugin uses, so the two cannot disagree
about what a play was.

It still reads a `.scrobbler.log` if the plugin has been run and left one. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
tab-separated line per track rated `L` for listened or `S` for skipped; only
the listened ones are sent. It runs **before** the copy, since the plays
already happened and a failed transfer is no reason to lose them.

Two things are unlike every other Last.fm call in these projects. Scrobbling is
a *write* method, so it needs `LASTFM_API_SECRET` and a session key obtained
once through the browser, not just the read-only key. And on a target with no
real-time clock Rockbox writes `/.scrobbler-timeless.log` with every timestamp
set to zero; those are counted and reported but never submitted, because
scrobbling them would mean inventing when they happened.

### Nothing played is thrown away

Two separate obligations, because a play that happened and never reached
Last.fm is gone for good.

**The original is renamed, never deleted.** If Last.fm quietly dropped
something, the evidence is still on the device as `playback.log.<ts>.submitted`.

**Anything not submitted is written back** into a live log for the next run:

| Outcome | What happens to it |
| ------------------------------ | ----------------------------------------- |
| Accepted by Last.fm | dropped from the live log |
| Not in the mirror yet | written back, tried again next run |
| In a batch that failed | written back, tried again next run |
| Nothing accepted at all | logs left completely untouched |
| A skip, or no usable timestamp | not retained — neither can ever be submitted, and the original still has it |

The batch boundary matters: submission is recorded as each batch is accepted,
so a failure partway through knows exactly what got through and writes back
only the remainder. No duplicates, no losses.

A file `ffprobe` cannot read costs one unidentified play, not the run.

`check_fat32.py` reports paths a FAT32 device will not accept — reserved
characters, trailing dots and spaces, over-long components and paths, and names
colliding case-insensitively. Run it against the mirror **before** an rsync:
rsync reports the failures too, but scattered through fifty thousand files where
they are easy to miss. Exits non-zero when it finds anything, so it can gate a
script.

## Tests

```sh
docker build --target test .   # what CI runs
pytest                         # needs ffmpeg and pytest on PATH
```

The ReplayGain tests need `rsgain` as well and skip without it.

The suite runs real ffmpeg encodes rather than mocking them. The interesting
failures are in what ffmpeg actually does with tags, cover art and container
formats, and a mock cannot fail that way — which is also why CI runs the tests
_inside the image_, against the ffmpeg that ships, rather than against whatever
the build runner provides. The published image is the `runtime` stage and
carries neither the tests nor pytest.

Run them directly instead if you prefer; they skip when ffmpeg is absent. On a
Nix machine:

```sh
nix shell nixpkgs#python3Packages.pytest nixpkgs#ffmpeg nixpkgs#rsgain -c pytest
```

## FAT32 and Rockbox

`--fat32-safe` names mirror files so a FAT32 device will accept them. Off by
default, because turning it on renames files and that should be a decision
rather than a surprise.

What it handles, per path component:

| Problem | Treatment |
| ------------------------------- | ------------------------------ |
| `< > : " \ \| ? *` and control characters | replaced with `_` |
| trailing dots and spaces | stripped — FAT eats them silently, so the name round-trips as a different name |
| a component left empty | becomes `_` |
| names differing only in case | detected and reported; one wins, as with any other collision |

`Dada Life - Kick Out the Epic Motherf**ker` is a real example from a real
library. Without this it simply never arrives on the device.

**Turning it on does not re-encode anything.** Every track whose name held a
reserved character changes path, and re-encoding those would be hours of work
producing files that already exist byte for byte. The run moves them instead,
and says so. Prune then finds nothing to remove because nothing was left
behind.

Renames are counted apart from encodes in the pass summary, and `--dry-run`
reports `would rename` rather than `would encode` — the difference between the
two is a minute against an afternoon, so a preview that conflated them would be
worse than no preview. A dry run also does not list the pre-rename files as
orphans: nothing was moved, so they are still there, but they are what a real
run would move rather than what it would delete.

### Path length

Rockbox's `MAX_PATH` is 260, from `firmware/include/fs_defines.h`, and it bounds
the path *as the device sees it*. The directory the mirror is copied into comes
out of the same budget, so `--device-prefix` (default `/Music`) is subtracted
from `--max-path` to get what a mirror-relative path may spend:

| Destination on the device | Mirror-relative budget |
| ------------------------- | ---------------------- |
| `/Music/` | 253 |
| the card root | 259 |

Worth being exact about, because a checker that measures mirror-relative paths
against the flat 260 quietly passes everything from 253 to 260 — and those are
the paths most likely to be near the edge in the first place.

Over-budget paths are shortened from the **deepest component outward**: the
track name carries the least navigational value and the artist directory the
most, so the filename goes first and the artist is touched only if nothing else
will do.

A component is cut **from the middle**, not the end, because of how these names
are built. Lidarr writes `Artist - Album - 07 - Flamethrower.mp3` inside a
directory already named for that artist and album, so a long album title
appears three times in one path and the informative part — the track number and
title — is at the very end. Cutting from the end throws exactly that away:

```
before  King Gizzard & the Lizard Wizard - PetroDragonic Apocalypse; or, Dawn of Eternal
        Night - An Annihilation of Planet Earth and the Beginning of Merciless
        Damnation - 07 - Flamethrower.mp3
after   King Gizzard & the Lizard~c526~ginning of Merciless Damnation - 07 - Flamethrower.mp3
```

Two thirds of the remaining room goes to the tail, since the head is usually a
restatement of the directory it sits in. A shortened component gains four hex
digits of the original name: two names sharing both a head and a tail would
otherwise produce the same string, and a silent collision between two tracks is
worse than an ugly filename.

The result is stable: the same source always produces the same shortened name,
so a pass does not rename what the previous pass wrote. A path too deeply
nested to fit without reducing every component to nonsense is left alone and
reported instead.

### Album art

Rockbox looks for cover art **on the filesystem** — `cover.jpg`, `folder.jpg`
and friends beside the track or in its parent — and that search never touches
the picture embedded in the tag. So a JPEG cover found beside the source is now
copied into the mirror as `cover.jpg`, in addition to being embedded. The iPod
firmware reads the embedded one; Rockbox reads the file. Both are satisfied.

A cover left behind in a directory whose tracks have all gone is pruned, or the
directory would never look empty and never be removed.

### Volume levelling

Rockbox can level the volume between tracks, but only from tags. It applies the
offset a ReplayGain tag carries and has no loudness analysis of its own, so a
mirror without those tags plays every album at whatever level it was mastered
to — and a 2008 remaster next to a 1972 pressing is a reach for the volume
wheel on every track change.

The tags are therefore written here, with `rsgain`, once an album's tracks are
in place. Both album gain and track gain are measured: album gain preserves the
quiet track that a record is supposed to have, track gain is the one that makes
sense on shuffle, and which of them is used is the device's decision, not this
one.

Turn it on at the player end under **Settings → Playback Settings →
Replaygain**:

| Setting          | Suggested            | Why                                                       |
| ---------------- | -------------------- | --------------------------------------------------------- |
| Replaygain type  | `Album Gain`, or `Track Gain if Shuffling` | Keeps a record's own dynamics; the second switches per mode |
| Prevent clipping | `Yes`                | Uses the peak tags to back off rather than distort         |
| Pre-amp          | `0 dB`               | The reference level is already −18 LUFS; move it only if the result is too quiet |

Measuring costs a full decode of every track, so the first pass after enabling
it takes roughly as long as the original encode did. After that only albums
that gained, lost or replaced a track are re-measured. An album is re-measured
as a whole, because album gain is a property of all of its tracks and one new
track makes the value stored on every sibling wrong.

Tagging rewrites the file, and staleness here is an mtime comparison, so
`rsgain` is run with `--preserve-mtimes`. Without it every levelled track would
look newer than its source and the next pass would re-encode the entire
library.

## Getting the result onto an iPod

The mirror is just a directory of MP3s, so any client will do:

- **macOS.** Add the mirror's SMB share to the Music app with _Copy files to
  Music Media folder_ and _Keep Media folder organised_ both **off**. The Mac
  then stores a library database and nothing else. Keep the share mounted at a
  stable path — if it is missing when Music opens, every track shows `!`.
- **Linux.** Rhythmbox links `libgpod` and handles iPod sync. An iPod Video
  (5th generation) predates the models whose database has to be signed, so no
  firmware-hash trickery is needed.
- **Rockbox.** No database to write at all — it reads a plain directory tree
  and builds its own index from tags. Copy the mirror across with rsync:

  ```sh
  python3 tools/check_fat32.py /mnt/tank/media/music-mp3   # before, not during
  rsync -rtv --delete --modify-window=2 \
      /mnt/tank/media/music-mp3/ /media/IPOD/Music/
  ```

  `--modify-window=2` because FAT stores modification times to two-second
  resolution; without it rsync re-copies the entire library on every run. `-rt`
  rather than `-a` because owners, groups and permissions mean nothing on FAT
  and asking for them only produces errors.

  Do not route this through Rhythmbox. Its `rb_ipod_helpers_is_ipod()` reads
  `access-protocols` from media-player-info first and returns true without
  looking at the filesystem at all, so an iPod in disk mode is identified by its
  USB id and managed as an iPod — writing a database Rockbox does not want.
  Deleting `iPod_Control` does not change this. Either use rsync, or untick
  Preferences → Plugins → Portable Players - iPod.

  Faster still for the first bulk copy: take the card out of the iFlash adapter
  and use a card reader. Fifty thousand files over USB 2.0 through an iPod is a
  long evening, and it avoids Rockbox's USB stack entirely.

Neither client transcodes at sync time; they copy finished MP3s.

Two device-side details worth knowing: the iPod reads cover art from the file's
tags and ignores `folder.jpg`, which is why art is embedded here; and the Apple
firmware levels volume from iTunes' Soundcheck tag rather than ReplayGain, so
the tags written here do nothing until the iPod is running Rockbox.