Merge pull request 'feat: level the mirror's volume with ReplayGain tags' (#12) from feat/replaygain into main

Reviewed-on: #12

Dockerfile

@@ -7,8 +7,10 @@ FROM python:3.13-alpine AS runtime
 
 ENV PYTHONUNBUFFERED=1
 
-# ffmpeg does the encoding; the application itself has no Python dependencies.
+# ffmpeg does the encoding and rsgain the volume levelling; the application
-RUN apk add --no-cache ffmpeg
+# itself has no Python dependencies. rsgain lives in the community repository,
+# which the official Python images already have enabled.
+RUN apk add --no-cache ffmpeg rsgain
 
 WORKDIR /app
 COPY pyproject.toml README.md ./

README.md

@@ -22,6 +22,7 @@ remembering to do anything.
 | Mirror up to date                | skip                                     |
 | Source is already MP3            | copy verbatim                            |
 | Source gone                      | delete the mirror file, prune empty dirs |
+| An album gained or lost a track  | re-measure its ReplayGain, tags only     |
 
 Freshness is modification time: an encoded file is stamped with its source's
 mtime, so a file is stale exactly when the two differ. There is no database to
@@ -94,6 +95,7 @@ music-mirror --source /music --mirror /music-mp3 --subdir "Artist/Album"
 | `--interval` | `MUSIC_MIRROR_INTERVAL` | unset     | Repeat forever, e.g. `45m`, `6h`, `1d`         |
 | `--subdir`   | —                       | unset     | Limit the pass to one directory; skips pruning |
 | `--fat32-safe` | `MUSIC_MIRROR_FAT32_SAFE` | off     | Name files so a FAT32 device accepts them      |
+| `--no-replaygain` | `MUSIC_MIRROR_REPLAYGAIN` | on   | Write ReplayGain tags; set the variable to `0` to skip |
 | `--no-prune` | —                       | off       | Keep mirror files whose source has gone        |
 | `--dry-run`  | —                       | off       | Report what would change, write nothing        |
 
@@ -114,7 +116,9 @@ The first full pass is the expensive one; after that only new and changed files
 are touched. Lower `MUSIC_MIRROR_JOBS` if you would rather the NAS stayed
 responsive than finished sooner.
 
-Requires `ffmpeg` and `ffprobe` on `PATH`. The container image provides both.
+Requires `ffmpeg` and `ffprobe` on `PATH`, and `rsgain` for volume levelling.
+The container image provides all three. A missing `rsgain` is reported once per
+pass and costs the ReplayGain tags; everything else still runs.
 
 ## Running it on TrueNAS Scale
 
@@ -193,6 +197,16 @@ flight; only the two figures that needed the second walk are missing.
 Piped to a log it prints a plain line every thirty seconds instead, with no
 carriage returns, and a summary at the end either way.
 
+### Albums that changed are copied whole
+
+After the main pass, any album that gained or lost a track has the rest of its
+tracks copied again with `--ignore-times`. Their ReplayGain tags were rewritten
+in place when the album was re-levelled, which changes neither their size nor
+their mtime — the only two things rsync compares — so nothing else would ever
+send them. `touched_albums.py` works out the list from rsync's own report of
+what it moved, so this costs no extra traversal, and tracks the main pass has
+already copied are left out of it. See [Volume levelling](#volume-levelling).
+
 ### The Rockbox database
 
 Point `MUSIC_MIRROR_DATABASE_TOOL` at Rockbox's host-side builder and the sync
@@ -347,6 +361,8 @@ docker build --target test .   # what CI runs
 pytest                         # needs ffmpeg and pytest on PATH
 ```
 
+The ReplayGain tests need `rsgain` as well and skip without it.
+
 The suite runs real ffmpeg encodes rather than mocking them. The interesting
 failures are in what ffmpeg actually does with tags, cover art and container
 formats, and a mock cannot fail that way — which is also why CI runs the tests
@@ -358,7 +374,7 @@ Run them directly instead if you prefer; they skip when ffmpeg is absent. On a
 Nix machine:
 
 ```sh
-nix shell nixpkgs#python3Packages.pytest nixpkgs#ffmpeg -c pytest
+nix shell nixpkgs#python3Packages.pytest nixpkgs#ffmpeg nixpkgs#rsgain -c pytest
 ```
 
 ## FAT32 and Rockbox
@@ -448,6 +464,64 @@ firmware reads the embedded one; Rockbox reads the file. Both are satisfied.
 A cover left behind in a directory whose tracks have all gone is pruned, or the
 directory would never look empty and never be removed.
 
+### Volume levelling
+
+Rockbox can level the volume between tracks, but only from tags. It applies the
+offset a ReplayGain tag carries and has no loudness analysis of its own, so a
+mirror without those tags plays every album at whatever level it was mastered
+to — and a 2008 remaster next to a 1972 pressing is a reach for the volume
+wheel on every track change.
+
+The tags are therefore written here, with `rsgain`, once an album's tracks are
+in place. Both album gain and track gain are measured: album gain preserves the
+quiet track that a record is supposed to have, track gain is the one that makes
+sense on shuffle, and which of them is used is the device's decision, not this
+one.
+
+Turn it on at the player end under **Settings → Playback Settings →
+Replaygain**:
+
+| Setting          | Suggested                    | Why                                                        |
+| ---------------- | ---------------------------- | ---------------------------------------------------------- |
+| Replaygain type  | `Track Gain if Shuffling`    | Album gain while playing a record, track gain once shuffle is on — the only setting that is right in both cases |
+| Prevent clipping | `Yes`                        | Backs the gain off using the peak tags rather than distorting |
+| Pre-amp          | `0 dB`                       | The reference is already −18 LUFS; raise it only if everything ends up too quiet |
+
+`Track Gain if Shuffling` is not a compromise between the other two — Rockbox
+reads the shuffle setting and picks whole-hog album or track gain from it
+(`apps/misc.c`, `replaygain_setting_mode`). It is also Rockbox's default, so on
+a fresh install there may be nothing to change but `Prevent clipping`.
+
+Measuring costs a full decode of every track, so the first pass after enabling
+it takes roughly as long as the original encode did. After that only albums
+that gained, lost or replaced a track are re-measured. An album is re-measured
+as a whole, because album gain is a property of all of its tracks and one new
+track makes the value stored on every sibling wrong.
+
+Tagging rewrites the file, and staleness here is an mtime comparison, so
+`rsgain` is run with `--preserve-mtimes`. Without it every levelled track would
+look newer than its source and the next pass would re-encode the entire
+library.
+
+That has a consequence for the sync, and it is not obvious. The first time a
+track is levelled its tag grows by about a kilobyte, so its size changes and
+rsync copies it — the whole library goes across once, and there is no way
+around that; the tag sits at the head of the file and every byte after it
+moves. But `rsgain` leaves padding behind, so a *later* re-level fits inside it
+and changes neither the size nor the mtime:
+
+```
+before re-level: 277757 bytes, mtime 1577880000, album gain 3.75 dB
+after  re-level: 277757 bytes, mtime 1577880000, album gain 6.25 dB
+```
+
+Those are the two things rsync's quick check compares, so it would see nothing
+to do and the device would keep the old gains. `sync-to-ipod.sh` handles it: the
+track that arrived or left is always visible, so any album the main pass
+touched has the rest of its tracks copied again with `--ignore-times`. Albums
+whose file set has not changed are not touched, which is what keeps this from
+being a full re-copy.
+
 ## Getting the result onto an iPod
 
 The mirror is just a directory of MP3s, so any client will do:
@@ -487,6 +561,6 @@ The mirror is just a directory of MP3s, so any client will do:
 Neither client transcodes at sync time; they copy finished MP3s.
 
 Two device-side details worth knowing: the iPod reads cover art from the file's
-tags and ignores `folder.jpg`, which is why art is embedded here; and volume
+tags and ignores `folder.jpg`, which is why art is embedded here; and the Apple
-levelling on the device uses iTunes' Soundcheck tag, not ReplayGain, so
+firmware levels volume from iTunes' Soundcheck tag rather than ReplayGain, so
-ReplayGain tags in the source are not carried over as such.
+the tags written here do nothing until the iPod is running Rockbox.

compose.yaml

@@ -31,6 +31,11 @@ services:
       # enabling it is quick -- but it is a one-way change to every such path,
       # so decide before running it rather than after.
       MUSIC_MIRROR_FAT32_SAFE: "true"
+      # ReplayGain tags, so Rockbox can level the volume between albums. On by
+      # default; set to 0 to skip the measuring pass. The first pass after
+      # enabling it levels the whole library, which costs a decode of every
+      # track, so expect it to take about as long as the original encode.
+      # MUSIC_MIRROR_REPLAYGAIN: "0"
     volumes:
       - /mnt/tank/media/music:/music:ro
       - /mnt/tank/media/music-mp3:/mirror

music_mirror.py

@@ -11,6 +11,11 @@ writes to the source library.
 Staleness is tracked by modification time: an encoded file is given its
 source's mtime, so a file is out of date exactly when the two differ. That
 makes runs idempotent without a database to keep in step.
+
+Finished albums are levelled with rsgain, which writes ReplayGain tags into the
+mirror. Rockbox applies the offset those tags carry but has no loudness
+analysis of its own, so without them every album plays at whatever level it was
+mastered to.
 """
 
 import argparse
@@ -81,6 +86,21 @@ MIRROR_COVER = "cover.jpg"
 # Files the mirror is allowed to contain, and therefore allowed to delete.
 MIRROR_SUFFIX = ".mp3"
 
+# Measures loudness and writes the ReplayGain tags. Not a hard requirement: a
+# pass without it still produces a correct mirror, only one the player cannot
+# level, so a missing binary is a warning rather than a failure.
+REPLAYGAIN_TOOL = "rsgain"
+
+# Looked for in a file's ID3v2 tag to tell a levelled track from an unlevelled
+# one. Album gain rather than track gain because the album value is the one
+# this writes for; a file carrying only track gain came from somewhere else and
+# should be rescanned.
+REPLAYGAIN_TAG = b"replaygain_album_gain"
+
+# Enough of a TXXX frame body to hold the encoding byte and the description.
+# The value after it says what the gain is, which is not the question here.
+TXXX_DESCRIPTION_BYTES = 128
+
 # Filesystems disagree about mtime precision; SMB in particular rounds.
 MTIME_TOLERANCE_SECONDS = 2
 
@@ -541,8 +561,13 @@ def prune(mirror_root, expected, dry_run):
     Driven by the set of paths the pass expects to exist rather than by
     probing the source tree for names, which would disagree with it over
     letter case and over any extension the walker does not collect.
+
+    Returns the number of files removed and the directories they came out of.
+    Losing a track changes an album's loudness, so those directories need
+    levelling again even though nothing was written into them.
     """
     removed = 0
+    emptied = set()
 
     for mirror in sorted(mirror_root.rglob(f"*{MIRROR_SUFFIX}")):
         if mirror in expected:
@@ -553,6 +578,7 @@ def prune(mirror_root, expected, dry_run):
             continue
         logger.info("removing orphan %s", mirror)
         mirror.unlink(missing_ok=True)
+        emptied.add(mirror.parent)
 
     if not dry_run:
         # A cover copied for an album whose tracks have all gone is an orphan
@@ -567,7 +593,164 @@ def prune(mirror_root, expected, dry_run):
             if directory.is_dir() and not any(directory.iterdir()):
                 directory.rmdir()
 
-    return removed
+    return removed, emptied
+
+
+def syncsafe(data):
+    """Return the integer held in syncsafe bytes: seven bits of each."""
+    value = 0
+    for byte in data:
+        value = (value << 7) | (byte & 0x7F)
+    return value
+
+
+def has_replaygain(path):
+    """Return whether an MP3 already carries ReplayGain tags.
+
+    Walks the ID3v2 frame headers and seeks over the bodies rather than reading
+    the tag whole. Every file in this mirror has its cover art embedded, so the
+    tag is routinely half a megabyte; reading all of it for every track on
+    every pass would turn an idle pass into a full read of the library.
+    """
+    try:
+        with open(path, "rb") as handle:
+            header = handle.read(10)
+            if len(header) < 10 or header[:3] != b"ID3" or header[3] not in (3, 4):
+                return False
+            remaining = syncsafe(header[6:10])
+
+            # Unsynchronisation shifts every offset in the tag, and the two
+            # versions describe an extended header differently. Nothing that
+            # writes this mirror emits either, so reading the tag whole is a
+            # cheaper answer than the code to walk one that does.
+            if header[5] & 0xC0:
+                return REPLAYGAIN_TAG in handle.read(remaining).lower()
+
+            while remaining >= 10:
+                frame = handle.read(10)
+                remaining -= 10
+                # Frame ids are upper-case letters and digits, so anything else
+                # is the padding that follows the last frame.
+                if len(frame) < 10 or not frame[:4].isalnum():
+                    return False
+                # 2.3 sizes count all eight bits per byte; 2.4 made them
+                # syncsafe like the tag length above.
+                length = (
+                    int.from_bytes(frame[4:8], "big")
+                    if header[3] == 3
+                    else syncsafe(frame[4:8])
+                )
+                if length <= 0 or length > remaining:
+                    return False
+                if frame[:4] == b"TXXX":
+                    body = handle.read(min(length, TXXX_DESCRIPTION_BYTES))
+                    handle.seek(length - len(body), os.SEEK_CUR)
+                    if REPLAYGAIN_TAG in body.lower():
+                        return True
+                else:
+                    handle.seek(length, os.SEEK_CUR)
+                remaining -= length
+    except OSError:
+        return False
+    return False
+
+
+def replaygain_albums(expected, written):
+    """Return the album directories needing a scan, each with its tracks.
+
+    A directory is scanned when this pass changed what is in it, because album
+    gain is a property of the whole album: one track added, replaced or removed
+    makes the value stored on every one of its siblings wrong. It is also
+    scanned when a track in it has never been levelled, which is what backfills
+    a mirror built before any of this existed.
+    """
+    albums = {}
+    for mirror in expected:
+        albums.setdefault(mirror.parent, []).append(mirror)
+
+    needed = {}
+    for directory, tracks in sorted(albums.items()):
+        # A dry run reaches here before anything has been encoded, so the
+        # tracks a changed album is going to hold do not exist yet.
+        present = sorted(track for track in tracks if track.is_file())
+        if directory in written:
+            needed[directory] = present
+        elif present and not all(map(has_replaygain, present)):
+            needed[directory] = present
+    return needed
+
+
+def replaygain_command(tracks):
+    """Return the rsgain command that levels one album directory."""
+    return [
+        REPLAYGAIN_TOOL,
+        "custom",
+        # Album mode writes the per-track tags as well as the album ones, so
+        # the device is left to choose between them -- Rockbox can apply track
+        # gain when shuffling and album gain otherwise, and only if both are
+        # present.
+        "--album",
+        "--tagmode=i",
+        # The mirror is ID3v2.3 for the iPod firmware's sake. rsgain would
+        # otherwise keep whatever version it found, and "whatever it found" is
+        # not a guarantee.
+        "--id3v2-version=3",
+        # Staleness here is an mtime comparison and tagging rewrites the file.
+        # Without this every levelled track would look newer than its source
+        # and the next pass would re-encode the entire library, forever.
+        "--preserve-mtimes",
+        "--quiet",
+        *[str(track) for track in tracks],
+    ]
+
+
+def scan_album(directory, tracks):
+    """Write ReplayGain tags across one album. Returns whether it worked."""
+    completed = subprocess.run(replaygain_command(tracks), capture_output=True, text=True)
+    if completed.returncode != 0:
+        lines = completed.stderr.strip().splitlines()
+        logger.warning("could not level %s: %s", directory, lines[-1] if lines else "rsgain failed")
+        return False
+    logger.info("levelled %s", directory)
+    return True
+
+
+def replaygain(expected, written, jobs, dry_run):
+    """Write ReplayGain tags into the albums that need them. Returns how many.
+
+    A failure here is reported and then left alone. The mirror is still correct
+    audio in the right place; it just plays at the level it was mastered to,
+    which is what every pass before this one produced.
+    """
+    albums = {
+        directory: tracks
+        for directory, tracks in replaygain_albums(expected, written).items()
+        if tracks or dry_run
+    }
+    if not albums:
+        return 0
+
+    if dry_run:
+        logger.info("would level %d album%s", len(albums), "" if len(albums) == 1 else "s")
+        return len(albums)
+
+    if shutil.which(REPLAYGAIN_TOOL) is None:
+        logger.warning(
+            "%s is not on PATH; %d albums are left without ReplayGain tags",
+            REPLAYGAIN_TOOL,
+            len(albums),
+        )
+        return 0
+
+    levelled = 0
+    with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool:
+        futures = [
+            pool.submit(scan_album, directory, tracks) for directory, tracks in albums.items()
+        ]
+        for future in concurrent.futures.as_completed(futures):
+            if future.result():
+                levelled += 1
+    return levelled
 
 
 def run_once(
@@ -580,6 +763,7 @@ def run_once(
     do_prune,
     safe=False,
     budget=0,
+    do_replaygain=True,
 ):
     """Run a single pass. Returns the number of failures.
 
@@ -590,6 +774,7 @@ def run_once(
     logger.info("pass starting with %d concurrent encoders", jobs)
     counts = {"encoded": 0, "copied": 0, "renamed": 0, "skipped": 0, "failed": 0}
     failures = []
+    written = set()
 
     work = plan(scan_root, source_root, mirror_root, safe, budget)
 
@@ -617,6 +802,8 @@ def run_once(
             counts[result.action] += 1
             if result.action == "failed":
                 failures.append(result)
+            elif result.action != "skipped":
+                written.add(result.path.parent)
 
     expected = set(work)
     if safe and dry_run:
@@ -627,20 +814,25 @@ def run_once(
         for source in work.values():
             expected.add(mirror_path_for(source, source_root, mirror_root))
             expected.add(mirror_path_for(source, source_root, mirror_root, True))
-    removed = prune(mirror_root, expected, dry_run) if do_prune else 0
+    removed, emptied = prune(mirror_root, expected, dry_run) if do_prune else (0, set())
+
+    # After pruning, so an album is not measured with a track in it that is
+    # about to be deleted.
+    levelled = replaygain(expected, written | emptied, jobs, dry_run) if do_replaygain else 0
 
     for failure in failures:
         logger.error("failed: %s: %s", failure.path, failure.error)
 
     logger.info(
         "pass complete in %.1fs: %d encoded, %d copied, %d renamed, %d up to date,"
-        " %d removed, %d failed",
+        " %d removed, %d levelled, %d failed",
         time.monotonic() - started,
         counts["encoded"],
         counts["copied"],
         counts["renamed"],
         counts["skipped"],
         removed,
+        levelled,
         counts["failed"],
     )
     return counts["failed"]
@@ -740,6 +932,13 @@ def build_parser():
         action="store_true",
         help="keep mirror files whose source has been deleted",
     )
+    parser.add_argument(
+        "--no-replaygain",
+        action="store_true",
+        default=os.getenv("MUSIC_MIRROR_REPLAYGAIN", "").lower() in ("0", "false", "no"),
+        help="do not write ReplayGain tags; skips the rsgain pass over changed"
+        " albums (env MUSIC_MIRROR_REPLAYGAIN=0)",
+    )
     parser.add_argument(
         "--dry-run",
         action="store_true",
@@ -831,6 +1030,7 @@ def main(argv=None):
                 do_prune,
                 args.fat32_safe,
                 budget,
+                not args.no_replaygain,
             )
             if interval is None or stopping:
                 return 1 if failures else 0

pyproject.toml

@@ -8,8 +8,8 @@ version = "0.4.0"
 description = "Maintain a lossy MP3 mirror of a lossless music library"
 readme = "README.md"
 requires-python = ">=3.11"
-# No runtime Python dependencies: the work is done by ffmpeg, which must be on
+# No runtime Python dependencies: the work is done by ffmpeg and rsgain, which
-# PATH.
+# must be on PATH. A missing rsgain costs the ReplayGain tags and nothing else.
 dependencies = []
 
 [project.scripts]

tests/conftest.py

@@ -19,6 +19,13 @@ def require_ffmpeg():
             pytest.skip(f"{tool} is not on PATH", allow_module_level=True)
 
 
+@pytest.fixture
+def require_rsgain():
+    """Skip a test that measures loudness for real rather than faking it."""
+    if shutil.which("rsgain") is None:
+        pytest.skip("rsgain is not on PATH")
+
+
 @pytest.fixture
 def tight_umask():
     """Run a test under a umask that would otherwise make the mirror private."""
@@ -46,7 +53,14 @@ def owner_hostile_umask():
 def make_flac():
     """Return a factory writing a short tagged FLAC file."""
 
-    def factory(path, title="Test Title", artist="Test Artist", album="Test Album", seconds=1):
+    def factory(
+        path,
+        title="Test Title",
+        artist="Test Artist",
+        album="Test Album",
+        seconds=1,
+        gain=0,
+    ):
         path.parent.mkdir(parents=True, exist_ok=True)
         subprocess.run(
             [
@@ -60,6 +74,9 @@ def make_flac():
                 "lavfi",
                 "-i",
                 f"sine=frequency=440:duration={seconds}",
+                # Quieter or louder than the default, for tests that need two
+                # tracks at different levels.
+                *(["-af", f"volume={gain}dB"] if gain else []),
                 "-metadata",
                 f"title={title}",
                 "-metadata",

tests/test_music_mirror.py

@@ -14,6 +14,16 @@ def run(source, mirror, *extra):
     return music_mirror.main(["--source", str(source), "--mirror", str(mirror), *extra])
 
 
+def audio_frames(path):
+    """Return a hash of a file's audio frames, ignoring its tags."""
+    return subprocess.run(
+        ["ffmpeg", "-v", "error", "-i", str(path), "-map", "0:a", "-c", "copy", "-f", "md5", "-"],
+        check=True,
+        capture_output=True,
+        text=True,
+    ).stdout.strip()
+
+
 def test_parse_quality_accepts_vbr_and_cbr():
     assert music_mirror.parse_quality("V0") == ["-q:a", "0"]
     assert music_mirror.parse_quality("v2") == ["-q:a", "2"]
@@ -61,8 +71,10 @@ def test_output_is_mp3(tmp_path, make_flac):
             "a:0",
             "-show_entries",
             "stream=codec_name",
+            # Not csv: a levelled file carries ReplayGain side data, which the
+            # csv writer renders as a trailing empty field.
             "-of",
-            "csv=p=0",
+            "default=noprint_wrappers=1:nokey=1",
             str(mirror / "a.mp3"),
         ],
         check=True,
@@ -140,6 +152,9 @@ def test_no_prune_keeps_orphans(tmp_path, make_flac):
 
 
 def test_existing_mp3_is_copied_not_re_encoded(tmp_path, make_flac):
+    """Compared by the audio frames rather than the whole file: the copy is
+    tagged with its ReplayGain values afterwards, so the two differ in the
+    container while carrying identical audio."""
     source = tmp_path / "src"
     mirror = tmp_path / "dst"
     flac = make_flac(source / "a.flac")
@@ -152,7 +167,7 @@ def test_existing_mp3_is_copied_not_re_encoded(tmp_path, make_flac):
 
     run(source, mirror)
 
-    assert (mirror / "b.mp3").read_bytes() == (source / "b.mp3").read_bytes()
+    assert audio_frames(mirror / "b.mp3") == audio_frames(source / "b.mp3")
 
 
 def test_interrupted_copy_leaves_nothing_behind(tmp_path, make_flac, monkeypatch):
@@ -786,3 +801,181 @@ def test_the_budget_is_reported_so_it_can_be_checked(tmp_path, make_flac, caplog
         run(source, mirror, "--fat32-safe")
 
     assert "limited to 253 characters" in caplog.text
+
+
+# Rockbox applies the offset a ReplayGain tag carries but never measures
+# loudness itself, so an untagged mirror plays each album at whatever level it
+# was mastered to. The tags have to be written here or nowhere.
+
+
+def test_replaygain_tags_are_written(tmp_path, make_flac, probe_tag, require_rsgain):
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+
+    run(source, mirror)
+
+    track = mirror / "Album" / "a.mp3"
+    assert probe_tag(track, "REPLAYGAIN_TRACK_GAIN").endswith("dB")
+    assert probe_tag(track, "REPLAYGAIN_ALBUM_GAIN").endswith("dB")
+    assert probe_tag(track, "REPLAYGAIN_TRACK_PEAK")
+
+
+def test_the_album_shares_one_gain_and_the_tracks_keep_their_own(
+    tmp_path, make_flac, probe_tag, require_rsgain
+):
+    """Album gain is what keeps a quiet track quiet within a record it belongs
+    to. Track gain is written alongside it so the device can pick the other
+    behaviour when shuffling."""
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "loud.flac")
+    make_flac(source / "Album" / "quiet.flac", gain=-12)
+
+    run(source, mirror)
+
+    loud = mirror / "Album" / "loud.mp3"
+    quiet = mirror / "Album" / "quiet.mp3"
+    assert probe_tag(loud, "REPLAYGAIN_ALBUM_GAIN") == probe_tag(quiet, "REPLAYGAIN_ALBUM_GAIN")
+    assert probe_tag(loud, "REPLAYGAIN_TRACK_GAIN") != probe_tag(quiet, "REPLAYGAIN_TRACK_GAIN")
+
+
+def test_levelling_does_not_make_the_next_pass_re_encode(
+    tmp_path, make_flac, caplog, require_rsgain
+):
+    """Writing tags rewrites the file, and staleness here is an mtime
+    comparison. Without --preserve-mtimes every pass would re-encode the whole
+    library and then level it again, forever."""
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+
+    run(source, mirror)
+    before = (mirror / "Album" / "a.mp3").stat().st_mtime
+
+    with caplog.at_level("INFO"):
+        run(source, mirror)
+
+    assert "0 encoded" in caplog.text
+    assert (mirror / "Album" / "a.mp3").stat().st_mtime == before
+
+
+def test_an_already_levelled_album_is_not_measured_again(
+    tmp_path, make_flac, caplog, require_rsgain
+):
+    """Measuring costs a decode of every track. A pass that changed nothing
+    must not pay it."""
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+    run(source, mirror)
+
+    with caplog.at_level("INFO"):
+        run(source, mirror)
+
+    assert "0 levelled" in caplog.text
+
+
+def test_a_new_track_relevels_the_album_around_it(
+    tmp_path, make_flac, probe_tag, require_rsgain
+):
+    """Album gain is a property of the whole album, so a track arriving late
+    makes the value stored on every one of its siblings wrong.
+
+    -10 dB rather than something more dramatic: R128 gates quiet passages out
+    of the measurement, and a track far enough below the rest of the record is
+    excluded from it entirely."""
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+    run(source, mirror)
+    first = probe_tag(mirror / "Album" / "a.mp3", "REPLAYGAIN_ALBUM_GAIN")
+
+    make_flac(source / "Album" / "b.flac", gain=-10)
+    run(source, mirror)
+
+    assert probe_tag(mirror / "Album" / "a.mp3", "REPLAYGAIN_ALBUM_GAIN") != first
+
+
+def test_a_removed_track_relevels_the_album_behind_it(
+    tmp_path, make_flac, caplog, require_rsgain
+):
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+    make_flac(source / "Album" / "b.flac", gain=-20)
+    run(source, mirror)
+
+    (source / "Album" / "b.flac").unlink()
+    with caplog.at_level("INFO"):
+        run(source, mirror)
+
+    assert "1 levelled" in caplog.text
+
+
+def test_embedded_cover_art_does_not_hide_the_tags(
+    tmp_path, make_flac, make_cover, require_rsgain
+):
+    """The check walks ID3v2 frame headers and seeks over the bodies. Cover art
+    sits between the text frames and the ones rsgain appends, so a check that
+    only read the start of the tag would never reach them -- and would measure
+    every album with a cover on every pass."""
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+    make_cover(source / "Album" / "cover.jpg")
+
+    run(source, mirror)
+
+    assert music_mirror.has_replaygain(mirror / "Album" / "a.mp3")
+
+
+def test_no_replaygain_leaves_the_tags_off(tmp_path, make_flac, probe_tag):
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+
+    run(source, mirror, "--no-replaygain")
+
+    assert not probe_tag(mirror / "Album" / "a.mp3", "REPLAYGAIN_ALBUM_GAIN")
+    assert not music_mirror.has_replaygain(mirror / "Album" / "a.mp3")
+
+
+def test_an_unlevelled_mirror_is_backfilled(tmp_path, make_flac, probe_tag, require_rsgain):
+    """A mirror built before any of this existed has correct mtimes, so no pass
+    would ever revisit those files on its own."""
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+    run(source, mirror, "--no-replaygain")
+
+    run(source, mirror)
+
+    assert probe_tag(mirror / "Album" / "a.mp3", "REPLAYGAIN_ALBUM_GAIN").endswith("dB")
+
+
+def test_a_missing_scanner_is_reported_and_not_fatal(tmp_path, make_flac, caplog, monkeypatch):
+    """The mirror is still correct audio in the right place. It just plays at
+    the level it was mastered to."""
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+    monkeypatch.setattr(music_mirror.shutil, "which", lambda name: None)
+
+    with caplog.at_level("INFO"):
+        assert run(source, mirror) == 0
+
+    assert "rsgain is not on PATH" in caplog.text
+    assert (mirror / "Album" / "a.mp3").is_file()
+
+
+def test_a_dry_run_measures_nothing(tmp_path, make_flac, caplog, require_rsgain):
+    source = tmp_path / "src"
+    mirror = tmp_path / "dst"
+    make_flac(source / "Album" / "a.flac")
+
+    with caplog.at_level("INFO"):
+        run(source, mirror, "--dry-run")
+
+    assert "would level 1 album" in caplog.text
+    assert not mirror.joinpath("Album").exists()

tests/test_sync_to_ipod.py

@@ -5,6 +5,7 @@ protecting against emptying the wrong directory -- a mistake that does not
 announce itself.
 """
 
+import os
 import shutil
 import subprocess
 from pathlib import Path
@@ -346,3 +347,88 @@ def test_counting_can_be_asked_for(mirror, tmp_path):
 
     assert result.returncode == 0, result.stderr
     assert "files to copy" in result.stderr
+
+
+# A ReplayGain re-level rewrites a track's tags in the padding the previous
+# write left behind, so neither the size nor the mtime changes -- and those are
+# the two things rsync's quick check compares.
+
+
+def stale_copy(mirror, destination, relative, current, previous):
+    """Put a file on the device that differs only in content from the mirror's."""
+    source = mirror / relative
+    source.parent.mkdir(parents=True, exist_ok=True)
+    source.write_bytes(current)
+    device = destination / relative
+    device.parent.mkdir(parents=True, exist_ok=True)
+    device.write_bytes(previous)
+    os.utime(device, (source.stat().st_atime, source.stat().st_mtime))
+    return device
+
+
+def test_a_tag_only_change_reaches_the_device_with_the_track_that_caused_it(
+    mirror, tmp_path
+):
+    """The new track is visible to rsync; its re-levelled sibling is not, and
+    would otherwise keep the old album gain on the device forever."""
+    destination = tmp_path / "dest"
+    destination.mkdir()
+    sibling = stale_copy(mirror, destination, "Album/sibling.mp3", b"NEW", b"OLD")
+
+    result = run("-f", "-S", "-U", str(mirror), str(destination))
+
+    assert result.returncode == 0, result.stderr
+    assert sibling.read_bytes() == b"NEW"
+    assert (destination / "Album" / "track.mp3").is_file()
+
+
+def test_a_deletion_also_relevels_what_is_left_behind(mirror, tmp_path):
+    destination = tmp_path / "dest"
+    destination.mkdir()
+    sibling = stale_copy(mirror, destination, "Album/sibling.mp3", b"NEW", b"OLD")
+    (destination / "Album" / "gone.mp3").write_bytes(b"old")
+
+    result = run("-f", "-S", "-U", str(mirror), str(destination))
+
+    assert result.returncode == 0, result.stderr
+    assert sibling.read_bytes() == b"NEW"
+    assert not (destination / "Album" / "gone.mp3").exists()
+
+
+def test_an_untouched_album_is_not_copied_again(mirror, tmp_path):
+    """The second pass is scoped to albums that changed. An album whose file
+    set is the same is left where it is, which is the whole point of not
+    running --ignore-times over the library."""
+    destination = tmp_path / "dest"
+    destination.mkdir()
+    quiet = stale_copy(mirror, destination, "Quiet/only.mp3", b"NEW", b"OLD")
+    (destination / "Album").mkdir()
+    shutil.copy2(mirror / "Album" / "track.mp3", destination / "Album" / "track.mp3")
+
+    result = run("-f", "-S", "-U", str(mirror), str(destination))
+
+    assert result.returncode == 0, result.stderr
+    assert quiet.read_bytes() == b"OLD"
+
+
+def test_a_dry_run_says_how_many_extra_tracks_are_involved(mirror, tmp_path):
+    destination = tmp_path / "dest"
+    destination.mkdir()
+    stale_copy(mirror, destination, "Album/sibling.mp3", b"NEW", b"OLD")
+
+    result = run("-f", "-S", "-U", "-n", str(mirror), str(destination))
+
+    assert result.returncode == 0, result.stderr
+    assert "and 1 more in those albums" in result.stderr
+
+
+def test_a_first_sync_does_not_copy_anything_twice(mirror, tmp_path):
+    """Everything is transferred by the main pass, so there is nothing left for
+    the second one and it must not announce itself."""
+    destination = tmp_path / "dest"
+    destination.mkdir()
+
+    result = run("-f", "-S", "-U", str(mirror), str(destination))
+
+    assert result.returncode == 0, result.stderr
+    assert "re-levelled" not in result.stderr

tests/test_touched_albums.py

@@ -0,0 +1,128 @@
+"""A ReplayGain re-level rewrites a track's tags without changing its size or
+its mtime, which is precisely the pair rsync's quick check compares. These
+cover the list that is fed back to rsync to copy those tracks anyway."""
+
+import io
+import subprocess
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
+
+import touched_albums  # noqa: E402
+
+
+def album(root, name, *tracks):
+    directory = root / name
+    directory.mkdir(parents=True, exist_ok=True)
+    for track in tracks:
+        (directory / track).write_bytes(b"x")
+    return directory
+
+
+def listing(mirror, *lines):
+    return touched_albums.remaining(mirror, *touched_albums.touched(list(lines)))
+
+
+def test_the_siblings_of_a_new_track_are_listed(tmp_path):
+    album(tmp_path, "Artist/Album", "01.mp3", "02.mp3", "03.mp3")
+
+    assert listing(tmp_path, "4096 Artist/Album/03.mp3") == [
+        "Artist/Album/01.mp3",
+        "Artist/Album/02.mp3",
+    ]
+
+
+def test_a_track_rsync_just_copied_is_not_copied_twice(tmp_path):
+    """A quality upgrade replaces every track on the record. Listing them again
+    would send the album across twice."""
+    album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
+
+    assert listing(tmp_path, "4096 Artist/Album/01.mp3", "4096 Artist/Album/02.mp3") == []
+
+
+def test_a_deletion_relevels_what_is_left(tmp_path):
+    album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
+
+    assert listing(tmp_path, "deleting Artist/Album/03.mp3") == [
+        "Artist/Album/01.mp3",
+        "Artist/Album/02.mp3",
+    ]
+
+
+def test_an_album_deleted_outright_lists_nothing(tmp_path):
+    assert listing(tmp_path, "deleting Artist/Gone/01.mp3") == []
+
+
+def test_untouched_albums_are_left_alone(tmp_path):
+    album(tmp_path, "Artist/Changed", "01.mp3", "02.mp3")
+    album(tmp_path, "Artist/Quiet", "01.mp3", "02.mp3")
+
+    assert listing(tmp_path, "4096 Artist/Changed/01.mp3") == ["Artist/Changed/02.mp3"]
+
+
+def test_a_replaced_cover_is_not_an_album_change(tmp_path):
+    """Only a track can change an album's gains, and covers are replaced often
+    enough that treating one as a re-level would copy records for nothing."""
+    album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
+
+    assert listing(tmp_path, "17408 Artist/Album/cover.jpg") == []
+
+
+def test_directories_are_not_mistaken_for_tracks(tmp_path):
+    album(tmp_path, "Artist/Album", "01.mp3")
+
+    assert listing(tmp_path, "4096 Artist/Album/", "deleting Artist/Old/") == []
+
+
+def test_rsync_talking_to_the_operator_is_not_a_path(tmp_path):
+    album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
+
+    assert (
+        listing(
+            tmp_path,
+            "sending incremental file list",
+            "",
+            "sent 1,234 bytes  received 56 bytes  2,580.00 bytes/sec",
+            "total size is 7,890  speedup is 6.12",
+        )
+        == []
+    )
+
+
+def test_a_track_at_the_mirror_root_does_not_pull_in_the_whole_tree(tmp_path):
+    """Nothing writes a mirror this way, but the directory of a root-level file
+    is the root, and recursing from there would be the whole library."""
+    (tmp_path / "loose.mp3").write_bytes(b"x")
+    (tmp_path / "other.mp3").write_bytes(b"x")
+    album(tmp_path, "Artist/Album", "01.mp3")
+
+    assert listing(tmp_path, "4096 loose.mp3") == ["other.mp3"]
+
+
+def test_the_paths_are_written_one_per_line(tmp_path):
+    """They are fed straight back to rsync as --files-from."""
+    album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
+    out = io.StringIO()
+
+    touched_albums.main(
+        ["--mirror", str(tmp_path)],
+        stream=["4096 Artist/Album/01.mp3"],
+        out=out,
+    )
+
+    assert out.getvalue() == "Artist/Album/02.mp3\n"
+
+
+def test_it_runs_as_a_script(tmp_path):
+    album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
+
+    completed = subprocess.run(
+        [sys.executable, touched_albums.__file__, "--mirror", str(tmp_path)],
+        input="4096 Artist/Album/01.mp3\n",
+        capture_output=True,
+        text=True,
+    )
+
+    assert completed.returncode == 0
+    assert completed.stdout == "Artist/Album/02.mp3\n"

tools/sync-to-ipod.sh

@@ -56,6 +56,10 @@ The destination must be on a mounted FAT filesystem. That check is also what
 catches an unmounted device: /media/IPOD/Music then resolves to the host's own
 root filesystem, and this refuses to empty that.
 
+An album that gained or lost a track is copied again in full afterwards. Its
+surviving tracks have had their ReplayGain tags rewritten in place, which
+changes neither their size nor their mtime, so the main pass cannot see them.
+
 Progress is one line that rewrites itself, showing the album currently going
 across, how far through the transfer is, the rate, and an estimate of what is
 left. Working the totals out first means a second pass over the tree, which is
@@ -193,8 +197,23 @@ done
 
 printf 'sync-to-ipod: %s -> %s\n' "$mirror" "$destination" >&2
 
+# Both passes below feed their file list through touched_albums.py, which reads
+# rsync's own report of what it moved. --files-from separates paths by newline,
+# so a filename containing one would be read as two -- which is a path FAT32
+# will not take either, and check_fat32.py above has already refused the run
+# unless -f was given to skip it.
+changed=$(mktemp)
+relevelled=$(mktemp)
+trap 'rm -f "$changed" "$relevelled"' EXIT
+
 if $dry_run; then
-    rsync "${options[@]}" --dry-run --verbose "$mirror/" "$destination/"
+    rsync "${options[@]}" --dry-run --verbose --out-format='%l %n' \
+        "$mirror/" "$destination/" | tee "$changed"
+    also=$(python3 "$here/touched_albums.py" --mirror "$mirror" <"$changed" | wc -l)
+    if [ "$also" -gt 0 ]; then
+        printf 'sync-to-ipod: and %s more in those albums, whose ReplayGain tags\n' "$also" >&2
+        printf 'sync-to-ipod: change without changing their size or their mtime\n' >&2
+    fi
     printf 'sync-to-ipod: dry run, nothing was written\n' >&2
     exit 0
 fi
@@ -256,10 +275,33 @@ interrupted() {
 trap interrupted INT TERM
 
 rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" |
+    tee "$changed" |
     python3 "$here/rsync_progress.py" --total "$total" --bytes "$total_bytes"
 status=${PIPESTATUS[0]}
 [ "$status" -eq 0 ] || die "rsync exited $status"
 
+# A ReplayGain album gain belongs to the whole record, so a track arriving or
+# leaving rewrites the tags on all of its siblings. rsgain fits the new values
+# into the padding its previous write left behind, which changes neither the
+# size nor the mtime -- the only two things the pass above compares. Those
+# tracks are invisible to it, and the device would keep the old gains.
+#
+# The track that arrived or left is visible, though. So every album the pass
+# touched has the rest of its tracks copied again, with --ignore-times to
+# defeat the same quick check. No --delete: the pass above has already settled
+# what should be on the device, and --delete aimed at an explicit file list
+# does not mean what it looks like it means.
+python3 "$here/touched_albums.py" --mirror "$mirror" <"$changed" >"$relevelled"
+if [ -s "$relevelled" ]; then
+    also=$(wc -l <"$relevelled")
+    printf 'sync-to-ipod: re-copying %s tracks whose album was re-levelled\n' "$also" >&2
+    rsync --times --modify-window=2 --whole-file --omit-dir-times --ignore-times \
+        --files-from="$relevelled" --out-format='%l %n' "$mirror/" "$destination/" |
+        python3 "$here/rsync_progress.py" --total "$also"
+    status=${PIPESTATUS[0]}
+    [ "$status" -eq 0 ] || die "the re-levelled tracks failed to copy: rsync exited $status"
+fi
+
 # Rockbox reads its database from .tcd files in .rockbox. Building them here
 # rather than on the device is not just faster: the on-device commit sorts the
 # whole index in whatever memory it can scrape together, and on a large library

tools/touched_albums.py

@@ -0,0 +1,97 @@
+#!/usr/bin/env python3
+"""List the tracks a sync has to copy again because their album was re-levelled.
+
+Reads rsync's `--out-format='%l %n'` output on stdin and writes mirror-relative
+paths on stdout, one per line, for feeding straight back to rsync as
+`--files-from`.
+
+The problem it solves: a ReplayGain album gain is a property of every track on
+the record, so one track arriving or leaving changes the tags on all of its
+siblings. rsgain writes the new values into the padding its previous write left
+behind, which leaves both the file's size and its mtime untouched -- and size
+and mtime are exactly what rsync's quick check compares. It sees nothing to do,
+and the device keeps the old gains.
+
+What rsync always can see is the track that arrived or left. So any album it
+touched has the rest of its tracks copied again, and nothing else does.
+
+Tracks rsync has already dealt with are left out of the list. After a quality
+upgrade that replaces every track on a record, listing them again would send
+the whole album twice.
+"""
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+# Under --out-format='%l %n' a transfer is reported as the size, a space and
+# the path. A removal is reported as "deleting <path>" whatever the format is.
+# Everything else on the stream is rsync talking to the operator -- the file
+# list preamble, the byte totals -- and is not a path.
+TRANSFER = re.compile(r"^(\d+) (.+)$")
+DELETION = re.compile(r"^deleting (.+)$")
+
+# What the mirror is made of, and so the only thing worth copying again. A
+# cover is not rewritten by a re-level.
+MIRROR_SUFFIX = ".mp3"
+
+
+def touched(lines):
+    """Return the paths rsync transferred and the directories it changed."""
+    transferred = set()
+    directories = set()
+
+    for line in lines:
+        line = line.rstrip("\n")
+        deletion = DELETION.match(line)
+        transfer = None if deletion else TRANSFER.match(line)
+        if deletion:
+            path = deletion.group(1)
+        elif transfer:
+            path = transfer.group(2)
+        else:
+            continue
+        # Only a track can change an album's gains. rsync reports the
+        # directories it creates and removes too, and a cover replaced on its
+        # own is no reason to send the record again.
+        if not path.endswith(MIRROR_SUFFIX):
+            continue
+        if transfer:
+            transferred.add(path)
+        directories.add(path.rpartition("/")[0])
+
+    return transferred, directories
+
+
+def remaining(mirror, transferred, directories):
+    """Return the tracks in those directories that rsync has not just copied."""
+    paths = []
+    for directory in sorted(directories):
+        album = mirror / directory if directory else mirror
+        if not album.is_dir():
+            # Removed along with the last of its tracks. Nothing to copy.
+            continue
+        for track in sorted(album.glob(f"*{MIRROR_SUFFIX}")):
+            relative = f"{directory}/{track.name}" if directory else track.name
+            if relative not in transferred:
+                paths.append(relative)
+    return paths
+
+
+def main(argv=None, stream=None, out=None):
+    parser = argparse.ArgumentParser(
+        prog="touched_albums.py",
+        description="List the tracks to copy again after an album was re-levelled.",
+    )
+    parser.add_argument("--mirror", required=True, type=Path, help="root of the MP3 mirror")
+    args = parser.parse_args(argv)
+
+    transferred, directories = touched(stream if stream is not None else sys.stdin)
+    for path in remaining(args.mirror, transferred, directories):
+        print(path, file=out if out is not None else sys.stdout)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())