Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Emma ThorpeandClaude Opus 5
|
c12d2328e5 |
+2
-4
@@ -7,10 +7,8 @@ FROM python:3.13-alpine AS runtime
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
# ffmpeg does the encoding and rsgain the volume levelling; the application
|
||||
# itself has no Python dependencies. rsgain lives in the community repository,
|
||||
# which the official Python images already have enabled.
|
||||
RUN apk add --no-cache ffmpeg rsgain
|
||||
# ffmpeg does the encoding; the application itself has no Python dependencies.
|
||||
RUN apk add --no-cache ffmpeg
|
||||
|
||||
WORKDIR /app
|
||||
COPY pyproject.toml README.md ./
|
||||
|
||||
@@ -22,7 +22,6 @@ remembering to do anything.
|
||||
| Mirror up to date | skip |
|
||||
| Source is already MP3 | copy verbatim |
|
||||
| Source gone | delete the mirror file, prune empty dirs |
|
||||
| An album gained or lost a track | re-measure its ReplayGain, tags only |
|
||||
|
||||
Freshness is modification time: an encoded file is stamped with its source's
|
||||
mtime, so a file is stale exactly when the two differ. There is no database to
|
||||
@@ -95,7 +94,6 @@ music-mirror --source /music --mirror /music-mp3 --subdir "Artist/Album"
|
||||
| `--interval` | `MUSIC_MIRROR_INTERVAL` | unset | Repeat forever, e.g. `45m`, `6h`, `1d` |
|
||||
| `--subdir` | — | unset | Limit the pass to one directory; skips pruning |
|
||||
| `--fat32-safe` | `MUSIC_MIRROR_FAT32_SAFE` | off | Name files so a FAT32 device accepts them |
|
||||
| `--no-replaygain` | `MUSIC_MIRROR_REPLAYGAIN` | on | Write ReplayGain tags; set the variable to `0` to skip |
|
||||
| `--no-prune` | — | off | Keep mirror files whose source has gone |
|
||||
| `--dry-run` | — | off | Report what would change, write nothing |
|
||||
|
||||
@@ -116,9 +114,7 @@ The first full pass is the expensive one; after that only new and changed files
|
||||
are touched. Lower `MUSIC_MIRROR_JOBS` if you would rather the NAS stayed
|
||||
responsive than finished sooner.
|
||||
|
||||
Requires `ffmpeg` and `ffprobe` on `PATH`, and `rsgain` for volume levelling.
|
||||
The container image provides all three. A missing `rsgain` is reported once per
|
||||
pass and costs the ReplayGain tags; everything else still runs.
|
||||
Requires `ffmpeg` and `ffprobe` on `PATH`. The container image provides both.
|
||||
|
||||
## Running it on TrueNAS Scale
|
||||
|
||||
@@ -197,16 +193,6 @@ flight; only the two figures that needed the second walk are missing.
|
||||
Piped to a log it prints a plain line every thirty seconds instead, with no
|
||||
carriage returns, and a summary at the end either way.
|
||||
|
||||
### Albums that changed are copied whole
|
||||
|
||||
After the main pass, any album that gained or lost a track has the rest of its
|
||||
tracks copied again with `--ignore-times`. Their ReplayGain tags were rewritten
|
||||
in place when the album was re-levelled, which changes neither their size nor
|
||||
their mtime — the only two things rsync compares — so nothing else would ever
|
||||
send them. `touched_albums.py` works out the list from rsync's own report of
|
||||
what it moved, so this costs no extra traversal, and tracks the main pass has
|
||||
already copied are left out of it. See [Volume levelling](#volume-levelling).
|
||||
|
||||
### The Rockbox database
|
||||
|
||||
Point `MUSIC_MIRROR_DATABASE_TOOL` at Rockbox's host-side builder and the sync
|
||||
@@ -323,6 +309,47 @@ real-time clock Rockbox writes `/.scrobbler-timeless.log` with every timestamp
|
||||
set to zero; those are counted and reported but never submitted, because
|
||||
scrobbling them would mean inventing when they happened.
|
||||
|
||||
### Timestamps are local wall clock, and are corrected here
|
||||
|
||||
Rockbox has no concept of a timezone. Its clock is set to local time, and it
|
||||
builds log timestamps with `mktime(get_time())` — but
|
||||
[its `mktime`](https://git.rockbox.org/cgit/rockbox.git/tree/firmware/libc/mktime.c)
|
||||
is plain calendar arithmetic that applies no offset, so the RTC's local fields
|
||||
come out as if they were UTC. The number in the log is therefore ahead of the
|
||||
real instant by whatever the offset was. Last.fm stores UTC, so submitting it
|
||||
raw puts every play an hour into the future for the half of the year the UK is
|
||||
on BST.
|
||||
|
||||
Rockbox is candid about this: its scrobbler plugin writes `#TZ/UNKNOWN` in the
|
||||
log header, and the AUDIOSCROBBLER spec says a device may claim `#TZ/UTC` only
|
||||
if it actually converted. The correction is the consumer's job.
|
||||
|
||||
Each timestamp is decoded back into the wall-clock fields it came from and
|
||||
reinterpreted in the player's zone. Doing it **per play** rather than applying
|
||||
one offset to the whole log matters: a week's listening can straddle a daylight
|
||||
saving change, and the two sides need different offsets. A log that declares
|
||||
`#TZ/UTC` is left alone, so a client that already converted is not shifted
|
||||
twice.
|
||||
|
||||
The zone defaults to this machine's. Set `ROCKBOX_TIMEZONE` (or pass
|
||||
`--device-timezone`) to an IANA name if the player's clock is keeping a
|
||||
different one.
|
||||
|
||||
Because Rockbox cannot adjust for daylight saving itself, **you have to change
|
||||
the player's clock by hand twice a year**. If you forget, its times are an hour
|
||||
out and no amount of zone arithmetic recovers them. The submitter reports any
|
||||
play that converts to a time in the future, which is what a forgotten
|
||||
adjustment looks like:
|
||||
|
||||
```
|
||||
37 plays are timestamped up to 58 minutes in the future, converting from
|
||||
Europe/London. Either the player's clock is wrong or that is not the zone
|
||||
it is set to.
|
||||
```
|
||||
|
||||
`--dry-run` prints each play's local time beside the epoch, so the conversion
|
||||
can be checked against when you actually remember listening.
|
||||
|
||||
### Nothing played is thrown away
|
||||
|
||||
Two separate obligations, because a play that happened and never reached
|
||||
@@ -361,8 +388,6 @@ docker build --target test . # what CI runs
|
||||
pytest # needs ffmpeg and pytest on PATH
|
||||
```
|
||||
|
||||
The ReplayGain tests need `rsgain` as well and skip without it.
|
||||
|
||||
The suite runs real ffmpeg encodes rather than mocking them. The interesting
|
||||
failures are in what ffmpeg actually does with tags, cover art and container
|
||||
formats, and a mock cannot fail that way — which is also why CI runs the tests
|
||||
@@ -374,7 +399,7 @@ Run them directly instead if you prefer; they skip when ffmpeg is absent. On a
|
||||
Nix machine:
|
||||
|
||||
```sh
|
||||
nix shell nixpkgs#python3Packages.pytest nixpkgs#ffmpeg nixpkgs#rsgain -c pytest
|
||||
nix shell nixpkgs#python3Packages.pytest nixpkgs#ffmpeg -c pytest
|
||||
```
|
||||
|
||||
## FAT32 and Rockbox
|
||||
@@ -464,64 +489,6 @@ firmware reads the embedded one; Rockbox reads the file. Both are satisfied.
|
||||
A cover left behind in a directory whose tracks have all gone is pruned, or the
|
||||
directory would never look empty and never be removed.
|
||||
|
||||
### Volume levelling
|
||||
|
||||
Rockbox can level the volume between tracks, but only from tags. It applies the
|
||||
offset a ReplayGain tag carries and has no loudness analysis of its own, so a
|
||||
mirror without those tags plays every album at whatever level it was mastered
|
||||
to — and a 2008 remaster next to a 1972 pressing is a reach for the volume
|
||||
wheel on every track change.
|
||||
|
||||
The tags are therefore written here, with `rsgain`, once an album's tracks are
|
||||
in place. Both album gain and track gain are measured: album gain preserves the
|
||||
quiet track that a record is supposed to have, track gain is the one that makes
|
||||
sense on shuffle, and which of them is used is the device's decision, not this
|
||||
one.
|
||||
|
||||
Turn it on at the player end under **Settings → Playback Settings →
|
||||
Replaygain**:
|
||||
|
||||
| Setting | Suggested | Why |
|
||||
| ---------------- | ---------------------------- | ---------------------------------------------------------- |
|
||||
| Replaygain type | `Track Gain if Shuffling` | Album gain while playing a record, track gain once shuffle is on — the only setting that is right in both cases |
|
||||
| Prevent clipping | `Yes` | Backs the gain off using the peak tags rather than distorting |
|
||||
| Pre-amp | `0 dB` | The reference is already −18 LUFS; raise it only if everything ends up too quiet |
|
||||
|
||||
`Track Gain if Shuffling` is not a compromise between the other two — Rockbox
|
||||
reads the shuffle setting and picks whole-hog album or track gain from it
|
||||
(`apps/misc.c`, `replaygain_setting_mode`). It is also Rockbox's default, so on
|
||||
a fresh install there may be nothing to change but `Prevent clipping`.
|
||||
|
||||
Measuring costs a full decode of every track, so the first pass after enabling
|
||||
it takes roughly as long as the original encode did. After that only albums
|
||||
that gained, lost or replaced a track are re-measured. An album is re-measured
|
||||
as a whole, because album gain is a property of all of its tracks and one new
|
||||
track makes the value stored on every sibling wrong.
|
||||
|
||||
Tagging rewrites the file, and staleness here is an mtime comparison, so
|
||||
`rsgain` is run with `--preserve-mtimes`. Without it every levelled track would
|
||||
look newer than its source and the next pass would re-encode the entire
|
||||
library.
|
||||
|
||||
That has a consequence for the sync, and it is not obvious. The first time a
|
||||
track is levelled its tag grows by about a kilobyte, so its size changes and
|
||||
rsync copies it — the whole library goes across once, and there is no way
|
||||
around that; the tag sits at the head of the file and every byte after it
|
||||
moves. But `rsgain` leaves padding behind, so a *later* re-level fits inside it
|
||||
and changes neither the size nor the mtime:
|
||||
|
||||
```
|
||||
before re-level: 277757 bytes, mtime 1577880000, album gain 3.75 dB
|
||||
after re-level: 277757 bytes, mtime 1577880000, album gain 6.25 dB
|
||||
```
|
||||
|
||||
Those are the two things rsync's quick check compares, so it would see nothing
|
||||
to do and the device would keep the old gains. `sync-to-ipod.sh` handles it: the
|
||||
track that arrived or left is always visible, so any album the main pass
|
||||
touched has the rest of its tracks copied again with `--ignore-times`. Albums
|
||||
whose file set has not changed are not touched, which is what keeps this from
|
||||
being a full re-copy.
|
||||
|
||||
## Getting the result onto an iPod
|
||||
|
||||
The mirror is just a directory of MP3s, so any client will do:
|
||||
@@ -561,6 +528,6 @@ The mirror is just a directory of MP3s, so any client will do:
|
||||
Neither client transcodes at sync time; they copy finished MP3s.
|
||||
|
||||
Two device-side details worth knowing: the iPod reads cover art from the file's
|
||||
tags and ignores `folder.jpg`, which is why art is embedded here; and the Apple
|
||||
firmware levels volume from iTunes' Soundcheck tag rather than ReplayGain, so
|
||||
the tags written here do nothing until the iPod is running Rockbox.
|
||||
tags and ignores `folder.jpg`, which is why art is embedded here; and volume
|
||||
levelling on the device uses iTunes' Soundcheck tag, not ReplayGain, so
|
||||
ReplayGain tags in the source are not carried over as such.
|
||||
|
||||
@@ -31,11 +31,6 @@ services:
|
||||
# enabling it is quick -- but it is a one-way change to every such path,
|
||||
# so decide before running it rather than after.
|
||||
MUSIC_MIRROR_FAT32_SAFE: "true"
|
||||
# ReplayGain tags, so Rockbox can level the volume between albums. On by
|
||||
# default; set to 0 to skip the measuring pass. The first pass after
|
||||
# enabling it levels the whole library, which costs a decode of every
|
||||
# track, so expect it to take about as long as the original encode.
|
||||
# MUSIC_MIRROR_REPLAYGAIN: "0"
|
||||
volumes:
|
||||
- /mnt/tank/media/music:/music:ro
|
||||
- /mnt/tank/media/music-mp3:/mirror
|
||||
|
||||
+3
-203
@@ -11,11 +11,6 @@ writes to the source library.
|
||||
Staleness is tracked by modification time: an encoded file is given its
|
||||
source's mtime, so a file is out of date exactly when the two differ. That
|
||||
makes runs idempotent without a database to keep in step.
|
||||
|
||||
Finished albums are levelled with rsgain, which writes ReplayGain tags into the
|
||||
mirror. Rockbox applies the offset those tags carry but has no loudness
|
||||
analysis of its own, so without them every album plays at whatever level it was
|
||||
mastered to.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
@@ -86,21 +81,6 @@ MIRROR_COVER = "cover.jpg"
|
||||
# Files the mirror is allowed to contain, and therefore allowed to delete.
|
||||
MIRROR_SUFFIX = ".mp3"
|
||||
|
||||
# Measures loudness and writes the ReplayGain tags. Not a hard requirement: a
|
||||
# pass without it still produces a correct mirror, only one the player cannot
|
||||
# level, so a missing binary is a warning rather than a failure.
|
||||
REPLAYGAIN_TOOL = "rsgain"
|
||||
|
||||
# Looked for in a file's ID3v2 tag to tell a levelled track from an unlevelled
|
||||
# one. Album gain rather than track gain because the album value is the one
|
||||
# this writes for; a file carrying only track gain came from somewhere else and
|
||||
# should be rescanned.
|
||||
REPLAYGAIN_TAG = b"replaygain_album_gain"
|
||||
|
||||
# Enough of a TXXX frame body to hold the encoding byte and the description.
|
||||
# The value after it says what the gain is, which is not the question here.
|
||||
TXXX_DESCRIPTION_BYTES = 128
|
||||
|
||||
# Filesystems disagree about mtime precision; SMB in particular rounds.
|
||||
MTIME_TOLERANCE_SECONDS = 2
|
||||
|
||||
@@ -561,13 +541,8 @@ def prune(mirror_root, expected, dry_run):
|
||||
Driven by the set of paths the pass expects to exist rather than by
|
||||
probing the source tree for names, which would disagree with it over
|
||||
letter case and over any extension the walker does not collect.
|
||||
|
||||
Returns the number of files removed and the directories they came out of.
|
||||
Losing a track changes an album's loudness, so those directories need
|
||||
levelling again even though nothing was written into them.
|
||||
"""
|
||||
removed = 0
|
||||
emptied = set()
|
||||
|
||||
for mirror in sorted(mirror_root.rglob(f"*{MIRROR_SUFFIX}")):
|
||||
if mirror in expected:
|
||||
@@ -578,7 +553,6 @@ def prune(mirror_root, expected, dry_run):
|
||||
continue
|
||||
logger.info("removing orphan %s", mirror)
|
||||
mirror.unlink(missing_ok=True)
|
||||
emptied.add(mirror.parent)
|
||||
|
||||
if not dry_run:
|
||||
# A cover copied for an album whose tracks have all gone is an orphan
|
||||
@@ -593,164 +567,7 @@ def prune(mirror_root, expected, dry_run):
|
||||
if directory.is_dir() and not any(directory.iterdir()):
|
||||
directory.rmdir()
|
||||
|
||||
return removed, emptied
|
||||
|
||||
|
||||
def syncsafe(data):
|
||||
"""Return the integer held in syncsafe bytes: seven bits of each."""
|
||||
value = 0
|
||||
for byte in data:
|
||||
value = (value << 7) | (byte & 0x7F)
|
||||
return value
|
||||
|
||||
|
||||
def has_replaygain(path):
|
||||
"""Return whether an MP3 already carries ReplayGain tags.
|
||||
|
||||
Walks the ID3v2 frame headers and seeks over the bodies rather than reading
|
||||
the tag whole. Every file in this mirror has its cover art embedded, so the
|
||||
tag is routinely half a megabyte; reading all of it for every track on
|
||||
every pass would turn an idle pass into a full read of the library.
|
||||
"""
|
||||
try:
|
||||
with open(path, "rb") as handle:
|
||||
header = handle.read(10)
|
||||
if len(header) < 10 or header[:3] != b"ID3" or header[3] not in (3, 4):
|
||||
return False
|
||||
remaining = syncsafe(header[6:10])
|
||||
|
||||
# Unsynchronisation shifts every offset in the tag, and the two
|
||||
# versions describe an extended header differently. Nothing that
|
||||
# writes this mirror emits either, so reading the tag whole is a
|
||||
# cheaper answer than the code to walk one that does.
|
||||
if header[5] & 0xC0:
|
||||
return REPLAYGAIN_TAG in handle.read(remaining).lower()
|
||||
|
||||
while remaining >= 10:
|
||||
frame = handle.read(10)
|
||||
remaining -= 10
|
||||
# Frame ids are upper-case letters and digits, so anything else
|
||||
# is the padding that follows the last frame.
|
||||
if len(frame) < 10 or not frame[:4].isalnum():
|
||||
return False
|
||||
# 2.3 sizes count all eight bits per byte; 2.4 made them
|
||||
# syncsafe like the tag length above.
|
||||
length = (
|
||||
int.from_bytes(frame[4:8], "big")
|
||||
if header[3] == 3
|
||||
else syncsafe(frame[4:8])
|
||||
)
|
||||
if length <= 0 or length > remaining:
|
||||
return False
|
||||
if frame[:4] == b"TXXX":
|
||||
body = handle.read(min(length, TXXX_DESCRIPTION_BYTES))
|
||||
handle.seek(length - len(body), os.SEEK_CUR)
|
||||
if REPLAYGAIN_TAG in body.lower():
|
||||
return True
|
||||
else:
|
||||
handle.seek(length, os.SEEK_CUR)
|
||||
remaining -= length
|
||||
except OSError:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def replaygain_albums(expected, written):
|
||||
"""Return the album directories needing a scan, each with its tracks.
|
||||
|
||||
A directory is scanned when this pass changed what is in it, because album
|
||||
gain is a property of the whole album: one track added, replaced or removed
|
||||
makes the value stored on every one of its siblings wrong. It is also
|
||||
scanned when a track in it has never been levelled, which is what backfills
|
||||
a mirror built before any of this existed.
|
||||
"""
|
||||
albums = {}
|
||||
for mirror in expected:
|
||||
albums.setdefault(mirror.parent, []).append(mirror)
|
||||
|
||||
needed = {}
|
||||
for directory, tracks in sorted(albums.items()):
|
||||
# A dry run reaches here before anything has been encoded, so the
|
||||
# tracks a changed album is going to hold do not exist yet.
|
||||
present = sorted(track for track in tracks if track.is_file())
|
||||
if directory in written:
|
||||
needed[directory] = present
|
||||
elif present and not all(map(has_replaygain, present)):
|
||||
needed[directory] = present
|
||||
return needed
|
||||
|
||||
|
||||
def replaygain_command(tracks):
|
||||
"""Return the rsgain command that levels one album directory."""
|
||||
return [
|
||||
REPLAYGAIN_TOOL,
|
||||
"custom",
|
||||
# Album mode writes the per-track tags as well as the album ones, so
|
||||
# the device is left to choose between them -- Rockbox can apply track
|
||||
# gain when shuffling and album gain otherwise, and only if both are
|
||||
# present.
|
||||
"--album",
|
||||
"--tagmode=i",
|
||||
# The mirror is ID3v2.3 for the iPod firmware's sake. rsgain would
|
||||
# otherwise keep whatever version it found, and "whatever it found" is
|
||||
# not a guarantee.
|
||||
"--id3v2-version=3",
|
||||
# Staleness here is an mtime comparison and tagging rewrites the file.
|
||||
# Without this every levelled track would look newer than its source
|
||||
# and the next pass would re-encode the entire library, forever.
|
||||
"--preserve-mtimes",
|
||||
"--quiet",
|
||||
*[str(track) for track in tracks],
|
||||
]
|
||||
|
||||
|
||||
def scan_album(directory, tracks):
|
||||
"""Write ReplayGain tags across one album. Returns whether it worked."""
|
||||
completed = subprocess.run(replaygain_command(tracks), capture_output=True, text=True)
|
||||
if completed.returncode != 0:
|
||||
lines = completed.stderr.strip().splitlines()
|
||||
logger.warning("could not level %s: %s", directory, lines[-1] if lines else "rsgain failed")
|
||||
return False
|
||||
logger.info("levelled %s", directory)
|
||||
return True
|
||||
|
||||
|
||||
def replaygain(expected, written, jobs, dry_run):
|
||||
"""Write ReplayGain tags into the albums that need them. Returns how many.
|
||||
|
||||
A failure here is reported and then left alone. The mirror is still correct
|
||||
audio in the right place; it just plays at the level it was mastered to,
|
||||
which is what every pass before this one produced.
|
||||
"""
|
||||
albums = {
|
||||
directory: tracks
|
||||
for directory, tracks in replaygain_albums(expected, written).items()
|
||||
if tracks or dry_run
|
||||
}
|
||||
if not albums:
|
||||
return 0
|
||||
|
||||
if dry_run:
|
||||
logger.info("would level %d album%s", len(albums), "" if len(albums) == 1 else "s")
|
||||
return len(albums)
|
||||
|
||||
if shutil.which(REPLAYGAIN_TOOL) is None:
|
||||
logger.warning(
|
||||
"%s is not on PATH; %d albums are left without ReplayGain tags",
|
||||
REPLAYGAIN_TOOL,
|
||||
len(albums),
|
||||
)
|
||||
return 0
|
||||
|
||||
levelled = 0
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool:
|
||||
futures = [
|
||||
pool.submit(scan_album, directory, tracks) for directory, tracks in albums.items()
|
||||
]
|
||||
for future in concurrent.futures.as_completed(futures):
|
||||
if future.result():
|
||||
levelled += 1
|
||||
return levelled
|
||||
return removed
|
||||
|
||||
|
||||
def run_once(
|
||||
@@ -763,7 +580,6 @@ def run_once(
|
||||
do_prune,
|
||||
safe=False,
|
||||
budget=0,
|
||||
do_replaygain=True,
|
||||
):
|
||||
"""Run a single pass. Returns the number of failures.
|
||||
|
||||
@@ -774,7 +590,6 @@ def run_once(
|
||||
logger.info("pass starting with %d concurrent encoders", jobs)
|
||||
counts = {"encoded": 0, "copied": 0, "renamed": 0, "skipped": 0, "failed": 0}
|
||||
failures = []
|
||||
written = set()
|
||||
|
||||
work = plan(scan_root, source_root, mirror_root, safe, budget)
|
||||
|
||||
@@ -802,8 +617,6 @@ def run_once(
|
||||
counts[result.action] += 1
|
||||
if result.action == "failed":
|
||||
failures.append(result)
|
||||
elif result.action != "skipped":
|
||||
written.add(result.path.parent)
|
||||
|
||||
expected = set(work)
|
||||
if safe and dry_run:
|
||||
@@ -814,25 +627,20 @@ def run_once(
|
||||
for source in work.values():
|
||||
expected.add(mirror_path_for(source, source_root, mirror_root))
|
||||
expected.add(mirror_path_for(source, source_root, mirror_root, True))
|
||||
removed, emptied = prune(mirror_root, expected, dry_run) if do_prune else (0, set())
|
||||
|
||||
# After pruning, so an album is not measured with a track in it that is
|
||||
# about to be deleted.
|
||||
levelled = replaygain(expected, written | emptied, jobs, dry_run) if do_replaygain else 0
|
||||
removed = prune(mirror_root, expected, dry_run) if do_prune else 0
|
||||
|
||||
for failure in failures:
|
||||
logger.error("failed: %s: %s", failure.path, failure.error)
|
||||
|
||||
logger.info(
|
||||
"pass complete in %.1fs: %d encoded, %d copied, %d renamed, %d up to date,"
|
||||
" %d removed, %d levelled, %d failed",
|
||||
" %d removed, %d failed",
|
||||
time.monotonic() - started,
|
||||
counts["encoded"],
|
||||
counts["copied"],
|
||||
counts["renamed"],
|
||||
counts["skipped"],
|
||||
removed,
|
||||
levelled,
|
||||
counts["failed"],
|
||||
)
|
||||
return counts["failed"]
|
||||
@@ -932,13 +740,6 @@ def build_parser():
|
||||
action="store_true",
|
||||
help="keep mirror files whose source has been deleted",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-replaygain",
|
||||
action="store_true",
|
||||
default=os.getenv("MUSIC_MIRROR_REPLAYGAIN", "").lower() in ("0", "false", "no"),
|
||||
help="do not write ReplayGain tags; skips the rsgain pass over changed"
|
||||
" albums (env MUSIC_MIRROR_REPLAYGAIN=0)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
@@ -1030,7 +831,6 @@ def main(argv=None):
|
||||
do_prune,
|
||||
args.fat32_safe,
|
||||
budget,
|
||||
not args.no_replaygain,
|
||||
)
|
||||
if interval is None or stopping:
|
||||
return 1 if failures else 0
|
||||
|
||||
+3
-3
@@ -4,12 +4,12 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "music-mirror"
|
||||
version = "0.5.0"
|
||||
version = "0.4.0"
|
||||
description = "Maintain a lossy MP3 mirror of a lossless music library"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
# No runtime Python dependencies: the work is done by ffmpeg and rsgain, which
|
||||
# must be on PATH. A missing rsgain costs the ReplayGain tags and nothing else.
|
||||
# No runtime Python dependencies: the work is done by ffmpeg, which must be on
|
||||
# PATH.
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
|
||||
+1
-18
@@ -19,13 +19,6 @@ def require_ffmpeg():
|
||||
pytest.skip(f"{tool} is not on PATH", allow_module_level=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def require_rsgain():
|
||||
"""Skip a test that measures loudness for real rather than faking it."""
|
||||
if shutil.which("rsgain") is None:
|
||||
pytest.skip("rsgain is not on PATH")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tight_umask():
|
||||
"""Run a test under a umask that would otherwise make the mirror private."""
|
||||
@@ -53,14 +46,7 @@ def owner_hostile_umask():
|
||||
def make_flac():
|
||||
"""Return a factory writing a short tagged FLAC file."""
|
||||
|
||||
def factory(
|
||||
path,
|
||||
title="Test Title",
|
||||
artist="Test Artist",
|
||||
album="Test Album",
|
||||
seconds=1,
|
||||
gain=0,
|
||||
):
|
||||
def factory(path, title="Test Title", artist="Test Artist", album="Test Album", seconds=1):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
[
|
||||
@@ -74,9 +60,6 @@ def make_flac():
|
||||
"lavfi",
|
||||
"-i",
|
||||
f"sine=frequency=440:duration={seconds}",
|
||||
# Quieter or louder than the default, for tests that need two
|
||||
# tracks at different levels.
|
||||
*(["-af", f"volume={gain}dB"] if gain else []),
|
||||
"-metadata",
|
||||
f"title={title}",
|
||||
"-metadata",
|
||||
|
||||
+2
-195
@@ -14,16 +14,6 @@ def run(source, mirror, *extra):
|
||||
return music_mirror.main(["--source", str(source), "--mirror", str(mirror), *extra])
|
||||
|
||||
|
||||
def audio_frames(path):
|
||||
"""Return a hash of a file's audio frames, ignoring its tags."""
|
||||
return subprocess.run(
|
||||
["ffmpeg", "-v", "error", "-i", str(path), "-map", "0:a", "-c", "copy", "-f", "md5", "-"],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout.strip()
|
||||
|
||||
|
||||
def test_parse_quality_accepts_vbr_and_cbr():
|
||||
assert music_mirror.parse_quality("V0") == ["-q:a", "0"]
|
||||
assert music_mirror.parse_quality("v2") == ["-q:a", "2"]
|
||||
@@ -71,10 +61,8 @@ def test_output_is_mp3(tmp_path, make_flac):
|
||||
"a:0",
|
||||
"-show_entries",
|
||||
"stream=codec_name",
|
||||
# Not csv: a levelled file carries ReplayGain side data, which the
|
||||
# csv writer renders as a trailing empty field.
|
||||
"-of",
|
||||
"default=noprint_wrappers=1:nokey=1",
|
||||
"csv=p=0",
|
||||
str(mirror / "a.mp3"),
|
||||
],
|
||||
check=True,
|
||||
@@ -152,9 +140,6 @@ def test_no_prune_keeps_orphans(tmp_path, make_flac):
|
||||
|
||||
|
||||
def test_existing_mp3_is_copied_not_re_encoded(tmp_path, make_flac):
|
||||
"""Compared by the audio frames rather than the whole file: the copy is
|
||||
tagged with its ReplayGain values afterwards, so the two differ in the
|
||||
container while carrying identical audio."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
flac = make_flac(source / "a.flac")
|
||||
@@ -167,7 +152,7 @@ def test_existing_mp3_is_copied_not_re_encoded(tmp_path, make_flac):
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
assert audio_frames(mirror / "b.mp3") == audio_frames(source / "b.mp3")
|
||||
assert (mirror / "b.mp3").read_bytes() == (source / "b.mp3").read_bytes()
|
||||
|
||||
|
||||
def test_interrupted_copy_leaves_nothing_behind(tmp_path, make_flac, monkeypatch):
|
||||
@@ -801,181 +786,3 @@ def test_the_budget_is_reported_so_it_can_be_checked(tmp_path, make_flac, caplog
|
||||
run(source, mirror, "--fat32-safe")
|
||||
|
||||
assert "limited to 253 characters" in caplog.text
|
||||
|
||||
|
||||
# Rockbox applies the offset a ReplayGain tag carries but never measures
|
||||
# loudness itself, so an untagged mirror plays each album at whatever level it
|
||||
# was mastered to. The tags have to be written here or nowhere.
|
||||
|
||||
|
||||
def test_replaygain_tags_are_written(tmp_path, make_flac, probe_tag, require_rsgain):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
track = mirror / "Album" / "a.mp3"
|
||||
assert probe_tag(track, "REPLAYGAIN_TRACK_GAIN").endswith("dB")
|
||||
assert probe_tag(track, "REPLAYGAIN_ALBUM_GAIN").endswith("dB")
|
||||
assert probe_tag(track, "REPLAYGAIN_TRACK_PEAK")
|
||||
|
||||
|
||||
def test_the_album_shares_one_gain_and_the_tracks_keep_their_own(
|
||||
tmp_path, make_flac, probe_tag, require_rsgain
|
||||
):
|
||||
"""Album gain is what keeps a quiet track quiet within a record it belongs
|
||||
to. Track gain is written alongside it so the device can pick the other
|
||||
behaviour when shuffling."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "loud.flac")
|
||||
make_flac(source / "Album" / "quiet.flac", gain=-12)
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
loud = mirror / "Album" / "loud.mp3"
|
||||
quiet = mirror / "Album" / "quiet.mp3"
|
||||
assert probe_tag(loud, "REPLAYGAIN_ALBUM_GAIN") == probe_tag(quiet, "REPLAYGAIN_ALBUM_GAIN")
|
||||
assert probe_tag(loud, "REPLAYGAIN_TRACK_GAIN") != probe_tag(quiet, "REPLAYGAIN_TRACK_GAIN")
|
||||
|
||||
|
||||
def test_levelling_does_not_make_the_next_pass_re_encode(
|
||||
tmp_path, make_flac, caplog, require_rsgain
|
||||
):
|
||||
"""Writing tags rewrites the file, and staleness here is an mtime
|
||||
comparison. Without --preserve-mtimes every pass would re-encode the whole
|
||||
library and then level it again, forever."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
before = (mirror / "Album" / "a.mp3").stat().st_mtime
|
||||
|
||||
with caplog.at_level("INFO"):
|
||||
run(source, mirror)
|
||||
|
||||
assert "0 encoded" in caplog.text
|
||||
assert (mirror / "Album" / "a.mp3").stat().st_mtime == before
|
||||
|
||||
|
||||
def test_an_already_levelled_album_is_not_measured_again(
|
||||
tmp_path, make_flac, caplog, require_rsgain
|
||||
):
|
||||
"""Measuring costs a decode of every track. A pass that changed nothing
|
||||
must not pay it."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
run(source, mirror)
|
||||
|
||||
with caplog.at_level("INFO"):
|
||||
run(source, mirror)
|
||||
|
||||
assert "0 levelled" in caplog.text
|
||||
|
||||
|
||||
def test_a_new_track_relevels_the_album_around_it(
|
||||
tmp_path, make_flac, probe_tag, require_rsgain
|
||||
):
|
||||
"""Album gain is a property of the whole album, so a track arriving late
|
||||
makes the value stored on every one of its siblings wrong.
|
||||
|
||||
-10 dB rather than something more dramatic: R128 gates quiet passages out
|
||||
of the measurement, and a track far enough below the rest of the record is
|
||||
excluded from it entirely."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
run(source, mirror)
|
||||
first = probe_tag(mirror / "Album" / "a.mp3", "REPLAYGAIN_ALBUM_GAIN")
|
||||
|
||||
make_flac(source / "Album" / "b.flac", gain=-10)
|
||||
run(source, mirror)
|
||||
|
||||
assert probe_tag(mirror / "Album" / "a.mp3", "REPLAYGAIN_ALBUM_GAIN") != first
|
||||
|
||||
|
||||
def test_a_removed_track_relevels_the_album_behind_it(
|
||||
tmp_path, make_flac, caplog, require_rsgain
|
||||
):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
make_flac(source / "Album" / "b.flac", gain=-20)
|
||||
run(source, mirror)
|
||||
|
||||
(source / "Album" / "b.flac").unlink()
|
||||
with caplog.at_level("INFO"):
|
||||
run(source, mirror)
|
||||
|
||||
assert "1 levelled" in caplog.text
|
||||
|
||||
|
||||
def test_embedded_cover_art_does_not_hide_the_tags(
|
||||
tmp_path, make_flac, make_cover, require_rsgain
|
||||
):
|
||||
"""The check walks ID3v2 frame headers and seeks over the bodies. Cover art
|
||||
sits between the text frames and the ones rsgain appends, so a check that
|
||||
only read the start of the tag would never reach them -- and would measure
|
||||
every album with a cover on every pass."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
make_cover(source / "Album" / "cover.jpg")
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
assert music_mirror.has_replaygain(mirror / "Album" / "a.mp3")
|
||||
|
||||
|
||||
def test_no_replaygain_leaves_the_tags_off(tmp_path, make_flac, probe_tag):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
|
||||
run(source, mirror, "--no-replaygain")
|
||||
|
||||
assert not probe_tag(mirror / "Album" / "a.mp3", "REPLAYGAIN_ALBUM_GAIN")
|
||||
assert not music_mirror.has_replaygain(mirror / "Album" / "a.mp3")
|
||||
|
||||
|
||||
def test_an_unlevelled_mirror_is_backfilled(tmp_path, make_flac, probe_tag, require_rsgain):
|
||||
"""A mirror built before any of this existed has correct mtimes, so no pass
|
||||
would ever revisit those files on its own."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
run(source, mirror, "--no-replaygain")
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
assert probe_tag(mirror / "Album" / "a.mp3", "REPLAYGAIN_ALBUM_GAIN").endswith("dB")
|
||||
|
||||
|
||||
def test_a_missing_scanner_is_reported_and_not_fatal(tmp_path, make_flac, caplog, monkeypatch):
|
||||
"""The mirror is still correct audio in the right place. It just plays at
|
||||
the level it was mastered to."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
monkeypatch.setattr(music_mirror.shutil, "which", lambda name: None)
|
||||
|
||||
with caplog.at_level("INFO"):
|
||||
assert run(source, mirror) == 0
|
||||
|
||||
assert "rsgain is not on PATH" in caplog.text
|
||||
assert (mirror / "Album" / "a.mp3").is_file()
|
||||
|
||||
|
||||
def test_a_dry_run_measures_nothing(tmp_path, make_flac, caplog, require_rsgain):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
|
||||
with caplog.at_level("INFO"):
|
||||
run(source, mirror, "--dry-run")
|
||||
|
||||
assert "would level 1 album" in caplog.text
|
||||
assert not mirror.joinpath("Album").exists()
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -289,6 +291,130 @@ def test_a_track_missing_from_the_mirror_is_counted_not_guessed(monkeypatch):
|
||||
assert len(result.retain) == 1
|
||||
|
||||
|
||||
LONDON = ZoneInfo("Europe/London")
|
||||
|
||||
# Rockbox builds timestamps with mktime(get_time()), and its mktime applies no
|
||||
# zone at all, so the number is the RTC's local wall clock read as if it were
|
||||
# UTC. Under BST that puts every play an hour ahead of when it happened.
|
||||
SUMMER_LOGGED = 1787839200 # 2026-08-27 14:00 written by the device
|
||||
SUMMER_TRUE = 1787835600 # the instant that actually was, 13:00 UTC
|
||||
WINTER_LOGGED = 1796479200 # 2026-12-05 14:00, when London is already UTC
|
||||
WINTER_TRUE = 1796479200
|
||||
|
||||
|
||||
def test_a_summer_timestamp_is_pulled_back_to_real_utc():
|
||||
"""The device logged 14:00 local. That was 13:00 UTC, and UTC is what
|
||||
Last.fm stores."""
|
||||
assert submit_scrobbles.device_time_to_utc(SUMMER_LOGGED, LONDON) == SUMMER_TRUE
|
||||
|
||||
|
||||
def test_a_winter_timestamp_is_left_alone():
|
||||
"""London keeps UTC for half the year, so there is nothing to correct and
|
||||
the correction must not invent an offset anyway."""
|
||||
assert submit_scrobbles.device_time_to_utc(WINTER_LOGGED, LONDON) == WINTER_TRUE
|
||||
|
||||
|
||||
def test_the_converted_time_reads_back_as_the_clock_the_device_showed():
|
||||
"""The round trip, which is the property that actually matters: whatever
|
||||
the player's screen said is what Last.fm should show in local time."""
|
||||
shown = datetime.fromtimestamp(SUMMER_LOGGED, timezone.utc)
|
||||
corrected = submit_scrobbles.device_time_to_utc(SUMMER_LOGGED, LONDON)
|
||||
|
||||
assert datetime.fromtimestamp(corrected, LONDON).strftime("%Y-%m-%d %H:%M") == (
|
||||
shown.strftime("%Y-%m-%d %H:%M")
|
||||
)
|
||||
|
||||
|
||||
def test_one_log_spanning_a_clock_change_converts_each_side_separately():
|
||||
"""A week's listening either side of the October change carries two
|
||||
different offsets. Correcting the log by a single figure would put half of
|
||||
it an hour out, which is why the offset is resolved per play."""
|
||||
before, after = 1792888200, 1792899000 # 2026-10-25, 00:30 BST and 03:30 GMT
|
||||
|
||||
assert submit_scrobbles.device_time_to_utc(before, LONDON) == before - 3600
|
||||
assert submit_scrobbles.device_time_to_utc(after, LONDON) == after
|
||||
|
||||
|
||||
def test_a_playback_log_is_corrected_before_submission(monkeypatch):
|
||||
monkeypatch.setattr(Path, "is_file", lambda self: True)
|
||||
log = f"{SUMMER_LOGGED}:180000:245000:/Music/Pendulum/Immersion/01.mp3\n"
|
||||
|
||||
played = submit_scrobbles.plays_from_playback_log(
|
||||
log, "/Music", "/mnt/mirror", runner=tags_of(), zone=LONDON
|
||||
).played
|
||||
|
||||
assert played[0]["timestamp"] == str(SUMMER_TRUE)
|
||||
|
||||
|
||||
def test_a_scrobbler_log_claiming_utc_is_not_corrected_twice():
|
||||
"""The format's header exists for exactly this. A client that already did
|
||||
the conversion says so, and correcting it again would break it."""
|
||||
log = LOG.replace("#TZ/UNKNOWN", "#TZ/UTC")
|
||||
|
||||
played, _, _ = submit_scrobbles.parse_log(log, LONDON)
|
||||
|
||||
assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"]
|
||||
|
||||
|
||||
def test_a_scrobbler_log_declaring_unknown_is_corrected():
|
||||
"""Rockbox writes UNKNOWN, meaning local wall clock, so the times are ours
|
||||
to fix."""
|
||||
played, _, _ = submit_scrobbles.parse_log(LOG, LONDON)
|
||||
|
||||
# 1700000100 and 1700000300 are November, when London is on UTC anyway;
|
||||
# the point is that the header did not exempt them from being looked at.
|
||||
assert submit_scrobbles.declares_utc(LOG) is False
|
||||
assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"]
|
||||
|
||||
|
||||
def test_the_system_zone_keeps_its_whole_name(monkeypatch, tmp_path):
|
||||
"""/etc/localtime points into the tzdata tree. Taking only the last
|
||||
component gives "London", which no database holds -- and the fallback for
|
||||
an unknown name is a fixed offset, which is wrong for half the year. The
|
||||
region has to survive."""
|
||||
zoneinfo_dir = tmp_path / "share" / "zoneinfo" / "Europe"
|
||||
zoneinfo_dir.mkdir(parents=True)
|
||||
(zoneinfo_dir / "London").write_bytes(b"TZif")
|
||||
monkeypatch.setattr(
|
||||
submit_scrobbles.Path,
|
||||
"read_text",
|
||||
lambda self, **kw: (_ for _ in ()).throw(OSError),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
submit_scrobbles.Path,
|
||||
"resolve",
|
||||
lambda self: zoneinfo_dir / "London",
|
||||
)
|
||||
|
||||
assert submit_scrobbles.system_zone_name() == "Europe/London"
|
||||
|
||||
|
||||
def test_a_named_zone_beats_the_machine_default():
|
||||
"""The player may not be in the same place as the laptop."""
|
||||
assert submit_scrobbles.device_zone("Asia/Tokyo") == ZoneInfo("Asia/Tokyo")
|
||||
|
||||
|
||||
def test_plays_dated_after_now_are_reported():
|
||||
"""A clock never put forward, or the wrong zone, produces plays that have
|
||||
not happened yet. Last.fm cannot tell those from real ones."""
|
||||
now = 1787835600
|
||||
played = [
|
||||
{"timestamp": str(now - 60)},
|
||||
{"timestamp": str(now + 3600)},
|
||||
]
|
||||
|
||||
ahead = submit_scrobbles.future_plays(played, now=now)
|
||||
|
||||
assert [entry["timestamp"] for entry in ahead] == [str(now + 3600)]
|
||||
|
||||
|
||||
def test_a_clock_a_minute_fast_is_not_reported():
|
||||
"""Devices drift. Only an offset large enough to be a zone error matters."""
|
||||
now = 1787835600
|
||||
|
||||
assert submit_scrobbles.future_plays([{"timestamp": str(now + 60)}], now=now) == []
|
||||
|
||||
|
||||
def test_playback_logs_are_found_including_rotations(tmp_path):
|
||||
"""Rockbox rotates the log once it passes half a megabyte."""
|
||||
rockbox = tmp_path / ".rockbox"
|
||||
|
||||
@@ -5,7 +5,6 @@ protecting against emptying the wrong directory -- a mistake that does not
|
||||
announce itself.
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
@@ -347,88 +346,3 @@ def test_counting_can_be_asked_for(mirror, tmp_path):
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "files to copy" in result.stderr
|
||||
|
||||
|
||||
# A ReplayGain re-level rewrites a track's tags in the padding the previous
|
||||
# write left behind, so neither the size nor the mtime changes -- and those are
|
||||
# the two things rsync's quick check compares.
|
||||
|
||||
|
||||
def stale_copy(mirror, destination, relative, current, previous):
|
||||
"""Put a file on the device that differs only in content from the mirror's."""
|
||||
source = mirror / relative
|
||||
source.parent.mkdir(parents=True, exist_ok=True)
|
||||
source.write_bytes(current)
|
||||
device = destination / relative
|
||||
device.parent.mkdir(parents=True, exist_ok=True)
|
||||
device.write_bytes(previous)
|
||||
os.utime(device, (source.stat().st_atime, source.stat().st_mtime))
|
||||
return device
|
||||
|
||||
|
||||
def test_a_tag_only_change_reaches_the_device_with_the_track_that_caused_it(
|
||||
mirror, tmp_path
|
||||
):
|
||||
"""The new track is visible to rsync; its re-levelled sibling is not, and
|
||||
would otherwise keep the old album gain on the device forever."""
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
sibling = stale_copy(mirror, destination, "Album/sibling.mp3", b"NEW", b"OLD")
|
||||
|
||||
result = run("-f", "-S", "-U", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert sibling.read_bytes() == b"NEW"
|
||||
assert (destination / "Album" / "track.mp3").is_file()
|
||||
|
||||
|
||||
def test_a_deletion_also_relevels_what_is_left_behind(mirror, tmp_path):
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
sibling = stale_copy(mirror, destination, "Album/sibling.mp3", b"NEW", b"OLD")
|
||||
(destination / "Album" / "gone.mp3").write_bytes(b"old")
|
||||
|
||||
result = run("-f", "-S", "-U", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert sibling.read_bytes() == b"NEW"
|
||||
assert not (destination / "Album" / "gone.mp3").exists()
|
||||
|
||||
|
||||
def test_an_untouched_album_is_not_copied_again(mirror, tmp_path):
|
||||
"""The second pass is scoped to albums that changed. An album whose file
|
||||
set is the same is left where it is, which is the whole point of not
|
||||
running --ignore-times over the library."""
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
quiet = stale_copy(mirror, destination, "Quiet/only.mp3", b"NEW", b"OLD")
|
||||
(destination / "Album").mkdir()
|
||||
shutil.copy2(mirror / "Album" / "track.mp3", destination / "Album" / "track.mp3")
|
||||
|
||||
result = run("-f", "-S", "-U", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert quiet.read_bytes() == b"OLD"
|
||||
|
||||
|
||||
def test_a_dry_run_says_how_many_extra_tracks_are_involved(mirror, tmp_path):
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
stale_copy(mirror, destination, "Album/sibling.mp3", b"NEW", b"OLD")
|
||||
|
||||
result = run("-f", "-S", "-U", "-n", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "and 1 more in those albums" in result.stderr
|
||||
|
||||
|
||||
def test_a_first_sync_does_not_copy_anything_twice(mirror, tmp_path):
|
||||
"""Everything is transferred by the main pass, so there is nothing left for
|
||||
the second one and it must not announce itself."""
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
|
||||
result = run("-f", "-S", "-U", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "re-levelled" not in result.stderr
|
||||
|
||||
@@ -1,128 +0,0 @@
|
||||
"""A ReplayGain re-level rewrites a track's tags without changing its size or
|
||||
its mtime, which is precisely the pair rsync's quick check compares. These
|
||||
cover the list that is fed back to rsync to copy those tracks anyway."""
|
||||
|
||||
import io
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
|
||||
|
||||
import touched_albums # noqa: E402
|
||||
|
||||
|
||||
def album(root, name, *tracks):
|
||||
directory = root / name
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
for track in tracks:
|
||||
(directory / track).write_bytes(b"x")
|
||||
return directory
|
||||
|
||||
|
||||
def listing(mirror, *lines):
|
||||
return touched_albums.remaining(mirror, *touched_albums.touched(list(lines)))
|
||||
|
||||
|
||||
def test_the_siblings_of_a_new_track_are_listed(tmp_path):
|
||||
album(tmp_path, "Artist/Album", "01.mp3", "02.mp3", "03.mp3")
|
||||
|
||||
assert listing(tmp_path, "4096 Artist/Album/03.mp3") == [
|
||||
"Artist/Album/01.mp3",
|
||||
"Artist/Album/02.mp3",
|
||||
]
|
||||
|
||||
|
||||
def test_a_track_rsync_just_copied_is_not_copied_twice(tmp_path):
|
||||
"""A quality upgrade replaces every track on the record. Listing them again
|
||||
would send the album across twice."""
|
||||
album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
|
||||
|
||||
assert listing(tmp_path, "4096 Artist/Album/01.mp3", "4096 Artist/Album/02.mp3") == []
|
||||
|
||||
|
||||
def test_a_deletion_relevels_what_is_left(tmp_path):
|
||||
album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
|
||||
|
||||
assert listing(tmp_path, "deleting Artist/Album/03.mp3") == [
|
||||
"Artist/Album/01.mp3",
|
||||
"Artist/Album/02.mp3",
|
||||
]
|
||||
|
||||
|
||||
def test_an_album_deleted_outright_lists_nothing(tmp_path):
|
||||
assert listing(tmp_path, "deleting Artist/Gone/01.mp3") == []
|
||||
|
||||
|
||||
def test_untouched_albums_are_left_alone(tmp_path):
|
||||
album(tmp_path, "Artist/Changed", "01.mp3", "02.mp3")
|
||||
album(tmp_path, "Artist/Quiet", "01.mp3", "02.mp3")
|
||||
|
||||
assert listing(tmp_path, "4096 Artist/Changed/01.mp3") == ["Artist/Changed/02.mp3"]
|
||||
|
||||
|
||||
def test_a_replaced_cover_is_not_an_album_change(tmp_path):
|
||||
"""Only a track can change an album's gains, and covers are replaced often
|
||||
enough that treating one as a re-level would copy records for nothing."""
|
||||
album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
|
||||
|
||||
assert listing(tmp_path, "17408 Artist/Album/cover.jpg") == []
|
||||
|
||||
|
||||
def test_directories_are_not_mistaken_for_tracks(tmp_path):
|
||||
album(tmp_path, "Artist/Album", "01.mp3")
|
||||
|
||||
assert listing(tmp_path, "4096 Artist/Album/", "deleting Artist/Old/") == []
|
||||
|
||||
|
||||
def test_rsync_talking_to_the_operator_is_not_a_path(tmp_path):
|
||||
album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
|
||||
|
||||
assert (
|
||||
listing(
|
||||
tmp_path,
|
||||
"sending incremental file list",
|
||||
"",
|
||||
"sent 1,234 bytes received 56 bytes 2,580.00 bytes/sec",
|
||||
"total size is 7,890 speedup is 6.12",
|
||||
)
|
||||
== []
|
||||
)
|
||||
|
||||
|
||||
def test_a_track_at_the_mirror_root_does_not_pull_in_the_whole_tree(tmp_path):
|
||||
"""Nothing writes a mirror this way, but the directory of a root-level file
|
||||
is the root, and recursing from there would be the whole library."""
|
||||
(tmp_path / "loose.mp3").write_bytes(b"x")
|
||||
(tmp_path / "other.mp3").write_bytes(b"x")
|
||||
album(tmp_path, "Artist/Album", "01.mp3")
|
||||
|
||||
assert listing(tmp_path, "4096 loose.mp3") == ["other.mp3"]
|
||||
|
||||
|
||||
def test_the_paths_are_written_one_per_line(tmp_path):
|
||||
"""They are fed straight back to rsync as --files-from."""
|
||||
album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
|
||||
out = io.StringIO()
|
||||
|
||||
touched_albums.main(
|
||||
["--mirror", str(tmp_path)],
|
||||
stream=["4096 Artist/Album/01.mp3"],
|
||||
out=out,
|
||||
)
|
||||
|
||||
assert out.getvalue() == "Artist/Album/02.mp3\n"
|
||||
|
||||
|
||||
def test_it_runs_as_a_script(tmp_path):
|
||||
album(tmp_path, "Artist/Album", "01.mp3", "02.mp3")
|
||||
|
||||
completed = subprocess.run(
|
||||
[sys.executable, touched_albums.__file__, "--mirror", str(tmp_path)],
|
||||
input="4096 Artist/Album/01.mp3\n",
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
assert completed.returncode == 0
|
||||
assert completed.stdout == "Artist/Album/02.mp3\n"
|
||||
+154
-7
@@ -21,7 +21,9 @@ import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
API_ROOT = "https://ws.audioscrobbler.com/2.0/"
|
||||
|
||||
@@ -48,6 +50,11 @@ LISTENED_FRACTION = 0.5
|
||||
# Rockbox logs ticks in milliseconds instead, which is not a date.
|
||||
EARLIEST_PLAUSIBLE = 1_000_000_000
|
||||
|
||||
# How far ahead of now a converted play may sit before it is reported. Some
|
||||
# slack absorbs a device clock drifting by a minute or two; an hour out means
|
||||
# the zone is wrong or the clock was never put forward.
|
||||
FUTURE_TOLERANCE_SECONDS = 300
|
||||
|
||||
SESSION_FILE = Path(
|
||||
os.getenv("XDG_CONFIG_HOME", Path.home() / ".config")
|
||||
) / "music-mirror" / "lastfm.json"
|
||||
@@ -57,14 +64,117 @@ class LastfmError(Exception):
|
||||
"""A Last.fm request that failed."""
|
||||
|
||||
|
||||
def parse_log(text):
|
||||
def system_zone_name():
|
||||
"""Return this machine's IANA zone name, or "" if nothing states it.
|
||||
|
||||
The name has to come out whole. /etc/localtime is a symlink into the
|
||||
tzdata tree, so the part after `zoneinfo/` is the name -- taking only the
|
||||
last component yields "London", which no database has, and falls back to a
|
||||
fixed offset that would then be wrong for half the year.
|
||||
"""
|
||||
try:
|
||||
named = Path("/etc/timezone").read_text(encoding="utf-8").strip()
|
||||
if named:
|
||||
return named
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
parts = Path("/etc/localtime").resolve().parts
|
||||
except OSError:
|
||||
return ""
|
||||
if "zoneinfo" in parts:
|
||||
return "/".join(parts[len(parts) - parts[::-1].index("zoneinfo"):])
|
||||
return ""
|
||||
|
||||
|
||||
def device_zone(name=None):
|
||||
"""Return the zone the device's clock is keeping.
|
||||
|
||||
Rockbox has no concept of a timezone, so its clock is set to local wall
|
||||
time and the zone has to be supplied from outside. Defaulting to this
|
||||
machine's zone is right whenever the player and the laptop are in the same
|
||||
place, which for a device synced by cable they are.
|
||||
"""
|
||||
if name:
|
||||
return ZoneInfo(name)
|
||||
for candidate in (os.getenv("TZ"), system_zone_name()):
|
||||
if not candidate:
|
||||
continue
|
||||
try:
|
||||
return ZoneInfo(candidate)
|
||||
except (ZoneInfoNotFoundError, ValueError):
|
||||
continue
|
||||
# Nothing named the zone, so the offset cannot be resolved per play: this
|
||||
# is today's offset applied to every timestamp, which is wrong either side
|
||||
# of a daylight saving change. Still better than pretending it logged UTC.
|
||||
print(
|
||||
"warning: no IANA timezone found for this machine; using its current"
|
||||
" offset for every play. Pass --device-timezone to fix older plays"
|
||||
" across a daylight saving change.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return datetime.now().astimezone().tzinfo
|
||||
|
||||
|
||||
def device_time_to_utc(stamp, zone):
|
||||
"""Return the true UTC epoch of a timestamp Rockbox wrote.
|
||||
|
||||
Rockbox builds its timestamps with `mktime(get_time())`, and its mktime
|
||||
(firmware/libc/mktime.c) is plain calendar arithmetic with no zone applied.
|
||||
Fed the RTC's local fields it yields local-wall-clock-as-if-UTC, so the
|
||||
number is ahead of real UTC by whatever the offset was. Its own scrobbler
|
||||
plugin admits this by writing `#TZ/UNKNOWN`, leaving the correction here.
|
||||
|
||||
Decoding the number back into those fields and reinterpreting them in the
|
||||
device's zone recovers the instant, and does so per play, so a log
|
||||
straddling a daylight-saving change converts each side by its own offset.
|
||||
"""
|
||||
fields = datetime.fromtimestamp(stamp, timezone.utc).replace(tzinfo=zone)
|
||||
return int(fields.timestamp())
|
||||
|
||||
|
||||
def future_plays(played, now=None):
|
||||
"""Return the plays timestamped later than now, which cannot have happened.
|
||||
|
||||
A device clock left on the wrong offset, or never adjusted across a
|
||||
daylight-saving change, shows up here. Last.fm has no way to tell such a
|
||||
scrobble from a real one, so it is worth saying out loud.
|
||||
"""
|
||||
now = time.time() if now is None else now
|
||||
return [
|
||||
entry for entry in played
|
||||
if int(entry["timestamp"]) > now + FUTURE_TOLERANCE_SECONDS
|
||||
]
|
||||
|
||||
|
||||
def declares_utc(text):
|
||||
"""Whether an AUDIOSCROBBLER log says its timestamps are already UTC.
|
||||
|
||||
The format's header carries `#TZ/UTC` or `#TZ/UNKNOWN`, and its spec is
|
||||
explicit that a device may only claim UTC if it converted. Rockbox writes
|
||||
UNKNOWN, meaning the times are local wall clock and want correcting; a log
|
||||
from anything that claims UTC must be left alone.
|
||||
"""
|
||||
for line in text.splitlines():
|
||||
if not line.startswith("#"):
|
||||
break
|
||||
if line.strip().upper().startswith("#TZ/"):
|
||||
return line.strip().upper() == "#TZ/UTC"
|
||||
return False
|
||||
|
||||
|
||||
def parse_log(text, zone=None):
|
||||
"""Return the listened tracks in an AUDIOSCROBBLER log, oldest first.
|
||||
|
||||
Fields are artist, album, title, track number, length, rating, timestamp
|
||||
and MusicBrainz id. Rockbox converts any tab inside a field to a space
|
||||
before writing, so splitting on tabs is safe.
|
||||
|
||||
Timestamps are corrected from the device's local wall clock to UTC unless
|
||||
the log's own header claims it did that already.
|
||||
"""
|
||||
played, skipped, timeless = [], 0, 0
|
||||
convert = zone is not None and not declares_utc(text)
|
||||
for line in text.splitlines():
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
@@ -85,6 +195,8 @@ def parse_log(text):
|
||||
continue
|
||||
if not artist or not title:
|
||||
continue
|
||||
if convert:
|
||||
when = device_time_to_utc(when, zone)
|
||||
played.append(
|
||||
{
|
||||
"artist": artist,
|
||||
@@ -173,11 +285,13 @@ class Conversion:
|
||||
self.retain = []
|
||||
|
||||
|
||||
def plays_from_playback_log(text, device_prefix, mirror, runner=None):
|
||||
def plays_from_playback_log(text, device_prefix, mirror, runner=None, zone=None):
|
||||
"""Return the conversion of a playback log.
|
||||
|
||||
Skips are decided by the same fraction the on-device plugin uses, so the
|
||||
two never disagree about what counted as a play.
|
||||
two never disagree about what counted as a play. Timestamps are corrected
|
||||
from the device's wall clock to UTC; the core log has no header to say so,
|
||||
but it is written the same way the plugin's UNKNOWN times are.
|
||||
"""
|
||||
result = Conversion(played=[])
|
||||
for line in text.splitlines():
|
||||
@@ -209,7 +323,9 @@ def plays_from_playback_log(text, device_prefix, mirror, runner=None):
|
||||
"album": tags.get("album", ""),
|
||||
"trackNumber": (tags.get("track") or "").split("/")[0],
|
||||
"duration": str(length // 1000) if length > 0 else "",
|
||||
"timestamp": str(stamp),
|
||||
"timestamp": str(
|
||||
device_time_to_utc(stamp, zone) if zone is not None else stamp
|
||||
),
|
||||
"mbid": tags.get("musicbrainz_trackid", ""),
|
||||
"line": line,
|
||||
}
|
||||
@@ -350,6 +466,14 @@ def main(argv=None, transport=http_post):
|
||||
help="where the music sits on the device, stripped when mapping a logged"
|
||||
" path back onto the mirror",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--device-timezone",
|
||||
default=os.getenv("ROCKBOX_TIMEZONE"),
|
||||
help="the zone the player's clock is set to, as an IANA name such as"
|
||||
" Europe/London. Rockbox keeps local wall time and cannot record an"
|
||||
" offset, so its timestamps need this to become the UTC Last.fm wants."
|
||||
" Defaults to this machine's zone",
|
||||
)
|
||||
parser.add_argument("--api-key", default=os.getenv("LASTFM_API_KEY"))
|
||||
parser.add_argument("--api-secret", default=os.getenv("LASTFM_API_SECRET"))
|
||||
parser.add_argument("--dry-run", action="store_true", help="parse and report only")
|
||||
@@ -358,6 +482,12 @@ def main(argv=None, transport=http_post):
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
zone = device_zone(args.device_timezone)
|
||||
except (ZoneInfoNotFoundError, ValueError) as error:
|
||||
print(f"unknown timezone {args.device_timezone!r}: {error}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
target = Path(args.device)
|
||||
log = target if target.is_file() else find_log(target)
|
||||
logs = []
|
||||
@@ -365,7 +495,7 @@ def main(argv=None, transport=http_post):
|
||||
conversion = None
|
||||
if log is not None:
|
||||
played, skipped, timeless = parse_log(
|
||||
log.read_text(encoding="utf-8", errors="replace")
|
||||
log.read_text(encoding="utf-8", errors="replace"), zone
|
||||
)
|
||||
unresolved = 0
|
||||
logs = [log]
|
||||
@@ -380,7 +510,9 @@ def main(argv=None, transport=http_post):
|
||||
text = "\n".join(
|
||||
path.read_text(encoding="utf-8", errors="replace") for path in logs
|
||||
)
|
||||
conversion = plays_from_playback_log(text, args.device_prefix, args.mirror)
|
||||
conversion = plays_from_playback_log(
|
||||
text, args.device_prefix, args.mirror, zone=zone
|
||||
)
|
||||
played = conversion.played
|
||||
skipped, unresolved, timeless = (
|
||||
conversion.skipped,
|
||||
@@ -413,9 +545,24 @@ def main(argv=None, transport=http_post):
|
||||
)
|
||||
if not played:
|
||||
return 0
|
||||
|
||||
ahead = future_plays(played)
|
||||
if ahead:
|
||||
newest = int(ahead[-1]["timestamp"]) - int(time.time())
|
||||
print(
|
||||
f" {len(ahead)} plays are timestamped up to {newest // 60} minutes in"
|
||||
f" the future, converting from {zone}. Either the player's clock is"
|
||||
" wrong or that is not the zone it is set to.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
if args.dry_run:
|
||||
for entry in played[:20]:
|
||||
print(f"{entry['timestamp']}\t{entry['artist']}\t{entry['track']}")
|
||||
when = datetime.fromtimestamp(int(entry["timestamp"]), zone)
|
||||
print(
|
||||
f"{entry['timestamp']}\t{when:%Y-%m-%d %H:%M %Z}"
|
||||
f"\t{entry['artist']}\t{entry['track']}"
|
||||
)
|
||||
return 0
|
||||
|
||||
if not args.api_key or not args.api_secret:
|
||||
|
||||
+12
-43
@@ -41,6 +41,12 @@ Submitting scrobbles needs LASTFM_API_KEY and LASTFM_API_SECRET; it is skipped
|
||||
with a note when they are unset. Scrobbling is a write method and needs the
|
||||
secret, unlike the read-only calls elsewhere in these projects.
|
||||
|
||||
Rockbox has no notion of a timezone: its clock holds local wall time and its
|
||||
logs record that, not UTC. Set ROCKBOX_TIMEZONE to the zone the player's clock
|
||||
is keeping (an IANA name, such as Europe/London) if it differs from this
|
||||
machine's, which is otherwise assumed. Getting it wrong shifts every scrobble
|
||||
by the difference.
|
||||
|
||||
The mirror is the directory holding the artist folders. The destination is
|
||||
where those folders should end up on the device -- not the card root, unless
|
||||
that is genuinely where you want them:
|
||||
@@ -56,10 +62,6 @@ The destination must be on a mounted FAT filesystem. That check is also what
|
||||
catches an unmounted device: /media/IPOD/Music then resolves to the host's own
|
||||
root filesystem, and this refuses to empty that.
|
||||
|
||||
An album that gained or lost a track is copied again in full afterwards. Its
|
||||
surviving tracks have had their ReplayGain tags rewritten in place, which
|
||||
changes neither their size nor their mtime, so the main pass cannot see them.
|
||||
|
||||
Progress is one line that rewrites itself, showing the album currently going
|
||||
across, how far through the transfer is, the rate, and an estimate of what is
|
||||
left. Working the totals out first means a second pass over the tree, which is
|
||||
@@ -162,6 +164,11 @@ if $scrobble; then
|
||||
else
|
||||
scrobble_options=()
|
||||
$dry_run && scrobble_options+=(--dry-run)
|
||||
# Rockbox keeps local wall time with no notion of a zone, so its
|
||||
# timestamps are not the UTC Last.fm expects. Naming the zone the
|
||||
# player's clock is set to lets them be corrected.
|
||||
[ -n "${ROCKBOX_TIMEZONE:-}" ] &&
|
||||
scrobble_options+=(--device-timezone "$ROCKBOX_TIMEZONE")
|
||||
# --mirror lets it convert Rockbox's own playback.log, so the on-device
|
||||
# scrobbler plugin never has to be run. The device root, not the music
|
||||
# directory: the logs live in .rockbox.
|
||||
@@ -197,23 +204,8 @@ done
|
||||
|
||||
printf 'sync-to-ipod: %s -> %s\n' "$mirror" "$destination" >&2
|
||||
|
||||
# Both passes below feed their file list through touched_albums.py, which reads
|
||||
# rsync's own report of what it moved. --files-from separates paths by newline,
|
||||
# so a filename containing one would be read as two -- which is a path FAT32
|
||||
# will not take either, and check_fat32.py above has already refused the run
|
||||
# unless -f was given to skip it.
|
||||
changed=$(mktemp)
|
||||
relevelled=$(mktemp)
|
||||
trap 'rm -f "$changed" "$relevelled"' EXIT
|
||||
|
||||
if $dry_run; then
|
||||
rsync "${options[@]}" --dry-run --verbose --out-format='%l %n' \
|
||||
"$mirror/" "$destination/" | tee "$changed"
|
||||
also=$(python3 "$here/touched_albums.py" --mirror "$mirror" <"$changed" | wc -l)
|
||||
if [ "$also" -gt 0 ]; then
|
||||
printf 'sync-to-ipod: and %s more in those albums, whose ReplayGain tags\n' "$also" >&2
|
||||
printf 'sync-to-ipod: change without changing their size or their mtime\n' >&2
|
||||
fi
|
||||
rsync "${options[@]}" --dry-run --verbose "$mirror/" "$destination/"
|
||||
printf 'sync-to-ipod: dry run, nothing was written\n' >&2
|
||||
exit 0
|
||||
fi
|
||||
@@ -275,33 +267,10 @@ interrupted() {
|
||||
trap interrupted INT TERM
|
||||
|
||||
rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" |
|
||||
tee "$changed" |
|
||||
python3 "$here/rsync_progress.py" --total "$total" --bytes "$total_bytes"
|
||||
status=${PIPESTATUS[0]}
|
||||
[ "$status" -eq 0 ] || die "rsync exited $status"
|
||||
|
||||
# A ReplayGain album gain belongs to the whole record, so a track arriving or
|
||||
# leaving rewrites the tags on all of its siblings. rsgain fits the new values
|
||||
# into the padding its previous write left behind, which changes neither the
|
||||
# size nor the mtime -- the only two things the pass above compares. Those
|
||||
# tracks are invisible to it, and the device would keep the old gains.
|
||||
#
|
||||
# The track that arrived or left is visible, though. So every album the pass
|
||||
# touched has the rest of its tracks copied again, with --ignore-times to
|
||||
# defeat the same quick check. No --delete: the pass above has already settled
|
||||
# what should be on the device, and --delete aimed at an explicit file list
|
||||
# does not mean what it looks like it means.
|
||||
python3 "$here/touched_albums.py" --mirror "$mirror" <"$changed" >"$relevelled"
|
||||
if [ -s "$relevelled" ]; then
|
||||
also=$(wc -l <"$relevelled")
|
||||
printf 'sync-to-ipod: re-copying %s tracks whose album was re-levelled\n' "$also" >&2
|
||||
rsync --times --modify-window=2 --whole-file --omit-dir-times --ignore-times \
|
||||
--files-from="$relevelled" --out-format='%l %n' "$mirror/" "$destination/" |
|
||||
python3 "$here/rsync_progress.py" --total "$also"
|
||||
status=${PIPESTATUS[0]}
|
||||
[ "$status" -eq 0 ] || die "the re-levelled tracks failed to copy: rsync exited $status"
|
||||
fi
|
||||
|
||||
# Rockbox reads its database from .tcd files in .rockbox. Building them here
|
||||
# rather than on the device is not just faster: the on-device commit sorts the
|
||||
# whole index in whatever memory it can scrape together, and on a large library
|
||||
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""List the tracks a sync has to copy again because their album was re-levelled.
|
||||
|
||||
Reads rsync's `--out-format='%l %n'` output on stdin and writes mirror-relative
|
||||
paths on stdout, one per line, for feeding straight back to rsync as
|
||||
`--files-from`.
|
||||
|
||||
The problem it solves: a ReplayGain album gain is a property of every track on
|
||||
the record, so one track arriving or leaving changes the tags on all of its
|
||||
siblings. rsgain writes the new values into the padding its previous write left
|
||||
behind, which leaves both the file's size and its mtime untouched -- and size
|
||||
and mtime are exactly what rsync's quick check compares. It sees nothing to do,
|
||||
and the device keeps the old gains.
|
||||
|
||||
What rsync always can see is the track that arrived or left. So any album it
|
||||
touched has the rest of its tracks copied again, and nothing else does.
|
||||
|
||||
Tracks rsync has already dealt with are left out of the list. After a quality
|
||||
upgrade that replaces every track on a record, listing them again would send
|
||||
the whole album twice.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Under --out-format='%l %n' a transfer is reported as the size, a space and
|
||||
# the path. A removal is reported as "deleting <path>" whatever the format is.
|
||||
# Everything else on the stream is rsync talking to the operator -- the file
|
||||
# list preamble, the byte totals -- and is not a path.
|
||||
TRANSFER = re.compile(r"^(\d+) (.+)$")
|
||||
DELETION = re.compile(r"^deleting (.+)$")
|
||||
|
||||
# What the mirror is made of, and so the only thing worth copying again. A
|
||||
# cover is not rewritten by a re-level.
|
||||
MIRROR_SUFFIX = ".mp3"
|
||||
|
||||
|
||||
def touched(lines):
|
||||
"""Return the paths rsync transferred and the directories it changed."""
|
||||
transferred = set()
|
||||
directories = set()
|
||||
|
||||
for line in lines:
|
||||
line = line.rstrip("\n")
|
||||
deletion = DELETION.match(line)
|
||||
transfer = None if deletion else TRANSFER.match(line)
|
||||
if deletion:
|
||||
path = deletion.group(1)
|
||||
elif transfer:
|
||||
path = transfer.group(2)
|
||||
else:
|
||||
continue
|
||||
# Only a track can change an album's gains. rsync reports the
|
||||
# directories it creates and removes too, and a cover replaced on its
|
||||
# own is no reason to send the record again.
|
||||
if not path.endswith(MIRROR_SUFFIX):
|
||||
continue
|
||||
if transfer:
|
||||
transferred.add(path)
|
||||
directories.add(path.rpartition("/")[0])
|
||||
|
||||
return transferred, directories
|
||||
|
||||
|
||||
def remaining(mirror, transferred, directories):
|
||||
"""Return the tracks in those directories that rsync has not just copied."""
|
||||
paths = []
|
||||
for directory in sorted(directories):
|
||||
album = mirror / directory if directory else mirror
|
||||
if not album.is_dir():
|
||||
# Removed along with the last of its tracks. Nothing to copy.
|
||||
continue
|
||||
for track in sorted(album.glob(f"*{MIRROR_SUFFIX}")):
|
||||
relative = f"{directory}/{track.name}" if directory else track.name
|
||||
if relative not in transferred:
|
||||
paths.append(relative)
|
||||
return paths
|
||||
|
||||
|
||||
def main(argv=None, stream=None, out=None):
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="touched_albums.py",
|
||||
description="List the tracks to copy again after an album was re-levelled.",
|
||||
)
|
||||
parser.add_argument("--mirror", required=True, type=Path, help="root of the MP3 mirror")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
transferred, directories = touched(stream if stream is not None else sys.stdin)
|
||||
for path in remaining(args.mirror, transferred, directories):
|
||||
print(path, file=out if out is not None else sys.stdout)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user