diff --git a/Dockerfile b/Dockerfile index b646258..9291180 100644 --- a/Dockerfile +++ b/Dockerfile @@ -27,5 +27,7 @@ FROM runtime AS test RUN pip install --no-cache-dir pytest COPY pytest.ini ./ +# Host-side tools; not in the runtime image, but the suite covers them. +COPY tools ./tools COPY tests ./tests RUN python -m pytest diff --git a/README.md b/README.md index aaae683..39f5639 100644 --- a/README.md +++ b/README.md @@ -93,6 +93,7 @@ music-mirror --source /music --mirror /music-mp3 --subdir "Artist/Album" | `--jobs` | `MUSIC_MIRROR_JOBS` | CPU count | Concurrent encodes | | `--interval` | `MUSIC_MIRROR_INTERVAL` | unset | Repeat forever, e.g. `45m`, `6h`, `1d` | | `--subdir` | — | unset | Limit the pass to one directory; skips pruning | +| `--fat32-safe` | `MUSIC_MIRROR_FAT32_SAFE` | off | Name files so a FAT32 device accepts them | | `--no-prune` | — | off | Keep mirror files whose source has gone | | `--dry-run` | — | off | Report what would change, write nothing | @@ -137,6 +138,52 @@ New Lidarr imports are picked up on the next pass. With `MUSIC_MIRROR_INTERVAL` at `6h` that is the worst case; run `--subdir` by hand if you want an album immediately. +## Tools + +Host-side scripts under `tools/`, not part of the container image. + +`sync-to-ipod.sh` does a whole transfer: submits the scrobbler log, checks the +mirror, rsyncs, syncs and unmounts. + +```sh +tools/sync-to-ipod.sh /mnt/tank/media/music-mp3 /media/IPOD/Music +tools/sync-to-ipod.sh -n /mnt/tank/media/music-mp3 /media/IPOD/Music # dry run +``` + +It refuses to start unless the destination is a mounted FAT filesystem that is +its own mount point, because `--delete` aimed at the wrong directory empties it +and does not announce itself. It also excludes `/.rockbox`, the scrobbler logs +and the various filesystem metadata directories from deletion — the mirror does +not contain them, and without the exclusion a sync to the card root would +remove the Rockbox install. + +The unmount is the point of doing this in a script. FAT32 has no journal and +the device is reached through disk mode, so an interrupted write is corruption +that needs `fsck.vfat` from another machine. + +`submit_scrobbles.py` sends the Rockbox scrobbler log to Last.fm and sets it +aside. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one +tab-separated line per track rated `L` for listened or `S` for skipped; only +the listened ones are sent. It runs **before** the copy, since the plays +already happened and a failed transfer is no reason to lose them. + +Two things are unlike every other Last.fm call in these projects. Scrobbling is +a *write* method, so it needs `LASTFM_API_SECRET` and a session key obtained +once through the browser, not just the read-only key. And on a target with no +real-time clock Rockbox writes `/.scrobbler-timeless.log` with every timestamp +set to zero; those are counted and reported but never submitted, because +scrobbling them would mean inventing when they happened. + +The log is renamed rather than deleted once accepted. If Last.fm quietly +dropped something, the evidence is still on the device. + +`check_fat32.py` reports paths a FAT32 device will not accept — reserved +characters, trailing dots and spaces, over-long components and paths, and names +colliding case-insensitively. Run it against the mirror **before** an rsync: +rsync reports the failures too, but scattered through fifty thousand files where +they are easy to miss. Exits non-zero when it finds anything, so it can gate a +script. + ## Tests ```sh @@ -158,6 +205,41 @@ Nix machine: nix shell nixpkgs#python3Packages.pytest nixpkgs#ffmpeg -c pytest ``` +## FAT32 and Rockbox + +`--fat32-safe` names mirror files so a FAT32 device will accept them. Off by +default, because turning it on renames files and that should be a decision +rather than a surprise. + +What it handles, per path component: + +| Problem | Treatment | +| ------------------------------- | ------------------------------ | +| `< > : " \ \| ? *` and control characters | replaced with `_` | +| trailing dots and spaces | stripped — FAT eats them silently, so the name round-trips as a different name | +| a component left empty | becomes `_` | +| names differing only in case | detected and reported; one wins, as with any other collision | + +`Dada Life - Kick Out the Epic Motherf**ker` is a real example from a real +library. Without this it simply never arrives on the device. + +**Turning it on does not re-encode anything.** Every track whose name held a +reserved character changes path, and re-encoding those would be hours of work +producing files that already exist byte for byte. The run moves them instead, +and says so. Prune then finds nothing to remove because nothing was left +behind. + +### Album art + +Rockbox looks for cover art **on the filesystem** — `cover.jpg`, `folder.jpg` +and friends beside the track or in its parent — and that search never touches +the picture embedded in the tag. So a JPEG cover found beside the source is now +copied into the mirror as `cover.jpg`, in addition to being embedded. The iPod +firmware reads the embedded one; Rockbox reads the file. Both are satisfied. + +A cover left behind in a directory whose tracks have all gone is pruned, or the +directory would never look empty and never be removed. + ## Getting the result onto an iPod The mirror is just a directory of MP3s, so any client will do: @@ -169,6 +251,30 @@ The mirror is just a directory of MP3s, so any client will do: - **Linux.** Rhythmbox links `libgpod` and handles iPod sync. An iPod Video (5th generation) predates the models whose database has to be signed, so no firmware-hash trickery is needed. +- **Rockbox.** No database to write at all — it reads a plain directory tree + and builds its own index from tags. Copy the mirror across with rsync: + + ```sh + python3 tools/check_fat32.py /mnt/tank/media/music-mp3 # before, not during + rsync -rtv --delete --modify-window=2 \ + /mnt/tank/media/music-mp3/ /media/IPOD/Music/ + ``` + + `--modify-window=2` because FAT stores modification times to two-second + resolution; without it rsync re-copies the entire library on every run. `-rt` + rather than `-a` because owners, groups and permissions mean nothing on FAT + and asking for them only produces errors. + + Do not route this through Rhythmbox. Its `rb_ipod_helpers_is_ipod()` reads + `access-protocols` from media-player-info first and returns true without + looking at the filesystem at all, so an iPod in disk mode is identified by its + USB id and managed as an iPod — writing a database Rockbox does not want. + Deleting `iPod_Control` does not change this. Either use rsync, or untick + Preferences → Plugins → Portable Players - iPod. + + Faster still for the first bulk copy: take the card out of the iFlash adapter + and use a card reader. Fifty thousand files over USB 2.0 through an iPod is a + long evening, and it avoids Rockbox's USB stack entirely. Neither client transcodes at sync time; they copy finished MP3s. diff --git a/compose.yaml b/compose.yaml index a157ab5..3222d57 100644 --- a/compose.yaml +++ b/compose.yaml @@ -22,6 +22,15 @@ services: # Concurrent encodes; defaults to the CPU count. Lower it to leave the # NAS responsive during the first full pass. # MUSIC_MIRROR_JOBS: "4" + # Name files so a FAT32 device will take them, for copying to a Rockbox + # player. Quoted, because an unquoted yes or true is a YAML boolean and + # compose rejects a boolean here. Accepts 1, true or yes. + # + # Turning this on renames every file whose name held a reserved + # character. They are moved, not re-encoded, so the first pass after + # enabling it is quick -- but it is a one-way change to every such path, + # so decide before running it rather than after. + MUSIC_MIRROR_FAT32_SAFE: "true" volumes: - /mnt/tank/media/music:/music:ro - /mnt/tank/media/music-mp3:/mirror diff --git a/music_mirror.py b/music_mirror.py index 7b4c263..306f745 100644 --- a/music_mirror.py +++ b/music_mirror.py @@ -71,12 +71,23 @@ SOURCE_PRIORITY = [ # Looked for in the source directory when a file has no embedded picture. COVER_NAMES = ("cover.jpg", "folder.jpg", "front.jpg", "cover.png", "folder.png") +# What a copied cover is called in the mirror. Rockbox looks for album art on +# the filesystem -- cover.jpg, folder.jpg and friends beside the track -- and +# its search never touches the picture embedded in the tag, so a mirror that +# only embeds art shows none of it on the device. +MIRROR_COVER = "cover.jpg" + # Files the mirror is allowed to contain, and therefore allowed to delete. MIRROR_SUFFIX = ".mp3" # Filesystems disagree about mtime precision; SMB in particular rounds. MTIME_TOLERANCE_SECONDS = 2 +# What FAT32 refuses in a filename, plus the control characters. The mirror is +# copied onto a FAT32 device, and one of these in a path means a track that +# never arrives -- "Kick Out the Epic Motherf**ker" is a real example. +FAT32_RESERVED = re.compile(r'[<>:"/\\|?*\x00-\x1f]') + # The mirror exists to be read back by something else -- an SMB share, another # account on the box -- so everything written into it has to be group-readable. # Neither writer manages that unaided: tempfile.mkstemp forces 0600 whatever the @@ -126,9 +137,30 @@ def parse_interval(interval): return value * {"": 1, "s": 1, "m": 60, "h": 3600, "d": 86400}[match.group(2)] -def mirror_path_for(source, source_root, mirror_root): +def fat32_safe(component): + """Return a single path component FAT32 will accept. + + The mirror exists to be copied onto a FAT32 device, and a name that device + will not take is a track that silently does not arrive. Cheaper to produce + an acceptable name here than to discover the problem partway through + copying fifty thousand files. + + Handled: the reserved characters, control characters, and the trailing dots + and spaces that FAT quietly eats -- a name ending in one round-trips as a + different name, which is worse than being rejected outright. + """ + cleaned = FAT32_RESERVED.sub("_", component).rstrip(". ") + # Stripping can empty a component outright: a directory named "..." is + # legal on ext4 and nothing at all on FAT. + return cleaned or "_" + + +def mirror_path_for(source, source_root, mirror_root, safe=False): """Return the mirror path corresponding to a source file.""" - return (mirror_root / source.relative_to(source_root)).with_suffix(MIRROR_SUFFIX) + relative = source.relative_to(source_root).with_suffix(MIRROR_SUFFIX) + if safe: + relative = Path(*(fat32_safe(part) for part in relative.parts)) + return mirror_root / relative def is_current(source, mirror): @@ -138,6 +170,30 @@ def is_current(source, mirror): return abs(source.stat().st_mtime - mirror.stat().st_mtime) <= MTIME_TOLERANCE_SECONDS +@functools.lru_cache(maxsize=4096) +def mirror_cover(source_directory, mirror_directory): + """Put a copy of the album's cover beside its tracks in the mirror. + + Cached per directory: an album's tracks all ask for the same file, and the + answer cannot change within a pass. + """ + cover = find_cover(source_directory) + if cover is None or cover.suffix.lower() not in (".jpg", ".jpeg"): + # Only JPEG is copied. Rockbox will read a BMP too, but converting a + # PNG is ffmpeg work for a file nothing else in the pass needs. + return None + destination = mirror_directory / MIRROR_COVER + try: + if destination.is_file() and destination.stat().st_size == cover.stat().st_size: + return destination + shutil.copy2(cover, destination) + make_group_readable(destination) + except OSError as error: + logger.warning("could not copy cover for %s: %s", source_directory, error) + return None + return destination + + def make_group_readable(path): """Add the group-read bit to a mirror file, leaving the rest of the mode alone.""" mode = path.stat().st_mode @@ -228,6 +284,7 @@ def encode(source, mirror, quality_args, dry_run): return Result("encoded", mirror) mirror.parent.mkdir(parents=True, exist_ok=True) + mirror_cover(source.parent, mirror.parent) # Probing costs an ffprobe process per file, so only ask when the answer # can change the command. With no cover file beside the track, `-map # 0:v:0?` carries embedded art if there is any and shrugs if there is not. @@ -275,6 +332,7 @@ def copy(source, mirror, dry_run): return Result("copied", mirror) mirror.parent.mkdir(parents=True, exist_ok=True) + mirror_cover(source.parent, mirror.parent) # Through a temporary file and a rename, for the same reason encodes go # that way, and a sharper one: copy2 reproduces the source's mtime as well @@ -299,8 +357,26 @@ def copy(source, mirror, dry_run): return Result("copied", mirror) -def process(source, mirror, quality_args, dry_run): +def adopt_existing(source, mirror, previous): + """Move an already-encoded file to its new name. Returns whether it moved. + + Turning on FAT32-safe naming changes the path of every track whose name + held a reserved character. Without this the run would encode them all again + and then prune the originals -- hours of work to produce files that already + exist, byte for byte, under the old name. + """ + if previous == mirror or not previous.is_file() or not is_current(source, previous): + return False + mirror.parent.mkdir(parents=True, exist_ok=True) + os.replace(previous, mirror) + logger.info("renamed %s -> %s", previous.name, mirror.name) + return True + + +def process(source, mirror, quality_args, dry_run, previous=None): """Bring one source file's mirror entry up to date.""" + if previous is not None and not dry_run and not mirror.exists(): + adopt_existing(source, mirror, previous) if is_current(source, mirror): # A mirror written before this bit was set has a correct mtime, so # nothing else in the pass would ever revisit it. Top it up here @@ -324,7 +400,7 @@ def find_sources(root): yield path -def plan(scan_root, source_root, mirror_root): +def plan(scan_root, source_root, mirror_root, safe=False): """Map each mirror path to the one source that should produce it. Two sources can want the same mirror path -- `01 Song.flac` alongside a @@ -335,12 +411,20 @@ def plan(scan_root, source_root, mirror_root): the outcome stable and predictable instead. """ chosen = {} + # Keyed case-insensitively when the target is FAT32, because two names + # differing only in case are two files here and one file there. Detecting + # that now beats discovering it as a silent overwrite during the copy. + seen = {} for source in find_sources(scan_root): - mirror = mirror_path_for(source, source_root, mirror_root) - rival = chosen.get(mirror) + mirror = mirror_path_for(source, source_root, mirror_root, safe) + key = str(mirror).casefold() if safe else str(mirror) + rival_path = seen.get(key) + rival = chosen.get(rival_path) if rival_path else None if rival is None: + seen[key] = mirror chosen[mirror] = source continue + mirror = rival_path winner, loser = sorted((source, rival), key=source_rank) logger.warning("%s and %s both map to %s; using %s", rival, source, mirror, winner) chosen[mirror] = winner @@ -374,6 +458,13 @@ def prune(mirror_root, expected, dry_run): mirror.unlink(missing_ok=True) if not dry_run: + # A cover copied for an album whose tracks have all gone is an orphan + # too, and while it sits there the directory never looks empty. + for cover in sorted(mirror_root.rglob(MIRROR_COVER)): + if not any(cover.parent.glob(f"*{MIRROR_SUFFIX}")): + logger.info("removing orphan %s", cover) + cover.unlink(missing_ok=True) + # Deepest first, so a directory emptied by the loop above is caught. for directory in sorted(mirror_root.rglob("*"), reverse=True): if directory.is_dir() and not any(directory.iterdir()): @@ -382,7 +473,9 @@ def prune(mirror_root, expected, dry_run): return removed -def run_once(scan_root, source_root, mirror_root, quality_args, jobs, dry_run, do_prune): +def run_once( + scan_root, source_root, mirror_root, quality_args, jobs, dry_run, do_prune, safe=False +): """Run a single pass. Returns the number of failures. `scan_root` is what gets walked and `source_root` is what mirror paths are @@ -393,11 +486,18 @@ def run_once(scan_root, source_root, mirror_root, quality_args, jobs, dry_run, d counts = {"encoded": 0, "copied": 0, "skipped": 0, "failed": 0} failures = [] - work = plan(scan_root, source_root, mirror_root) + work = plan(scan_root, source_root, mirror_root, safe) with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool: futures = [ - pool.submit(process, source, mirror, quality_args, dry_run) + pool.submit( + process, + source, + mirror, + quality_args, + dry_run, + mirror_path_for(source, source_root, mirror_root) if safe else None, + ) for mirror, source in work.items() ] for future in concurrent.futures.as_completed(futures): @@ -492,6 +592,13 @@ def build_parser(): default=None, help="limit the pass to one directory below --source; skips pruning", ) + parser.add_argument( + "--fat32-safe", + action="store_true", + default=os.getenv("MUSIC_MIRROR_FAT32_SAFE", "").lower() in ("1", "true", "yes"), + help="name mirror files so a FAT32 device will accept them" + " (env MUSIC_MIRROR_FAT32_SAFE)", + ) parser.add_argument( "--no-prune", action="store_true", @@ -575,6 +682,7 @@ def main(argv=None): args.jobs, args.dry_run, do_prune, + args.fat32_safe, ) if interval is None or stopping: return 1 if failures else 0 diff --git a/tests/conftest.py b/tests/conftest.py index 753d029..c672c11 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -78,6 +78,23 @@ def make_flac(): return factory +@pytest.fixture +def make_cover(): + """Return a factory writing a small JPEG beside an album's tracks.""" + + def factory(path): + path.parent.mkdir(parents=True, exist_ok=True) + subprocess.run( + ["ffmpeg", "-nostdin", "-hide_banner", "-loglevel", "error", "-y", + "-f", "lavfi", "-i", "color=c=red:s=64x64:d=1", "-frames:v", "1", str(path)], + check=True, + capture_output=True, + ) + return path + + return factory + + @pytest.fixture def probe_tag(): """Return a helper reading a single metadata tag from a file.""" diff --git a/tests/test_check_fat32.py b/tests/test_check_fat32.py new file mode 100644 index 0000000..f0c9da3 --- /dev/null +++ b/tests/test_check_fat32.py @@ -0,0 +1,64 @@ +import subprocess +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools")) + +import check_fat32 # noqa: E402 + + +def problems(name): + return check_fat32.problems_with(Path(name)) + + +def test_a_reserved_character_is_a_problem(): + assert any("reserved" in p for p in problems("Album/Motherf**ker.mp3")) + + +def test_a_trailing_dot_or_space_is_a_problem(): + """FAT eats them silently, so the name round-trips as a different name.""" + assert any("trailing" in p for p in problems("Trailing Dot./x.mp3")) + assert any("trailing" in p for p in problems("Space /x.mp3")) + + +def test_an_ordinary_name_is_fine(): + assert problems("Artist/Album/01 Fine Track.mp3") == [] + + +def test_accents_are_fine(): + assert problems("Mötley Crüe/Album/Track.mp3") == [] + + +def test_an_over_long_path_is_a_problem(): + deep = "/".join("d" * 40 for _ in range(10)) + "/track.mp3" + assert any("path of" in p for p in problems(deep)) + + +def test_case_collisions_are_reported(tmp_path): + album = tmp_path / "Album" + album.mkdir() + (album / "Song.mp3").write_bytes(b"x") + (album / "SONG.mp3").write_bytes(b"x") + + completed = subprocess.run( + [sys.executable, str(Path(check_fat32.__file__)), str(tmp_path)], + capture_output=True, + text=True, + ) + + assert completed.returncode == 1 + assert "collides case-insensitively" in completed.stdout + + +def test_a_clean_tree_exits_zero(tmp_path): + (tmp_path / "Album").mkdir() + (tmp_path / "Album" / "Fine.mp3").write_bytes(b"x") + + completed = subprocess.run( + [sys.executable, str(Path(check_fat32.__file__)), str(tmp_path)], + capture_output=True, + text=True, + ) + + assert completed.returncode == 0 + assert "0 problems" in completed.stderr diff --git a/tests/test_music_mirror.py b/tests/test_music_mirror.py index b609985..d01e0c8 100644 --- a/tests/test_music_mirror.py +++ b/tests/test_music_mirror.py @@ -443,3 +443,118 @@ def test_mirror_inside_source_is_refused(tmp_path, make_flac): def test_missing_source_is_refused(tmp_path): assert run(tmp_path / "nope", tmp_path / "dst") == 2 + + +@pytest.mark.parametrize( + ("name", "expected"), + [ + ("Kick Out the Epic Motherf**ker", "Kick Out the Epic Motherf__ker"), + ("Where Are You?", "Where Are You_"), + ("Song: Part 2", "Song_ Part 2"), + ('"Heroes"', "_Heroes_"), + ("trailing dot.", "trailing dot"), + ("trailing space ", "trailing space"), + ("...", "_"), + ("Mötley Crüe", "Mötley Crüe"), + ("perfectly ordinary", "perfectly ordinary"), + ], +) +def test_fat32_safe_names(name, expected): + """A name FAT32 will not take is a track that silently does not arrive.""" + assert music_mirror.fat32_safe(name) == expected + + +def test_a_reserved_character_is_replaced_in_the_mirror_path(tmp_path, make_flac): + source = tmp_path / "src" + mirror = tmp_path / "dst" + make_flac(source / "Dada Life" / "Album" / "Kick Out the Epic Motherf**ker.flac") + + run(source, mirror, "--fat32-safe") + + assert (mirror / "Dada Life" / "Album" / "Kick Out the Epic Motherf__ker.mp3").is_file() + + +def test_without_the_flag_the_name_is_left_alone(tmp_path, make_flac): + source = tmp_path / "src" + mirror = tmp_path / "dst" + make_flac(source / "Album" / "Where Are You?.flac") + + run(source, mirror) + + assert (mirror / "Album" / "Where Are You?.mp3").is_file() + + +def test_an_existing_mirror_is_renamed_not_re_encoded(tmp_path, make_flac): + """Turning the flag on changes the path of every track holding a reserved + character. Re-encoding those would be hours of work to produce files that + already exist byte for byte.""" + source = tmp_path / "src" + mirror = tmp_path / "dst" + make_flac(source / "Album" / "Where Are You?.flac") + + run(source, mirror) + before = mirror / "Album" / "Where Are You?.mp3" + contents = before.read_bytes() + stamp = before.stat().st_mtime_ns + + run(source, mirror, "--fat32-safe") + + after = mirror / "Album" / "Where Are You_.mp3" + assert after.is_file() + assert not before.exists() + assert after.read_bytes() == contents, "it was re-encoded rather than moved" + assert after.stat().st_mtime_ns == stamp + + +def test_names_colliding_only_by_case_are_caught(tmp_path, make_flac): + """Two files here, one file on FAT32. Better found now than as a silent + overwrite partway through the copy.""" + source = tmp_path / "src" + mirror = tmp_path / "dst" + make_flac(source / "Album" / "Song.flac") + make_flac(source / "Album" / "SONG.flac") + + run(source, mirror, "--fat32-safe") + + written = sorted(p.name for p in (mirror / "Album").glob("*.mp3")) + assert len(written) == 1, written + + +def test_the_album_cover_is_copied_beside_the_tracks(tmp_path, make_flac, make_cover): + """Rockbox looks for art on the filesystem; its search never touches the + picture embedded in the tag.""" + source = tmp_path / "src" + mirror = tmp_path / "dst" + make_flac(source / "Album" / "a.flac") + make_cover(source / "Album" / "cover.jpg") + + run(source, mirror) + + assert (mirror / "Album" / "cover.jpg").is_file() + + +def test_a_copied_cover_is_group_readable(tmp_path, make_flac, make_cover, tight_umask): + source = tmp_path / "src" + mirror = tmp_path / "dst" + make_flac(source / "Album" / "a.flac") + make_cover(source / "Album" / "cover.jpg") + + run(source, mirror) + + assert (mirror / "Album" / "cover.jpg").stat().st_mode & stat.S_IRGRP + + +def test_a_cover_left_without_tracks_is_pruned(tmp_path, make_flac, make_cover): + """Otherwise the directory never looks empty and never goes.""" + source = tmp_path / "src" + mirror = tmp_path / "dst" + make_flac(source / "Gone" / "a.flac") + make_cover(source / "Gone" / "cover.jpg") + + run(source, mirror) + assert (mirror / "Gone" / "cover.jpg").is_file() + + shutil.rmtree(source / "Gone") + run(source, mirror) + + assert not (mirror / "Gone").exists() diff --git a/tests/test_submit_scrobbles.py b/tests/test_submit_scrobbles.py new file mode 100644 index 0000000..d91e8b5 --- /dev/null +++ b/tests/test_submit_scrobbles.py @@ -0,0 +1,189 @@ +import json +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools")) + +import submit_scrobbles # noqa: E402 + +LOG = """#AUDIOSCROBBLER/1.1 +#TZ/UNKNOWN +#CLIENT/Rockbox ipodvideo 4.0 +#ARTIST\t#ALBUM\t#TITLE\t#TRACKNUM\t#LENGTH\t#RATING\t#TIMESTAMP\t#MUSICBRAINZ_TRACKID +Pendulum\tImmersion\tWatercolour\t3\t245\tL\t1700000300\t +Green Day\tDookie\tBasket Case\t7\t180\tS\t1700000200\t +Mötley Crüe\tDr. Feelgood\tKickstart My Heart\t2\t283\tL\t1700000100\tmb-1 +""" + + +def test_only_listened_tracks_are_submitted(): + """A skip is not a play.""" + played, skipped, timeless = submit_scrobbles.parse_log(LOG) + + assert [entry["track"] for entry in played] == ["Kickstart My Heart", "Watercolour"] + assert skipped == 1 + assert timeless == 0 + + +def test_entries_come_back_oldest_first(): + played, _, _ = submit_scrobbles.parse_log(LOG) + + assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"] + + +def test_a_timeless_log_is_counted_and_not_submitted(): + """Without a real-time clock Rockbox writes every timestamp as zero. Those + cannot be scrobbled without inventing when they happened.""" + log = LOG + "Band\tAlbum\tTrack\t-1\t100\tL\t0\t\n" + + played, _, timeless = submit_scrobbles.parse_log(log) + + assert timeless == 1 + assert all(int(entry["timestamp"]) > 0 for entry in played) + + +def test_absent_optional_fields_are_dropped(): + played, _, _ = submit_scrobbles.parse_log(LOG) + params = submit_scrobbles.batch_params(played) + + assert "mbid[0]" in params # Kickstart My Heart has one + assert "mbid[1]" not in params # Watercolour does not + assert params["trackNumber[0]"] == "2" + + +def test_a_track_number_of_minus_one_is_not_sent(): + """Rockbox writes -1 when it does not know, which is not a track number.""" + played, _, _ = submit_scrobbles.parse_log( + "Band\tAlbum\tTrack\t-1\t100\tL\t1700000000\t\n" + ) + + assert submit_scrobbles.batch_params(played).get("trackNumber[0]") is None + + +def test_the_signature_sorts_names_by_ascii_not_by_number(): + """Last.fm sorts parameter names as strings, so artist[10] precedes + artist[1]. Sorting numerically produces an invalid signature and nothing + else.""" + params = {"artist[1]": "b", "artist[10]": "a", "api_key": "k"} + + expected = submit_scrobbles.hashlib.md5( + ("api_keyk" + "artist[1]b" + "artist[10]a" + "s").encode() + ).hexdigest() + assert submit_scrobbles.sign(params, "s") != expected + + correct = submit_scrobbles.hashlib.md5( + ("api_keyk" + "artist[10]a" + "artist[1]b" + "s").encode() + ).hexdigest() + assert submit_scrobbles.sign(params, "s") == correct + + +def fake_transport(responses): + """Return a transport serving canned responses and recording requests.""" + sent = [] + + def transport(request, timeout=None): + sent.append(dict(submit_scrobbles.urllib.parse.parse_qsl(request.data.decode()))) + return json.dumps(responses[len(sent) - 1]) + + transport.sent = sent + return transport + + +def test_scrobbles_are_sent_in_batches_of_fifty(): + entries = [ + {"artist": "A", "track": f"T{i}", "timestamp": str(1700000000 + i)} + for i in range(120) + ] + accepted = {"scrobbles": {"@attr": {"accepted": 50, "ignored": 0}}} + transport = fake_transport([accepted, accepted, accepted]) + + submit_scrobbles.submit(entries, "k", "s", "sk", transport, delay=0) + + assert len(transport.sent) == 3 + assert transport.sent[0]["method"] == "track.scrobble" + assert "artist[49]" in transport.sent[0] + assert "artist[50]" not in transport.sent[0] + + +def test_every_request_carries_a_signature_and_session(): + entries = [{"artist": "A", "track": "T", "timestamp": "1700000000"}] + transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}]) + + submit_scrobbles.submit(entries, "k", "s", "session-key", transport, delay=0) + + assert transport.sent[0]["sk"] == "session-key" + assert len(transport.sent[0]["api_sig"]) == 32 + + +def test_a_service_error_is_raised_not_swallowed(): + entries = [{"artist": "A", "track": "T", "timestamp": "1700000000"}] + transport = fake_transport([{"error": 9, "message": "Invalid session key"}]) + + with pytest.raises(submit_scrobbles.LastfmError, match="error 9"): + submit_scrobbles.submit(entries, "k", "s", "sk", transport, delay=0) + + +def test_the_log_is_set_aside_after_a_successful_submission(tmp_path, monkeypatch): + device = tmp_path / "IPOD" + device.mkdir() + (device / ".scrobbler.log").write_text(LOG, encoding="utf-8") + transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 2, "ignored": 0}}}]) + monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk") + + submit_scrobbles.main( + [str(device), "--api-key", "k", "--api-secret", "s"], transport=transport + ) + + assert not (device / ".scrobbler.log").exists() + # Renamed, not deleted: if Last.fm quietly dropped one, the evidence remains. + assert list(device.glob(".scrobbler.log.*.submitted")) + + +def test_a_failed_submission_leaves_the_log_alone(tmp_path, monkeypatch): + device = tmp_path / "IPOD" + device.mkdir() + (device / ".scrobbler.log").write_text(LOG, encoding="utf-8") + transport = fake_transport([{"error": 29, "message": "Rate limit"}]) + monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk") + + code = submit_scrobbles.main( + [str(device), "--api-key", "k", "--api-secret", "s"], transport=transport + ) + + assert code == 1 + assert (device / ".scrobbler.log").is_file() + + +def test_a_dry_run_submits_nothing(tmp_path): + device = tmp_path / "IPOD" + device.mkdir() + (device / ".scrobbler.log").write_text(LOG, encoding="utf-8") + transport = fake_transport([]) + + submit_scrobbles.main([str(device), "--dry-run"], transport=transport) + + assert transport.sent == [] + assert (device / ".scrobbler.log").is_file() + + +def test_no_log_is_not_an_error(tmp_path): + device = tmp_path / "IPOD" + device.mkdir() + + assert submit_scrobbles.main([str(device)], transport=fake_transport([])) == 0 + + +def test_write_credentials_are_required(tmp_path, capsys, monkeypatch): + """The read-only key used elsewhere is not enough for a write method.""" + monkeypatch.delenv("LASTFM_API_KEY", raising=False) + monkeypatch.delenv("LASTFM_API_SECRET", raising=False) + device = tmp_path / "IPOD" + device.mkdir() + (device / ".scrobbler.log").write_text(LOG, encoding="utf-8") + + code = submit_scrobbles.main([str(device)], transport=fake_transport([])) + + assert code == 2 + assert "LASTFM_API_SECRET" in capsys.readouterr().err diff --git a/tools/check_fat32.py b/tools/check_fat32.py new file mode 100755 index 0000000..d0baf30 --- /dev/null +++ b/tools/check_fat32.py @@ -0,0 +1,93 @@ +#!/usr/bin/env python3 +"""Report paths a FAT32 device will not accept, before copying rather than during. + +Run this against the mirror before an rsync to a Rockbox iPod. rsync will +report the failures too, but scattered through a run of fifty thousand files, +where they are easy to lose. + +Checks the four ways a name fails on FAT32: reserved characters, trailing dots +or spaces that FAT silently eats, components longer than 255 characters, and +names that differ only in case -- two files here, one file there, and the +second silently overwrites the first. +""" + +import argparse +import os +import re +import sys +import unicodedata +from collections import defaultdict +from pathlib import Path + +RESERVED = re.compile(r'[<>:"\\|?*\x00-\x1f]') +COMPONENT_LIMIT = 255 +# Rockbox builds paths into a fixed buffer; long trees fail on the device even +# when every individual component is legal. +PATH_LIMIT = 260 + + +def problems_with(relative): + """Return every reason this relative path is unfit for FAT32.""" + found = [] + for part in relative.parts: + if RESERVED.search(part): + found.append(f"reserved character in {part!r}") + if part != part.rstrip(". "): + found.append(f"trailing dot or space in {part!r}") + if len(part) > COMPONENT_LIMIT: + found.append(f"component of {len(part)} characters") + if len(str(relative)) > PATH_LIMIT: + found.append(f"path of {len(str(relative))} characters") + return found + + +def walk(root): + """Yield every file below a root, as a path relative to it.""" + for base, _, names in os.walk(root): + for name in names: + yield Path(os.path.join(base, name)).relative_to(root) + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("root", help="directory to check, e.g. the mirror") + parser.add_argument("--limit", type=int, default=0, help="show at most this many") + args = parser.parse_args(argv) + + root = Path(args.root) + if not root.is_dir(): + print(f"{root} is not a directory", file=sys.stderr) + return 2 + + faults = [] + by_case = defaultdict(list) + total = 0 + for relative in walk(root): + total += 1 + # NFC first: the same name written by two systems is otherwise two + # different strings, and the collision check would miss it. + key = unicodedata.normalize("NFC", str(relative)).casefold() + by_case[key].append(relative) + for problem in problems_with(relative): + faults.append((relative, problem)) + + for relative, group in sorted(by_case.items()): + if len(group) > 1: + names = ", ".join(str(path) for path in sorted(group)) + faults.append((group[0], f"collides case-insensitively with: {names}")) + + for relative, problem in faults[: args.limit or None]: + print(f"{relative}\t{problem}") + + print(f"\n{len(faults)} problems across {total} files", file=sys.stderr) + if faults: + print( + "Run music-mirror with --fat32-safe to have the mirror named" + " acceptably in the first place.", + file=sys.stderr, + ) + return 1 if faults else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/submit_scrobbles.py b/tools/submit_scrobbles.py new file mode 100755 index 0000000..3468852 --- /dev/null +++ b/tools/submit_scrobbles.py @@ -0,0 +1,255 @@ +#!/usr/bin/env python3 +"""Submit a Rockbox scrobbler log to Last.fm, then set it aside. + +Rockbox writes /.scrobbler.log on the device in AUDIOSCROBBLER 1.1 format: one +tab-separated line per track, rated `L` for listened or `S` for skipped. Only +the listened ones are submitted; a skip is not a play. + +Scrobbling is a write method, so unlike everything else here it needs the API +secret and a session key, obtained once through the browser. Read-only calls +elsewhere in these projects need neither. +""" + +import argparse +import hashlib +import json +import os +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path + +API_ROOT = "https://ws.audioscrobbler.com/2.0/" + +# Last.fm's documented ceiling for one track.scrobble call. +BATCH = 50 + +# Rockbox names the log for whether the target has a real-time clock. Without +# one every timestamp it writes is zero, which is not a time anything can +# scrobble. +LOG_NAMES = (".scrobbler.log", ".scrobbler-timeless.log") + +SESSION_FILE = Path( + os.getenv("XDG_CONFIG_HOME", Path.home() / ".config") +) / "music-mirror" / "lastfm.json" + + +class LastfmError(Exception): + """A Last.fm request that failed.""" + + +def parse_log(text): + """Return the listened tracks in an AUDIOSCROBBLER log, oldest first. + + Fields are artist, album, title, track number, length, rating, timestamp + and MusicBrainz id. Rockbox converts any tab inside a field to a space + before writing, so splitting on tabs is safe. + """ + played, skipped, timeless = [], 0, 0 + for line in text.splitlines(): + if not line or line.startswith("#"): + continue + fields = line.split("\t") + if len(fields) < 7: + continue + artist, album, title, number, length, rating, timestamp = fields[:7] + mbid = fields[7] if len(fields) > 7 else "" + if rating.strip().upper() != "L": + skipped += 1 + continue + try: + when = int(timestamp) + except ValueError: + continue + if when <= 0: + timeless += 1 + continue + if not artist or not title: + continue + played.append( + { + "artist": artist, + "album": album, + "track": title, + "trackNumber": number if number not in ("", "-1") else "", + "duration": length if length.isdigit() and int(length) > 0 else "", + "timestamp": str(when), + "mbid": mbid, + } + ) + played.sort(key=lambda entry: int(entry["timestamp"])) + return played, skipped, timeless + + +def sign(params, secret): + """Return Last.fm's method signature for a set of parameters. + + Names are sorted by the ASCII table rather than numerically, which is why + `artist[10]` comes before `artist[1]`. Getting that wrong produces an + invalid signature and nothing else. + """ + joined = "".join(f"{name}{params[name]}" for name in sorted(params)) + return hashlib.md5((joined + secret).encode("utf-8")).hexdigest() # noqa: S324 + + +def post(params, transport): + """Sign, post, and return the decoded response.""" + body = urllib.parse.urlencode(params).encode("utf-8") + request = urllib.request.Request(API_ROOT, data=body) + try: + payload = json.loads(transport(request)) + except urllib.error.HTTPError as error: + detail = error.read().decode("utf-8", "replace")[:300] + raise LastfmError(f"HTTP {error.code}: {detail}") from error + except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as error: + raise LastfmError(str(error)) from error + if payload.get("error"): + raise LastfmError(f"error {payload['error']}: {payload.get('message', '')}") + return payload + + +def call(method, params, key, secret, session, transport): + """Make one signed, authenticated call.""" + full = {**params, "method": method, "api_key": key} + if session: + full["sk"] = session + full["api_sig"] = sign(full, secret) + full["format"] = "json" + return post(full, transport) + + +def authorise(key, secret, transport, opener=print): + """Walk the one-time browser authorisation and return a session key.""" + token = call("auth.getToken", {}, key, secret, None, transport)["token"] + url = f"https://www.last.fm/api/auth/?api_key={key}&token={token}" + opener(f"Open this, approve the application, then press Enter:\n\n {url}\n") + input() + session = call("auth.getSession", {"token": token}, key, secret, None, transport) + return session["session"]["key"] + + +def load_session(): + if SESSION_FILE.is_file(): + return json.loads(SESSION_FILE.read_text(encoding="utf-8")).get("session") + return None + + +def save_session(session): + SESSION_FILE.parent.mkdir(parents=True, exist_ok=True) + SESSION_FILE.write_text(json.dumps({"session": session}), encoding="utf-8") + SESSION_FILE.chmod(0o600) + + +def batch_params(entries): + """Return the indexed parameters for one track.scrobble call.""" + params = {} + for index, entry in enumerate(entries): + for name in ("artist", "track", "timestamp", "album", "trackNumber", "duration", "mbid"): + if entry.get(name): + params[f"{name}[{index}]"] = entry[name] + return params + + +def submit(entries, key, secret, session, transport, delay=1.0): + """Submit every entry. Returns how many the service accepted. + + Batches are counted as they succeed rather than at the end, so a failure + partway through leaves an honest number and the caller can keep the rest of + the log instead of losing it. + """ + accepted = 0 + for start in range(0, len(entries), BATCH): + chunk = entries[start : start + BATCH] + payload = call( + "track.scrobble", batch_params(chunk), key, secret, session, transport + ) + block = payload.get("scrobbles", {}) + summary = block.get("@attr", block) + accepted += int(summary.get("accepted", len(chunk))) + ignored = int(summary.get("ignored", 0)) + if ignored: + print(f" {ignored} of {len(chunk)} ignored by Last.fm", file=sys.stderr) + if start + BATCH < len(entries): + time.sleep(delay) + return accepted + + +def http_post(request, timeout=30): + with urllib.request.urlopen(request, timeout=timeout) as response: # noqa: S310 + return response.read().decode("utf-8") + + +def find_log(device): + """Return the scrobbler log on a mounted device, or None.""" + for name in LOG_NAMES: + candidate = Path(device) / name + if candidate.is_file() and candidate.stat().st_size: + return candidate + return None + + +def main(argv=None, transport=http_post): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("device", help="the mounted device, or a scrobbler log file") + parser.add_argument("--api-key", default=os.getenv("LASTFM_API_KEY")) + parser.add_argument("--api-secret", default=os.getenv("LASTFM_API_SECRET")) + parser.add_argument("--dry-run", action="store_true", help="parse and report only") + parser.add_argument( + "--keep", action="store_true", help="do not set the log aside afterwards" + ) + args = parser.parse_args(argv) + + target = Path(args.device) + log = target if target.is_file() else find_log(target) + if log is None: + print("no scrobbler log to submit", file=sys.stderr) + return 0 + + played, skipped, timeless = parse_log(log.read_text(encoding="utf-8", errors="replace")) + print(f"{log}: {len(played)} listened, {skipped} skipped", file=sys.stderr) + if timeless: + print( + f" {timeless} entries have no timestamp, so this target has no clock." + " They cannot be scrobbled without inventing when they happened.", + file=sys.stderr, + ) + if not played: + return 0 + if args.dry_run: + for entry in played[:20]: + print(f"{entry['timestamp']}\t{entry['artist']}\t{entry['track']}") + return 0 + + if not args.api_key or not args.api_secret: + print( + "scrobbling is a write method: it needs LASTFM_API_KEY and" + " LASTFM_API_SECRET, not just the read-only key", + file=sys.stderr, + ) + return 2 + + session = load_session() + if not session: + session = authorise(args.api_key, args.api_secret, transport) + save_session(session) + + try: + accepted = submit(played, args.api_key, args.api_secret, session, transport) + except LastfmError as error: + print(f"submission failed: {error}", file=sys.stderr) + return 1 + + print(f"{accepted} scrobbles accepted", file=sys.stderr) + if not args.keep and accepted: + # Renamed rather than deleted: if Last.fm quietly dropped something, + # the evidence is still on the device. + aside = log.with_name(f"{log.name}.{played[-1]['timestamp']}.submitted") + log.rename(aside) + print(f"log moved to {aside.name}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/sync-to-ipod.sh b/tools/sync-to-ipod.sh new file mode 100755 index 0000000..2253c31 --- /dev/null +++ b/tools/sync-to-ipod.sh @@ -0,0 +1,136 @@ +#!/usr/bin/env bash +# Copy the mirror onto a Rockbox device, then unmount it cleanly. +# +# The device is FAT32 with no journal, reached through the Apple firmware's +# disk mode because Rockbox's own mass storage is unreliable on an iFlash. An +# interrupted write is corruption that needs fsck.vfat from another machine, so +# this syncs and unmounts rather than leaving that to whoever pulls the cable. +# +# rsync --delete is pointed at a whole filesystem, so the checks below are the +# point of the script rather than decoration. + +set -euo pipefail + +usage() { + cat >&2 <<'USAGE' +usage: sync-to-ipod.sh [options] + + -n dry run; show what would change and touch nothing + -f copy even if the FAT32 check finds unacceptable paths + -S skip submitting the Rockbox scrobbler log to Last.fm + -U leave the destination mounted afterwards + +Submitting scrobbles needs LASTFM_API_KEY and LASTFM_API_SECRET; it is skipped +with a note when they are unset. Scrobbling is a write method and needs the +secret, unlike the read-only calls elsewhere in these projects. + +The destination must be a mounted FAT filesystem. Reach it with the Apple +firmware's disk mode: Menu+Select to reboot, then immediately Select+Play. +USAGE + exit 2 +} + +dry_run=false +force=false +unmount=true +scrobble=true +while getopts ":nfSUh" option; do + case "$option" in + n) dry_run=true ;; + f) force=true ;; + S) scrobble=false ;; + U) unmount=false ;; + *) usage ;; + esac +done +shift $((OPTIND - 1)) +[ $# -eq 2 ] || usage + +mirror=${1%/} +destination=${2%/} +here=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) + +die() { + printf 'sync-to-ipod: %s\n' "$1" >&2 + exit 1 +} + +[ -d "$mirror" ] || die "mirror $mirror is not a directory" +[ -n "$(ls -A "$mirror")" ] || die "mirror $mirror is empty; refusing to mirror nothing" +[ -d "$destination" ] || die "destination $destination is not a directory" + +# --delete makes every one of these load-bearing. A destination that is not its +# own mount point means the path is wrong, and emptying the wrong directory is +# not a mistake that announces itself. +case "$destination" in + "" | "/" | "$HOME") die "refusing to sync onto $destination" ;; +esac +[ "$(readlink -f "$mirror")" != "$(readlink -f "$destination")" ] || + die "mirror and destination are the same directory" +mountpoint -q -- "$destination" || die "$destination is not a mount point" + +filesystem=$(findmnt -no FSTYPE --target "$destination") +case "$filesystem" in + vfat | exfat) ;; + *) + $force || die "$destination is $filesystem, not FAT; pass -f if that is deliberate" + printf 'sync-to-ipod: destination is %s, not FAT\n' "$filesystem" >&2 + ;; +esac + +if $force; then + printf 'sync-to-ipod: skipping the FAT32 check\n' >&2 +elif ! python3 "$here/check_fat32.py" "$mirror"; then + die "the mirror holds paths FAT32 will not take; run music-mirror with --fat32-safe" +fi + +# Before the copy, not after: the plays already happened, and if the transfer +# then fails there is no reason to have lost them too. +if $scrobble; then + if [ -z "${LASTFM_API_KEY:-}" ] || [ -z "${LASTFM_API_SECRET:-}" ]; then + printf 'sync-to-ipod: no Last.fm credentials, skipping the scrobbler log\n' >&2 + else + scrobble_options=() + $dry_run && scrobble_options+=(--dry-run) + python3 "$here/submit_scrobbles.py" "${scrobble_options[@]}" "$destination" || + die "submitting scrobbles failed; nothing has been copied" + fi +fi + +# -rt rather than -a: owners, groups and permissions mean nothing on FAT, and +# asking for them produces a screenful of errors and a non-zero exit. +# --modify-window=2 because FAT stores mtimes to two-second resolution, without +# which every file looks changed and the whole library is copied every time. +# --delete removes tracks whose source has gone, which is the point. It would +# also remove everything on the device that the mirror does not contain -- and +# if the destination is the card root that means /.rockbox, the Rockbox install +# itself. Excluded paths are not deleted unless --delete-excluded is given, +# which it never is here. +options=(--recursive --times --delete --modify-window=2 --human-readable --info=progress2) +for owned in "/.rockbox" "/.scrobbler.log" "/.scrobbler.log.*" "/.playlist_control" \ + "/System Volume Information" "/.Spotlight-V100" "/.Trashes" "/.fseventsd"; do + options+=(--exclude "$owned") +done +$dry_run && options+=(--dry-run --verbose) + +printf 'sync-to-ipod: %s -> %s\n' "$mirror" "$destination" >&2 +rsync "${options[@]}" "$mirror/" "$destination/" + +if $dry_run; then + printf 'sync-to-ipod: dry run, nothing was written\n' >&2 + exit 0 +fi + +sync +if $unmount; then + device=$(findmnt -no SOURCE --target "$destination") + printf 'sync-to-ipod: unmounting %s\n' "$device" >&2 + if command -v udisksctl >/dev/null 2>&1; then + udisksctl unmount -b "$device" + else + umount -- "$destination" + fi + printf 'sync-to-ipod: safe to disconnect\n' >&2 +else + printf 'sync-to-ipod: still mounted; unmount before disconnecting\n' >&2 +fi