feat: a sync script that submits scrobbles, copies, and unmounts cleanly

tools/sync-to-ipod.sh does the whole transfer to a Rockbox device, so the only
manual part left is the disk-mode button sequence.

The guards are the substance rather than decoration. rsync --delete is being
aimed at a whole filesystem, so the destination must exist, be its own mount
point, and be a FAT filesystem; the mirror must be non-empty and must not be
the destination. Emptying the wrong directory is not a mistake that announces
itself.

It also excludes /.rockbox, the scrobbler logs and the usual filesystem
metadata directories. The mirror does not contain them, so a sync to the card
root would otherwise have deleted the Rockbox installation -- which the first
draft of this script would have done.

The unmount is why this is a script at all. FAT32 has no journal, the device is
reached through the Apple firmware's disk mode because Rockbox's own mass
storage is unreliable on an iFlash, and an interrupted write is corruption that
needs fsck.vfat from another machine.

tools/submit_scrobbles.py sends the Rockbox scrobbler log to Last.fm and sets
it aside. Rockbox writes it in AUDIOSCROBBLER 1.1: tab-separated, one line per
track, rated L for listened or S for skipped, and only the listened ones are a
play. It runs before the copy, because the plays already happened and a failed
transfer is no reason to lose them as well.

Two things there differ from every other Last.fm call in these projects.
Scrobbling is a write method, so it needs the API secret and a session key
obtained once through the browser rather than the read-only key. And a target
with no real-time clock gets /.scrobbler-timeless.log with every timestamp set
to zero; those are counted and reported but never sent, since submitting them
would mean inventing when they happened.

Signature generation sorts parameter names by the ASCII table rather than
numerically, so artist[10] precedes artist[1]. Sorting them the obvious way
produces an invalid signature and no other symptom, so there is a test for it.

The log is renamed rather than deleted once accepted, so that if Last.fm
quietly dropped something the evidence is still on the device.

tools/submit_scrobbles.py

@@ -0,0 +1,255 @@
+#!/usr/bin/env python3
+"""Submit a Rockbox scrobbler log to Last.fm, then set it aside.
+
+Rockbox writes /.scrobbler.log on the device in AUDIOSCROBBLER 1.1 format: one
+tab-separated line per track, rated `L` for listened or `S` for skipped. Only
+the listened ones are submitted; a skip is not a play.
+
+Scrobbling is a write method, so unlike everything else here it needs the API
+secret and a session key, obtained once through the browser. Read-only calls
+elsewhere in these projects need neither.
+"""
+
+import argparse
+import hashlib
+import json
+import os
+import sys
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+from pathlib import Path
+
+API_ROOT = "https://ws.audioscrobbler.com/2.0/"
+
+# Last.fm's documented ceiling for one track.scrobble call.
+BATCH = 50
+
+# Rockbox names the log for whether the target has a real-time clock. Without
+# one every timestamp it writes is zero, which is not a time anything can
+# scrobble.
+LOG_NAMES = (".scrobbler.log", ".scrobbler-timeless.log")
+
+SESSION_FILE = Path(
+    os.getenv("XDG_CONFIG_HOME", Path.home() / ".config")
+) / "music-mirror" / "lastfm.json"
+
+
+class LastfmError(Exception):
+    """A Last.fm request that failed."""
+
+
+def parse_log(text):
+    """Return the listened tracks in an AUDIOSCROBBLER log, oldest first.
+
+    Fields are artist, album, title, track number, length, rating, timestamp
+    and MusicBrainz id. Rockbox converts any tab inside a field to a space
+    before writing, so splitting on tabs is safe.
+    """
+    played, skipped, timeless = [], 0, 0
+    for line in text.splitlines():
+        if not line or line.startswith("#"):
+            continue
+        fields = line.split("\t")
+        if len(fields) < 7:
+            continue
+        artist, album, title, number, length, rating, timestamp = fields[:7]
+        mbid = fields[7] if len(fields) > 7 else ""
+        if rating.strip().upper() != "L":
+            skipped += 1
+            continue
+        try:
+            when = int(timestamp)
+        except ValueError:
+            continue
+        if when <= 0:
+            timeless += 1
+            continue
+        if not artist or not title:
+            continue
+        played.append(
+            {
+                "artist": artist,
+                "album": album,
+                "track": title,
+                "trackNumber": number if number not in ("", "-1") else "",
+                "duration": length if length.isdigit() and int(length) > 0 else "",
+                "timestamp": str(when),
+                "mbid": mbid,
+            }
+        )
+    played.sort(key=lambda entry: int(entry["timestamp"]))
+    return played, skipped, timeless
+
+
+def sign(params, secret):
+    """Return Last.fm's method signature for a set of parameters.
+
+    Names are sorted by the ASCII table rather than numerically, which is why
+    `artist[10]` comes before `artist[1]`. Getting that wrong produces an
+    invalid signature and nothing else.
+    """
+    joined = "".join(f"{name}{params[name]}" for name in sorted(params))
+    return hashlib.md5((joined + secret).encode("utf-8")).hexdigest()  # noqa: S324
+
+
+def post(params, transport):
+    """Sign, post, and return the decoded response."""
+    body = urllib.parse.urlencode(params).encode("utf-8")
+    request = urllib.request.Request(API_ROOT, data=body)
+    try:
+        payload = json.loads(transport(request))
+    except urllib.error.HTTPError as error:
+        detail = error.read().decode("utf-8", "replace")[:300]
+        raise LastfmError(f"HTTP {error.code}: {detail}") from error
+    except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as error:
+        raise LastfmError(str(error)) from error
+    if payload.get("error"):
+        raise LastfmError(f"error {payload['error']}: {payload.get('message', '')}")
+    return payload
+
+
+def call(method, params, key, secret, session, transport):
+    """Make one signed, authenticated call."""
+    full = {**params, "method": method, "api_key": key}
+    if session:
+        full["sk"] = session
+    full["api_sig"] = sign(full, secret)
+    full["format"] = "json"
+    return post(full, transport)
+
+
+def authorise(key, secret, transport, opener=print):
+    """Walk the one-time browser authorisation and return a session key."""
+    token = call("auth.getToken", {}, key, secret, None, transport)["token"]
+    url = f"https://www.last.fm/api/auth/?api_key={key}&token={token}"
+    opener(f"Open this, approve the application, then press Enter:\n\n  {url}\n")
+    input()
+    session = call("auth.getSession", {"token": token}, key, secret, None, transport)
+    return session["session"]["key"]
+
+
+def load_session():
+    if SESSION_FILE.is_file():
+        return json.loads(SESSION_FILE.read_text(encoding="utf-8")).get("session")
+    return None
+
+
+def save_session(session):
+    SESSION_FILE.parent.mkdir(parents=True, exist_ok=True)
+    SESSION_FILE.write_text(json.dumps({"session": session}), encoding="utf-8")
+    SESSION_FILE.chmod(0o600)
+
+
+def batch_params(entries):
+    """Return the indexed parameters for one track.scrobble call."""
+    params = {}
+    for index, entry in enumerate(entries):
+        for name in ("artist", "track", "timestamp", "album", "trackNumber", "duration", "mbid"):
+            if entry.get(name):
+                params[f"{name}[{index}]"] = entry[name]
+    return params
+
+
+def submit(entries, key, secret, session, transport, delay=1.0):
+    """Submit every entry. Returns how many the service accepted.
+
+    Batches are counted as they succeed rather than at the end, so a failure
+    partway through leaves an honest number and the caller can keep the rest of
+    the log instead of losing it.
+    """
+    accepted = 0
+    for start in range(0, len(entries), BATCH):
+        chunk = entries[start : start + BATCH]
+        payload = call(
+            "track.scrobble", batch_params(chunk), key, secret, session, transport
+        )
+        block = payload.get("scrobbles", {})
+        summary = block.get("@attr", block)
+        accepted += int(summary.get("accepted", len(chunk)))
+        ignored = int(summary.get("ignored", 0))
+        if ignored:
+            print(f"  {ignored} of {len(chunk)} ignored by Last.fm", file=sys.stderr)
+        if start + BATCH < len(entries):
+            time.sleep(delay)
+    return accepted
+
+
+def http_post(request, timeout=30):
+    with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310
+        return response.read().decode("utf-8")
+
+
+def find_log(device):
+    """Return the scrobbler log on a mounted device, or None."""
+    for name in LOG_NAMES:
+        candidate = Path(device) / name
+        if candidate.is_file() and candidate.stat().st_size:
+            return candidate
+    return None
+
+
+def main(argv=None, transport=http_post):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("device", help="the mounted device, or a scrobbler log file")
+    parser.add_argument("--api-key", default=os.getenv("LASTFM_API_KEY"))
+    parser.add_argument("--api-secret", default=os.getenv("LASTFM_API_SECRET"))
+    parser.add_argument("--dry-run", action="store_true", help="parse and report only")
+    parser.add_argument(
+        "--keep", action="store_true", help="do not set the log aside afterwards"
+    )
+    args = parser.parse_args(argv)
+
+    target = Path(args.device)
+    log = target if target.is_file() else find_log(target)
+    if log is None:
+        print("no scrobbler log to submit", file=sys.stderr)
+        return 0
+
+    played, skipped, timeless = parse_log(log.read_text(encoding="utf-8", errors="replace"))
+    print(f"{log}: {len(played)} listened, {skipped} skipped", file=sys.stderr)
+    if timeless:
+        print(
+            f"  {timeless} entries have no timestamp, so this target has no clock."
+            " They cannot be scrobbled without inventing when they happened.",
+            file=sys.stderr,
+        )
+    if not played:
+        return 0
+    if args.dry_run:
+        for entry in played[:20]:
+            print(f"{entry['timestamp']}\t{entry['artist']}\t{entry['track']}")
+        return 0
+
+    if not args.api_key or not args.api_secret:
+        print(
+            "scrobbling is a write method: it needs LASTFM_API_KEY and"
+            " LASTFM_API_SECRET, not just the read-only key",
+            file=sys.stderr,
+        )
+        return 2
+
+    session = load_session()
+    if not session:
+        session = authorise(args.api_key, args.api_secret, transport)
+        save_session(session)
+
+    try:
+        accepted = submit(played, args.api_key, args.api_secret, session, transport)
+    except LastfmError as error:
+        print(f"submission failed: {error}", file=sys.stderr)
+        return 1
+
+    print(f"{accepted} scrobbles accepted", file=sys.stderr)
+    if not args.keep and accepted:
+        # Renamed rather than deleted: if Last.fm quietly dropped something,
+        # the evidence is still on the device.
+        aside = log.with_name(f"{log.name}.{played[-1]['timestamp']}.submitted")
+        log.rename(aside)
+        print(f"log moved to {aside.name}", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tools/sync-to-ipod.sh

@@ -0,0 +1,136 @@
+#!/usr/bin/env bash
+# Copy the mirror onto a Rockbox device, then unmount it cleanly.
+#
+# The device is FAT32 with no journal, reached through the Apple firmware's
+# disk mode because Rockbox's own mass storage is unreliable on an iFlash. An
+# interrupted write is corruption that needs fsck.vfat from another machine, so
+# this syncs and unmounts rather than leaving that to whoever pulls the cable.
+#
+# rsync --delete is pointed at a whole filesystem, so the checks below are the
+# point of the script rather than decoration.
+
+set -euo pipefail
+
+usage() {
+    cat >&2 <<'USAGE'
+usage: sync-to-ipod.sh [options] <mirror> <destination>
+
+  -n  dry run; show what would change and touch nothing
+  -f  copy even if the FAT32 check finds unacceptable paths
+  -S  skip submitting the Rockbox scrobbler log to Last.fm
+  -U  leave the destination mounted afterwards
+
+Submitting scrobbles needs LASTFM_API_KEY and LASTFM_API_SECRET; it is skipped
+with a note when they are unset. Scrobbling is a write method and needs the
+secret, unlike the read-only calls elsewhere in these projects.
+
+The destination must be a mounted FAT filesystem. Reach it with the Apple
+firmware's disk mode: Menu+Select to reboot, then immediately Select+Play.
+USAGE
+    exit 2
+}
+
+dry_run=false
+force=false
+unmount=true
+scrobble=true
+while getopts ":nfSUh" option; do
+    case "$option" in
+        n) dry_run=true ;;
+        f) force=true ;;
+        S) scrobble=false ;;
+        U) unmount=false ;;
+        *) usage ;;
+    esac
+done
+shift $((OPTIND - 1))
+[ $# -eq 2 ] || usage
+
+mirror=${1%/}
+destination=${2%/}
+here=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
+
+die() {
+    printf 'sync-to-ipod: %s\n' "$1" >&2
+    exit 1
+}
+
+[ -d "$mirror" ] || die "mirror $mirror is not a directory"
+[ -n "$(ls -A "$mirror")" ] || die "mirror $mirror is empty; refusing to mirror nothing"
+[ -d "$destination" ] || die "destination $destination is not a directory"
+
+# --delete makes every one of these load-bearing. A destination that is not its
+# own mount point means the path is wrong, and emptying the wrong directory is
+# not a mistake that announces itself.
+case "$destination" in
+    "" | "/" | "$HOME") die "refusing to sync onto $destination" ;;
+esac
+[ "$(readlink -f "$mirror")" != "$(readlink -f "$destination")" ] ||
+    die "mirror and destination are the same directory"
+mountpoint -q -- "$destination" || die "$destination is not a mount point"
+
+filesystem=$(findmnt -no FSTYPE --target "$destination")
+case "$filesystem" in
+    vfat | exfat) ;;
+    *)
+        $force || die "$destination is $filesystem, not FAT; pass -f if that is deliberate"
+        printf 'sync-to-ipod: destination is %s, not FAT\n' "$filesystem" >&2
+        ;;
+esac
+
+if $force; then
+    printf 'sync-to-ipod: skipping the FAT32 check\n' >&2
+elif ! python3 "$here/check_fat32.py" "$mirror"; then
+    die "the mirror holds paths FAT32 will not take; run music-mirror with --fat32-safe"
+fi
+
+# Before the copy, not after: the plays already happened, and if the transfer
+# then fails there is no reason to have lost them too.
+if $scrobble; then
+    if [ -z "${LASTFM_API_KEY:-}" ] || [ -z "${LASTFM_API_SECRET:-}" ]; then
+        printf 'sync-to-ipod: no Last.fm credentials, skipping the scrobbler log\n' >&2
+    else
+        scrobble_options=()
+        $dry_run && scrobble_options+=(--dry-run)
+        python3 "$here/submit_scrobbles.py" "${scrobble_options[@]}" "$destination" ||
+            die "submitting scrobbles failed; nothing has been copied"
+    fi
+fi
+
+# -rt rather than -a: owners, groups and permissions mean nothing on FAT, and
+# asking for them produces a screenful of errors and a non-zero exit.
+# --modify-window=2 because FAT stores mtimes to two-second resolution, without
+# which every file looks changed and the whole library is copied every time.
+# --delete removes tracks whose source has gone, which is the point. It would
+# also remove everything on the device that the mirror does not contain -- and
+# if the destination is the card root that means /.rockbox, the Rockbox install
+# itself. Excluded paths are not deleted unless --delete-excluded is given,
+# which it never is here.
+options=(--recursive --times --delete --modify-window=2 --human-readable --info=progress2)
+for owned in "/.rockbox" "/.scrobbler.log" "/.scrobbler.log.*" "/.playlist_control" \
+    "/System Volume Information" "/.Spotlight-V100" "/.Trashes" "/.fseventsd"; do
+    options+=(--exclude "$owned")
+done
+$dry_run && options+=(--dry-run --verbose)
+
+printf 'sync-to-ipod: %s -> %s\n' "$mirror" "$destination" >&2
+rsync "${options[@]}" "$mirror/" "$destination/"
+
+if $dry_run; then
+    printf 'sync-to-ipod: dry run, nothing was written\n' >&2
+    exit 0
+fi
+
+sync
+if $unmount; then
+    device=$(findmnt -no SOURCE --target "$destination")
+    printf 'sync-to-ipod: unmounting %s\n' "$device" >&2
+    if command -v udisksctl >/dev/null 2>&1; then
+        udisksctl unmount -b "$device"
+    else
+        umount -- "$destination"
+    fi
+    printf 'sync-to-ipod: safe to disconnect\n' >&2
+else
+    printf 'sync-to-ipod: still mounted; unmount before disconnecting\n' >&2
+fi