feat: a sync script that submits scrobbles, copies, and unmounts cleanly
Build and publish container / build (pull_request) Canceled after 3m8s

tools/sync-to-ipod.sh does the whole transfer to a Rockbox device, so the only
manual part left is the disk-mode button sequence.

The guards are the substance rather than decoration. rsync --delete is being
aimed at a whole filesystem, so the destination must exist, be its own mount
point, and be a FAT filesystem; the mirror must be non-empty and must not be
the destination. Emptying the wrong directory is not a mistake that announces
itself.

It also excludes /.rockbox, the scrobbler logs and the usual filesystem
metadata directories. The mirror does not contain them, so a sync to the card
root would otherwise have deleted the Rockbox installation -- which the first
draft of this script would have done.

The unmount is why this is a script at all. FAT32 has no journal, the device is
reached through the Apple firmware's disk mode because Rockbox's own mass
storage is unreliable on an iFlash, and an interrupted write is corruption that
needs fsck.vfat from another machine.

tools/submit_scrobbles.py sends the Rockbox scrobbler log to Last.fm and sets
it aside. Rockbox writes it in AUDIOSCROBBLER 1.1: tab-separated, one line per
track, rated L for listened or S for skipped, and only the listened ones are a
play. It runs before the copy, because the plays already happened and a failed
transfer is no reason to lose them as well.

Two things there differ from every other Last.fm call in these projects.
Scrobbling is a write method, so it needs the API secret and a session key
obtained once through the browser rather than the read-only key. And a target
with no real-time clock gets /.scrobbler-timeless.log with every timestamp set
to zero; those are counted and reported but never sent, since submitting them
would mean inventing when they happened.

Signature generation sorts parameter names by the ASCII table rather than
numerically, so artist[10] precedes artist[1]. Sorting them the obvious way
produces an invalid signature and no other symptom, so there is a test for it.

The log is renamed rather than deleted once accepted, so that if Last.fm
quietly dropped something the evidence is still on the device.
This commit is contained in:
Emma Thorpe
2026-08-25 11:05:00 +01:00
parent 3141f7ca87
commit 802d91490f
4 changed files with 615 additions and 0 deletions
+35
View File
@@ -142,6 +142,41 @@ immediately.
Host-side scripts under `tools/`, not part of the container image. Host-side scripts under `tools/`, not part of the container image.
`sync-to-ipod.sh` does a whole transfer: submits the scrobbler log, checks the
mirror, rsyncs, syncs and unmounts.
```sh
tools/sync-to-ipod.sh /mnt/tank/media/music-mp3 /media/IPOD/Music
tools/sync-to-ipod.sh -n /mnt/tank/media/music-mp3 /media/IPOD/Music # dry run
```
It refuses to start unless the destination is a mounted FAT filesystem that is
its own mount point, because `--delete` aimed at the wrong directory empties it
and does not announce itself. It also excludes `/.rockbox`, the scrobbler logs
and the various filesystem metadata directories from deletion — the mirror does
not contain them, and without the exclusion a sync to the card root would
remove the Rockbox install.
The unmount is the point of doing this in a script. FAT32 has no journal and
the device is reached through disk mode, so an interrupted write is corruption
that needs `fsck.vfat` from another machine.
`submit_scrobbles.py` sends the Rockbox scrobbler log to Last.fm and sets it
aside. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
tab-separated line per track rated `L` for listened or `S` for skipped; only
the listened ones are sent. It runs **before** the copy, since the plays
already happened and a failed transfer is no reason to lose them.
Two things are unlike every other Last.fm call in these projects. Scrobbling is
a *write* method, so it needs `LASTFM_API_SECRET` and a session key obtained
once through the browser, not just the read-only key. And on a target with no
real-time clock Rockbox writes `/.scrobbler-timeless.log` with every timestamp
set to zero; those are counted and reported but never submitted, because
scrobbling them would mean inventing when they happened.
The log is renamed rather than deleted once accepted. If Last.fm quietly
dropped something, the evidence is still on the device.
`check_fat32.py` reports paths a FAT32 device will not accept — reserved `check_fat32.py` reports paths a FAT32 device will not accept — reserved
characters, trailing dots and spaces, over-long components and paths, and names characters, trailing dots and spaces, over-long components and paths, and names
colliding case-insensitively. Run it against the mirror **before** an rsync: colliding case-insensitively. Run it against the mirror **before** an rsync:
+189
View File
@@ -0,0 +1,189 @@
import json
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
import submit_scrobbles # noqa: E402
LOG = """#AUDIOSCROBBLER/1.1
#TZ/UNKNOWN
#CLIENT/Rockbox ipodvideo 4.0
#ARTIST\t#ALBUM\t#TITLE\t#TRACKNUM\t#LENGTH\t#RATING\t#TIMESTAMP\t#MUSICBRAINZ_TRACKID
Pendulum\tImmersion\tWatercolour\t3\t245\tL\t1700000300\t
Green Day\tDookie\tBasket Case\t7\t180\tS\t1700000200\t
Mötley Crüe\tDr. Feelgood\tKickstart My Heart\t2\t283\tL\t1700000100\tmb-1
"""
def test_only_listened_tracks_are_submitted():
"""A skip is not a play."""
played, skipped, timeless = submit_scrobbles.parse_log(LOG)
assert [entry["track"] for entry in played] == ["Kickstart My Heart", "Watercolour"]
assert skipped == 1
assert timeless == 0
def test_entries_come_back_oldest_first():
played, _, _ = submit_scrobbles.parse_log(LOG)
assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"]
def test_a_timeless_log_is_counted_and_not_submitted():
"""Without a real-time clock Rockbox writes every timestamp as zero. Those
cannot be scrobbled without inventing when they happened."""
log = LOG + "Band\tAlbum\tTrack\t-1\t100\tL\t0\t\n"
played, _, timeless = submit_scrobbles.parse_log(log)
assert timeless == 1
assert all(int(entry["timestamp"]) > 0 for entry in played)
def test_absent_optional_fields_are_dropped():
played, _, _ = submit_scrobbles.parse_log(LOG)
params = submit_scrobbles.batch_params(played)
assert "mbid[0]" in params # Kickstart My Heart has one
assert "mbid[1]" not in params # Watercolour does not
assert params["trackNumber[0]"] == "2"
def test_a_track_number_of_minus_one_is_not_sent():
"""Rockbox writes -1 when it does not know, which is not a track number."""
played, _, _ = submit_scrobbles.parse_log(
"Band\tAlbum\tTrack\t-1\t100\tL\t1700000000\t\n"
)
assert submit_scrobbles.batch_params(played).get("trackNumber[0]") is None
def test_the_signature_sorts_names_by_ascii_not_by_number():
"""Last.fm sorts parameter names as strings, so artist[10] precedes
artist[1]. Sorting numerically produces an invalid signature and nothing
else."""
params = {"artist[1]": "b", "artist[10]": "a", "api_key": "k"}
expected = submit_scrobbles.hashlib.md5(
("api_keyk" + "artist[1]b" + "artist[10]a" + "s").encode()
).hexdigest()
assert submit_scrobbles.sign(params, "s") != expected
correct = submit_scrobbles.hashlib.md5(
("api_keyk" + "artist[10]a" + "artist[1]b" + "s").encode()
).hexdigest()
assert submit_scrobbles.sign(params, "s") == correct
def fake_transport(responses):
"""Return a transport serving canned responses and recording requests."""
sent = []
def transport(request, timeout=None):
sent.append(dict(submit_scrobbles.urllib.parse.parse_qsl(request.data.decode())))
return json.dumps(responses[len(sent) - 1])
transport.sent = sent
return transport
def test_scrobbles_are_sent_in_batches_of_fifty():
entries = [
{"artist": "A", "track": f"T{i}", "timestamp": str(1700000000 + i)}
for i in range(120)
]
accepted = {"scrobbles": {"@attr": {"accepted": 50, "ignored": 0}}}
transport = fake_transport([accepted, accepted, accepted])
submit_scrobbles.submit(entries, "k", "s", "sk", transport, delay=0)
assert len(transport.sent) == 3
assert transport.sent[0]["method"] == "track.scrobble"
assert "artist[49]" in transport.sent[0]
assert "artist[50]" not in transport.sent[0]
def test_every_request_carries_a_signature_and_session():
entries = [{"artist": "A", "track": "T", "timestamp": "1700000000"}]
transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
submit_scrobbles.submit(entries, "k", "s", "session-key", transport, delay=0)
assert transport.sent[0]["sk"] == "session-key"
assert len(transport.sent[0]["api_sig"]) == 32
def test_a_service_error_is_raised_not_swallowed():
entries = [{"artist": "A", "track": "T", "timestamp": "1700000000"}]
transport = fake_transport([{"error": 9, "message": "Invalid session key"}])
with pytest.raises(submit_scrobbles.LastfmError, match="error 9"):
submit_scrobbles.submit(entries, "k", "s", "sk", transport, delay=0)
def test_the_log_is_set_aside_after_a_successful_submission(tmp_path, monkeypatch):
device = tmp_path / "IPOD"
device.mkdir()
(device / ".scrobbler.log").write_text(LOG, encoding="utf-8")
transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 2, "ignored": 0}}}])
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
submit_scrobbles.main(
[str(device), "--api-key", "k", "--api-secret", "s"], transport=transport
)
assert not (device / ".scrobbler.log").exists()
# Renamed, not deleted: if Last.fm quietly dropped one, the evidence remains.
assert list(device.glob(".scrobbler.log.*.submitted"))
def test_a_failed_submission_leaves_the_log_alone(tmp_path, monkeypatch):
device = tmp_path / "IPOD"
device.mkdir()
(device / ".scrobbler.log").write_text(LOG, encoding="utf-8")
transport = fake_transport([{"error": 29, "message": "Rate limit"}])
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
code = submit_scrobbles.main(
[str(device), "--api-key", "k", "--api-secret", "s"], transport=transport
)
assert code == 1
assert (device / ".scrobbler.log").is_file()
def test_a_dry_run_submits_nothing(tmp_path):
device = tmp_path / "IPOD"
device.mkdir()
(device / ".scrobbler.log").write_text(LOG, encoding="utf-8")
transport = fake_transport([])
submit_scrobbles.main([str(device), "--dry-run"], transport=transport)
assert transport.sent == []
assert (device / ".scrobbler.log").is_file()
def test_no_log_is_not_an_error(tmp_path):
device = tmp_path / "IPOD"
device.mkdir()
assert submit_scrobbles.main([str(device)], transport=fake_transport([])) == 0
def test_write_credentials_are_required(tmp_path, capsys, monkeypatch):
"""The read-only key used elsewhere is not enough for a write method."""
monkeypatch.delenv("LASTFM_API_KEY", raising=False)
monkeypatch.delenv("LASTFM_API_SECRET", raising=False)
device = tmp_path / "IPOD"
device.mkdir()
(device / ".scrobbler.log").write_text(LOG, encoding="utf-8")
code = submit_scrobbles.main([str(device)], transport=fake_transport([]))
assert code == 2
assert "LASTFM_API_SECRET" in capsys.readouterr().err
+255
View File
@@ -0,0 +1,255 @@
#!/usr/bin/env python3
"""Submit a Rockbox scrobbler log to Last.fm, then set it aside.
Rockbox writes /.scrobbler.log on the device in AUDIOSCROBBLER 1.1 format: one
tab-separated line per track, rated `L` for listened or `S` for skipped. Only
the listened ones are submitted; a skip is not a play.
Scrobbling is a write method, so unlike everything else here it needs the API
secret and a session key, obtained once through the browser. Read-only calls
elsewhere in these projects need neither.
"""
import argparse
import hashlib
import json
import os
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
from pathlib import Path
API_ROOT = "https://ws.audioscrobbler.com/2.0/"
# Last.fm's documented ceiling for one track.scrobble call.
BATCH = 50
# Rockbox names the log for whether the target has a real-time clock. Without
# one every timestamp it writes is zero, which is not a time anything can
# scrobble.
LOG_NAMES = (".scrobbler.log", ".scrobbler-timeless.log")
SESSION_FILE = Path(
os.getenv("XDG_CONFIG_HOME", Path.home() / ".config")
) / "music-mirror" / "lastfm.json"
class LastfmError(Exception):
"""A Last.fm request that failed."""
def parse_log(text):
"""Return the listened tracks in an AUDIOSCROBBLER log, oldest first.
Fields are artist, album, title, track number, length, rating, timestamp
and MusicBrainz id. Rockbox converts any tab inside a field to a space
before writing, so splitting on tabs is safe.
"""
played, skipped, timeless = [], 0, 0
for line in text.splitlines():
if not line or line.startswith("#"):
continue
fields = line.split("\t")
if len(fields) < 7:
continue
artist, album, title, number, length, rating, timestamp = fields[:7]
mbid = fields[7] if len(fields) > 7 else ""
if rating.strip().upper() != "L":
skipped += 1
continue
try:
when = int(timestamp)
except ValueError:
continue
if when <= 0:
timeless += 1
continue
if not artist or not title:
continue
played.append(
{
"artist": artist,
"album": album,
"track": title,
"trackNumber": number if number not in ("", "-1") else "",
"duration": length if length.isdigit() and int(length) > 0 else "",
"timestamp": str(when),
"mbid": mbid,
}
)
played.sort(key=lambda entry: int(entry["timestamp"]))
return played, skipped, timeless
def sign(params, secret):
"""Return Last.fm's method signature for a set of parameters.
Names are sorted by the ASCII table rather than numerically, which is why
`artist[10]` comes before `artist[1]`. Getting that wrong produces an
invalid signature and nothing else.
"""
joined = "".join(f"{name}{params[name]}" for name in sorted(params))
return hashlib.md5((joined + secret).encode("utf-8")).hexdigest() # noqa: S324
def post(params, transport):
"""Sign, post, and return the decoded response."""
body = urllib.parse.urlencode(params).encode("utf-8")
request = urllib.request.Request(API_ROOT, data=body)
try:
payload = json.loads(transport(request))
except urllib.error.HTTPError as error:
detail = error.read().decode("utf-8", "replace")[:300]
raise LastfmError(f"HTTP {error.code}: {detail}") from error
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as error:
raise LastfmError(str(error)) from error
if payload.get("error"):
raise LastfmError(f"error {payload['error']}: {payload.get('message', '')}")
return payload
def call(method, params, key, secret, session, transport):
"""Make one signed, authenticated call."""
full = {**params, "method": method, "api_key": key}
if session:
full["sk"] = session
full["api_sig"] = sign(full, secret)
full["format"] = "json"
return post(full, transport)
def authorise(key, secret, transport, opener=print):
"""Walk the one-time browser authorisation and return a session key."""
token = call("auth.getToken", {}, key, secret, None, transport)["token"]
url = f"https://www.last.fm/api/auth/?api_key={key}&token={token}"
opener(f"Open this, approve the application, then press Enter:\n\n {url}\n")
input()
session = call("auth.getSession", {"token": token}, key, secret, None, transport)
return session["session"]["key"]
def load_session():
if SESSION_FILE.is_file():
return json.loads(SESSION_FILE.read_text(encoding="utf-8")).get("session")
return None
def save_session(session):
SESSION_FILE.parent.mkdir(parents=True, exist_ok=True)
SESSION_FILE.write_text(json.dumps({"session": session}), encoding="utf-8")
SESSION_FILE.chmod(0o600)
def batch_params(entries):
"""Return the indexed parameters for one track.scrobble call."""
params = {}
for index, entry in enumerate(entries):
for name in ("artist", "track", "timestamp", "album", "trackNumber", "duration", "mbid"):
if entry.get(name):
params[f"{name}[{index}]"] = entry[name]
return params
def submit(entries, key, secret, session, transport, delay=1.0):
"""Submit every entry. Returns how many the service accepted.
Batches are counted as they succeed rather than at the end, so a failure
partway through leaves an honest number and the caller can keep the rest of
the log instead of losing it.
"""
accepted = 0
for start in range(0, len(entries), BATCH):
chunk = entries[start : start + BATCH]
payload = call(
"track.scrobble", batch_params(chunk), key, secret, session, transport
)
block = payload.get("scrobbles", {})
summary = block.get("@attr", block)
accepted += int(summary.get("accepted", len(chunk)))
ignored = int(summary.get("ignored", 0))
if ignored:
print(f" {ignored} of {len(chunk)} ignored by Last.fm", file=sys.stderr)
if start + BATCH < len(entries):
time.sleep(delay)
return accepted
def http_post(request, timeout=30):
with urllib.request.urlopen(request, timeout=timeout) as response: # noqa: S310
return response.read().decode("utf-8")
def find_log(device):
"""Return the scrobbler log on a mounted device, or None."""
for name in LOG_NAMES:
candidate = Path(device) / name
if candidate.is_file() and candidate.stat().st_size:
return candidate
return None
def main(argv=None, transport=http_post):
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("device", help="the mounted device, or a scrobbler log file")
parser.add_argument("--api-key", default=os.getenv("LASTFM_API_KEY"))
parser.add_argument("--api-secret", default=os.getenv("LASTFM_API_SECRET"))
parser.add_argument("--dry-run", action="store_true", help="parse and report only")
parser.add_argument(
"--keep", action="store_true", help="do not set the log aside afterwards"
)
args = parser.parse_args(argv)
target = Path(args.device)
log = target if target.is_file() else find_log(target)
if log is None:
print("no scrobbler log to submit", file=sys.stderr)
return 0
played, skipped, timeless = parse_log(log.read_text(encoding="utf-8", errors="replace"))
print(f"{log}: {len(played)} listened, {skipped} skipped", file=sys.stderr)
if timeless:
print(
f" {timeless} entries have no timestamp, so this target has no clock."
" They cannot be scrobbled without inventing when they happened.",
file=sys.stderr,
)
if not played:
return 0
if args.dry_run:
for entry in played[:20]:
print(f"{entry['timestamp']}\t{entry['artist']}\t{entry['track']}")
return 0
if not args.api_key or not args.api_secret:
print(
"scrobbling is a write method: it needs LASTFM_API_KEY and"
" LASTFM_API_SECRET, not just the read-only key",
file=sys.stderr,
)
return 2
session = load_session()
if not session:
session = authorise(args.api_key, args.api_secret, transport)
save_session(session)
try:
accepted = submit(played, args.api_key, args.api_secret, session, transport)
except LastfmError as error:
print(f"submission failed: {error}", file=sys.stderr)
return 1
print(f"{accepted} scrobbles accepted", file=sys.stderr)
if not args.keep and accepted:
# Renamed rather than deleted: if Last.fm quietly dropped something,
# the evidence is still on the device.
aside = log.with_name(f"{log.name}.{played[-1]['timestamp']}.submitted")
log.rename(aside)
print(f"log moved to {aside.name}", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main())
+136
View File
@@ -0,0 +1,136 @@
#!/usr/bin/env bash
# Copy the mirror onto a Rockbox device, then unmount it cleanly.
#
# The device is FAT32 with no journal, reached through the Apple firmware's
# disk mode because Rockbox's own mass storage is unreliable on an iFlash. An
# interrupted write is corruption that needs fsck.vfat from another machine, so
# this syncs and unmounts rather than leaving that to whoever pulls the cable.
#
# rsync --delete is pointed at a whole filesystem, so the checks below are the
# point of the script rather than decoration.
set -euo pipefail
usage() {
cat >&2 <<'USAGE'
usage: sync-to-ipod.sh [options] <mirror> <destination>
-n dry run; show what would change and touch nothing
-f copy even if the FAT32 check finds unacceptable paths
-S skip submitting the Rockbox scrobbler log to Last.fm
-U leave the destination mounted afterwards
Submitting scrobbles needs LASTFM_API_KEY and LASTFM_API_SECRET; it is skipped
with a note when they are unset. Scrobbling is a write method and needs the
secret, unlike the read-only calls elsewhere in these projects.
The destination must be a mounted FAT filesystem. Reach it with the Apple
firmware's disk mode: Menu+Select to reboot, then immediately Select+Play.
USAGE
exit 2
}
dry_run=false
force=false
unmount=true
scrobble=true
while getopts ":nfSUh" option; do
case "$option" in
n) dry_run=true ;;
f) force=true ;;
S) scrobble=false ;;
U) unmount=false ;;
*) usage ;;
esac
done
shift $((OPTIND - 1))
[ $# -eq 2 ] || usage
mirror=${1%/}
destination=${2%/}
here=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
die() {
printf 'sync-to-ipod: %s\n' "$1" >&2
exit 1
}
[ -d "$mirror" ] || die "mirror $mirror is not a directory"
[ -n "$(ls -A "$mirror")" ] || die "mirror $mirror is empty; refusing to mirror nothing"
[ -d "$destination" ] || die "destination $destination is not a directory"
# --delete makes every one of these load-bearing. A destination that is not its
# own mount point means the path is wrong, and emptying the wrong directory is
# not a mistake that announces itself.
case "$destination" in
"" | "/" | "$HOME") die "refusing to sync onto $destination" ;;
esac
[ "$(readlink -f "$mirror")" != "$(readlink -f "$destination")" ] ||
die "mirror and destination are the same directory"
mountpoint -q -- "$destination" || die "$destination is not a mount point"
filesystem=$(findmnt -no FSTYPE --target "$destination")
case "$filesystem" in
vfat | exfat) ;;
*)
$force || die "$destination is $filesystem, not FAT; pass -f if that is deliberate"
printf 'sync-to-ipod: destination is %s, not FAT\n' "$filesystem" >&2
;;
esac
if $force; then
printf 'sync-to-ipod: skipping the FAT32 check\n' >&2
elif ! python3 "$here/check_fat32.py" "$mirror"; then
die "the mirror holds paths FAT32 will not take; run music-mirror with --fat32-safe"
fi
# Before the copy, not after: the plays already happened, and if the transfer
# then fails there is no reason to have lost them too.
if $scrobble; then
if [ -z "${LASTFM_API_KEY:-}" ] || [ -z "${LASTFM_API_SECRET:-}" ]; then
printf 'sync-to-ipod: no Last.fm credentials, skipping the scrobbler log\n' >&2
else
scrobble_options=()
$dry_run && scrobble_options+=(--dry-run)
python3 "$here/submit_scrobbles.py" "${scrobble_options[@]}" "$destination" ||
die "submitting scrobbles failed; nothing has been copied"
fi
fi
# -rt rather than -a: owners, groups and permissions mean nothing on FAT, and
# asking for them produces a screenful of errors and a non-zero exit.
# --modify-window=2 because FAT stores mtimes to two-second resolution, without
# which every file looks changed and the whole library is copied every time.
# --delete removes tracks whose source has gone, which is the point. It would
# also remove everything on the device that the mirror does not contain -- and
# if the destination is the card root that means /.rockbox, the Rockbox install
# itself. Excluded paths are not deleted unless --delete-excluded is given,
# which it never is here.
options=(--recursive --times --delete --modify-window=2 --human-readable --info=progress2)
for owned in "/.rockbox" "/.scrobbler.log" "/.scrobbler.log.*" "/.playlist_control" \
"/System Volume Information" "/.Spotlight-V100" "/.Trashes" "/.fseventsd"; do
options+=(--exclude "$owned")
done
$dry_run && options+=(--dry-run --verbose)
printf 'sync-to-ipod: %s -> %s\n' "$mirror" "$destination" >&2
rsync "${options[@]}" "$mirror/" "$destination/"
if $dry_run; then
printf 'sync-to-ipod: dry run, nothing was written\n' >&2
exit 0
fi
sync
if $unmount; then
device=$(findmnt -no SOURCE --target "$destination")
printf 'sync-to-ipod: unmounting %s\n' "$device" >&2
if command -v udisksctl >/dev/null 2>&1; then
udisksctl unmount -b "$device"
else
umount -- "$destination"
fi
printf 'sync-to-ipod: safe to disconnect\n' >&2
else
printf 'sync-to-ipod: still mounted; unmount before disconnecting\n' >&2
fi