fix: correct Rockbox's local wall-clock timestamps to UTC

Rockbox has no concept of a timezone. Its clock holds local time, and it builds log timestamps with mktime(get_time()) -- but firmware/libc/mktime.c is plain calendar arithmetic applying no offset, so the RTC's local fields come out as though they were UTC. The number in the log is ahead of the real instant by whatever the offset was, and Last.fm stores UTC, so every play submitted during BST landed an hour in the future. Rockbox states this itself: its scrobbler plugin writes #TZ/UNKNOWN, and the AUDIOSCROBBLER spec allows #TZ/UTC only for a device that actually converted. The correction belongs to the consumer. Each timestamp is decoded back to its wall-clock fields and reinterpreted in the player's zone, per play rather than as one offset over the whole log, so a log spanning a daylight saving change converts each side correctly. A log declaring #TZ/UTC is left alone rather than shifted twice. The zone defaults to this machine's, overridable with --device-timezone or ROCKBOX_TIMEZONE. Deriving it needs the whole IANA name: /etc/localtime resolves into the tzdata tree, and taking only the final component yields "London", which no database holds, silently falling back to a fixed offset that is wrong for half the year. Since Rockbox cannot adjust for daylight saving on its own, the player's clock has to be changed by hand twice a year. Any play converting to a future time is now reported, which is what a forgotten adjustment looks like. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fix: never discard a play that has not been submitted
2026-08-27 14:18:52 +01:00 · 2026-08-26 18:32:05 +01:00 · 2026-08-26 18:24:12 +01:00 · 2026-08-26 13:57:49 +01:00 · 2026-08-26 13:30:26 +01:00 · 2026-08-26 12:28:36 +00:00
9 changed files with 1918 additions and 51 deletions
@@ -26,6 +26,9 @@ ENTRYPOINT ["music-mirror"]
 FROM runtime AS test
 RUN pip install --no-cache-dir pytest
 # sync-to-ipod.sh and its tests need these; the runtime image deliberately does
 # not carry them, and neither does the base.
 RUN apk add --no-cache bash rsync findmnt
 COPY pytest.ini ./
 # Host-side tools; not in the runtime image, but the suite covers them.
 COPY tools ./tools
@@ -150,19 +150,154 @@ tools/sync-to-ipod.sh /mnt/tank/media/music-mp3 /media/IPOD/Music
 tools/sync-to-ipod.sh -n /mnt/tank/media/music-mp3 /media/IPOD/Music   # dry run
 ```
-It refuses to start unless the destination is a mounted FAT filesystem that is
+The destination is where the artist folders should end up — normally a
-its own mount point, because `--delete` aimed at the wrong directory empties it
+subdirectory such as `/media/IPOD/Music`, not the card root. A subdirectory is
-and does not announce itself. It also excludes `/.rockbox`, the scrobbler logs
+the better target: `--delete` is confined to it, and the device path budget is
 derived from it rather than configured, so the two cannot disagree.
 It refuses to start unless the destination is on a mounted FAT filesystem. That
 check is also what catches an unmounted device — `/media/IPOD/Music` then
 resolves to the host's own root filesystem, and this refuses to empty that.
 `--help` says all of it. It also excludes `/.rockbox`, the scrobbler logs
 and the various filesystem metadata directories from deletion — the mirror does
 not contain them, and without the exclusion a sync to the card root would
 remove the Rockbox install.
 Progress is a single line that rewrites itself:
 ```
 [ 24%] 12,345/49,600  3.2 GiB/13.1 GiB  4.4 MiB/s  ETA 38m12s  King Gizzard / Petro…
 ```
 The estimate comes from rsync's `%l`, which gives each file's size as it
 completes. Bytes done over time elapsed is the same arithmetic rsync would do,
 and needs nothing it does not already print. The rate is measured over a
 trailing thirty seconds rather than the whole run, so it follows a device that
 slows down instead of averaging the slowdown away — and it is suppressed
 entirely for the first two seconds, where the window is microseconds wide and
 would report gigabytes per second.
 rsync says nothing at all while it builds its file list, which on fifty
 thousand files is minutes of apparent hang, and its own `progress2` percentage
 is computed against a list it has not finished discovering. So the script
 renders its own.
 **The percentage and the estimate are opt-in, via `-P`.** They need a total,
 the total needs a counting pass, and that pass walks and compares both trees in
 full exactly as the transfer does. Measured on a real card: read from the
 source at 35 MB/s and write to the card at 21 MB/s, yet the sync crawled —
 because the traversal, not the data, was the cost, and it was being paid twice.
 Without `-P` the line still shows the running count, the rate and the album in
 flight; only the two figures that needed the second walk are missing.
 Piped to a log it prints a plain line every thirty seconds instead, with no
 carriage returns, and a summary at the end either way.
 ### The Rockbox database
 Point `MUSIC_MIRROR_DATABASE_TOOL` at Rockbox's host-side builder and the sync
 rebuilds the database itself, so it never has to happen on the device.
 ```sh
 git clone --depth 1 https://github.com/Rockbox/rockbox.git
 cd rockbox && mkdir build-db && cd build-db
 ../tools/configure --target=ipodvideo --type=d && make -j$(nproc)
 ```
 It needs a native compiler and SDL2 development headers, not the ARM
 cross-toolchain, and `tools/configure` detects `__aarch64__` correctly. On a
 distribution without `/usr/bin/perl` or `gcc-ar` — NixOS, say — patch the
 shebangs in `tools/*.pl` and pass `AR=ar`.
 Building it here rather than on the device is not merely faster. The on-device
 commit sorts the whole index in whatever memory `core_alloc_maximum()` can
 scrape together; on a fifty-thousand-track library it runs for hours or aborts
 outright.
 The scan runs against a scratch root — a real `.rockbox` beside a symlink
 standing in for wherever the music lands on the device — so the paths recorded
 are the ones Rockbox will look up, while the **bytes are read from the mirror
 rather than over USB**. Only the dozen `.tcd` files cross to the card.
 Cost, measured: the parser makes about 49 reads and 43 seeks per file, probing
 the head for ID3v2 and the tail for ID3v1. On a local disk that is 2,000 files
 in half a second. Over SMB, readahead absorbs most of the reads but the opens
 and the head/tail split are real round trips, so a first full scan is minutes
 rather than seconds. It is a one-time cost: the builder is incremental, and the
 scratch root is kept between runs, so a later pass over unchanged files does no
 metadata reads at all.
 If minutes is still too many, run the builder where the mirror is local — on
 the NAS — and copy the `.tcd` files across. There is nothing to parallelise:
 the tool is single-threaded, and two instances cannot produce one database.
 ### If the sync is interrupted
 No partially copied track is ever left under a name Rockbox would play. rsync
 writes to a hidden temporary file and only renames it into place once the file
 is complete, and *"by default, rsync will delete any partially transferred file
 if the transfer is interrupted"*. `--partial` is deliberately not used, and
 there is a test asserting it never will be.
 After an unclean kill or a power cut a hidden `.track.mp3.XXXXXX` can survive.
 It is not playable, it is not in the source, and the next run's `--delete`
 removes it.
 Interrupting does not, by itself, endanger the filesystem. The kernel flushes
 dirty pages within `dirty_expire_centisecs` — thirty seconds by default — and
 `umount` always syncs before it returns. Losing data needs you to interrupt,
 *and* pull the card inside that window, *and* skip the unmount.
 The script still traps `INT` and `TERM` and flushes and unmounts on the way
 out, exiting 130. Not because a Ctrl-C is dangerous, but because it removes the
 manual step and makes the exit deterministic — you get the same "safe to
 disconnect" either way, rather than having to remember which path you took.
 Both paths call the same function, so they cannot drift apart.
 What genuinely does lose data is pulling the cable or the card without
 unmounting at all, interrupted or not. FAT32 has no journal. Wait for the
 unmount line.
 ### Making it faster over a network mount
 The transfer is metadata-bound, not throughput-bound: 49,600 files means 49,600
 round trips, and the counting pass doubles that. In rough order of what it is
 worth doing:
 | Lever | Why |
 | ----- | --- |
 | Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. |
 | Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. |
 | Counting is off by default | The percentage costs a second full traversal of both trees. On a FAT card of fifty thousand files that is slower than the transfer. `-P` asks for it. |
 | `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. |
 **NFS instead of SMB** is worth trying but is not the big win it looks like.
 Its attribute caching defaults are far more generous than SMB's — `acregmax` of
 sixty seconds against `actimeo=1` — which is precisely the gap that
 `actimeo=60` closes on the mount you already have. Bulk read throughput between
 the two is much of a muchness on a gigabit link. Try the mount option first; it
 is one line and needs no change on the NAS.
 And if the destination is the iPod rather than a card reader, none of this
 matters much: the source can feed data faster than USB 2.0 through an iPod will
 take it either way.
 The unmount is the point of doing this in a script. FAT32 has no journal and
 the device is reached through disk mode, so an interrupted write is corruption
 that needs `fsck.vfat` from another machine.
-`submit_scrobbles.py` sends the Rockbox scrobbler log to Last.fm and sets it
+`submit_scrobbles.py` sends what was played to Last.fm and sets the logs aside.
-aside. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
+
 It reads **Rockbox's own `playback.log`**, which core Rockbox writes whenever
 "play log" is enabled, with no plugin running. Each line is
 `timestamp:elapsed_ms:length_ms:path` — a path and nothing else, which is why
 the on-device scrobbler plugin exists at all: reading tags back off the player
 is slow. Off the mirror it is free, so `--mirror` lets the conversion happen
 here and the plugin never has to be run. A play counts as listened at half the
 track's length, the same fraction the plugin uses, so the two cannot disagree
 about what a play was.
 It still reads a `.scrobbler.log` if the plugin has been run and left one. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
 tab-separated line per track rated `L` for listened or `S` for skipped; only
 the listened ones are sent. It runs **before** the copy, since the plays
 already happened and a failed transfer is no reason to lose them.
@@ -174,8 +309,70 @@ real-time clock Rockbox writes `/.scrobbler-timeless.log` with every timestamp
 set to zero; those are counted and reported but never submitted, because
 scrobbling them would mean inventing when they happened.
-The log is renamed rather than deleted once accepted. If Last.fm quietly
+### Timestamps are local wall clock, and are corrected here
-dropped something, the evidence is still on the device.
+
 Rockbox has no concept of a timezone. Its clock is set to local time, and it
 builds log timestamps with `mktime(get_time())` — but
 [its `mktime`](https://git.rockbox.org/cgit/rockbox.git/tree/firmware/libc/mktime.c)
 is plain calendar arithmetic that applies no offset, so the RTC's local fields
 come out as if they were UTC. The number in the log is therefore ahead of the
 real instant by whatever the offset was. Last.fm stores UTC, so submitting it
 raw puts every play an hour into the future for the half of the year the UK is
 on BST.
 Rockbox is candid about this: its scrobbler plugin writes `#TZ/UNKNOWN` in the
 log header, and the AUDIOSCROBBLER spec says a device may claim `#TZ/UTC` only
 if it actually converted. The correction is the consumer's job.
 Each timestamp is decoded back into the wall-clock fields it came from and
 reinterpreted in the player's zone. Doing it **per play** rather than applying
 one offset to the whole log matters: a week's listening can straddle a daylight
 saving change, and the two sides need different offsets. A log that declares
 `#TZ/UTC` is left alone, so a client that already converted is not shifted
 twice.
 The zone defaults to this machine's. Set `ROCKBOX_TIMEZONE` (or pass
 `--device-timezone`) to an IANA name if the player's clock is keeping a
 different one.
 Because Rockbox cannot adjust for daylight saving itself, **you have to change
 the player's clock by hand twice a year**. If you forget, its times are an hour
 out and no amount of zone arithmetic recovers them. The submitter reports any
 play that converts to a time in the future, which is what a forgotten
 adjustment looks like:
 ```
  37 plays are timestamped up to 58 minutes in the future, converting from
  Europe/London. Either the player's clock is wrong or that is not the zone
  it is set to.
 ```
 `--dry-run` prints each play's local time beside the epoch, so the conversion
 can be checked against when you actually remember listening.
 ### Nothing played is thrown away
 Two separate obligations, because a play that happened and never reached
 Last.fm is gone for good.
 **The original is renamed, never deleted.** If Last.fm quietly dropped
 something, the evidence is still on the device as `playback.log.<ts>.submitted`.
 **Anything not submitted is written back** into a live log for the next run:
 | Outcome | What happens to it |
 | ------------------------------ | ----------------------------------------- |
 | Accepted by Last.fm | dropped from the live log |
 | Not in the mirror yet | written back, tried again next run |
 | In a batch that failed | written back, tried again next run |
 | Nothing accepted at all | logs left completely untouched |
 | A skip, or no usable timestamp | not retained — neither can ever be submitted, and the original still has it |
 The batch boundary matters: submission is recorded as each batch is accepted,
 so a failure partway through knows exactly what got through and writes back
 only the remainder. No duplicates, no losses.
 A file `ffprobe` cannot read costs one unidentified play, not the run.
 `check_fat32.py` reports paths a FAT32 device will not accept — reserved
 characters, trailing dots and spaces, over-long components and paths, and names
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "music-mirror"
-version = "0.2.1"
+version = "0.4.0"
 description = "Maintain a lossy MP3 mirror of a lossless music library"
 readme = "README.md"
 requires-python = ">=3.11"
@@ -0,0 +1,180 @@
 import io
 import sys
 from pathlib import Path
 import pytest
 sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
 import rsync_progress  # noqa: E402
 class NotATerminal(io.StringIO):
    def isatty(self):
        return False
 class Terminal(io.StringIO):
    def isatty(self):
        return True
 def run(lines, total=0, out=None, bytes_expected=0):
    out = out or NotATerminal()
    rsync_progress.main(
        ["--total", str(total), "--bytes", str(bytes_expected)],
        stream=io.StringIO(lines),
        out=out,
    )
    return out.getvalue()
 def test_the_artist_and_album_are_pulled_from_the_path():
    assert rsync_progress.album_of("Pendulum/Immersion/01 - Watercolour.mp3") == (
        "Pendulum / Immersion"
    )
 def test_a_shallower_path_degrades_rather_than_failing():
    assert rsync_progress.album_of("Pendulum/loose.mp3") == "Pendulum"
    assert rsync_progress.album_of("loose.mp3") == ""
 def test_directories_are_not_counted():
    """rsync reports them too, and counting them puts the percentage past 100."""
    output = run("Artist/\nArtist/Album/\nArtist/Album/track.mp3\n", total=1)
    assert "1/1" in output
    assert "100%" in output
 def test_the_percentage_tracks_the_total():
    output = run("".join(f"A/B/{i}.mp3\n" for i in range(5)), total=10)
    assert "5/10" in output
    assert "50%" in output
 def test_without_a_total_it_counts_instead_of_guessing():
    output = run("A/B/one.mp3\nA/B/two.mp3\n")
    assert "2 files" in output
    assert "%" not in output
 def test_a_final_line_is_always_printed():
    """Otherwise the last state of a rewriting line is whatever it happened to
    be when the interval last elapsed."""
    output = run("A/B/one.mp3\n", total=1)
    assert output.endswith("\n")
    assert "1/1" in output
 def test_nothing_transferred_still_reports():
    output = run("", total=0)
    assert "0 files" in output
 def test_a_log_gets_no_carriage_returns():
    """A non-terminal filling with \\r and escape codes is unreadable."""
    output = run("".join(f"A/B/{i}.mp3\n" for i in range(50)), total=50)
    assert "\r" not in output
    assert "\033" not in output
 def test_a_terminal_rewrites_one_line():
    output = run("".join(f"A/B/{i}.mp3\n" for i in range(50)), total=50, out=Terminal())
    assert "\r\033[2K" in output
@pytest.mark.parametrize(
    ("text", "width", "expected"),
    [
        ("short", 20, "short"),
        ("King Gizzard / PetroDragonic Apocalypse", 20, "…agonic Apocalypse"),
    ],
 )
 def test_long_labels_are_trimmed_from_the_left(text, width, expected):
    """The album is the informative end, so the artist is what gets cut."""
    trimmed = rsync_progress.fit(text, width)
    assert len(trimmed) <= width
    if len(text) > width:
        assert trimmed.startswith("…")
        assert text.endswith(trimmed.lstrip("…"))
 def test_the_size_and_path_are_parsed():
    assert rsync_progress.parse("5000 Artist/Album/Track.mp3\n") == (
        5000,
        "Artist/Album/Track.mp3",
    )
 def test_a_filename_containing_spaces_survives():
    """Splitting on every space would lose most of the library."""
    assert rsync_progress.parse("1234 Artist/An Album/A Track With Spaces.mp3") == (
        1234,
        "Artist/An Album/A Track With Spaces.mp3",
    )
 def test_a_bare_path_is_tolerated():
    """In case this is fed --out-format='%n' by something older."""
    assert rsync_progress.parse("Artist/Album/Track.mp3") == (0, "Artist/Album/Track.mp3")
 def test_directory_sizes_do_not_inflate_the_total():
    """rsync reports directories with a 4096 inode size, which is several
    megabytes of nothing across six thousand albums."""
    output = run("4096 Artist/\n4096 Artist/Album/\n5000 Artist/Album/t.mp3\n", total=1)
    assert "4.9 KiB" in output
    assert "12" not in output.split("Artist")[0]
 def test_a_rate_is_not_reported_until_it_means_something():
    """The first files arrive microseconds apart and would give a rate in the
    gigabytes per second and an ETA of zero."""
    rate = rsync_progress.Rate()
    rate.add(100.0, 0)
    rate.add(100.5, 5_000_000)
    assert rate.per_second() == 0.0
 def test_a_rate_over_a_long_enough_window_is_reported():
    rate = rsync_progress.Rate()
    rate.add(100.0, 0)
    rate.add(110.0, 10_000_000)
    assert rate.per_second() == pytest.approx(1_000_000)
 def test_the_window_forgets_the_distant_past():
    """So the estimate follows a device that slows down rather than averaging
    the slowdown away."""
    rate = rsync_progress.Rate(window=30.0)
    for second in range(0, 100, 10):
        rate.add(float(second), second * 1_000_000)
    rate.add(200.0, 100_000_000)
    assert rate.samples[0][0] >= 90.0
@pytest.mark.parametrize(
    ("seconds", "expected"),
    [(0, "0s"), (45, "45s"), (60, "1m00s"), (1092, "18m12s"), (7500, "2h05m")],
 )
 def test_durations_read_without_arithmetic(seconds, expected):
    assert rsync_progress.human_duration(seconds) == expected
 def test_a_summary_is_printed_at_the_end():
    output = run("5000000 A/B/one.mp3\n", total=1, bytes_expected=5000000)
    assert "copied 4.8 MiB in" in output
@@ -1,6 +1,9 @@
 import json
 import subprocess
 import sys
 from datetime import datetime, timezone
 from pathlib import Path
 from zoneinfo import ZoneInfo
 import pytest
@@ -187,3 +190,374 @@ def test_write_credentials_are_required(tmp_path, capsys, monkeypatch):
    assert code == 2
    assert "LASTFM_API_SECRET" in capsys.readouterr().err
 PLAYBACK_LOG = """1700000300:180000:245000:/Music/Pendulum/Immersion/01.mp3
 1700000200:9000:180000:/Music/Green Day/Dookie/07.mp3
 0:180000:245000:/Music/No/Clock/track.mp3
 malformed line without colons
 """
 def tags_of(artist="Pendulum", title="Watercolour", album="Immersion", track="1/11"):
    def runner(path):
        return json.dumps(
            {"format": {"tags": {"ARTIST": artist, "TITLE": title,
                                 "ALBUM": album, "track": track}}}
        )
    return runner
 def test_the_playback_log_format_is_four_fields():
    """timestamp:elapsed_ms:length_ms:path, written by Rockbox core."""
    plays = submit_scrobbles.parse_playback_log(PLAYBACK_LOG)
    assert len(plays) == 3  # the malformed line is dropped
    assert plays[0] == (1700000300, 180000, 245000, "/Music/Pendulum/Immersion/01.mp3")
 def test_a_device_path_maps_onto_the_mirror():
    assert submit_scrobbles.device_to_local(
        "/Music/Pendulum/Immersion/01.mp3", "/Music", "/mnt/mirror"
    ) == Path("/mnt/mirror/Pendulum/Immersion/01.mp3")
 def test_a_path_outside_the_prefix_is_not_mapped():
    """Something played from elsewhere on the card is not in the mirror."""
    assert submit_scrobbles.device_to_local(
        "/Podcasts/episode.mp3", "/Music", "/mnt/mirror"
    ) is None
 def test_a_path_at_the_card_root_maps_straight_across():
    assert submit_scrobbles.device_to_local(
        "/Pendulum/Immersion/01.mp3", "/", "/mnt/mirror"
    ) == Path("/mnt/mirror/Pendulum/Immersion/01.mp3")
 def test_a_short_play_is_a_skip_not_a_scrobble(tmp_path, monkeypatch):
    """Nine seconds of a three minute track. The on-device plugin uses the same
    fraction, so the two never disagree about what counted as a play."""
    monkeypatch.setattr(Path, "is_file", lambda self: True)
    result = submit_scrobbles.plays_from_playback_log(
        PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
    )
    assert result.skipped == 1
    assert [entry["timestamp"] for entry in result.played] == ["1700000300"]
 def test_a_zero_timestamp_is_refused(tmp_path, monkeypatch):
    """Without a real-time clock Rockbox logs ticks, not dates. Scrobbling
    those would mean inventing when they happened."""
    monkeypatch.setattr(Path, "is_file", lambda self: True)
    result = submit_scrobbles.plays_from_playback_log(
        PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
    )
    assert result.timeless == 1
 def test_tags_come_from_the_mirror(tmp_path, monkeypatch):
    """The log carries only a path -- which is precisely why the on-device
    plugin exists. Off the mirror the tags are free."""
    monkeypatch.setattr(Path, "is_file", lambda self: True)
    played = submit_scrobbles.plays_from_playback_log(
        PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
    ).played
    assert played[0]["artist"] == "Pendulum"
    assert played[0]["album"] == "Immersion"
    assert played[0]["trackNumber"] == "1"       # "1/11" -> "1"
    assert played[0]["duration"] == "245"        # milliseconds -> seconds
 def test_a_track_missing_from_the_mirror_is_counted_not_guessed(monkeypatch):
    monkeypatch.setattr(Path, "is_file", lambda self: False)
    result = submit_scrobbles.plays_from_playback_log(
        PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
    )
    assert result.played == []
    # One: the skip and the clockless entry are filtered before the file is
    # looked for, since neither would be submitted either way.
    assert result.unresolved == 1
    # And that one play is kept, so a later run can try it again.
    assert len(result.retain) == 1
 LONDON = ZoneInfo("Europe/London")
 # Rockbox builds timestamps with mktime(get_time()), and its mktime applies no
 # zone at all, so the number is the RTC's local wall clock read as if it were
 # UTC. Under BST that puts every play an hour ahead of when it happened.
 SUMMER_LOGGED = 1787839200   # 2026-08-27 14:00 written by the device
 SUMMER_TRUE = 1787835600     # the instant that actually was, 13:00 UTC
 WINTER_LOGGED = 1796479200   # 2026-12-05 14:00, when London is already UTC
 WINTER_TRUE = 1796479200
 def test_a_summer_timestamp_is_pulled_back_to_real_utc():
    """The device logged 14:00 local. That was 13:00 UTC, and UTC is what
    Last.fm stores."""
    assert submit_scrobbles.device_time_to_utc(SUMMER_LOGGED, LONDON) == SUMMER_TRUE
 def test_a_winter_timestamp_is_left_alone():
    """London keeps UTC for half the year, so there is nothing to correct and
    the correction must not invent an offset anyway."""
    assert submit_scrobbles.device_time_to_utc(WINTER_LOGGED, LONDON) == WINTER_TRUE
 def test_the_converted_time_reads_back_as_the_clock_the_device_showed():
    """The round trip, which is the property that actually matters: whatever
    the player's screen said is what Last.fm should show in local time."""
    shown = datetime.fromtimestamp(SUMMER_LOGGED, timezone.utc)
    corrected = submit_scrobbles.device_time_to_utc(SUMMER_LOGGED, LONDON)
    assert datetime.fromtimestamp(corrected, LONDON).strftime("%Y-%m-%d %H:%M") == (
        shown.strftime("%Y-%m-%d %H:%M")
    )
 def test_one_log_spanning_a_clock_change_converts_each_side_separately():
    """A week's listening either side of the October change carries two
    different offsets. Correcting the log by a single figure would put half of
    it an hour out, which is why the offset is resolved per play."""
    before, after = 1792888200, 1792899000  # 2026-10-25, 00:30 BST and 03:30 GMT
    assert submit_scrobbles.device_time_to_utc(before, LONDON) == before - 3600
    assert submit_scrobbles.device_time_to_utc(after, LONDON) == after
 def test_a_playback_log_is_corrected_before_submission(monkeypatch):
    monkeypatch.setattr(Path, "is_file", lambda self: True)
    log = f"{SUMMER_LOGGED}:180000:245000:/Music/Pendulum/Immersion/01.mp3\n"
    played = submit_scrobbles.plays_from_playback_log(
        log, "/Music", "/mnt/mirror", runner=tags_of(), zone=LONDON
    ).played
    assert played[0]["timestamp"] == str(SUMMER_TRUE)
 def test_a_scrobbler_log_claiming_utc_is_not_corrected_twice():
    """The format's header exists for exactly this. A client that already did
    the conversion says so, and correcting it again would break it."""
    log = LOG.replace("#TZ/UNKNOWN", "#TZ/UTC")
    played, _, _ = submit_scrobbles.parse_log(log, LONDON)
    assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"]
 def test_a_scrobbler_log_declaring_unknown_is_corrected():
    """Rockbox writes UNKNOWN, meaning local wall clock, so the times are ours
    to fix."""
    played, _, _ = submit_scrobbles.parse_log(LOG, LONDON)
    # 1700000100 and 1700000300 are November, when London is on UTC anyway;
    # the point is that the header did not exempt them from being looked at.
    assert submit_scrobbles.declares_utc(LOG) is False
    assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"]
 def test_the_system_zone_keeps_its_whole_name(monkeypatch, tmp_path):
    """/etc/localtime points into the tzdata tree. Taking only the last
    component gives "London", which no database holds -- and the fallback for
    an unknown name is a fixed offset, which is wrong for half the year. The
    region has to survive."""
    zoneinfo_dir = tmp_path / "share" / "zoneinfo" / "Europe"
    zoneinfo_dir.mkdir(parents=True)
    (zoneinfo_dir / "London").write_bytes(b"TZif")
    monkeypatch.setattr(
        submit_scrobbles.Path,
        "read_text",
        lambda self, **kw: (_ for _ in ()).throw(OSError),
    )
    monkeypatch.setattr(
        submit_scrobbles.Path,
        "resolve",
        lambda self: zoneinfo_dir / "London",
    )
    assert submit_scrobbles.system_zone_name() == "Europe/London"
 def test_a_named_zone_beats_the_machine_default():
    """The player may not be in the same place as the laptop."""
    assert submit_scrobbles.device_zone("Asia/Tokyo") == ZoneInfo("Asia/Tokyo")
 def test_plays_dated_after_now_are_reported():
    """A clock never put forward, or the wrong zone, produces plays that have
    not happened yet. Last.fm cannot tell those from real ones."""
    now = 1787835600
    played = [
        {"timestamp": str(now - 60)},
        {"timestamp": str(now + 3600)},
    ]
    ahead = submit_scrobbles.future_plays(played, now=now)
    assert [entry["timestamp"] for entry in ahead] == [str(now + 3600)]
 def test_a_clock_a_minute_fast_is_not_reported():
    """Devices drift. Only an offset large enough to be a zone error matters."""
    now = 1787835600
    assert submit_scrobbles.future_plays([{"timestamp": str(now + 60)}], now=now) == []
 def test_playback_logs_are_found_including_rotations(tmp_path):
    """Rockbox rotates the log once it passes half a megabyte."""
    rockbox = tmp_path / ".rockbox"
    rockbox.mkdir()
    for name in ("playback.log", "playback_0001.log", "playback_0002.log"):
        (rockbox / name).write_text("1700000000:1:1:/Music/a.mp3\n")
    (rockbox / "empty.log").write_text("")
    found = submit_scrobbles.find_playback_logs(tmp_path)
    assert [path.name for path in found] == [
        "playback.log", "playback_0001.log", "playback_0002.log"
    ]
 def test_without_a_mirror_it_says_what_is_needed(tmp_path, capsys):
    device = tmp_path / "IPOD"
    (device / ".rockbox").mkdir(parents=True)
    (device / ".rockbox" / "playback.log").write_text("1700000000:1:1:/Music/a.mp3\n")
    submit_scrobbles.main([str(device)], transport=fake_transport([]))
    assert "pass --mirror" in capsys.readouterr().err
 MIRROR = "/mnt/mirror"
 def stub_ffprobe(monkeypatch, **tags):
    """Answer for anything under the mirror without invoking ffprobe."""
    monkeypatch.setattr(submit_scrobbles, "_ffprobe", tags_of(**tags))
 def only_mirror_files_exist(monkeypatch, resolvable=None):
    """Make the mirror's files appear to exist, and nothing else.
    Patching is_file wholesale makes the device directory look like a log file,
    which sends main() down the .scrobbler.log path instead.
    """
    real = Path.is_file
    def patched(self):
        text = str(self)
        if text.startswith(MIRROR):
            return resolvable is None or text == resolvable
        return real(self)
    monkeypatch.setattr(Path, "is_file", patched)
 def playback_device(tmp_path, log=PLAYBACK_LOG):
    device = tmp_path / "IPOD"
    (device / ".rockbox").mkdir(parents=True)
    (device / ".rockbox" / "playback.log").write_text(log)
    return device
 def test_a_failed_submission_keeps_every_log(tmp_path, monkeypatch):
    """Nothing got through, so nothing may be set aside."""
    only_mirror_files_exist(monkeypatch)
    stub_ffprobe(monkeypatch)
    monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
    device = playback_device(tmp_path)
    transport = fake_transport([{"error": 29, "message": "Rate limit"}])
    code = submit_scrobbles.main(
        [str(device), "--mirror", MIRROR, "--api-key", "k", "--api-secret", "s"],
        transport=transport,
    )
    assert code == 1
    assert (device / ".rockbox" / "playback.log").is_file()
    assert not list((device / ".rockbox").glob("*.submitted"))
 def test_an_unmatched_play_is_written_back_not_lost(tmp_path, monkeypatch, capsys):
    """A track the mirror does not yet hold is still a play that happened. It
    is kept so a later run, after the file has been copied, can submit it."""
    monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
    # Only the first track resolves; the rest are absent from the mirror.
    only_mirror_files_exist(monkeypatch, f"{MIRROR}/Pendulum/Immersion/01.mp3")
    stub_ffprobe(monkeypatch)
    log = (
        "1700000300:180000:245000:/Music/Pendulum/Immersion/01.mp3\n"
        "1700000400:180000:245000:/Music/Missing/Album/09.mp3\n"
    )
    device = playback_device(tmp_path, log)
    transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
    code = submit_scrobbles.main(
        [str(device), "--mirror", MIRROR, "--api-key", "k", "--api-secret", "s"],
        transport=transport,
    )
    assert code == 0
    rockbox = device / ".rockbox"
    # The original is preserved untouched...
    assert list(rockbox.glob("playback.log.*.submitted"))
    # ...and the unmatched play is back in a live log for the next attempt.
    written = (rockbox / "playback.log").read_text()
    assert "Missing/Album/09.mp3" in written
    assert "Pendulum/Immersion/01.mp3" not in written
 def test_the_original_is_renamed_rather_than_deleted(tmp_path, monkeypatch):
    """If Last.fm quietly dropped something, the evidence stays on the device."""
    only_mirror_files_exist(monkeypatch)
    stub_ffprobe(monkeypatch)
    monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
    device = playback_device(tmp_path)
    transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
    submit_scrobbles.main(
        [str(device), "--mirror", MIRROR, "--api-key", "k", "--api-secret", "s"],
        transport=transport,
    )
    aside = list((device / ".rockbox").glob("playback.log.*.submitted"))
    assert len(aside) == 1
    assert PLAYBACK_LOG.splitlines()[0] in aside[0].read_text()
 def test_keep_leaves_everything_alone(tmp_path, monkeypatch):
    only_mirror_files_exist(monkeypatch)
    stub_ffprobe(monkeypatch)
    monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
    device = playback_device(tmp_path)
    transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
    submit_scrobbles.main(
        [str(device), "--mirror", MIRROR, "--keep",
         "--api-key", "k", "--api-secret", "s"],
        transport=transport,
    )
    assert (device / ".rockbox" / "playback.log").read_text() == PLAYBACK_LOG
    assert not list((device / ".rockbox").glob("*.submitted"))
 def test_a_file_ffprobe_cannot_read_does_not_abandon_the_rest(monkeypatch):
    """CalledProcessError is not an OSError, so one bad file used to take the
    whole submission with it."""
    def broken(path):
        raise subprocess.CalledProcessError(1, "ffprobe")
    assert submit_scrobbles.read_tags(Path("/mnt/mirror/x.mp3"), runner=broken) == {}
@@ -0,0 +1,348 @@
 """The guards on sync-to-ipod.sh, which are the substance of the script.
 rsync --delete is being aimed at a whole filesystem, so every refusal here is
 protecting against emptying the wrong directory -- a mistake that does not
 announce itself.
 """
 import shutil
 import subprocess
 from pathlib import Path
 import pytest
 SCRIPT = Path(__file__).resolve().parent.parent / "tools" / "sync-to-ipod.sh"
 # Skipped rather than failed where the tools are absent: this is a host-side
 # script, and a machine without rsync is not a machine that would run it.
 REQUIRED = ("bash", "rsync", "findmnt")
 pytestmark = pytest.mark.skipif(
    not all(shutil.which(tool) for tool in REQUIRED),
    reason=f"needs {', '.join(REQUIRED)} on PATH",
 )
 def run(*arguments):
    return subprocess.run(
        ["bash", str(SCRIPT), *arguments], capture_output=True, text=True
    )
@pytest.fixture
 def mirror(tmp_path):
    source = tmp_path / "mirror"
    (source / "Album").mkdir(parents=True)
    (source / "Album" / "track.mp3").write_bytes(b"x")
    return source
 def test_the_host_root_is_refused(mirror):
    """Stripping the trailing slash from "/" leaves an empty string, and an
    earlier version then reported it as "not a directory" instead."""
    result = run(str(mirror), "/")
    assert result.returncode == 1
    assert "refusing to sync onto /" in result.stderr
 def test_an_empty_mirror_is_refused(tmp_path):
    """Mirroring nothing onto the device would delete everything on it."""
    empty = tmp_path / "empty"
    empty.mkdir()
    destination = tmp_path / "dest"
    destination.mkdir()
    result = run(str(empty), str(destination))
    assert result.returncode == 1
    assert "refusing to mirror nothing" in result.stderr
 def test_syncing_a_directory_onto_itself_is_refused(mirror):
    result = run(str(mirror), str(mirror))
    assert result.returncode == 1
    assert "same directory" in result.stderr
 def test_a_non_fat_destination_is_refused(mirror, tmp_path):
    """Which is also how an unmounted device is caught: /media/IPOD/Music then
    resolves to the host's own root filesystem."""
    destination = tmp_path / "dest"
    destination.mkdir()
    result = run(str(mirror), str(destination))
    assert result.returncode == 1
    assert "not FAT" in result.stderr
    assert "Is the device mounted?" in result.stderr
 def test_a_missing_destination_is_refused(mirror, tmp_path):
    result = run(str(mirror), str(tmp_path / "nowhere"))
    assert result.returncode == 1
    assert "not a directory" in result.stderr
 def test_a_subdirectory_of_the_device_is_a_valid_target(mirror, tmp_path):
    """The better target, in fact: --delete is confined to it."""
    destination = tmp_path / "dest" / "Music"
    destination.mkdir(parents=True)
    result = run("-f", "-n", str(mirror), str(destination))
    assert result.returncode == 0, result.stderr
    assert "dry run, nothing was written" in result.stderr
 def test_the_device_prefix_is_derived_from_the_destination(mirror, tmp_path):
    """Derived rather than configured, so it cannot disagree with where the
    files are actually going -- and the device's path limit applies to it."""
    destination = tmp_path / "dest" / "Music"
    destination.mkdir(parents=True)
    result = run("-f", "-n", str(mirror), str(destination))
    assert "the device will see this as /" in result.stderr
 def test_a_dry_run_writes_nothing(mirror, tmp_path):
    destination = tmp_path / "dest"
    destination.mkdir()
    run("-f", "-n", str(mirror), str(destination))
    assert list(destination.iterdir()) == []
 def test_rockbox_is_never_deleted(mirror, tmp_path):
    """A sync to the card root would otherwise remove the Rockbox install,
    since the mirror does not contain it."""
    destination = tmp_path / "dest"
    destination.mkdir()
    (destination / ".rockbox").mkdir()
    (destination / ".rockbox" / "rockbox.ipod").write_bytes(b"firmware")
    (destination / ".scrobbler.log").write_bytes(b"#AUDIOSCROBBLER/1.1\n")
    (destination / "Stale.mp3").write_bytes(b"old")
    result = run("-f", "-S", "-U", str(mirror), str(destination))
    assert result.returncode == 0, result.stderr
    assert (destination / ".rockbox" / "rockbox.ipod").is_file()
    assert (destination / ".scrobbler.log").is_file()
    # But a track whose source has gone is still removed. That is the point.
    assert not (destination / "Stale.mp3").exists()
    assert (destination / "Album" / "track.mp3").is_file()
 def test_help_goes_to_stdout_and_exits_clean():
    """Asking for help is not an error; getting the arguments wrong is."""
    result = run("--help")
    assert result.returncode == 0
    assert result.stdout.startswith("usage:")
    assert result.stderr == ""
 def test_short_help_behaves_the_same():
    result = run("-h")
    assert result.returncode == 0
    assert result.stdout.startswith("usage:")
 def test_misuse_goes_to_stderr_and_does_not():
    result = run("only-one-argument")
    assert result.returncode == 2
    assert result.stderr.startswith("usage:")
    assert result.stdout == ""
 def test_the_help_explains_what_the_destination_should_be():
    """The question this script actually gets asked."""
    help_text = run("--help").stdout
    assert "/media/IPOD/Music" in help_text
    assert "artist folders" in help_text
    assert ".rockbox" in help_text
 def test_the_help_says_how_to_reach_and_leave_disk_mode():
    help_text = run("--help").stdout
    assert "Menu+Select" in help_text
    assert "holding Play" in help_text
 def test_counting_is_off_by_default(mirror, tmp_path):
    """The counting pass walks and compares both trees exactly as the transfer
    does. On a FAT card of fifty thousand files that costs more than moving the
    data, so the percentage has to be asked for."""
    destination = tmp_path / "dest"
    destination.mkdir()
    result = run("-f", "-S", "-U", str(mirror), str(destination))
    assert result.returncode == 0, result.stderr
    assert "files to copy" not in result.stderr
    assert (destination / "Album" / "track.mp3").is_file()
 def test_the_delta_algorithm_is_disabled(mirror, tmp_path):
    """It would read every destination file back over USB to checksum it, to
    avoid resending an MP3 that has changed in its entirety anyway."""
    script = SCRIPT.read_text()
    assert "--whole-file" in script
 def test_directory_timestamps_are_not_set(mirror, tmp_path):
    """One setattr round trip per directory, across six thousand albums, to set
    timestamps nothing reads."""
    script = SCRIPT.read_text()
    assert "--omit-dir-times" in script
 def test_an_interrupt_is_trapped_so_the_filesystem_is_flushed():
    """Ctrl-C during a transfer would otherwise skip the sync and the unmount,
    leaving a journal-less FAT filesystem with dirty buffers -- which is the
    corruption this script exists to prevent.
    Structural rather than timed: reproducing a mid-transfer signal needs a
    payload large enough to be slow, and a test that depends on losing a race
    is a test that fails in CI for no reason.
    """
    script = SCRIPT.read_text()
    assert "trap interrupted INT TERM" in script
    assert "exit 130" in script
 def test_the_flush_and_unmount_happen_on_every_exit_path():
    script = SCRIPT.read_text()
    # Both the normal path and the interrupt path go through the same function,
    # so one cannot drift from the other.
    assert script.count("finish\n") >= 2
    assert "--partial" not in script, "rsync must delete partial files, not keep them"
 def test_the_database_step_is_skipped_without_a_tool(mirror, tmp_path, monkeypatch):
    """Opt-in, like the scrobbler: absent configuration is not an error."""
    destination = tmp_path / "dest"
    destination.mkdir()
    monkeypatch.delenv("MUSIC_MIRROR_DATABASE_TOOL", raising=False)
    result = run("-f", "-S", "-U", str(mirror), str(destination))
    assert result.returncode == 0, result.stderr
    assert "no database tool configured" in result.stderr
 def test_a_missing_database_tool_is_refused(mirror, tmp_path, monkeypatch):
    destination = tmp_path / "dest"
    destination.mkdir()
    monkeypatch.setenv("MUSIC_MIRROR_DATABASE_TOOL", str(tmp_path / "nonexistent"))
    result = run("-f", "-S", "-U", str(mirror), str(destination))
    assert result.returncode == 1
    assert "not executable" in result.stderr
 def test_the_database_step_can_be_skipped(mirror, tmp_path, monkeypatch):
    destination = tmp_path / "dest"
    destination.mkdir()
    monkeypatch.setenv("MUSIC_MIRROR_DATABASE_TOOL", str(tmp_path / "nonexistent"))
    result = run("-f", "-S", "-U", "-B", str(mirror), str(destination))
    assert result.returncode == 0, result.stderr
    assert "not executable" not in result.stderr
 def test_the_scan_reads_from_the_mirror_not_the_device():
    """The whole point: tags come off the mirror, only the .tcd files go over
    USB. Reading 49,600 files through an iPod's USB bridge is the slow path."""
    script = SCRIPT.read_text()
    assert 'ln -s "$mirror"' in script
    assert 'cd "$scratch"' in script
 def can_bind_mount():
    """User namespaces let an unprivileged process bind mount. Not everywhere,
    notably not inside some containers, so the test that needs it skips."""
    return (
        subprocess.run(
            ["unshare", "-Umr", "true"], capture_output=True, check=False
        ).returncode
        == 0
    )
@pytest.mark.skipif(not can_bind_mount(), reason="needs unprivileged user namespaces")
 def test_the_database_lands_at_the_device_root_not_the_music_folder(tmp_path):
    """The two tools disagree about where the root is. rsync copies artist
    folders into <device>/Music; the database tool must run one level up, where
    .rockbox lives, and must record /Music/... paths while reading the bytes
    from the mirror. device_prefix is what reconciles them.
    """
    mirror = tmp_path / "mirror" / "Pendulum" / "Immersion"
    mirror.mkdir(parents=True)
    (mirror / "01.mp3").write_bytes(b"not really an mp3")
    card = tmp_path / "card"
    (card / ".rockbox").mkdir(parents=True)
    (card / "Music").mkdir()
    device = tmp_path / "device"
    device.mkdir()
    tool = tmp_path / "fake-database"
    # Records where it was run and what it could see, which is the whole
    # question; producing a real database needs Rockbox's builder.
    tool.write_text(
        "#!/bin/sh\n"
        "printf '%s\\n' \"$PWD\" > .rockbox/where.txt\n"
        "ls Music/ > .rockbox/saw.txt\n"
        "echo db > .rockbox/database_0.tcd\n"
    )
    tool.chmod(0o755)
    script = (
        f"mount --bind {card} {device} && "
        f"XDG_CACHE_HOME={tmp_path / 'cache'} MUSIC_MIRROR_DATABASE_TOOL={tool} "
        f"bash {SCRIPT} -f -S -U {tmp_path / 'mirror'} {device / 'Music'}"
    )
    result = subprocess.run(
        ["unshare", "-Umr", "sh", "-c", script], capture_output=True, text=True
    )
    assert result.returncode == 0, result.stderr
    # The database lands beside the device root, not inside Music.
    assert (card / ".rockbox" / "database_0.tcd").is_file()
    # Only *.tcd is copied across, so the markers stay in the scratch root --
    # which is itself the point: nothing else is written to the device.
    scratch = tmp_path / "cache" / "music-mirror" / "database" / ".rockbox"
    assert not (card / ".rockbox" / "where.txt").exists()
    # It ran in the scratch root, not on the card.
    where = (scratch / "where.txt").read_text().strip()
    assert where.endswith("music-mirror/database"), where
    # ...and could walk into the mirror through a symlink named for the device
    # prefix, which is how the paths come out as /Music/... while the bytes are
    # read from somewhere else entirely.
    assert "Pendulum" in (scratch / "saw.txt").read_text()
 def test_counting_can_be_asked_for(mirror, tmp_path):
    """When the destination is cheap to traverse, the percentage is worth it."""
    destination = tmp_path / "dest"
    destination.mkdir()
    result = run("-f", "-S", "-U", "-P", str(mirror), str(destination))
    assert result.returncode == 0, result.stderr
    assert "files to copy" in result.stderr
@@ -0,0 +1,190 @@
 #!/usr/bin/env python3
 """Render rsync's per-file output as a single updating status line.
 Fed the size and path rsync reports with --out-format='%l %n', one per line.
 The size is what makes an estimate possible: rsync's own rate is not exposed
 per file, but bytes completed over time elapsed is the same arithmetic and
 needs nothing rsync does not already print. Prints one
 line that rewrites itself, showing how far through the transfer is and which
 album is currently going across, rather than either scrolling fifty thousand
 filenames past or -- as rsync does while it builds its file list -- saying
 nothing at all for several minutes.
 Falls back to periodic plain lines when stderr is not a terminal, so a log does
 not fill up with carriage returns.
 """
 import argparse
 import collections
 import os
 import shutil
 import sys
 import time
 # The estimate is taken over a trailing window rather than the whole run, so it
 # follows a device that slows down instead of averaging the slowdown away.
 RATE_WINDOW_SECONDS = 30.0
 # Below this the window is too narrow to divide by: the first few files arrive
 # in microseconds and produce a rate in the gigabytes per second, and an ETA of
 # nothing at all. Better to show neither until the figure means something.
 RATE_MINIMUM_SPAN_SECONDS = 2.0
 def parse(line):
    """Return (bytes, path) for one line of rsync output.
    Tolerates a bare path, in case someone runs this against --out-format='%n'.
    """
    line = line.rstrip("\n")
    size, separator, path = line.partition(" ")
    if separator and size.isdigit():
        return int(size), path
    return 0, line
 def human_bytes(count):
    size = float(count)
    for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
        if size < 1024 or unit == "TiB":
            return f"{size:.1f} {unit}"
        size /= 1024
 def human_duration(seconds):
    """Return a duration nobody has to do arithmetic on."""
    seconds = int(seconds)
    if seconds < 60:
        return f"{seconds}s"
    if seconds < 3600:
        return f"{seconds // 60}m{seconds % 60:02d}s"
    return f"{seconds // 3600}h{(seconds % 3600) // 60:02d}m"
 class Rate:
    """Bytes per second over a trailing window."""
    def __init__(self, window=RATE_WINDOW_SECONDS):
        self.window = window
        self.samples = collections.deque()
    def add(self, when, total_bytes):
        self.samples.append((when, total_bytes))
        while len(self.samples) > 2 and when - self.samples[0][0] > self.window:
            self.samples.popleft()
    def per_second(self):
        if len(self.samples) < 2:
            return 0.0
        (first_time, first_bytes), (last_time, last_bytes) = (
            self.samples[0],
            self.samples[-1],
        )
        elapsed = last_time - first_time
        if elapsed < RATE_MINIMUM_SPAN_SECONDS:
            return 0.0
        return (last_bytes - first_bytes) / elapsed
 def album_of(path):
    """Return "Artist / Album" for a mirror-relative path."""
    parts = [part for part in path.strip("/").split("/") if part]
    if len(parts) >= 3:
        return f"{parts[0]} / {parts[1]}"
    if len(parts) == 2:
        return parts[0]
    return ""
 def fit(text, width):
    """Trim to the terminal, from the left: the album matters more than the artist."""
    if width <= 1 or len(text) <= width:
        return text
    return "…" + text[-(width - 1) :]
 def render(done, total, copied, expected, rate, label, width):
    """Build the status line, giving whatever room is left to the album."""
    if total > 0:
        share = min(100, done * 100 // total)
        head = f"[{share:>3}%] {done:,}/{total:,}"
    else:
        head = f"[{done:,} files]"
    if expected > 0:
        head += f"  {human_bytes(copied)}/{human_bytes(expected)}"
    if rate > 0:
        head += f"  {human_bytes(rate)}/s"
        remaining = expected - copied
        if remaining > 0:
            head += f"  ETA {human_duration(remaining / rate)}"
    head += "  "
    return head + fit(label, max(0, width - len(head)))
 def main(argv=None, stream=None, out=None):
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--total", type=int, default=0, help="files expected")
    parser.add_argument("--bytes", type=int, default=0, help="bytes expected")
    parser.add_argument("--interval", type=float, default=0.1, help="seconds between redraws")
    args = parser.parse_args(argv)
    stream = stream or sys.stdin
    out = out or sys.stderr
    interactive = out.isatty()
    width = shutil.get_terminal_size((100, 24)).columns - 1
    done = 0
    copied = 0
    rate = Rate()
    started = time.monotonic()
    rate.add(started, 0)
    last_drawn = 0.0
    label = ""
    for line in stream:
        size, path = parse(line)
        # rsync reports directories too, with a trailing slash and an inode
        # size. Counting them puts the percentage past a hundred and the byte
        # total well over what will actually be transferred.
        if not path or path.endswith("/"):
            continue
        done += 1
        copied += size
        label = album_of(path) or os.path.basename(path)
        now = time.monotonic()
        rate.add(now, copied)
        if interactive:
            if now - last_drawn >= args.interval:
                out.write(
                    "\r\033[2K"
                    + render(done, args.total, copied, args.bytes, rate.per_second(),
                             label, width)
                )
                out.flush()
                last_drawn = now
        elif now - last_drawn >= 30:
            out.write(
                render(done, args.total, copied, args.bytes, rate.per_second(), label, width)
                + "\n"
            )
            out.flush()
            last_drawn = now
    elapsed = max(1e-9, time.monotonic() - started)
    if interactive:
        out.write("\r\033[2K")
    summary = render(done, args.total, copied, args.bytes, 0, label, width).rstrip()
    out.write(f"{summary}\n")
    if copied:
        out.write(
            f"copied {human_bytes(copied)} in {human_duration(elapsed)}"
            f" at {human_bytes(copied / elapsed)}/s\n"
        )
    out.flush()
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -14,12 +14,16 @@ import argparse
 import hashlib
 import json
 import os
 import subprocess
 import sys
 import time
 import urllib.error
 import urllib.parse
 import urllib.request
 from dataclasses import dataclass
 from datetime import datetime, timezone
 from pathlib import Path
 from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
 API_ROOT = "https://ws.audioscrobbler.com/2.0/"
@@ -31,6 +35,26 @@ BATCH = 50
 # scrobble.
 LOG_NAMES = (".scrobbler.log", ".scrobbler-timeless.log")
 # Rockbox core writes this whenever "play log" is on, with no plugin running:
 #     timestamp:elapsed_ms:length_ms:/Music/Artist/Album/Track.mp3
 # It is rotated once it grows past half a megabyte. Converting it needs tags,
 # which is why the on-device plugin exists -- reading them back off the player
 # is slow. Off the mirror it is free, so the plugin can be skipped entirely.
 PLAYBACK_LOG_NAMES = ("playback.log", "playback_*.log")
 # The plugin counts a track as listened at savepct of its length, defaulting to
 # fifty. Same rule here, or the two disagree about what a play is.
 LISTENED_FRACTION = 0.5
 # Below this a timestamp is not a wall-clock time. Without a real-time clock
 # Rockbox logs ticks in milliseconds instead, which is not a date.
 EARLIEST_PLAUSIBLE = 1_000_000_000
 # How far ahead of now a converted play may sit before it is reported. Some
 # slack absorbs a device clock drifting by a minute or two; an hour out means
 # the zone is wrong or the clock was never put forward.
 FUTURE_TOLERANCE_SECONDS = 300
 SESSION_FILE = Path(
    os.getenv("XDG_CONFIG_HOME", Path.home() / ".config")
 ) / "music-mirror" / "lastfm.json"
@@ -40,14 +64,117 @@ class LastfmError(Exception):
    """A Last.fm request that failed."""
-def parse_log(text):
+def system_zone_name():
    """Return this machine's IANA zone name, or "" if nothing states it.
    The name has to come out whole. /etc/localtime is a symlink into the
    tzdata tree, so the part after `zoneinfo/` is the name -- taking only the
    last component yields "London", which no database has, and falls back to a
    fixed offset that would then be wrong for half the year.
    """
    try:
        named = Path("/etc/timezone").read_text(encoding="utf-8").strip()
        if named:
            return named
    except OSError:
        pass
    try:
        parts = Path("/etc/localtime").resolve().parts
    except OSError:
        return ""
    if "zoneinfo" in parts:
        return "/".join(parts[len(parts) - parts[::-1].index("zoneinfo"):])
    return ""
 def device_zone(name=None):
    """Return the zone the device's clock is keeping.
    Rockbox has no concept of a timezone, so its clock is set to local wall
    time and the zone has to be supplied from outside. Defaulting to this
    machine's zone is right whenever the player and the laptop are in the same
    place, which for a device synced by cable they are.
    """
    if name:
        return ZoneInfo(name)
    for candidate in (os.getenv("TZ"), system_zone_name()):
        if not candidate:
            continue
        try:
            return ZoneInfo(candidate)
        except (ZoneInfoNotFoundError, ValueError):
            continue
    # Nothing named the zone, so the offset cannot be resolved per play: this
    # is today's offset applied to every timestamp, which is wrong either side
    # of a daylight saving change. Still better than pretending it logged UTC.
    print(
        "warning: no IANA timezone found for this machine; using its current"
        " offset for every play. Pass --device-timezone to fix older plays"
        " across a daylight saving change.",
        file=sys.stderr,
    )
    return datetime.now().astimezone().tzinfo
 def device_time_to_utc(stamp, zone):
    """Return the true UTC epoch of a timestamp Rockbox wrote.
    Rockbox builds its timestamps with `mktime(get_time())`, and its mktime
    (firmware/libc/mktime.c) is plain calendar arithmetic with no zone applied.
    Fed the RTC's local fields it yields local-wall-clock-as-if-UTC, so the
    number is ahead of real UTC by whatever the offset was. Its own scrobbler
    plugin admits this by writing `#TZ/UNKNOWN`, leaving the correction here.
    Decoding the number back into those fields and reinterpreting them in the
    device's zone recovers the instant, and does so per play, so a log
    straddling a daylight-saving change converts each side by its own offset.
    """
    fields = datetime.fromtimestamp(stamp, timezone.utc).replace(tzinfo=zone)
    return int(fields.timestamp())
 def future_plays(played, now=None):
    """Return the plays timestamped later than now, which cannot have happened.
    A device clock left on the wrong offset, or never adjusted across a
    daylight-saving change, shows up here. Last.fm has no way to tell such a
    scrobble from a real one, so it is worth saying out loud.
    """
    now = time.time() if now is None else now
    return [
        entry for entry in played
        if int(entry["timestamp"]) > now + FUTURE_TOLERANCE_SECONDS
    ]
 def declares_utc(text):
    """Whether an AUDIOSCROBBLER log says its timestamps are already UTC.
    The format's header carries `#TZ/UTC` or `#TZ/UNKNOWN`, and its spec is
    explicit that a device may only claim UTC if it converted. Rockbox writes
    UNKNOWN, meaning the times are local wall clock and want correcting; a log
    from anything that claims UTC must be left alone.
    """
    for line in text.splitlines():
        if not line.startswith("#"):
            break
        if line.strip().upper().startswith("#TZ/"):
            return line.strip().upper() == "#TZ/UTC"
    return False
 def parse_log(text, zone=None):
    """Return the listened tracks in an AUDIOSCROBBLER log, oldest first.
    Fields are artist, album, title, track number, length, rating, timestamp
    and MusicBrainz id. Rockbox converts any tab inside a field to a space
    before writing, so splitting on tabs is safe.
    Timestamps are corrected from the device's local wall clock to UTC unless
    the log's own header claims it did that already.
    """
    played, skipped, timeless = [], 0, 0
    convert = zone is not None and not declares_utc(text)
    for line in text.splitlines():
        if not line or line.startswith("#"):
            continue
@@ -68,6 +195,8 @@ def parse_log(text):
            continue
        if not artist or not title:
            continue
        if convert:
            when = device_time_to_utc(when, zone)
        played.append(
            {
                "artist": artist,
@@ -83,6 +212,136 @@ def parse_log(text):
    return played, skipped, timeless
 def parse_playback_log(text):
    """Return (timestamp, elapsed_ms, length_ms, path) for each logged play."""
    plays = []
    for line in text.splitlines():
        fields = line.strip().split(":", 3)
        if len(fields) != 4:
            continue
        stamp, elapsed, length, path = fields
        try:
            plays.append((int(stamp), int(elapsed), int(length), path))
        except ValueError:
            continue
    return plays
 def device_to_local(path, device_prefix, mirror):
    """Map a path as the player sees it onto the mirror it was copied from."""
    prefix = "/" + device_prefix.strip("/")
    if prefix != "/":
        if not path.startswith(prefix + "/"):
            return None
        path = path[len(prefix) :]
    return Path(mirror) / path.lstrip("/")
 def read_tags(path, runner=None):
    """Return the tags of a local file, via ffprobe. Empty if it cannot be read.
    A file ffprobe chokes on is one play left unidentified, not a reason to
    abandon the rest -- and CalledProcessError is not an OSError, so catching
    the obvious things is not enough.
    """
    runner = runner or _ffprobe
    try:
        payload = json.loads(runner(path))
    except (OSError, ValueError, subprocess.SubprocessError):
        return {}
    return {
        key.lower(): value
        for key, value in (payload.get("format", {}).get("tags") or {}).items()
    }
 def _ffprobe(path):
    return subprocess.run(
        ["ffprobe", "-v", "error", "-show_entries", "format_tags",
         "-of", "json", str(path)],
        capture_output=True, text=True, check=True,
    ).stdout
@dataclass
 class Conversion:
    """What a playback log turned into, and what must not be thrown away.
    `retain` holds the raw lines of plays that were real but could not be
    submitted -- a track absent from the mirror, usually because the sync had
    not copied it yet. Those are written back so a later run can try again.
    Skips and clockless entries are not retained: neither can ever be
    submitted, and the untouched original is set aside regardless.
    """
    played: list
    skipped: int = 0
    unresolved: int = 0
    timeless: int = 0
    retain: list = None
    def __post_init__(self):
        if self.retain is None:
            self.retain = []
 def plays_from_playback_log(text, device_prefix, mirror, runner=None, zone=None):
    """Return the conversion of a playback log.
    Skips are decided by the same fraction the on-device plugin uses, so the
    two never disagree about what counted as a play. Timestamps are corrected
    from the device's wall clock to UTC; the core log has no header to say so,
    but it is written the same way the plugin's UNKNOWN times are.
    """
    result = Conversion(played=[])
    for line in text.splitlines():
        parsed = parse_playback_log(line)
        if not parsed:
            continue
        stamp, elapsed, length, device_path = parsed[0]
        if stamp < EARLIEST_PLAUSIBLE:
            result.timeless += 1
            continue
        if length > 0 and elapsed < length * LISTENED_FRACTION:
            result.skipped += 1
            continue
        local = device_to_local(device_path, device_prefix, mirror)
        tags = read_tags(local, runner) if local and local.is_file() else {}
        artist = tags.get("artist") or tags.get("album_artist") or ""
        title = tags.get("title") or ""
        if not artist or not title:
            # A real play of a track this run could not identify. Kept, so a
            # later run -- after the file has been copied, or the tags fixed --
            # can submit it rather than the play being lost.
            result.unresolved += 1
            result.retain.append(line)
            continue
        result.played.append(
            {
                "artist": artist,
                "track": title,
                "album": tags.get("album", ""),
                "trackNumber": (tags.get("track") or "").split("/")[0],
                "duration": str(length // 1000) if length > 0 else "",
                "timestamp": str(
                    device_time_to_utc(stamp, zone) if zone is not None else stamp
                ),
                "mbid": tags.get("musicbrainz_trackid", ""),
                "line": line,
            }
        )
    result.played.sort(key=lambda entry: int(entry["timestamp"]))
    return result
 def find_playback_logs(device):
    """Return every playback log on a device, oldest first."""
    found = []
    for pattern in PLAYBACK_LOG_NAMES:
        found.extend(sorted(Path(device).glob(f".rockbox/{pattern}")))
    return [path for path in found if path.is_file() and path.stat().st_size]
 def sign(params, secret):
    """Return Last.fm's method signature for a set of parameters.
@@ -152,7 +411,7 @@ def batch_params(entries):
    return params
-def submit(entries, key, secret, session, transport, delay=1.0):
+def submit(entries, key, secret, session, transport, delay=1.0, on_sent=None):
    """Submit every entry. Returns how many the service accepted.
    Batches are counted as they succeed rather than at the end, so a failure
@@ -168,6 +427,8 @@ def submit(entries, key, secret, session, transport, delay=1.0):
        block = payload.get("scrobbles", {})
        summary = block.get("@attr", block)
        accepted += int(summary.get("accepted", len(chunk)))
        if on_sent is not None:
            on_sent(chunk)
        ignored = int(summary.get("ignored", 0))
        if ignored:
            print(f"  {ignored} of {len(chunk)} ignored by Last.fm", file=sys.stderr)
@@ -193,6 +454,26 @@ def find_log(device):
 def main(argv=None, transport=http_post):
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("device", help="the mounted device, or a scrobbler log file")
    parser.add_argument(
        "--mirror",
        help="the mirror the device was copied from. Given this, Rockbox's own"
        " playback.log is converted here rather than needing the on-device"
        " plugin run first",
    )
    parser.add_argument(
        "--device-prefix",
        default="/Music",
        help="where the music sits on the device, stripped when mapping a logged"
        " path back onto the mirror",
    )
    parser.add_argument(
        "--device-timezone",
        default=os.getenv("ROCKBOX_TIMEZONE"),
        help="the zone the player's clock is set to, as an IANA name such as"
        " Europe/London. Rockbox keeps local wall time and cannot record an"
        " offset, so its timestamps need this to become the UTC Last.fm wants."
        " Defaults to this machine's zone",
    )
    parser.add_argument("--api-key", default=os.getenv("LASTFM_API_KEY"))
    parser.add_argument("--api-secret", default=os.getenv("LASTFM_API_SECRET"))
    parser.add_argument("--dry-run", action="store_true", help="parse and report only")
@@ -201,14 +482,61 @@ def main(argv=None, transport=http_post):
    )
    args = parser.parse_args(argv)
    try:
        zone = device_zone(args.device_timezone)
    except (ZoneInfoNotFoundError, ValueError) as error:
        print(f"unknown timezone {args.device_timezone!r}: {error}", file=sys.stderr)
        return 2
    target = Path(args.device)
    log = target if target.is_file() else find_log(target)
-    if log is None:
+    logs = []
    conversion = None
    if log is not None:
        played, skipped, timeless = parse_log(
            log.read_text(encoding="utf-8", errors="replace"), zone
        )
        unresolved = 0
        logs = [log]
        print(f"{log}: {len(played)} listened, {skipped} skipped", file=sys.stderr)
    elif args.mirror:
        # No plugin has been run, but the core log is there. Tags come off the
        # mirror, which is the only reason the plugin was needed at all.
        logs = find_playback_logs(target)
        if not logs:
            print("no scrobbler log to submit", file=sys.stderr)
            return 0
        text = "\n".join(
            path.read_text(encoding="utf-8", errors="replace") for path in logs
        )
        conversion = plays_from_playback_log(
            text, args.device_prefix, args.mirror, zone=zone
        )
        played = conversion.played
        skipped, unresolved, timeless = (
            conversion.skipped,
            conversion.unresolved,
            conversion.timeless,
        )
        print(
            f"{len(logs)} playback log(s): {len(played)} listened, {skipped} skipped",
            file=sys.stderr,
        )
        if unresolved:
            print(
                f"  {unresolved} could not be matched to a file in the mirror."
                " Those plays are kept for a later run rather than discarded.",
                file=sys.stderr,
            )
    else:
        print(
            "no scrobbler log to submit. Rockbox's own playback.log can be used"
            " instead -- pass --mirror so tags can be read from it.",
            file=sys.stderr,
        )
        return 0
    played, skipped, timeless = parse_log(log.read_text(encoding="utf-8", errors="replace"))
    print(f"{log}: {len(played)} listened, {skipped} skipped", file=sys.stderr)
    if timeless:
        print(
            f"  {timeless} entries have no timestamp, so this target has no clock."
@@ -217,9 +545,24 @@ def main(argv=None, transport=http_post):
        )
    if not played:
        return 0
    ahead = future_plays(played)
    if ahead:
        newest = int(ahead[-1]["timestamp"]) - int(time.time())
        print(
            f"  {len(ahead)} plays are timestamped up to {newest // 60} minutes in"
            f" the future, converting from {zone}. Either the player's clock is"
            " wrong or that is not the zone it is set to.",
            file=sys.stderr,
        )
    if args.dry_run:
        for entry in played[:20]:
-            print(f"{entry['timestamp']}\t{entry['artist']}\t{entry['track']}")
+            when = datetime.fromtimestamp(int(entry["timestamp"]), zone)
            print(
                f"{entry['timestamp']}\t{when:%Y-%m-%d %H:%M %Z}"
                f"\t{entry['artist']}\t{entry['track']}"
            )
        return 0
    if not args.api_key or not args.api_secret:
@@ -235,21 +578,66 @@ def main(argv=None, transport=http_post):
        session = authorise(args.api_key, args.api_secret, transport)
        save_session(session)
    # Recorded as each batch is accepted, so a failure partway through knows
    # exactly what got through and what did not.
    sent = []
    try:
-        accepted = submit(played, args.api_key, args.api_secret, session, transport)
+        accepted = submit(
            played, args.api_key, args.api_secret, session, transport,
            on_sent=sent.extend,
        )
    except LastfmError as error:
-        print(f"submission failed: {error}", file=sys.stderr)
+        print(f"submission failed after {len(sent)} scrobbles: {error}", file=sys.stderr)
        keep_history(logs, played, sent, conversion, args.keep)
        return 1
    print(f"{accepted} scrobbles accepted", file=sys.stderr)
-    if not args.keep and accepted:
+    keep_history(logs, played, sent, conversion, args.keep)
        # Renamed rather than deleted: if Last.fm quietly dropped something,
        # the evidence is still on the device.
        aside = log.with_name(f"{log.name}.{played[-1]['timestamp']}.submitted")
        log.rename(aside)
        print(f"log moved to {aside.name}", file=sys.stderr)
    return 0
 def keep_history(logs, played, sent, conversion, keep):
    """Set the logs aside, writing back anything still owed a submission.
    Two separate obligations. The original is preserved untouched, renamed
    rather than deleted, so a play is never lost to a mistake here. And any
    play that was not submitted -- unmatched, or in a batch that failed -- is
    written back into a live log, so the next run tries it again instead of it
    quietly vanishing with the rest.
    """
    if keep or not logs:
        if keep:
            print("logs left in place", file=sys.stderr)
        return
    submitted = {id(entry) for entry in sent}
    # A .scrobbler.log was converted by the on-device plugin and carries no
    # per-line record, so there is nothing to write back for it -- only the
    # rename below, which loses nothing.
    pending = list(conversion.retain) if conversion is not None else []
    pending += [
        entry["line"] for entry in played
        if "line" in entry and id(entry) not in submitted
    ]
    if not sent:
        print("nothing was accepted; logs left untouched", file=sys.stderr)
        return
    stamp = played[-1]["timestamp"] if played else "0"
    for path in logs:
        path.rename(path.with_name(f"{path.name}.{stamp}.submitted"))
    if pending:
        live = logs[0].with_name("playback.log")
        live.write_text("\n".join(pending) + "\n", encoding="utf-8")
        print(
            f"{len(pending)} plays not submitted were written back to"
            f" {live.name} for the next run",
            file=sys.stderr,
        )
    print(f"{len(logs)} log(s) set aside as .submitted", file=sys.stderr)
 if __name__ == "__main__":
    sys.exit(main())
@@ -12,46 +12,105 @@
 set -euo pipefail
 usage() {
-    cat >&2 <<'USAGE'
+    # Help goes to stdout and exits clean; misuse goes to stderr and does not.
    local stream=2 code=2
    if [ "${1:-}" = "help" ]; then
        stream=1
        code=0
    fi
    cat >&"$stream" <<'USAGE'
 usage: sync-to-ipod.sh [options] <mirror> <destination>
  -n  dry run; show what would change and touch nothing
  -P  count what needs copying first, so progress can show a percentage and
      an estimate. Costs a second full traversal of both trees, which on a
      FAT card of fifty thousand files is slower than the transfer itself
  -f  copy even if the FAT32 check finds unacceptable paths
  -S  skip submitting the Rockbox scrobbler log to Last.fm
  -B  skip rebuilding the Rockbox database
  -U  leave the destination mounted afterwards
 Rebuilding the database needs MUSIC_MIRROR_DATABASE_TOOL pointing at Rockbox's
 host-side builder (tools/database, built with ./tools/configure --type=d). It
 is skipped with a note when unset. The scan reads tags from the mirror rather
 than from the device, so it costs seconds rather than the hours an on-device
 commit takes -- and on a large library the on-device commit may not finish at
 all.
 Submitting scrobbles needs LASTFM_API_KEY and LASTFM_API_SECRET; it is skipped
 with a note when they are unset. Scrobbling is a write method and needs the
 secret, unlike the read-only calls elsewhere in these projects.
-The destination must be a mounted FAT filesystem. Reach it with the Apple
+Rockbox has no notion of a timezone: its clock holds local wall time and its
-firmware's disk mode: Menu+Select to reboot, then immediately Select+Play.
+logs record that, not UTC. Set ROCKBOX_TIMEZONE to the zone the player's clock
 is keeping (an IANA name, such as Europe/London) if it differs from this
 machine's, which is otherwise assumed. Getting it wrong shifts every scrobble
 by the difference.
 The mirror is the directory holding the artist folders. The destination is
 where those folders should end up on the device -- not the card root, unless
 that is genuinely where you want them:
  sync-to-ipod.sh /mnt/tank/media/music-mp3 /media/IPOD/Music
 A subdirectory is the better target: --delete is confined to it, and the path
 budget is derived from it, since the device's 260-character limit counts the
 whole path as the device sees it. /.rockbox and the scrobbler logs are never
 deleted wherever you point this.
 The destination must be on a mounted FAT filesystem. That check is also what
 catches an unmounted device: /media/IPOD/Music then resolves to the host's own
 root filesystem, and this refuses to empty that.
 Progress is one line that rewrites itself, showing the album currently going
 across, how far through the transfer is, the rate, and an estimate of what is
 left. Working the totals out first means a second pass over the tree, which is
 the price of figures that mean something; rsync's own percentage is computed
 against a file list it is still building.
 Reach the device with the Apple firmware's disk mode: Menu+Select to reboot,
 then immediately Select+Play. Power off afterwards by holding Play.
 USAGE
-    exit 2
+    exit "$code"
 }
 dry_run=false
 counting=false
 force=false
 unmount=true
 scrobble=true
-while getopts ":nfSUh" option; do
+database=true
 for argument in "$@"; do
    [ "$argument" = "--help" ] && usage help
 done
 while getopts ":nPfSBUh" option; do
    case "$option" in
        n) dry_run=true ;;
        P) counting=true ;;
        f) force=true ;;
        S) scrobble=false ;;
        B) database=false ;;
        U) unmount=false ;;
        h) usage help ;;
        *) usage ;;
    esac
 done
 shift $((OPTIND - 1))
 [ $# -eq 2 ] || usage
 # Trailing slashes are stripped for tidiness, but stripping one from "/" leaves
 # an empty string, and the guard below would then never see the root it is
 # there to refuse.
 mirror=${1%/}
 mirror=${mirror:-/}
 destination=${2%/}
 destination=${destination:-/}
 here=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
 die() {
-    printf 'sync-to-ipod: %s\n' "$1" >&2
+    # Every argument, not just the first: the second half of a message is
    # usually the half that says what to do about it.
    printf 'sync-to-ipod: %s\n' "$*" >&2
    exit 1
 }
@@ -59,28 +118,41 @@ die() {
 [ -n "$(ls -A "$mirror")" ] || die "mirror $mirror is empty; refusing to mirror nothing"
 [ -d "$destination" ] || die "destination $destination is not a directory"
-# --delete makes every one of these load-bearing. A destination that is not its
+# --delete makes every one of these load-bearing. Emptying the wrong directory
-# own mount point means the path is wrong, and emptying the wrong directory is
+# is not a mistake that announces itself.
 # not a mistake that announces itself.
 case "$destination" in
    "" | "/" | "$HOME") die "refusing to sync onto $destination" ;;
 esac
 [ "$(readlink -f "$mirror")" != "$(readlink -f "$destination")" ] ||
    die "mirror and destination are the same directory"
 mountpoint -q -- "$destination" || die "$destination is not a mount point"
 # The filesystem the destination sits on, which is the check that matters: a
 # subdirectory of the card is a perfectly good target, and is the better one,
 # because --delete is then confined to it. Being FAT is also what proves the
 # card is mounted at all -- an unmounted /media/IPOD/Music resolves to the
 # host's own root filesystem, and this refuses to empty that.
 filesystem=$(findmnt -no FSTYPE --target "$destination")
 mounted_on=$(findmnt -no TARGET --target "$destination")
 case "$filesystem" in
    vfat | exfat) ;;
    *)
-        $force || die "$destination is $filesystem, not FAT; pass -f if that is deliberate"
+        $force ||
            die "$destination is on a $filesystem filesystem, not FAT." \
                "Is the device mounted? Pass -f if this is deliberate."
        printf 'sync-to-ipod: destination is %s, not FAT\n' "$filesystem" >&2
        ;;
 esac
 # What the device will call this directory, which is what its path limit
 # applies to. Derived rather than configured, so it cannot disagree with where
 # the files are actually going.
 device_prefix=${destination#"$mounted_on"}
 device_prefix="/${device_prefix#/}"
 printf 'sync-to-ipod: the device will see this as %s\n' "$device_prefix" >&2
 if $force; then
    printf 'sync-to-ipod: skipping the FAT32 check\n' >&2
-elif ! python3 "$here/check_fat32.py" "$mirror"; then
+elif ! python3 "$here/check_fat32.py" --device-prefix "$device_prefix" "$mirror"; then
    die "the mirror holds paths FAT32 will not take; run music-mirror with --fat32-safe"
 fi
@@ -92,7 +164,16 @@ if $scrobble; then
    else
        scrobble_options=()
        $dry_run && scrobble_options+=(--dry-run)
-        python3 "$here/submit_scrobbles.py" "${scrobble_options[@]}" "$destination" ||
+        # Rockbox keeps local wall time with no notion of a zone, so its
        # timestamps are not the UTC Last.fm expects. Naming the zone the
        # player's clock is set to lets them be corrected.
        [ -n "${ROCKBOX_TIMEZONE:-}" ] &&
            scrobble_options+=(--device-timezone "$ROCKBOX_TIMEZONE")
        # --mirror lets it convert Rockbox's own playback.log, so the on-device
        # scrobbler plugin never has to be run. The device root, not the music
        # directory: the logs live in .rockbox.
        python3 "$here/submit_scrobbles.py" "${scrobble_options[@]}" \
            --mirror "$mirror" --device-prefix "$device_prefix" "$mounted_on" ||
            die "submitting scrobbles failed; nothing has been copied"
    fi
 fi
@@ -101,36 +182,142 @@ fi
 # asking for them produces a screenful of errors and a non-zero exit.
 # --modify-window=2 because FAT stores mtimes to two-second resolution, without
 # which every file looks changed and the whole library is copied every time.
 # --whole-file is already the default when both ends are local paths, and an
 # SMB or FAT mount counts as one, but stating it documents that the delta
 # algorithm is deliberately not wanted: it would read every destination file
 # back over USB to compute a checksum, to save sending an MP3 that has changed
 # entirely anyway.
 #
 # --omit-dir-times drops a setattr round trip per directory. Across six
 # thousand album folders on a FAT card that is six thousand operations to set
 # timestamps nothing reads.
 options=(--recursive --times --delete --modify-window=2 --whole-file --omit-dir-times)
 # --delete removes tracks whose source has gone, which is the point. It would
 # also remove everything on the device that the mirror does not contain -- and
 # if the destination is the card root that means /.rockbox, the Rockbox install
 # itself. Excluded paths are not deleted unless --delete-excluded is given,
 # which it never is here.
 options=(--recursive --times --delete --modify-window=2 --human-readable --info=progress2)
 for owned in "/.rockbox" "/.scrobbler.log" "/.scrobbler.log.*" "/.playlist_control" \
    "/System Volume Information" "/.Spotlight-V100" "/.Trashes" "/.fseventsd"; do
    options+=(--exclude "$owned")
 done
 $dry_run && options+=(--dry-run --verbose)
 printf 'sync-to-ipod: %s -> %s\n' "$mirror" "$destination" >&2
 rsync "${options[@]}" "$mirror/" "$destination/"
 if $dry_run; then
    rsync "${options[@]}" --dry-run --verbose "$mirror/" "$destination/"
    printf 'sync-to-ipod: dry run, nothing was written\n' >&2
    exit 0
 fi
-sync
+# rsync says nothing at all while it builds its file list, which on fifty
-if $unmount; then
+# thousand files over USB is minutes of apparent hang. Counting first costs a
-    device=$(findmnt -no SOURCE --target "$destination")
+# second pass over the tree but means the transfer can show a real percentage
 # rather than a number that grows as rsync discovers more work.
 # Counting is opt-in because it is not cheap. It walks and compares both trees
 # in full, exactly as the transfer does, and on a FAT card holding fifty
 # thousand files that traversal costs more than moving the data. Without it the
 # progress line still shows the running count, the rate and the album in
 # flight; only the percentage and the estimate are lost, and those were the
 # least useful part of it.
 total=0
 total_bytes=0
 if $counting; then
    printf 'sync-to-ipod: working out what needs copying...\n' >&2
    # %l is the file's size, which is what makes an estimate possible.
    # Directories are dropped: rsync reports those too, with an inode size that
    # would inflate the total by several megabytes of nothing.
    counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
        awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }')
    total=${counted% *}
    total_bytes=${counted#* }
    printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
 fi
 # Flushing and unmounting is the whole reason this is a script, so it has to
 # happen on the way out whichever way that is. Ctrl-C during a transfer would
 # otherwise leave a FAT filesystem with dirty buffers and no journal, which is
 # the corruption this exists to avoid.
 finish() {
    sync
    if $unmount; then
        device=$(findmnt -no SOURCE --target "$destination" 2>/dev/null || true)
        if [ -n "$device" ]; then
            printf 'sync-to-ipod: unmounting %s\n' "$device" >&2
            if command -v udisksctl >/dev/null 2>&1; then
-        udisksctl unmount -b "$device"
+                udisksctl unmount -b "$device" || umount -- "$destination" || true
            else
-        umount -- "$destination"
+                umount -- "$destination" || true
            fi
            printf 'sync-to-ipod: safe to disconnect\n' >&2
-else
+        fi
    else
        printf 'sync-to-ipod: still mounted; unmount before disconnecting\n' >&2
-fi
+    fi
 }
 interrupted() {
    trap - INT TERM
    printf '\nsync-to-ipod: interrupted -- rsync leaves no partial files, but the\n' >&2
    printf 'sync-to-ipod: filesystem still needs flushing before you pull anything\n' >&2
    finish
    exit 130
 }
 trap interrupted INT TERM
 rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" |
    python3 "$here/rsync_progress.py" --total "$total" --bytes "$total_bytes"
 status=${PIPESTATUS[0]}
 [ "$status" -eq 0 ] || die "rsync exited $status"
 # Rockbox reads its database from .tcd files in .rockbox. Building them here
 # rather than on the device is not just faster: the on-device commit sorts the
 # whole index in whatever memory it can scrape together, and on a large library
 # it runs for hours or dies outright.
 #
 # The scan reads tags through a scratch root -- a real .rockbox beside a symlink
 # standing in for where the music lands on the device -- so the paths recorded
 # match what Rockbox will look up, while the bytes are read from the mirror
 # instead of over USB. The scratch is kept between runs because the builder is
 # incremental: a second pass over unchanged files does no work at all.
 rebuild_database() {
    local tool=${MUSIC_MIRROR_DATABASE_TOOL:-}
    if [ -z "$tool" ]; then
        printf 'sync-to-ipod: no database tool configured, skipping the database\n' >&2
        return 0
    fi
    [ -x "$tool" ] || die "$tool is not executable"
    local device_rockbox="$mounted_on/.rockbox"
    if [ ! -d "$device_rockbox" ]; then
        printf 'sync-to-ipod: no .rockbox on the device, skipping the database\n' >&2
        return 0
    fi
    local scratch="${XDG_CACHE_HOME:-$HOME/.cache}/music-mirror/database"
    mkdir -p "$scratch/.rockbox"
    # Rebuild the symlink layout each time; the mirror path or the device
    # prefix may have changed since the last run.
    find "$scratch" -maxdepth 1 -type l -delete
    if [ "$device_prefix" = "/" ]; then
        ln -s "$mirror"/* "$scratch/" 2>/dev/null || true
    else
        local under=${device_prefix#/}
        rm -rf "${scratch:?}/${under%%/*}"
        mkdir -p "$scratch/$(dirname "$under")"
        ln -s "$mirror" "$scratch/$under"
    fi
    printf 'sync-to-ipod: building the database from the mirror...\n' >&2
    ( cd "$scratch" && "$tool" ) >/dev/null || die "the database build failed"
    cp -- "$scratch"/.rockbox/*.tcd "$device_rockbox/" ||
        die "could not copy the database onto the device"
    printf 'sync-to-ipod: database copied to %s\n' "$device_rockbox" >&2
 }
 $database && rebuild_database
 finish
Author	SHA1	Message	Date
Emma ThorpeandClaude Opus 5	9942e1a920	fix: correct Rockbox's local wall-clock timestamps to UTC Rockbox has no concept of a timezone. Its clock holds local time, and it builds log timestamps with mktime(get_time()) -- but firmware/libc/mktime.c is plain calendar arithmetic applying no offset, so the RTC's local fields come out as though they were UTC. The number in the log is ahead of the real instant by whatever the offset was, and Last.fm stores UTC, so every play submitted during BST landed an hour in the future. Rockbox states this itself: its scrobbler plugin writes #TZ/UNKNOWN, and the AUDIOSCROBBLER spec allows #TZ/UTC only for a device that actually converted. The correction belongs to the consumer. Each timestamp is decoded back to its wall-clock fields and reinterpreted in the player's zone, per play rather than as one offset over the whole log, so a log spanning a daylight saving change converts each side correctly. A log declaring #TZ/UTC is left alone rather than shifted twice. The zone defaults to this machine's, overridable with --device-timezone or ROCKBOX_TIMEZONE. Deriving it needs the whole IANA name: /etc/localtime resolves into the tzdata tree, and taking only the final component yields "London", which no database holds, silently falling back to a fixed offset that is wrong for half the year. Since Rockbox cannot adjust for daylight saving on its own, the player's clock has to be changed by hand twice a year. Any play converting to a future time is now reported, which is what a forgotten adjustment looks like. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-27 14:18:52 +01:00
Emma Thorpe	374a17474f	fix: never discard a play that has not been submitted Build and publish container / build (pull_request) Successful in 5m11s Details Setting the logs aside after a successful submission threw away more than it had submitted. A play of a track missing from the mirror -- not yet copied, or its tags unreadable -- was counted as unresolved and then carried off with the rest, with nothing to retry it. The play happened and was lost. Two obligations now, kept separate. The original log is renamed rather than deleted, so a mistake here cannot destroy the record. And every play that was not submitted is written back into a live log, so the next run attempts it again: the unmatched ones, and anything in a batch that failed. Submission is recorded batch by batch as each is accepted, so a failure partway through knows exactly what got through. The remainder is written back and nothing is sent twice. When nothing at all is accepted the logs are left untouched. Skips and clockless entries are deliberately not retained. Neither can ever be submitted, so keeping them would mean reprocessing them for ever, and the untouched original holds them regardless. Also fixes a way to lose the lot: read_tags caught OSError and ValueError, but ffprobe failing raises CalledProcessError, which is neither. A single unreadable file aborted the whole submission rather than costing one unidentified play.	2026-08-26 18:32:05 +01:00
Emma Thorpe	f324b1b720	feat: convert Rockbox's playback log on the laptop, skipping the plugin Build and publish container / build (pull_request) Successful in 5m6s Details Scrobbling previously needed the on-device Last.fm plugin run by hand before each sync, to turn Rockbox's playback log into AUDIOSCROBBLER format. Forgetting that step means the sync submits nothing and quietly appears not to work. Core Rockbox writes ROCKBOX_DIR/playback.log whenever "play log" is enabled, with no plugin running at all. Each line is timestamp:elapsed_ms:length_ms:path. The only thing missing is tags, and that is exactly why the plugin exists: reading them back off the player is slow. Off the mirror it is free, because the same files are already there -- so the conversion belongs on the laptop, and the plugin can be skipped entirely. A play counts as listened at half the track's length, matching the plugin's savepct default, so the two cannot disagree about what a play was. A short play is a skip. An entry with no usable timestamp is refused rather than invented, which is the clockless case again. A path that maps to nothing in the mirror is counted and reported instead of guessed at. Rotated logs are picked up too; Rockbox starts a new one past half a megabyte. All of them are renamed aside together once Last.fm has accepted the batch. A .scrobbler.log is still read when the plugin has been run and left one.	2026-08-26 18:24:12 +01:00
Emma Thorpe	19ac9e5d92	fix: stop counting by default; the pass costs more than the transfer Build and publish container / build (pull_request) Successful in 3m32s Details The counting pass was added so the progress line could show a percentage and an estimate, and on a real card it turned out to dominate the run. Measured against the device: reading a track from the SMB mirror ran at 35 MB/s and writing to the card at 21 MB/s, while the sync itself managed tens of kilobytes per second. Neither end was slow. The cost was traversing fifty thousand files across six thousand directories on FAT, and the counting pass does that a second time, comparing both trees in full exactly as the transfer does. Counting is now opt-in behind -P. Without it the progress line still shows the running count, the transfer rate and the album in flight; the percentage and the estimate are what needed the extra walk, and they were the least useful part of the display. That the fix for "it looks hung" was itself making it slow is the sort of thing only measuring catches. The line still answers the question it was added for -- whether anything is happening -- without paying for the part that merely made it prettier.	2026-08-26 13:57:49 +01:00
Emma Thorpe	633fbbaf91	test: prove the database lands at the device root, not in the music folder Build and publish container / build (pull_request) Successful in 4m30s Details The two tools disagree about where the root is. rsync copies artist folders into <device>/Music, while the database builder must run one level up, where .rockbox lives, and must record /Music/... paths despite reading the bytes from the mirror. device_prefix is what reconciles them, and until now that was only argued rather than demonstrated. Unprivileged user namespaces make a real bind mount possible, so the test can create an actual mount point and exercise the derivation instead of asserting the shape of the script. A stub builder records its working directory and what it could see. The test asserts the .tcd file arrives beside .rockbox rather than inside Music, that the build ran in the scratch root and not on the card, and that it could walk into the mirror through the symlink -- which is the mechanism that produces device paths from mirror bytes. It skips where user namespaces are unavailable, which includes the CI container.	2026-08-26 13:30:26 +01:00
lyrathorpe	cdbe471166	chore(release): v0.4.0	2026-08-26 12:28:36 +00:00
lyrathorpe	61e0031262	Merge pull request 'feat: destination handling, live progress and interrupt safety for sync-to-ipod' (#9 ) from fix/sync-destination-subdirectory into main Build and publish container / build (push) Successful in 5m59s Details Reviewed-on: #9	2026-08-26 13:22:38 +01:00
Emma Thorpe	c99b423b72	feat: rebuild the Rockbox database during the sync, off the mirror Build and publish container / build (pull_request) Successful in 6m36s Details The on-device database commit does not work at this library size. It sorts the whole index in whatever memory core_alloc_maximum() can scrape together, and on fifty thousand tracks it runs for hours or aborts with a data abort -- observed across several builds including stable. Rockbox ships a host-side builder for exactly this, and the sync is the moment the library changes, so it belongs here. MUSIC_MIRROR_DATABASE_TOOL points at it; the step is skipped with a note when unset, as the scrobbler step is. The scan runs against a scratch root -- a real .rockbox beside a symlink standing in for wherever the music lands on the device -- so the paths recorded are the ones Rockbox will look up, while the bytes are read from the mirror rather than over USB. Only the dozen .tcd files cross to the card. Verified: scanning through the symlink records /Music/... paths while reading from somewhere else entirely. The scratch root is kept between runs because the builder is incremental. A second pass over unchanged files performs no metadata reads and finishes in a fraction of a second, so only the first build pays the full cost. That cost, measured rather than guessed: about 49 reads and 43 seeks per file, the parser probing the head for ID3v2 and the tail for ID3v1. Two thousand files in half a second on local disk. Over SMB the opens and the head/tail split are real round trips, making a first full scan minutes rather than seconds -- still preferable to an on-device commit that does not finish. There is nothing to parallelise: the tool is single-threaded and two instances cannot produce one database.	2026-08-26 13:20:20 +01:00
Emma Thorpe	9091c4d049	docs: stop overstating the risk of interrupting a sync Build and publish container / build (pull_request) Successful in 3m16s Details The README claimed that interrupting left buffers unwritten and therefore corruption. That is wrong. The kernel flushes dirty pages within dirty_expire_centisecs, thirty seconds by default, and umount syncs before it returns, so losing data requires interrupting and pulling the card inside that window and skipping the unmount. The trap is still worth having, for a smaller and more honest reason: it removes a manual step and makes the exit deterministic, so the same "safe to disconnect" appears whichever way the run ends. What actually loses data is pulling the card without unmounting at all, which has nothing to do with whether the transfer was interrupted.	2026-08-25 12:54:19 +01:00
Emma Thorpe	8228c81b5c	fix: flush and unmount even when the sync is interrupted Build and publish container / build (pull_request) Canceled after 2m10s Details rsync itself is safe under interruption. It writes to a hidden temporary file and renames it into place only once complete, and by default deletes any partial file when interrupted -- verified both in the manual and by killing a transfer and inspecting what was left, which was nothing. --partial is deliberately absent and there is now a test asserting it stays that way. An unclean kill can leave a hidden .track.mp3.XXXXXX behind; it is unplayable, it is not in the source, and the next run's --delete removes it. The script was not safe. Ctrl-C killed it before the sync and the unmount, leaving a journal-less FAT filesystem holding dirty buffers -- which is the exact corruption the script exists to prevent, arrived at by the most likely route a person would take. INT and TERM are now trapped. Both the normal path and the interrupt path call the same finish function, so the flush and the unmount cannot drift apart, and an interrupted run exits 130 rather than pretending to have succeeded. The test is structural rather than timed. Reproducing a mid-transfer signal needs a payload large enough to be slow, and a test that depends on winning a race is a test that fails in CI for reasons that have nothing to do with the code. The behaviour was verified by hand: SIGTERM mid-transfer gave exit 130, the flush ran, and the destination held no short files and no leftover temporaries.	2026-08-25 12:51:42 +01:00
Emma Thorpe	d6ef70922c	perf: cut round trips over a network mount, and allow skipping the count Build and publish container / build (pull_request) Successful in 2m39s Details The transfer is metadata-bound rather than throughput-bound. Fifty thousand files is fifty thousand round trips, and the counting pass added for the percentage doubles that. --whole-file is already implied when both ends are local paths, which an SMB or FAT mount is, but stating it records that the delta algorithm is deliberately unwanted here: it would read every destination file back over USB to checksum it, in order to avoid resending an MP3 that has changed in its entirety anyway. --omit-dir-times drops one setattr per directory. Across six thousand album folders on a FAT card that is six thousand operations spent on timestamps nothing reads. -Q skips the counting pass. The percentage and the estimate are worth a second walk of a local tree and frequently are not worth one of a network mount, so that is now a choice rather than a fixed cost. The README covers the part that is not an rsync flag at all: SMB defaults to a one second attribute cache, so nearly every stat goes to the wire, twice. An actimeo of sixty on the mount does more than any of the above, and closes most of the gap that would otherwise argue for moving to NFS.	2026-08-25 12:44:53 +01:00
Emma Thorpe	131c80f5de	feat: estimate the time remaining from bytes and observed rate Build and publish container / build (pull_request) Successful in 2m13s Details rsync reports each file's size with %l as it completes, which is all an estimate needs: bytes done over time elapsed is the same arithmetic rsync would do internally, and requires nothing it does not already print. The scan pass now sums those sizes as well as counting files, so both a percentage and an estimate have a real denominator. The rate is measured over a trailing thirty seconds rather than the whole run, so it follows a device that slows down instead of averaging the slowdown away -- which for a card reader that thermally throttles, or a USB link that renegotiates after an hour, is the difference between a useful estimate and a reassuring one. Below two seconds no rate is reported at all. The first handful of files arrive microseconds apart, and dividing by that window produces a rate in the gigabytes per second and an estimate of zero, which is worse than showing nothing. Directory entries are excluded from the byte total as well as the file count. rsync reports them with a 4096 inode size, which across six thousand album directories is several megabytes of transfer that never happens.	2026-08-25 12:40:32 +01:00
Emma Thorpe	37b841f009	feat: show which album is copying, and how far through Build and publish container / build (pull_request) Successful in 2m33s Details The transfer looked hung. rsync prints nothing while it builds its file list, which on fifty thousand files over USB is several minutes of silence, and --info=progress2 does not help: with incremental recursion its percentage is computed against a list rsync has not finished discovering, so it moves backwards as often as forwards. The script now counts what needs copying first and says so, then renders its own single line that rewrites in place, showing the album currently going across and a percentage against a total that is actually known. Counting costs a second pass over the tree. That is the price of a percentage meaning something, and it is cheaper than staring at a blank terminal wondering whether the thing has died. Directories are excluded from the count. rsync reports those too, and including them puts the figure past a hundred per cent. Piped to a log the line becomes a plain one every thirty seconds, because a log full of carriage returns and escape codes is not a log anybody reads.	2026-08-25 12:31:13 +01:00
Emma Thorpe	d5dce9c769	fix: accept a destination inside the device, and say so in --help Build and publish container / build (pull_request) Successful in 2m27s Details The script required the destination to be its own mount point, while the documentation and its own usage text both told the user to pass /media/IPOD/Music. The documented invocation was rejected. A subdirectory is the better target, so the guard was what was wrong. --delete is confined to it, and the device path budget is now derived from it -- the part of the destination below its mount point -- rather than configured, so the budget cannot disagree with where the files are actually going. The check that matters is that the destination sits on a FAT filesystem, which is also what catches an unmounted device: /media/IPOD/Music then resolves to the host's own root filesystem, and emptying that is the outcome all of these guards exist to prevent. Three further faults found by testing the guards rather than reasoning about them: Stripping the trailing slash from "/" left an empty string, so the guard refusing the host root never fired and the user got "destination is not a directory" instead. die() printed only its first argument, so the second half of the non-FAT message -- the half saying to check whether the device is mounted -- was silently dropped. --help was not handled at all. Only -h reached the usage text, and it exited 2 to stderr, which is right for misuse and wrong for someone asking a question. Help now goes to stdout and exits zero, and carries the guidance rather than leaving it to the README, since the question it answers is asked at a terminal. The test stage installs bash, rsync and findmnt, none of which are in the base image, and the tests skip rather than fail where they are absent -- a machine without rsync is not a machine that would run this script.	2026-08-25 12:23:52 +01:00
lyrathorpe	8f53a24e1c	chore(release): v0.3.0	2026-08-25 10:57:25 +00:00