Author SHA1 Message Date
Emma ThorpeandClaude Opus 5 c12d2328e5 fix: correct Rockbox's local wall-clock timestamps to UTC
Build and publish container / build (pull_request) Successful in 4m23s
Rockbox has no concept of a timezone. Its clock holds local time, and it
builds log timestamps with mktime(get_time()) -- but firmware/libc/mktime.c
is plain calendar arithmetic applying no offset, so the RTC's local fields
come out as though they were UTC. The number in the log is ahead of the real
instant by whatever the offset was, and Last.fm stores UTC, so every play
submitted during BST landed an hour in the future.

Rockbox states this itself: its scrobbler plugin writes #TZ/UNKNOWN, and the
AUDIOSCROBBLER spec allows #TZ/UTC only for a device that actually converted.
The correction belongs to the consumer.

Each timestamp is decoded back to its wall-clock fields and reinterpreted in
the player's zone, per play rather than as one offset over the whole log, so
a log spanning a daylight saving change converts each side correctly. A log
declaring #TZ/UTC is left alone rather than shifted twice.

The zone defaults to this machine's, overridable with --device-timezone or
ROCKBOX_TIMEZONE. Deriving it needs the whole IANA name: /etc/localtime
resolves into the tzdata tree, and taking only the final component yields
"London", which no database holds, silently falling back to a fixed offset
that is wrong for half the year.

Since Rockbox cannot adjust for daylight saving on its own, the player's
clock has to be changed by hand twice a year. Any play converting to a future
time is now reported, which is what a forgotten adjustment looks like.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-27 15:32:54 +01:00
lyrathorpe 8d6885c46a Merge pull request 'test: prove the database lands at the device root, not in the music folder' (#10) from feat/database-in-sync into main
Reviewed-on: #10
2026-08-26 20:38:53 +01:00
Emma Thorpe 374a17474f fix: never discard a play that has not been submitted
Build and publish container / build (pull_request) Successful in 5m11s
Setting the logs aside after a successful submission threw away more than it
had submitted. A play of a track missing from the mirror -- not yet copied, or
its tags unreadable -- was counted as unresolved and then carried off with the
rest, with nothing to retry it. The play happened and was lost.

Two obligations now, kept separate. The original log is renamed rather than
deleted, so a mistake here cannot destroy the record. And every play that was
not submitted is written back into a live log, so the next run attempts it
again: the unmatched ones, and anything in a batch that failed.

Submission is recorded batch by batch as each is accepted, so a failure partway
through knows exactly what got through. The remainder is written back and
nothing is sent twice. When nothing at all is accepted the logs are left
untouched.

Skips and clockless entries are deliberately not retained. Neither can ever be
submitted, so keeping them would mean reprocessing them for ever, and the
untouched original holds them regardless.

Also fixes a way to lose the lot: read_tags caught OSError and ValueError, but
ffprobe failing raises CalledProcessError, which is neither. A single unreadable
file aborted the whole submission rather than costing one unidentified play.
2026-08-26 18:32:05 +01:00
Emma Thorpe f324b1b720 feat: convert Rockbox's playback log on the laptop, skipping the plugin
Build and publish container / build (pull_request) Successful in 5m6s
Scrobbling previously needed the on-device Last.fm plugin run by hand before
each sync, to turn Rockbox's playback log into AUDIOSCROBBLER format. Forgetting
that step means the sync submits nothing and quietly appears not to work.

Core Rockbox writes ROCKBOX_DIR/playback.log whenever "play log" is enabled,
with no plugin running at all. Each line is timestamp:elapsed_ms:length_ms:path.
The only thing missing is tags, and that is exactly why the plugin exists:
reading them back off the player is slow. Off the mirror it is free, because the
same files are already there -- so the conversion belongs on the laptop, and the
plugin can be skipped entirely.

A play counts as listened at half the track's length, matching the plugin's
savepct default, so the two cannot disagree about what a play was. A short play
is a skip. An entry with no usable timestamp is refused rather than invented,
which is the clockless case again. A path that maps to nothing in the mirror is
counted and reported instead of guessed at.

Rotated logs are picked up too; Rockbox starts a new one past half a megabyte.
All of them are renamed aside together once Last.fm has accepted the batch.

A .scrobbler.log is still read when the plugin has been run and left one.
2026-08-26 18:24:12 +01:00
Emma Thorpe 19ac9e5d92 fix: stop counting by default; the pass costs more than the transfer
Build and publish container / build (pull_request) Successful in 3m32s
The counting pass was added so the progress line could show a percentage and an
estimate, and on a real card it turned out to dominate the run. Measured
against the device: reading a track from the SMB mirror ran at 35 MB/s and
writing to the card at 21 MB/s, while the sync itself managed tens of kilobytes
per second. Neither end was slow. The cost was traversing fifty thousand files
across six thousand directories on FAT, and the counting pass does that a
second time, comparing both trees in full exactly as the transfer does.

Counting is now opt-in behind -P. Without it the progress line still shows the
running count, the transfer rate and the album in flight; the percentage and
the estimate are what needed the extra walk, and they were the least useful
part of the display.

That the fix for "it looks hung" was itself making it slow is the sort of thing
only measuring catches. The line still answers the question it was added for --
whether anything is happening -- without paying for the part that merely made
it prettier.
2026-08-26 13:57:49 +01:00
Emma Thorpe 633fbbaf91 test: prove the database lands at the device root, not in the music folder
Build and publish container / build (pull_request) Successful in 4m30s
The two tools disagree about where the root is. rsync copies artist folders
into <device>/Music, while the database builder must run one level up, where
.rockbox lives, and must record /Music/... paths despite reading the bytes from
the mirror. device_prefix is what reconciles them, and until now that was only
argued rather than demonstrated.

Unprivileged user namespaces make a real bind mount possible, so the test can
create an actual mount point and exercise the derivation instead of asserting
the shape of the script. A stub builder records its working directory and what
it could see. The test asserts the .tcd file arrives beside .rockbox rather
than inside Music, that the build ran in the scratch root and not on the card,
and that it could walk into the mirror through the symlink -- which is the
mechanism that produces device paths from mirror bytes.

It skips where user namespaces are unavailable, which includes the CI
container.
2026-08-26 13:30:26 +01:00
5 changed files with 977 additions and 40 deletions
+91 -11
View File
@@ -178,12 +178,20 @@ entirely for the first two seconds, where the window is microseconds wide and
would report gigabytes per second.
rsync says nothing at all while it builds its file list, which on fifty
thousand files over USB is minutes of apparent hang, and its own `progress2`
percentage is computed against a list it has not finished discovering. So the
script counts first — files and bytes both, a second pass over the tree, which
is what a percentage and an estimate that mean something cost — and renders the
rest itself. Piped to a log it prints a plain line every thirty seconds
instead, with no carriage returns, and a summary at the end either way.
thousand files is minutes of apparent hang, and its own `progress2` percentage
is computed against a list it has not finished discovering. So the script
renders its own.
**The percentage and the estimate are opt-in, via `-P`.** They need a total,
the total needs a counting pass, and that pass walks and compares both trees in
full exactly as the transfer does. Measured on a real card: read from the
source at 35 MB/s and write to the card at 21 MB/s, yet the sync crawled —
because the traversal, not the data, was the cost, and it was being paid twice.
Without `-P` the line still shows the running count, the rate and the album in
flight; only the two figures that needed the second walk are missing.
Piped to a log it prints a plain line every thirty seconds instead, with no
carriage returns, and a summary at the end either way.
### The Rockbox database
@@ -260,7 +268,7 @@ worth doing:
| ----- | --- |
| Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. |
| Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. |
| `-Q` | Skips the counting pass entirely. Costs the percentage and the estimate, saves a whole walk of the tree. |
| Counting is off by default | The percentage costs a second full traversal of both trees. On a FAT card of fifty thousand files that is slower than the transfer. `-P` asks for it. |
| `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. |
**NFS instead of SMB** is worth trying but is not the big win it looks like.
@@ -278,8 +286,18 @@ The unmount is the point of doing this in a script. FAT32 has no journal and
the device is reached through disk mode, so an interrupted write is corruption
that needs `fsck.vfat` from another machine.
`submit_scrobbles.py` sends the Rockbox scrobbler log to Last.fm and sets it
aside. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
`submit_scrobbles.py` sends what was played to Last.fm and sets the logs aside.
It reads **Rockbox's own `playback.log`**, which core Rockbox writes whenever
"play log" is enabled, with no plugin running. Each line is
`timestamp:elapsed_ms:length_ms:path` — a path and nothing else, which is why
the on-device scrobbler plugin exists at all: reading tags back off the player
is slow. Off the mirror it is free, so `--mirror` lets the conversion happen
here and the plugin never has to be run. A play counts as listened at half the
track's length, the same fraction the plugin uses, so the two cannot disagree
about what a play was.
It still reads a `.scrobbler.log` if the plugin has been run and left one. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
tab-separated line per track rated `L` for listened or `S` for skipped; only
the listened ones are sent. It runs **before** the copy, since the plays
already happened and a failed transfer is no reason to lose them.
@@ -291,8 +309,70 @@ real-time clock Rockbox writes `/.scrobbler-timeless.log` with every timestamp
set to zero; those are counted and reported but never submitted, because
scrobbling them would mean inventing when they happened.
The log is renamed rather than deleted once accepted. If Last.fm quietly
dropped something, the evidence is still on the device.
### Timestamps are local wall clock, and are corrected here
Rockbox has no concept of a timezone. Its clock is set to local time, and it
builds log timestamps with `mktime(get_time())` — but
[its `mktime`](https://git.rockbox.org/cgit/rockbox.git/tree/firmware/libc/mktime.c)
is plain calendar arithmetic that applies no offset, so the RTC's local fields
come out as if they were UTC. The number in the log is therefore ahead of the
real instant by whatever the offset was. Last.fm stores UTC, so submitting it
raw puts every play an hour into the future for the half of the year the UK is
on BST.
Rockbox is candid about this: its scrobbler plugin writes `#TZ/UNKNOWN` in the
log header, and the AUDIOSCROBBLER spec says a device may claim `#TZ/UTC` only
if it actually converted. The correction is the consumer's job.
Each timestamp is decoded back into the wall-clock fields it came from and
reinterpreted in the player's zone. Doing it **per play** rather than applying
one offset to the whole log matters: a week's listening can straddle a daylight
saving change, and the two sides need different offsets. A log that declares
`#TZ/UTC` is left alone, so a client that already converted is not shifted
twice.
The zone defaults to this machine's. Set `ROCKBOX_TIMEZONE` (or pass
`--device-timezone`) to an IANA name if the player's clock is keeping a
different one.
Because Rockbox cannot adjust for daylight saving itself, **you have to change
the player's clock by hand twice a year**. If you forget, its times are an hour
out and no amount of zone arithmetic recovers them. The submitter reports any
play that converts to a time in the future, which is what a forgotten
adjustment looks like:
```
37 plays are timestamped up to 58 minutes in the future, converting from
Europe/London. Either the player's clock is wrong or that is not the zone
it is set to.
```
`--dry-run` prints each play's local time beside the epoch, so the conversion
can be checked against when you actually remember listening.
### Nothing played is thrown away
Two separate obligations, because a play that happened and never reached
Last.fm is gone for good.
**The original is renamed, never deleted.** If Last.fm quietly dropped
something, the evidence is still on the device as `playback.log.<ts>.submitted`.
**Anything not submitted is written back** into a live log for the next run:
| Outcome | What happens to it |
| ------------------------------ | ----------------------------------------- |
| Accepted by Last.fm | dropped from the live log |
| Not in the mirror yet | written back, tried again next run |
| In a batch that failed | written back, tried again next run |
| Nothing accepted at all | logs left completely untouched |
| A skip, or no usable timestamp | not retained — neither can ever be submitted, and the original still has it |
The batch boundary matters: submission is recorded as each batch is accepted,
so a failure partway through knows exactly what got through and writes back
only the remainder. No duplicates, no losses.
A file `ffprobe` cannot read costs one unidentified play, not the run.
`check_fat32.py` reports paths a FAT32 device will not accept — reserved
characters, trailing dots and spaces, over-long components and paths, and names
+374
View File
@@ -1,6 +1,9 @@
import json
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path
from zoneinfo import ZoneInfo
import pytest
@@ -187,3 +190,374 @@ def test_write_credentials_are_required(tmp_path, capsys, monkeypatch):
assert code == 2
assert "LASTFM_API_SECRET" in capsys.readouterr().err
PLAYBACK_LOG = """1700000300:180000:245000:/Music/Pendulum/Immersion/01.mp3
1700000200:9000:180000:/Music/Green Day/Dookie/07.mp3
0:180000:245000:/Music/No/Clock/track.mp3
malformed line without colons
"""
def tags_of(artist="Pendulum", title="Watercolour", album="Immersion", track="1/11"):
def runner(path):
return json.dumps(
{"format": {"tags": {"ARTIST": artist, "TITLE": title,
"ALBUM": album, "track": track}}}
)
return runner
def test_the_playback_log_format_is_four_fields():
"""timestamp:elapsed_ms:length_ms:path, written by Rockbox core."""
plays = submit_scrobbles.parse_playback_log(PLAYBACK_LOG)
assert len(plays) == 3 # the malformed line is dropped
assert plays[0] == (1700000300, 180000, 245000, "/Music/Pendulum/Immersion/01.mp3")
def test_a_device_path_maps_onto_the_mirror():
assert submit_scrobbles.device_to_local(
"/Music/Pendulum/Immersion/01.mp3", "/Music", "/mnt/mirror"
) == Path("/mnt/mirror/Pendulum/Immersion/01.mp3")
def test_a_path_outside_the_prefix_is_not_mapped():
"""Something played from elsewhere on the card is not in the mirror."""
assert submit_scrobbles.device_to_local(
"/Podcasts/episode.mp3", "/Music", "/mnt/mirror"
) is None
def test_a_path_at_the_card_root_maps_straight_across():
assert submit_scrobbles.device_to_local(
"/Pendulum/Immersion/01.mp3", "/", "/mnt/mirror"
) == Path("/mnt/mirror/Pendulum/Immersion/01.mp3")
def test_a_short_play_is_a_skip_not_a_scrobble(tmp_path, monkeypatch):
"""Nine seconds of a three minute track. The on-device plugin uses the same
fraction, so the two never disagree about what counted as a play."""
monkeypatch.setattr(Path, "is_file", lambda self: True)
result = submit_scrobbles.plays_from_playback_log(
PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
)
assert result.skipped == 1
assert [entry["timestamp"] for entry in result.played] == ["1700000300"]
def test_a_zero_timestamp_is_refused(tmp_path, monkeypatch):
"""Without a real-time clock Rockbox logs ticks, not dates. Scrobbling
those would mean inventing when they happened."""
monkeypatch.setattr(Path, "is_file", lambda self: True)
result = submit_scrobbles.plays_from_playback_log(
PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
)
assert result.timeless == 1
def test_tags_come_from_the_mirror(tmp_path, monkeypatch):
"""The log carries only a path -- which is precisely why the on-device
plugin exists. Off the mirror the tags are free."""
monkeypatch.setattr(Path, "is_file", lambda self: True)
played = submit_scrobbles.plays_from_playback_log(
PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
).played
assert played[0]["artist"] == "Pendulum"
assert played[0]["album"] == "Immersion"
assert played[0]["trackNumber"] == "1" # "1/11" -> "1"
assert played[0]["duration"] == "245" # milliseconds -> seconds
def test_a_track_missing_from_the_mirror_is_counted_not_guessed(monkeypatch):
monkeypatch.setattr(Path, "is_file", lambda self: False)
result = submit_scrobbles.plays_from_playback_log(
PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
)
assert result.played == []
# One: the skip and the clockless entry are filtered before the file is
# looked for, since neither would be submitted either way.
assert result.unresolved == 1
# And that one play is kept, so a later run can try it again.
assert len(result.retain) == 1
LONDON = ZoneInfo("Europe/London")
# Rockbox builds timestamps with mktime(get_time()), and its mktime applies no
# zone at all, so the number is the RTC's local wall clock read as if it were
# UTC. Under BST that puts every play an hour ahead of when it happened.
SUMMER_LOGGED = 1787839200 # 2026-08-27 14:00 written by the device
SUMMER_TRUE = 1787835600 # the instant that actually was, 13:00 UTC
WINTER_LOGGED = 1796479200 # 2026-12-05 14:00, when London is already UTC
WINTER_TRUE = 1796479200
def test_a_summer_timestamp_is_pulled_back_to_real_utc():
"""The device logged 14:00 local. That was 13:00 UTC, and UTC is what
Last.fm stores."""
assert submit_scrobbles.device_time_to_utc(SUMMER_LOGGED, LONDON) == SUMMER_TRUE
def test_a_winter_timestamp_is_left_alone():
"""London keeps UTC for half the year, so there is nothing to correct and
the correction must not invent an offset anyway."""
assert submit_scrobbles.device_time_to_utc(WINTER_LOGGED, LONDON) == WINTER_TRUE
def test_the_converted_time_reads_back_as_the_clock_the_device_showed():
"""The round trip, which is the property that actually matters: whatever
the player's screen said is what Last.fm should show in local time."""
shown = datetime.fromtimestamp(SUMMER_LOGGED, timezone.utc)
corrected = submit_scrobbles.device_time_to_utc(SUMMER_LOGGED, LONDON)
assert datetime.fromtimestamp(corrected, LONDON).strftime("%Y-%m-%d %H:%M") == (
shown.strftime("%Y-%m-%d %H:%M")
)
def test_one_log_spanning_a_clock_change_converts_each_side_separately():
"""A week's listening either side of the October change carries two
different offsets. Correcting the log by a single figure would put half of
it an hour out, which is why the offset is resolved per play."""
before, after = 1792888200, 1792899000 # 2026-10-25, 00:30 BST and 03:30 GMT
assert submit_scrobbles.device_time_to_utc(before, LONDON) == before - 3600
assert submit_scrobbles.device_time_to_utc(after, LONDON) == after
def test_a_playback_log_is_corrected_before_submission(monkeypatch):
monkeypatch.setattr(Path, "is_file", lambda self: True)
log = f"{SUMMER_LOGGED}:180000:245000:/Music/Pendulum/Immersion/01.mp3\n"
played = submit_scrobbles.plays_from_playback_log(
log, "/Music", "/mnt/mirror", runner=tags_of(), zone=LONDON
).played
assert played[0]["timestamp"] == str(SUMMER_TRUE)
def test_a_scrobbler_log_claiming_utc_is_not_corrected_twice():
"""The format's header exists for exactly this. A client that already did
the conversion says so, and correcting it again would break it."""
log = LOG.replace("#TZ/UNKNOWN", "#TZ/UTC")
played, _, _ = submit_scrobbles.parse_log(log, LONDON)
assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"]
def test_a_scrobbler_log_declaring_unknown_is_corrected():
"""Rockbox writes UNKNOWN, meaning local wall clock, so the times are ours
to fix."""
played, _, _ = submit_scrobbles.parse_log(LOG, LONDON)
# 1700000100 and 1700000300 are November, when London is on UTC anyway;
# the point is that the header did not exempt them from being looked at.
assert submit_scrobbles.declares_utc(LOG) is False
assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"]
def test_the_system_zone_keeps_its_whole_name(monkeypatch, tmp_path):
"""/etc/localtime points into the tzdata tree. Taking only the last
component gives "London", which no database holds -- and the fallback for
an unknown name is a fixed offset, which is wrong for half the year. The
region has to survive."""
zoneinfo_dir = tmp_path / "share" / "zoneinfo" / "Europe"
zoneinfo_dir.mkdir(parents=True)
(zoneinfo_dir / "London").write_bytes(b"TZif")
monkeypatch.setattr(
submit_scrobbles.Path,
"read_text",
lambda self, **kw: (_ for _ in ()).throw(OSError),
)
monkeypatch.setattr(
submit_scrobbles.Path,
"resolve",
lambda self: zoneinfo_dir / "London",
)
assert submit_scrobbles.system_zone_name() == "Europe/London"
def test_a_named_zone_beats_the_machine_default():
"""The player may not be in the same place as the laptop."""
assert submit_scrobbles.device_zone("Asia/Tokyo") == ZoneInfo("Asia/Tokyo")
def test_plays_dated_after_now_are_reported():
"""A clock never put forward, or the wrong zone, produces plays that have
not happened yet. Last.fm cannot tell those from real ones."""
now = 1787835600
played = [
{"timestamp": str(now - 60)},
{"timestamp": str(now + 3600)},
]
ahead = submit_scrobbles.future_plays(played, now=now)
assert [entry["timestamp"] for entry in ahead] == [str(now + 3600)]
def test_a_clock_a_minute_fast_is_not_reported():
"""Devices drift. Only an offset large enough to be a zone error matters."""
now = 1787835600
assert submit_scrobbles.future_plays([{"timestamp": str(now + 60)}], now=now) == []
def test_playback_logs_are_found_including_rotations(tmp_path):
"""Rockbox rotates the log once it passes half a megabyte."""
rockbox = tmp_path / ".rockbox"
rockbox.mkdir()
for name in ("playback.log", "playback_0001.log", "playback_0002.log"):
(rockbox / name).write_text("1700000000:1:1:/Music/a.mp3\n")
(rockbox / "empty.log").write_text("")
found = submit_scrobbles.find_playback_logs(tmp_path)
assert [path.name for path in found] == [
"playback.log", "playback_0001.log", "playback_0002.log"
]
def test_without_a_mirror_it_says_what_is_needed(tmp_path, capsys):
device = tmp_path / "IPOD"
(device / ".rockbox").mkdir(parents=True)
(device / ".rockbox" / "playback.log").write_text("1700000000:1:1:/Music/a.mp3\n")
submit_scrobbles.main([str(device)], transport=fake_transport([]))
assert "pass --mirror" in capsys.readouterr().err
MIRROR = "/mnt/mirror"
def stub_ffprobe(monkeypatch, **tags):
"""Answer for anything under the mirror without invoking ffprobe."""
monkeypatch.setattr(submit_scrobbles, "_ffprobe", tags_of(**tags))
def only_mirror_files_exist(monkeypatch, resolvable=None):
"""Make the mirror's files appear to exist, and nothing else.
Patching is_file wholesale makes the device directory look like a log file,
which sends main() down the .scrobbler.log path instead.
"""
real = Path.is_file
def patched(self):
text = str(self)
if text.startswith(MIRROR):
return resolvable is None or text == resolvable
return real(self)
monkeypatch.setattr(Path, "is_file", patched)
def playback_device(tmp_path, log=PLAYBACK_LOG):
device = tmp_path / "IPOD"
(device / ".rockbox").mkdir(parents=True)
(device / ".rockbox" / "playback.log").write_text(log)
return device
def test_a_failed_submission_keeps_every_log(tmp_path, monkeypatch):
"""Nothing got through, so nothing may be set aside."""
only_mirror_files_exist(monkeypatch)
stub_ffprobe(monkeypatch)
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
device = playback_device(tmp_path)
transport = fake_transport([{"error": 29, "message": "Rate limit"}])
code = submit_scrobbles.main(
[str(device), "--mirror", MIRROR, "--api-key", "k", "--api-secret", "s"],
transport=transport,
)
assert code == 1
assert (device / ".rockbox" / "playback.log").is_file()
assert not list((device / ".rockbox").glob("*.submitted"))
def test_an_unmatched_play_is_written_back_not_lost(tmp_path, monkeypatch, capsys):
"""A track the mirror does not yet hold is still a play that happened. It
is kept so a later run, after the file has been copied, can submit it."""
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
# Only the first track resolves; the rest are absent from the mirror.
only_mirror_files_exist(monkeypatch, f"{MIRROR}/Pendulum/Immersion/01.mp3")
stub_ffprobe(monkeypatch)
log = (
"1700000300:180000:245000:/Music/Pendulum/Immersion/01.mp3\n"
"1700000400:180000:245000:/Music/Missing/Album/09.mp3\n"
)
device = playback_device(tmp_path, log)
transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
code = submit_scrobbles.main(
[str(device), "--mirror", MIRROR, "--api-key", "k", "--api-secret", "s"],
transport=transport,
)
assert code == 0
rockbox = device / ".rockbox"
# The original is preserved untouched...
assert list(rockbox.glob("playback.log.*.submitted"))
# ...and the unmatched play is back in a live log for the next attempt.
written = (rockbox / "playback.log").read_text()
assert "Missing/Album/09.mp3" in written
assert "Pendulum/Immersion/01.mp3" not in written
def test_the_original_is_renamed_rather_than_deleted(tmp_path, monkeypatch):
"""If Last.fm quietly dropped something, the evidence stays on the device."""
only_mirror_files_exist(monkeypatch)
stub_ffprobe(monkeypatch)
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
device = playback_device(tmp_path)
transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
submit_scrobbles.main(
[str(device), "--mirror", MIRROR, "--api-key", "k", "--api-secret", "s"],
transport=transport,
)
aside = list((device / ".rockbox").glob("playback.log.*.submitted"))
assert len(aside) == 1
assert PLAYBACK_LOG.splitlines()[0] in aside[0].read_text()
def test_keep_leaves_everything_alone(tmp_path, monkeypatch):
only_mirror_files_exist(monkeypatch)
stub_ffprobe(monkeypatch)
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
device = playback_device(tmp_path)
transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
submit_scrobbles.main(
[str(device), "--mirror", MIRROR, "--keep",
"--api-key", "k", "--api-secret", "s"],
transport=transport,
)
assert (device / ".rockbox" / "playback.log").read_text() == PLAYBACK_LOG
assert not list((device / ".rockbox").glob("*.submitted"))
def test_a_file_ffprobe_cannot_read_does_not_abandon_the_rest(monkeypatch):
"""CalledProcessError is not an OSError, so one bad file used to take the
whole submission with it."""
def broken(path):
raise subprocess.CalledProcessError(1, "ffprobe")
assert submit_scrobbles.read_tags(Path("/mnt/mirror/x.mp3"), runner=broken) == {}
+80 -5
View File
@@ -176,16 +176,16 @@ def test_the_help_says_how_to_reach_and_leave_disk_mode():
assert "holding Play" in help_text
def test_quick_mode_skips_the_counting_pass(mirror, tmp_path):
"""Over SMB the walk is the expensive part, and doing it twice for a
percentage is not always the trade you want."""
def test_counting_is_off_by_default(mirror, tmp_path):
"""The counting pass walks and compares both trees exactly as the transfer
does. On a FAT card of fifty thousand files that costs more than moving the
data, so the percentage has to be asked for."""
destination = tmp_path / "dest"
destination.mkdir()
result = run("-f", "-S", "-U", "-Q", str(mirror), str(destination))
result = run("-f", "-S", "-U", str(mirror), str(destination))
assert result.returncode == 0, result.stderr
assert "skipping the count" in result.stderr
assert "files to copy" not in result.stderr
assert (destination / "Album" / "track.mp3").is_file()
@@ -271,3 +271,78 @@ def test_the_scan_reads_from_the_mirror_not_the_device():
assert 'ln -s "$mirror"' in script
assert 'cd "$scratch"' in script
def can_bind_mount():
"""User namespaces let an unprivileged process bind mount. Not everywhere,
notably not inside some containers, so the test that needs it skips."""
return (
subprocess.run(
["unshare", "-Umr", "true"], capture_output=True, check=False
).returncode
== 0
)
@pytest.mark.skipif(not can_bind_mount(), reason="needs unprivileged user namespaces")
def test_the_database_lands_at_the_device_root_not_the_music_folder(tmp_path):
"""The two tools disagree about where the root is. rsync copies artist
folders into <device>/Music; the database tool must run one level up, where
.rockbox lives, and must record /Music/... paths while reading the bytes
from the mirror. device_prefix is what reconciles them.
"""
mirror = tmp_path / "mirror" / "Pendulum" / "Immersion"
mirror.mkdir(parents=True)
(mirror / "01.mp3").write_bytes(b"not really an mp3")
card = tmp_path / "card"
(card / ".rockbox").mkdir(parents=True)
(card / "Music").mkdir()
device = tmp_path / "device"
device.mkdir()
tool = tmp_path / "fake-database"
# Records where it was run and what it could see, which is the whole
# question; producing a real database needs Rockbox's builder.
tool.write_text(
"#!/bin/sh\n"
"printf '%s\\n' \"$PWD\" > .rockbox/where.txt\n"
"ls Music/ > .rockbox/saw.txt\n"
"echo db > .rockbox/database_0.tcd\n"
)
tool.chmod(0o755)
script = (
f"mount --bind {card} {device} && "
f"XDG_CACHE_HOME={tmp_path / 'cache'} MUSIC_MIRROR_DATABASE_TOOL={tool} "
f"bash {SCRIPT} -f -S -U {tmp_path / 'mirror'} {device / 'Music'}"
)
result = subprocess.run(
["unshare", "-Umr", "sh", "-c", script], capture_output=True, text=True
)
assert result.returncode == 0, result.stderr
# The database lands beside the device root, not inside Music.
assert (card / ".rockbox" / "database_0.tcd").is_file()
# Only *.tcd is copied across, so the markers stay in the scratch root --
# which is itself the point: nothing else is written to the device.
scratch = tmp_path / "cache" / "music-mirror" / "database" / ".rockbox"
assert not (card / ".rockbox" / "where.txt").exists()
# It ran in the scratch root, not on the card.
where = (scratch / "where.txt").read_text().strip()
assert where.endswith("music-mirror/database"), where
# ...and could walk into the mirror through a symlink named for the device
# prefix, which is how the paths come out as /Music/... while the bytes are
# read from somewhere else entirely.
assert "Pendulum" in (scratch / "saw.txt").read_text()
def test_counting_can_be_asked_for(mirror, tmp_path):
"""When the destination is cheap to traverse, the percentage is worth it."""
destination = tmp_path / "dest"
destination.mkdir()
result = run("-f", "-S", "-U", "-P", str(mirror), str(destination))
assert result.returncode == 0, result.stderr
assert "files to copy" in result.stderr
+403 -15
View File
@@ -14,12 +14,16 @@ import argparse
import hashlib
import json
import os
import subprocess
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
API_ROOT = "https://ws.audioscrobbler.com/2.0/"
@@ -31,6 +35,26 @@ BATCH = 50
# scrobble.
LOG_NAMES = (".scrobbler.log", ".scrobbler-timeless.log")
# Rockbox core writes this whenever "play log" is on, with no plugin running:
# timestamp:elapsed_ms:length_ms:/Music/Artist/Album/Track.mp3
# It is rotated once it grows past half a megabyte. Converting it needs tags,
# which is why the on-device plugin exists -- reading them back off the player
# is slow. Off the mirror it is free, so the plugin can be skipped entirely.
PLAYBACK_LOG_NAMES = ("playback.log", "playback_*.log")
# The plugin counts a track as listened at savepct of its length, defaulting to
# fifty. Same rule here, or the two disagree about what a play is.
LISTENED_FRACTION = 0.5
# Below this a timestamp is not a wall-clock time. Without a real-time clock
# Rockbox logs ticks in milliseconds instead, which is not a date.
EARLIEST_PLAUSIBLE = 1_000_000_000
# How far ahead of now a converted play may sit before it is reported. Some
# slack absorbs a device clock drifting by a minute or two; an hour out means
# the zone is wrong or the clock was never put forward.
FUTURE_TOLERANCE_SECONDS = 300
SESSION_FILE = Path(
os.getenv("XDG_CONFIG_HOME", Path.home() / ".config")
) / "music-mirror" / "lastfm.json"
@@ -40,14 +64,117 @@ class LastfmError(Exception):
"""A Last.fm request that failed."""
def parse_log(text):
def system_zone_name():
"""Return this machine's IANA zone name, or "" if nothing states it.
The name has to come out whole. /etc/localtime is a symlink into the
tzdata tree, so the part after `zoneinfo/` is the name -- taking only the
last component yields "London", which no database has, and falls back to a
fixed offset that would then be wrong for half the year.
"""
try:
named = Path("/etc/timezone").read_text(encoding="utf-8").strip()
if named:
return named
except OSError:
pass
try:
parts = Path("/etc/localtime").resolve().parts
except OSError:
return ""
if "zoneinfo" in parts:
return "/".join(parts[len(parts) - parts[::-1].index("zoneinfo"):])
return ""
def device_zone(name=None):
"""Return the zone the device's clock is keeping.
Rockbox has no concept of a timezone, so its clock is set to local wall
time and the zone has to be supplied from outside. Defaulting to this
machine's zone is right whenever the player and the laptop are in the same
place, which for a device synced by cable they are.
"""
if name:
return ZoneInfo(name)
for candidate in (os.getenv("TZ"), system_zone_name()):
if not candidate:
continue
try:
return ZoneInfo(candidate)
except (ZoneInfoNotFoundError, ValueError):
continue
# Nothing named the zone, so the offset cannot be resolved per play: this
# is today's offset applied to every timestamp, which is wrong either side
# of a daylight saving change. Still better than pretending it logged UTC.
print(
"warning: no IANA timezone found for this machine; using its current"
" offset for every play. Pass --device-timezone to fix older plays"
" across a daylight saving change.",
file=sys.stderr,
)
return datetime.now().astimezone().tzinfo
def device_time_to_utc(stamp, zone):
"""Return the true UTC epoch of a timestamp Rockbox wrote.
Rockbox builds its timestamps with `mktime(get_time())`, and its mktime
(firmware/libc/mktime.c) is plain calendar arithmetic with no zone applied.
Fed the RTC's local fields it yields local-wall-clock-as-if-UTC, so the
number is ahead of real UTC by whatever the offset was. Its own scrobbler
plugin admits this by writing `#TZ/UNKNOWN`, leaving the correction here.
Decoding the number back into those fields and reinterpreting them in the
device's zone recovers the instant, and does so per play, so a log
straddling a daylight-saving change converts each side by its own offset.
"""
fields = datetime.fromtimestamp(stamp, timezone.utc).replace(tzinfo=zone)
return int(fields.timestamp())
def future_plays(played, now=None):
"""Return the plays timestamped later than now, which cannot have happened.
A device clock left on the wrong offset, or never adjusted across a
daylight-saving change, shows up here. Last.fm has no way to tell such a
scrobble from a real one, so it is worth saying out loud.
"""
now = time.time() if now is None else now
return [
entry for entry in played
if int(entry["timestamp"]) > now + FUTURE_TOLERANCE_SECONDS
]
def declares_utc(text):
"""Whether an AUDIOSCROBBLER log says its timestamps are already UTC.
The format's header carries `#TZ/UTC` or `#TZ/UNKNOWN`, and its spec is
explicit that a device may only claim UTC if it converted. Rockbox writes
UNKNOWN, meaning the times are local wall clock and want correcting; a log
from anything that claims UTC must be left alone.
"""
for line in text.splitlines():
if not line.startswith("#"):
break
if line.strip().upper().startswith("#TZ/"):
return line.strip().upper() == "#TZ/UTC"
return False
def parse_log(text, zone=None):
"""Return the listened tracks in an AUDIOSCROBBLER log, oldest first.
Fields are artist, album, title, track number, length, rating, timestamp
and MusicBrainz id. Rockbox converts any tab inside a field to a space
before writing, so splitting on tabs is safe.
Timestamps are corrected from the device's local wall clock to UTC unless
the log's own header claims it did that already.
"""
played, skipped, timeless = [], 0, 0
convert = zone is not None and not declares_utc(text)
for line in text.splitlines():
if not line or line.startswith("#"):
continue
@@ -68,6 +195,8 @@ def parse_log(text):
continue
if not artist or not title:
continue
if convert:
when = device_time_to_utc(when, zone)
played.append(
{
"artist": artist,
@@ -83,6 +212,136 @@ def parse_log(text):
return played, skipped, timeless
def parse_playback_log(text):
"""Return (timestamp, elapsed_ms, length_ms, path) for each logged play."""
plays = []
for line in text.splitlines():
fields = line.strip().split(":", 3)
if len(fields) != 4:
continue
stamp, elapsed, length, path = fields
try:
plays.append((int(stamp), int(elapsed), int(length), path))
except ValueError:
continue
return plays
def device_to_local(path, device_prefix, mirror):
"""Map a path as the player sees it onto the mirror it was copied from."""
prefix = "/" + device_prefix.strip("/")
if prefix != "/":
if not path.startswith(prefix + "/"):
return None
path = path[len(prefix) :]
return Path(mirror) / path.lstrip("/")
def read_tags(path, runner=None):
"""Return the tags of a local file, via ffprobe. Empty if it cannot be read.
A file ffprobe chokes on is one play left unidentified, not a reason to
abandon the rest -- and CalledProcessError is not an OSError, so catching
the obvious things is not enough.
"""
runner = runner or _ffprobe
try:
payload = json.loads(runner(path))
except (OSError, ValueError, subprocess.SubprocessError):
return {}
return {
key.lower(): value
for key, value in (payload.get("format", {}).get("tags") or {}).items()
}
def _ffprobe(path):
return subprocess.run(
["ffprobe", "-v", "error", "-show_entries", "format_tags",
"-of", "json", str(path)],
capture_output=True, text=True, check=True,
).stdout
@dataclass
class Conversion:
"""What a playback log turned into, and what must not be thrown away.
`retain` holds the raw lines of plays that were real but could not be
submitted -- a track absent from the mirror, usually because the sync had
not copied it yet. Those are written back so a later run can try again.
Skips and clockless entries are not retained: neither can ever be
submitted, and the untouched original is set aside regardless.
"""
played: list
skipped: int = 0
unresolved: int = 0
timeless: int = 0
retain: list = None
def __post_init__(self):
if self.retain is None:
self.retain = []
def plays_from_playback_log(text, device_prefix, mirror, runner=None, zone=None):
"""Return the conversion of a playback log.
Skips are decided by the same fraction the on-device plugin uses, so the
two never disagree about what counted as a play. Timestamps are corrected
from the device's wall clock to UTC; the core log has no header to say so,
but it is written the same way the plugin's UNKNOWN times are.
"""
result = Conversion(played=[])
for line in text.splitlines():
parsed = parse_playback_log(line)
if not parsed:
continue
stamp, elapsed, length, device_path = parsed[0]
if stamp < EARLIEST_PLAUSIBLE:
result.timeless += 1
continue
if length > 0 and elapsed < length * LISTENED_FRACTION:
result.skipped += 1
continue
local = device_to_local(device_path, device_prefix, mirror)
tags = read_tags(local, runner) if local and local.is_file() else {}
artist = tags.get("artist") or tags.get("album_artist") or ""
title = tags.get("title") or ""
if not artist or not title:
# A real play of a track this run could not identify. Kept, so a
# later run -- after the file has been copied, or the tags fixed --
# can submit it rather than the play being lost.
result.unresolved += 1
result.retain.append(line)
continue
result.played.append(
{
"artist": artist,
"track": title,
"album": tags.get("album", ""),
"trackNumber": (tags.get("track") or "").split("/")[0],
"duration": str(length // 1000) if length > 0 else "",
"timestamp": str(
device_time_to_utc(stamp, zone) if zone is not None else stamp
),
"mbid": tags.get("musicbrainz_trackid", ""),
"line": line,
}
)
result.played.sort(key=lambda entry: int(entry["timestamp"]))
return result
def find_playback_logs(device):
"""Return every playback log on a device, oldest first."""
found = []
for pattern in PLAYBACK_LOG_NAMES:
found.extend(sorted(Path(device).glob(f".rockbox/{pattern}")))
return [path for path in found if path.is_file() and path.stat().st_size]
def sign(params, secret):
"""Return Last.fm's method signature for a set of parameters.
@@ -152,7 +411,7 @@ def batch_params(entries):
return params
def submit(entries, key, secret, session, transport, delay=1.0):
def submit(entries, key, secret, session, transport, delay=1.0, on_sent=None):
"""Submit every entry. Returns how many the service accepted.
Batches are counted as they succeed rather than at the end, so a failure
@@ -168,6 +427,8 @@ def submit(entries, key, secret, session, transport, delay=1.0):
block = payload.get("scrobbles", {})
summary = block.get("@attr", block)
accepted += int(summary.get("accepted", len(chunk)))
if on_sent is not None:
on_sent(chunk)
ignored = int(summary.get("ignored", 0))
if ignored:
print(f" {ignored} of {len(chunk)} ignored by Last.fm", file=sys.stderr)
@@ -193,6 +454,26 @@ def find_log(device):
def main(argv=None, transport=http_post):
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("device", help="the mounted device, or a scrobbler log file")
parser.add_argument(
"--mirror",
help="the mirror the device was copied from. Given this, Rockbox's own"
" playback.log is converted here rather than needing the on-device"
" plugin run first",
)
parser.add_argument(
"--device-prefix",
default="/Music",
help="where the music sits on the device, stripped when mapping a logged"
" path back onto the mirror",
)
parser.add_argument(
"--device-timezone",
default=os.getenv("ROCKBOX_TIMEZONE"),
help="the zone the player's clock is set to, as an IANA name such as"
" Europe/London. Rockbox keeps local wall time and cannot record an"
" offset, so its timestamps need this to become the UTC Last.fm wants."
" Defaults to this machine's zone",
)
parser.add_argument("--api-key", default=os.getenv("LASTFM_API_KEY"))
parser.add_argument("--api-secret", default=os.getenv("LASTFM_API_SECRET"))
parser.add_argument("--dry-run", action="store_true", help="parse and report only")
@@ -201,14 +482,61 @@ def main(argv=None, transport=http_post):
)
args = parser.parse_args(argv)
try:
zone = device_zone(args.device_timezone)
except (ZoneInfoNotFoundError, ValueError) as error:
print(f"unknown timezone {args.device_timezone!r}: {error}", file=sys.stderr)
return 2
target = Path(args.device)
log = target if target.is_file() else find_log(target)
if log is None:
print("no scrobbler log to submit", file=sys.stderr)
logs = []
conversion = None
if log is not None:
played, skipped, timeless = parse_log(
log.read_text(encoding="utf-8", errors="replace"), zone
)
unresolved = 0
logs = [log]
print(f"{log}: {len(played)} listened, {skipped} skipped", file=sys.stderr)
elif args.mirror:
# No plugin has been run, but the core log is there. Tags come off the
# mirror, which is the only reason the plugin was needed at all.
logs = find_playback_logs(target)
if not logs:
print("no scrobbler log to submit", file=sys.stderr)
return 0
text = "\n".join(
path.read_text(encoding="utf-8", errors="replace") for path in logs
)
conversion = plays_from_playback_log(
text, args.device_prefix, args.mirror, zone=zone
)
played = conversion.played
skipped, unresolved, timeless = (
conversion.skipped,
conversion.unresolved,
conversion.timeless,
)
print(
f"{len(logs)} playback log(s): {len(played)} listened, {skipped} skipped",
file=sys.stderr,
)
if unresolved:
print(
f" {unresolved} could not be matched to a file in the mirror."
" Those plays are kept for a later run rather than discarded.",
file=sys.stderr,
)
else:
print(
"no scrobbler log to submit. Rockbox's own playback.log can be used"
" instead -- pass --mirror so tags can be read from it.",
file=sys.stderr,
)
return 0
played, skipped, timeless = parse_log(log.read_text(encoding="utf-8", errors="replace"))
print(f"{log}: {len(played)} listened, {skipped} skipped", file=sys.stderr)
if timeless:
print(
f" {timeless} entries have no timestamp, so this target has no clock."
@@ -217,9 +545,24 @@ def main(argv=None, transport=http_post):
)
if not played:
return 0
ahead = future_plays(played)
if ahead:
newest = int(ahead[-1]["timestamp"]) - int(time.time())
print(
f" {len(ahead)} plays are timestamped up to {newest // 60} minutes in"
f" the future, converting from {zone}. Either the player's clock is"
" wrong or that is not the zone it is set to.",
file=sys.stderr,
)
if args.dry_run:
for entry in played[:20]:
print(f"{entry['timestamp']}\t{entry['artist']}\t{entry['track']}")
when = datetime.fromtimestamp(int(entry["timestamp"]), zone)
print(
f"{entry['timestamp']}\t{when:%Y-%m-%d %H:%M %Z}"
f"\t{entry['artist']}\t{entry['track']}"
)
return 0
if not args.api_key or not args.api_secret:
@@ -235,21 +578,66 @@ def main(argv=None, transport=http_post):
session = authorise(args.api_key, args.api_secret, transport)
save_session(session)
# Recorded as each batch is accepted, so a failure partway through knows
# exactly what got through and what did not.
sent = []
try:
accepted = submit(played, args.api_key, args.api_secret, session, transport)
accepted = submit(
played, args.api_key, args.api_secret, session, transport,
on_sent=sent.extend,
)
except LastfmError as error:
print(f"submission failed: {error}", file=sys.stderr)
print(f"submission failed after {len(sent)} scrobbles: {error}", file=sys.stderr)
keep_history(logs, played, sent, conversion, args.keep)
return 1
print(f"{accepted} scrobbles accepted", file=sys.stderr)
if not args.keep and accepted:
# Renamed rather than deleted: if Last.fm quietly dropped something,
# the evidence is still on the device.
aside = log.with_name(f"{log.name}.{played[-1]['timestamp']}.submitted")
log.rename(aside)
print(f"log moved to {aside.name}", file=sys.stderr)
keep_history(logs, played, sent, conversion, args.keep)
return 0
def keep_history(logs, played, sent, conversion, keep):
"""Set the logs aside, writing back anything still owed a submission.
Two separate obligations. The original is preserved untouched, renamed
rather than deleted, so a play is never lost to a mistake here. And any
play that was not submitted -- unmatched, or in a batch that failed -- is
written back into a live log, so the next run tries it again instead of it
quietly vanishing with the rest.
"""
if keep or not logs:
if keep:
print("logs left in place", file=sys.stderr)
return
submitted = {id(entry) for entry in sent}
# A .scrobbler.log was converted by the on-device plugin and carries no
# per-line record, so there is nothing to write back for it -- only the
# rename below, which loses nothing.
pending = list(conversion.retain) if conversion is not None else []
pending += [
entry["line"] for entry in played
if "line" in entry and id(entry) not in submitted
]
if not sent:
print("nothing was accepted; logs left untouched", file=sys.stderr)
return
stamp = played[-1]["timestamp"] if played else "0"
for path in logs:
path.rename(path.with_name(f"{path.name}.{stamp}.submitted"))
if pending:
live = logs[0].with_name("playback.log")
live.write_text("\n".join(pending) + "\n", encoding="utf-8")
print(
f"{len(pending)} plays not submitted were written back to"
f" {live.name} for the next run",
file=sys.stderr,
)
print(f"{len(logs)} log(s) set aside as .submitted", file=sys.stderr)
if __name__ == "__main__":
sys.exit(main())
+29 -9
View File
@@ -22,8 +22,9 @@ usage() {
usage: sync-to-ipod.sh [options] <mirror> <destination>
-n dry run; show what would change and touch nothing
-Q skip the counting pass; no percentage or estimate, but one less walk
of the source tree, which over SMB is the expensive part
-P count what needs copying first, so progress can show a percentage and
an estimate. Costs a second full traversal of both trees, which on a
FAT card of fifty thousand files is slower than the transfer itself
-f copy even if the FAT32 check finds unacceptable paths
-S skip submitting the Rockbox scrobbler log to Last.fm
-B skip rebuilding the Rockbox database
@@ -40,6 +41,12 @@ Submitting scrobbles needs LASTFM_API_KEY and LASTFM_API_SECRET; it is skipped
with a note when they are unset. Scrobbling is a write method and needs the
secret, unlike the read-only calls elsewhere in these projects.
Rockbox has no notion of a timezone: its clock holds local wall time and its
logs record that, not UTC. Set ROCKBOX_TIMEZONE to the zone the player's clock
is keeping (an IANA name, such as Europe/London) if it differs from this
machine's, which is otherwise assumed. Getting it wrong shifts every scrobble
by the difference.
The mirror is the directory holding the artist folders. The destination is
where those folders should end up on the device -- not the card root, unless
that is genuinely where you want them:
@@ -68,7 +75,7 @@ USAGE
}
dry_run=false
quick=false
counting=false
force=false
unmount=true
scrobble=true
@@ -76,10 +83,10 @@ database=true
for argument in "$@"; do
[ "$argument" = "--help" ] && usage help
done
while getopts ":nQfSBUh" option; do
while getopts ":nPfSBUh" option; do
case "$option" in
n) dry_run=true ;;
Q) quick=true ;;
P) counting=true ;;
f) force=true ;;
S) scrobble=false ;;
B) database=false ;;
@@ -157,7 +164,16 @@ if $scrobble; then
else
scrobble_options=()
$dry_run && scrobble_options+=(--dry-run)
python3 "$here/submit_scrobbles.py" "${scrobble_options[@]}" "$destination" ||
# Rockbox keeps local wall time with no notion of a zone, so its
# timestamps are not the UTC Last.fm expects. Naming the zone the
# player's clock is set to lets them be corrected.
[ -n "${ROCKBOX_TIMEZONE:-}" ] &&
scrobble_options+=(--device-timezone "$ROCKBOX_TIMEZONE")
# --mirror lets it convert Rockbox's own playback.log, so the on-device
# scrobbler plugin never has to be run. The device root, not the music
# directory: the logs live in .rockbox.
python3 "$here/submit_scrobbles.py" "${scrobble_options[@]}" \
--mirror "$mirror" --device-prefix "$device_prefix" "$mounted_on" ||
die "submitting scrobbles failed; nothing has been copied"
fi
fi
@@ -198,11 +214,15 @@ fi
# thousand files over USB is minutes of apparent hang. Counting first costs a
# second pass over the tree but means the transfer can show a real percentage
# rather than a number that grows as rsync discovers more work.
# Counting is opt-in because it is not cheap. It walks and compares both trees
# in full, exactly as the transfer does, and on a FAT card holding fifty
# thousand files that traversal costs more than moving the data. Without it the
# progress line still shows the running count, the rate and the album in
# flight; only the percentage and the estimate are lost, and those were the
# least useful part of it.
total=0
total_bytes=0
if $quick; then
printf 'sync-to-ipod: skipping the count; no percentage or estimate\n' >&2
else
if $counting; then
printf 'sync-to-ipod: working out what needs copying...\n' >&2
# %l is the file's size, which is what makes an estimate possible.
# Directories are dropped: rsync reports those too, with an inode size that