Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Emma Thorpe
|
19b5c750b7 |
@@ -26,9 +26,6 @@ ENTRYPOINT ["music-mirror"]
|
||||
FROM runtime AS test
|
||||
|
||||
RUN pip install --no-cache-dir pytest
|
||||
# sync-to-ipod.sh and its tests need these; the runtime image deliberately does
|
||||
# not carry them, and neither does the base.
|
||||
RUN apk add --no-cache bash rsync findmnt
|
||||
COPY pytest.ini ./
|
||||
# Host-side tools; not in the runtime image, but the suite covers them.
|
||||
COPY tools ./tools
|
||||
|
||||
@@ -163,141 +163,12 @@ and the various filesystem metadata directories from deletion — the mirror doe
|
||||
not contain them, and without the exclusion a sync to the card root would
|
||||
remove the Rockbox install.
|
||||
|
||||
Progress is a single line that rewrites itself:
|
||||
|
||||
```
|
||||
[ 24%] 12,345/49,600 3.2 GiB/13.1 GiB 4.4 MiB/s ETA 38m12s King Gizzard / Petro…
|
||||
```
|
||||
|
||||
The estimate comes from rsync's `%l`, which gives each file's size as it
|
||||
completes. Bytes done over time elapsed is the same arithmetic rsync would do,
|
||||
and needs nothing it does not already print. The rate is measured over a
|
||||
trailing thirty seconds rather than the whole run, so it follows a device that
|
||||
slows down instead of averaging the slowdown away — and it is suppressed
|
||||
entirely for the first two seconds, where the window is microseconds wide and
|
||||
would report gigabytes per second.
|
||||
|
||||
rsync says nothing at all while it builds its file list, which on fifty
|
||||
thousand files is minutes of apparent hang, and its own `progress2` percentage
|
||||
is computed against a list it has not finished discovering. So the script
|
||||
renders its own.
|
||||
|
||||
**The percentage and the estimate are opt-in, via `-P`.** They need a total,
|
||||
the total needs a counting pass, and that pass walks and compares both trees in
|
||||
full exactly as the transfer does. Measured on a real card: read from the
|
||||
source at 35 MB/s and write to the card at 21 MB/s, yet the sync crawled —
|
||||
because the traversal, not the data, was the cost, and it was being paid twice.
|
||||
Without `-P` the line still shows the running count, the rate and the album in
|
||||
flight; only the two figures that needed the second walk are missing.
|
||||
|
||||
Piped to a log it prints a plain line every thirty seconds instead, with no
|
||||
carriage returns, and a summary at the end either way.
|
||||
|
||||
### The Rockbox database
|
||||
|
||||
Point `MUSIC_MIRROR_DATABASE_TOOL` at Rockbox's host-side builder and the sync
|
||||
rebuilds the database itself, so it never has to happen on the device.
|
||||
|
||||
```sh
|
||||
git clone --depth 1 https://github.com/Rockbox/rockbox.git
|
||||
cd rockbox && mkdir build-db && cd build-db
|
||||
../tools/configure --target=ipodvideo --type=d && make -j$(nproc)
|
||||
```
|
||||
|
||||
It needs a native compiler and SDL2 development headers, not the ARM
|
||||
cross-toolchain, and `tools/configure` detects `__aarch64__` correctly. On a
|
||||
distribution without `/usr/bin/perl` or `gcc-ar` — NixOS, say — patch the
|
||||
shebangs in `tools/*.pl` and pass `AR=ar`.
|
||||
|
||||
Building it here rather than on the device is not merely faster. The on-device
|
||||
commit sorts the whole index in whatever memory `core_alloc_maximum()` can
|
||||
scrape together; on a fifty-thousand-track library it runs for hours or aborts
|
||||
outright.
|
||||
|
||||
The scan runs against a scratch root — a real `.rockbox` beside a symlink
|
||||
standing in for wherever the music lands on the device — so the paths recorded
|
||||
are the ones Rockbox will look up, while the **bytes are read from the mirror
|
||||
rather than over USB**. Only the dozen `.tcd` files cross to the card.
|
||||
|
||||
Cost, measured: the parser makes about 49 reads and 43 seeks per file, probing
|
||||
the head for ID3v2 and the tail for ID3v1. On a local disk that is 2,000 files
|
||||
in half a second. Over SMB, readahead absorbs most of the reads but the opens
|
||||
and the head/tail split are real round trips, so a first full scan is minutes
|
||||
rather than seconds. It is a one-time cost: the builder is incremental, and the
|
||||
scratch root is kept between runs, so a later pass over unchanged files does no
|
||||
metadata reads at all.
|
||||
|
||||
If minutes is still too many, run the builder where the mirror is local — on
|
||||
the NAS — and copy the `.tcd` files across. There is nothing to parallelise:
|
||||
the tool is single-threaded, and two instances cannot produce one database.
|
||||
|
||||
### If the sync is interrupted
|
||||
|
||||
No partially copied track is ever left under a name Rockbox would play. rsync
|
||||
writes to a hidden temporary file and only renames it into place once the file
|
||||
is complete, and *"by default, rsync will delete any partially transferred file
|
||||
if the transfer is interrupted"*. `--partial` is deliberately not used, and
|
||||
there is a test asserting it never will be.
|
||||
|
||||
After an unclean kill or a power cut a hidden `.track.mp3.XXXXXX` can survive.
|
||||
It is not playable, it is not in the source, and the next run's `--delete`
|
||||
removes it.
|
||||
|
||||
Interrupting does not, by itself, endanger the filesystem. The kernel flushes
|
||||
dirty pages within `dirty_expire_centisecs` — thirty seconds by default — and
|
||||
`umount` always syncs before it returns. Losing data needs you to interrupt,
|
||||
*and* pull the card inside that window, *and* skip the unmount.
|
||||
|
||||
The script still traps `INT` and `TERM` and flushes and unmounts on the way
|
||||
out, exiting 130. Not because a Ctrl-C is dangerous, but because it removes the
|
||||
manual step and makes the exit deterministic — you get the same "safe to
|
||||
disconnect" either way, rather than having to remember which path you took.
|
||||
Both paths call the same function, so they cannot drift apart.
|
||||
|
||||
What genuinely does lose data is pulling the cable or the card without
|
||||
unmounting at all, interrupted or not. FAT32 has no journal. Wait for the
|
||||
unmount line.
|
||||
|
||||
### Making it faster over a network mount
|
||||
|
||||
The transfer is metadata-bound, not throughput-bound: 49,600 files means 49,600
|
||||
round trips, and the counting pass doubles that. In rough order of what it is
|
||||
worth doing:
|
||||
|
||||
| Lever | Why |
|
||||
| ----- | --- |
|
||||
| Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. |
|
||||
| Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. |
|
||||
| Counting is off by default | The percentage costs a second full traversal of both trees. On a FAT card of fifty thousand files that is slower than the transfer. `-P` asks for it. |
|
||||
| `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. |
|
||||
|
||||
**NFS instead of SMB** is worth trying but is not the big win it looks like.
|
||||
Its attribute caching defaults are far more generous than SMB's — `acregmax` of
|
||||
sixty seconds against `actimeo=1` — which is precisely the gap that
|
||||
`actimeo=60` closes on the mount you already have. Bulk read throughput between
|
||||
the two is much of a muchness on a gigabit link. Try the mount option first; it
|
||||
is one line and needs no change on the NAS.
|
||||
|
||||
And if the destination is the iPod rather than a card reader, none of this
|
||||
matters much: the source can feed data faster than USB 2.0 through an iPod will
|
||||
take it either way.
|
||||
|
||||
The unmount is the point of doing this in a script. FAT32 has no journal and
|
||||
the device is reached through disk mode, so an interrupted write is corruption
|
||||
that needs `fsck.vfat` from another machine.
|
||||
|
||||
`submit_scrobbles.py` sends what was played to Last.fm and sets the logs aside.
|
||||
|
||||
It reads **Rockbox's own `playback.log`**, which core Rockbox writes whenever
|
||||
"play log" is enabled, with no plugin running. Each line is
|
||||
`timestamp:elapsed_ms:length_ms:path` — a path and nothing else, which is why
|
||||
the on-device scrobbler plugin exists at all: reading tags back off the player
|
||||
is slow. Off the mirror it is free, so `--mirror` lets the conversion happen
|
||||
here and the plugin never has to be run. A play counts as listened at half the
|
||||
track's length, the same fraction the plugin uses, so the two cannot disagree
|
||||
about what a play was.
|
||||
|
||||
It still reads a `.scrobbler.log` if the plugin has been run and left one. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
|
||||
`submit_scrobbles.py` sends the Rockbox scrobbler log to Last.fm and sets it
|
||||
aside. Rockbox writes `/.scrobbler.log` in AUDIOSCROBBLER 1.1 format, one
|
||||
tab-separated line per track rated `L` for listened or `S` for skipped; only
|
||||
the listened ones are sent. It runs **before** the copy, since the plays
|
||||
already happened and a failed transfer is no reason to lose them.
|
||||
@@ -309,70 +180,8 @@ real-time clock Rockbox writes `/.scrobbler-timeless.log` with every timestamp
|
||||
set to zero; those are counted and reported but never submitted, because
|
||||
scrobbling them would mean inventing when they happened.
|
||||
|
||||
### Timestamps are local wall clock, and are corrected here
|
||||
|
||||
Rockbox has no concept of a timezone. Its clock is set to local time, and it
|
||||
builds log timestamps with `mktime(get_time())` — but
|
||||
[its `mktime`](https://git.rockbox.org/cgit/rockbox.git/tree/firmware/libc/mktime.c)
|
||||
is plain calendar arithmetic that applies no offset, so the RTC's local fields
|
||||
come out as if they were UTC. The number in the log is therefore ahead of the
|
||||
real instant by whatever the offset was. Last.fm stores UTC, so submitting it
|
||||
raw puts every play an hour into the future for the half of the year the UK is
|
||||
on BST.
|
||||
|
||||
Rockbox is candid about this: its scrobbler plugin writes `#TZ/UNKNOWN` in the
|
||||
log header, and the AUDIOSCROBBLER spec says a device may claim `#TZ/UTC` only
|
||||
if it actually converted. The correction is the consumer's job.
|
||||
|
||||
Each timestamp is decoded back into the wall-clock fields it came from and
|
||||
reinterpreted in the player's zone. Doing it **per play** rather than applying
|
||||
one offset to the whole log matters: a week's listening can straddle a daylight
|
||||
saving change, and the two sides need different offsets. A log that declares
|
||||
`#TZ/UTC` is left alone, so a client that already converted is not shifted
|
||||
twice.
|
||||
|
||||
The zone defaults to this machine's. Set `ROCKBOX_TIMEZONE` (or pass
|
||||
`--device-timezone`) to an IANA name if the player's clock is keeping a
|
||||
different one.
|
||||
|
||||
Because Rockbox cannot adjust for daylight saving itself, **you have to change
|
||||
the player's clock by hand twice a year**. If you forget, its times are an hour
|
||||
out and no amount of zone arithmetic recovers them. The submitter reports any
|
||||
play that converts to a time in the future, which is what a forgotten
|
||||
adjustment looks like:
|
||||
|
||||
```
|
||||
37 plays are timestamped up to 58 minutes in the future, converting from
|
||||
Europe/London. Either the player's clock is wrong or that is not the zone
|
||||
it is set to.
|
||||
```
|
||||
|
||||
`--dry-run` prints each play's local time beside the epoch, so the conversion
|
||||
can be checked against when you actually remember listening.
|
||||
|
||||
### Nothing played is thrown away
|
||||
|
||||
Two separate obligations, because a play that happened and never reached
|
||||
Last.fm is gone for good.
|
||||
|
||||
**The original is renamed, never deleted.** If Last.fm quietly dropped
|
||||
something, the evidence is still on the device as `playback.log.<ts>.submitted`.
|
||||
|
||||
**Anything not submitted is written back** into a live log for the next run:
|
||||
|
||||
| Outcome | What happens to it |
|
||||
| ------------------------------ | ----------------------------------------- |
|
||||
| Accepted by Last.fm | dropped from the live log |
|
||||
| Not in the mirror yet | written back, tried again next run |
|
||||
| In a batch that failed | written back, tried again next run |
|
||||
| Nothing accepted at all | logs left completely untouched |
|
||||
| A skip, or no usable timestamp | not retained — neither can ever be submitted, and the original still has it |
|
||||
|
||||
The batch boundary matters: submission is recorded as each batch is accepted,
|
||||
so a failure partway through knows exactly what got through and writes back
|
||||
only the remainder. No duplicates, no losses.
|
||||
|
||||
A file `ffprobe` cannot read costs one unidentified play, not the run.
|
||||
The log is renamed rather than deleted once accepted. If Last.fm quietly
|
||||
dropped something, the evidence is still on the device.
|
||||
|
||||
`check_fat32.py` reports paths a FAT32 device will not accept — reserved
|
||||
characters, trailing dots and spaces, over-long components and paths, and names
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "music-mirror"
|
||||
version = "0.4.0"
|
||||
version = "0.3.0"
|
||||
description = "Maintain a lossy MP3 mirror of a lossless music library"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
|
||||
@@ -1,180 +0,0 @@
|
||||
import io
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
|
||||
|
||||
import rsync_progress # noqa: E402
|
||||
|
||||
|
||||
class NotATerminal(io.StringIO):
|
||||
def isatty(self):
|
||||
return False
|
||||
|
||||
|
||||
class Terminal(io.StringIO):
|
||||
def isatty(self):
|
||||
return True
|
||||
|
||||
|
||||
def run(lines, total=0, out=None, bytes_expected=0):
|
||||
out = out or NotATerminal()
|
||||
rsync_progress.main(
|
||||
["--total", str(total), "--bytes", str(bytes_expected)],
|
||||
stream=io.StringIO(lines),
|
||||
out=out,
|
||||
)
|
||||
return out.getvalue()
|
||||
|
||||
|
||||
def test_the_artist_and_album_are_pulled_from_the_path():
|
||||
assert rsync_progress.album_of("Pendulum/Immersion/01 - Watercolour.mp3") == (
|
||||
"Pendulum / Immersion"
|
||||
)
|
||||
|
||||
|
||||
def test_a_shallower_path_degrades_rather_than_failing():
|
||||
assert rsync_progress.album_of("Pendulum/loose.mp3") == "Pendulum"
|
||||
assert rsync_progress.album_of("loose.mp3") == ""
|
||||
|
||||
|
||||
def test_directories_are_not_counted():
|
||||
"""rsync reports them too, and counting them puts the percentage past 100."""
|
||||
output = run("Artist/\nArtist/Album/\nArtist/Album/track.mp3\n", total=1)
|
||||
|
||||
assert "1/1" in output
|
||||
assert "100%" in output
|
||||
|
||||
|
||||
def test_the_percentage_tracks_the_total():
|
||||
output = run("".join(f"A/B/{i}.mp3\n" for i in range(5)), total=10)
|
||||
|
||||
assert "5/10" in output
|
||||
assert "50%" in output
|
||||
|
||||
|
||||
def test_without_a_total_it_counts_instead_of_guessing():
|
||||
output = run("A/B/one.mp3\nA/B/two.mp3\n")
|
||||
|
||||
assert "2 files" in output
|
||||
assert "%" not in output
|
||||
|
||||
|
||||
def test_a_final_line_is_always_printed():
|
||||
"""Otherwise the last state of a rewriting line is whatever it happened to
|
||||
be when the interval last elapsed."""
|
||||
output = run("A/B/one.mp3\n", total=1)
|
||||
|
||||
assert output.endswith("\n")
|
||||
assert "1/1" in output
|
||||
|
||||
|
||||
def test_nothing_transferred_still_reports():
|
||||
output = run("", total=0)
|
||||
|
||||
assert "0 files" in output
|
||||
|
||||
|
||||
def test_a_log_gets_no_carriage_returns():
|
||||
"""A non-terminal filling with \\r and escape codes is unreadable."""
|
||||
output = run("".join(f"A/B/{i}.mp3\n" for i in range(50)), total=50)
|
||||
|
||||
assert "\r" not in output
|
||||
assert "\033" not in output
|
||||
|
||||
|
||||
def test_a_terminal_rewrites_one_line():
|
||||
output = run("".join(f"A/B/{i}.mp3\n" for i in range(50)), total=50, out=Terminal())
|
||||
|
||||
assert "\r\033[2K" in output
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("text", "width", "expected"),
|
||||
[
|
||||
("short", 20, "short"),
|
||||
("King Gizzard / PetroDragonic Apocalypse", 20, "…agonic Apocalypse"),
|
||||
],
|
||||
)
|
||||
def test_long_labels_are_trimmed_from_the_left(text, width, expected):
|
||||
"""The album is the informative end, so the artist is what gets cut."""
|
||||
trimmed = rsync_progress.fit(text, width)
|
||||
|
||||
assert len(trimmed) <= width
|
||||
if len(text) > width:
|
||||
assert trimmed.startswith("…")
|
||||
assert text.endswith(trimmed.lstrip("…"))
|
||||
|
||||
|
||||
def test_the_size_and_path_are_parsed():
|
||||
assert rsync_progress.parse("5000 Artist/Album/Track.mp3\n") == (
|
||||
5000,
|
||||
"Artist/Album/Track.mp3",
|
||||
)
|
||||
|
||||
|
||||
def test_a_filename_containing_spaces_survives():
|
||||
"""Splitting on every space would lose most of the library."""
|
||||
assert rsync_progress.parse("1234 Artist/An Album/A Track With Spaces.mp3") == (
|
||||
1234,
|
||||
"Artist/An Album/A Track With Spaces.mp3",
|
||||
)
|
||||
|
||||
|
||||
def test_a_bare_path_is_tolerated():
|
||||
"""In case this is fed --out-format='%n' by something older."""
|
||||
assert rsync_progress.parse("Artist/Album/Track.mp3") == (0, "Artist/Album/Track.mp3")
|
||||
|
||||
|
||||
def test_directory_sizes_do_not_inflate_the_total():
|
||||
"""rsync reports directories with a 4096 inode size, which is several
|
||||
megabytes of nothing across six thousand albums."""
|
||||
output = run("4096 Artist/\n4096 Artist/Album/\n5000 Artist/Album/t.mp3\n", total=1)
|
||||
|
||||
assert "4.9 KiB" in output
|
||||
assert "12" not in output.split("Artist")[0]
|
||||
|
||||
|
||||
def test_a_rate_is_not_reported_until_it_means_something():
|
||||
"""The first files arrive microseconds apart and would give a rate in the
|
||||
gigabytes per second and an ETA of zero."""
|
||||
rate = rsync_progress.Rate()
|
||||
rate.add(100.0, 0)
|
||||
rate.add(100.5, 5_000_000)
|
||||
|
||||
assert rate.per_second() == 0.0
|
||||
|
||||
|
||||
def test_a_rate_over_a_long_enough_window_is_reported():
|
||||
rate = rsync_progress.Rate()
|
||||
rate.add(100.0, 0)
|
||||
rate.add(110.0, 10_000_000)
|
||||
|
||||
assert rate.per_second() == pytest.approx(1_000_000)
|
||||
|
||||
|
||||
def test_the_window_forgets_the_distant_past():
|
||||
"""So the estimate follows a device that slows down rather than averaging
|
||||
the slowdown away."""
|
||||
rate = rsync_progress.Rate(window=30.0)
|
||||
for second in range(0, 100, 10):
|
||||
rate.add(float(second), second * 1_000_000)
|
||||
rate.add(200.0, 100_000_000)
|
||||
|
||||
assert rate.samples[0][0] >= 90.0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("seconds", "expected"),
|
||||
[(0, "0s"), (45, "45s"), (60, "1m00s"), (1092, "18m12s"), (7500, "2h05m")],
|
||||
)
|
||||
def test_durations_read_without_arithmetic(seconds, expected):
|
||||
assert rsync_progress.human_duration(seconds) == expected
|
||||
|
||||
|
||||
def test_a_summary_is_printed_at_the_end():
|
||||
output = run("5000000 A/B/one.mp3\n", total=1, bytes_expected=5000000)
|
||||
|
||||
assert "copied 4.8 MiB in" in output
|
||||
@@ -1,9 +1,6 @@
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import pytest
|
||||
|
||||
@@ -190,374 +187,3 @@ def test_write_credentials_are_required(tmp_path, capsys, monkeypatch):
|
||||
|
||||
assert code == 2
|
||||
assert "LASTFM_API_SECRET" in capsys.readouterr().err
|
||||
|
||||
|
||||
PLAYBACK_LOG = """1700000300:180000:245000:/Music/Pendulum/Immersion/01.mp3
|
||||
1700000200:9000:180000:/Music/Green Day/Dookie/07.mp3
|
||||
0:180000:245000:/Music/No/Clock/track.mp3
|
||||
malformed line without colons
|
||||
"""
|
||||
|
||||
|
||||
def tags_of(artist="Pendulum", title="Watercolour", album="Immersion", track="1/11"):
|
||||
def runner(path):
|
||||
return json.dumps(
|
||||
{"format": {"tags": {"ARTIST": artist, "TITLE": title,
|
||||
"ALBUM": album, "track": track}}}
|
||||
)
|
||||
|
||||
return runner
|
||||
|
||||
|
||||
def test_the_playback_log_format_is_four_fields():
|
||||
"""timestamp:elapsed_ms:length_ms:path, written by Rockbox core."""
|
||||
plays = submit_scrobbles.parse_playback_log(PLAYBACK_LOG)
|
||||
|
||||
assert len(plays) == 3 # the malformed line is dropped
|
||||
assert plays[0] == (1700000300, 180000, 245000, "/Music/Pendulum/Immersion/01.mp3")
|
||||
|
||||
|
||||
def test_a_device_path_maps_onto_the_mirror():
|
||||
assert submit_scrobbles.device_to_local(
|
||||
"/Music/Pendulum/Immersion/01.mp3", "/Music", "/mnt/mirror"
|
||||
) == Path("/mnt/mirror/Pendulum/Immersion/01.mp3")
|
||||
|
||||
|
||||
def test_a_path_outside_the_prefix_is_not_mapped():
|
||||
"""Something played from elsewhere on the card is not in the mirror."""
|
||||
assert submit_scrobbles.device_to_local(
|
||||
"/Podcasts/episode.mp3", "/Music", "/mnt/mirror"
|
||||
) is None
|
||||
|
||||
|
||||
def test_a_path_at_the_card_root_maps_straight_across():
|
||||
assert submit_scrobbles.device_to_local(
|
||||
"/Pendulum/Immersion/01.mp3", "/", "/mnt/mirror"
|
||||
) == Path("/mnt/mirror/Pendulum/Immersion/01.mp3")
|
||||
|
||||
|
||||
def test_a_short_play_is_a_skip_not_a_scrobble(tmp_path, monkeypatch):
|
||||
"""Nine seconds of a three minute track. The on-device plugin uses the same
|
||||
fraction, so the two never disagree about what counted as a play."""
|
||||
monkeypatch.setattr(Path, "is_file", lambda self: True)
|
||||
|
||||
result = submit_scrobbles.plays_from_playback_log(
|
||||
PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
|
||||
)
|
||||
|
||||
assert result.skipped == 1
|
||||
assert [entry["timestamp"] for entry in result.played] == ["1700000300"]
|
||||
|
||||
|
||||
def test_a_zero_timestamp_is_refused(tmp_path, monkeypatch):
|
||||
"""Without a real-time clock Rockbox logs ticks, not dates. Scrobbling
|
||||
those would mean inventing when they happened."""
|
||||
monkeypatch.setattr(Path, "is_file", lambda self: True)
|
||||
|
||||
result = submit_scrobbles.plays_from_playback_log(
|
||||
PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
|
||||
)
|
||||
|
||||
assert result.timeless == 1
|
||||
|
||||
|
||||
def test_tags_come_from_the_mirror(tmp_path, monkeypatch):
|
||||
"""The log carries only a path -- which is precisely why the on-device
|
||||
plugin exists. Off the mirror the tags are free."""
|
||||
monkeypatch.setattr(Path, "is_file", lambda self: True)
|
||||
|
||||
played = submit_scrobbles.plays_from_playback_log(
|
||||
PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
|
||||
).played
|
||||
|
||||
assert played[0]["artist"] == "Pendulum"
|
||||
assert played[0]["album"] == "Immersion"
|
||||
assert played[0]["trackNumber"] == "1" # "1/11" -> "1"
|
||||
assert played[0]["duration"] == "245" # milliseconds -> seconds
|
||||
|
||||
|
||||
def test_a_track_missing_from_the_mirror_is_counted_not_guessed(monkeypatch):
|
||||
monkeypatch.setattr(Path, "is_file", lambda self: False)
|
||||
|
||||
result = submit_scrobbles.plays_from_playback_log(
|
||||
PLAYBACK_LOG, "/Music", "/mnt/mirror", runner=tags_of()
|
||||
)
|
||||
|
||||
assert result.played == []
|
||||
# One: the skip and the clockless entry are filtered before the file is
|
||||
# looked for, since neither would be submitted either way.
|
||||
assert result.unresolved == 1
|
||||
# And that one play is kept, so a later run can try it again.
|
||||
assert len(result.retain) == 1
|
||||
|
||||
|
||||
LONDON = ZoneInfo("Europe/London")
|
||||
|
||||
# Rockbox builds timestamps with mktime(get_time()), and its mktime applies no
|
||||
# zone at all, so the number is the RTC's local wall clock read as if it were
|
||||
# UTC. Under BST that puts every play an hour ahead of when it happened.
|
||||
SUMMER_LOGGED = 1787839200 # 2026-08-27 14:00 written by the device
|
||||
SUMMER_TRUE = 1787835600 # the instant that actually was, 13:00 UTC
|
||||
WINTER_LOGGED = 1796479200 # 2026-12-05 14:00, when London is already UTC
|
||||
WINTER_TRUE = 1796479200
|
||||
|
||||
|
||||
def test_a_summer_timestamp_is_pulled_back_to_real_utc():
|
||||
"""The device logged 14:00 local. That was 13:00 UTC, and UTC is what
|
||||
Last.fm stores."""
|
||||
assert submit_scrobbles.device_time_to_utc(SUMMER_LOGGED, LONDON) == SUMMER_TRUE
|
||||
|
||||
|
||||
def test_a_winter_timestamp_is_left_alone():
|
||||
"""London keeps UTC for half the year, so there is nothing to correct and
|
||||
the correction must not invent an offset anyway."""
|
||||
assert submit_scrobbles.device_time_to_utc(WINTER_LOGGED, LONDON) == WINTER_TRUE
|
||||
|
||||
|
||||
def test_the_converted_time_reads_back_as_the_clock_the_device_showed():
|
||||
"""The round trip, which is the property that actually matters: whatever
|
||||
the player's screen said is what Last.fm should show in local time."""
|
||||
shown = datetime.fromtimestamp(SUMMER_LOGGED, timezone.utc)
|
||||
corrected = submit_scrobbles.device_time_to_utc(SUMMER_LOGGED, LONDON)
|
||||
|
||||
assert datetime.fromtimestamp(corrected, LONDON).strftime("%Y-%m-%d %H:%M") == (
|
||||
shown.strftime("%Y-%m-%d %H:%M")
|
||||
)
|
||||
|
||||
|
||||
def test_one_log_spanning_a_clock_change_converts_each_side_separately():
|
||||
"""A week's listening either side of the October change carries two
|
||||
different offsets. Correcting the log by a single figure would put half of
|
||||
it an hour out, which is why the offset is resolved per play."""
|
||||
before, after = 1792888200, 1792899000 # 2026-10-25, 00:30 BST and 03:30 GMT
|
||||
|
||||
assert submit_scrobbles.device_time_to_utc(before, LONDON) == before - 3600
|
||||
assert submit_scrobbles.device_time_to_utc(after, LONDON) == after
|
||||
|
||||
|
||||
def test_a_playback_log_is_corrected_before_submission(monkeypatch):
|
||||
monkeypatch.setattr(Path, "is_file", lambda self: True)
|
||||
log = f"{SUMMER_LOGGED}:180000:245000:/Music/Pendulum/Immersion/01.mp3\n"
|
||||
|
||||
played = submit_scrobbles.plays_from_playback_log(
|
||||
log, "/Music", "/mnt/mirror", runner=tags_of(), zone=LONDON
|
||||
).played
|
||||
|
||||
assert played[0]["timestamp"] == str(SUMMER_TRUE)
|
||||
|
||||
|
||||
def test_a_scrobbler_log_claiming_utc_is_not_corrected_twice():
|
||||
"""The format's header exists for exactly this. A client that already did
|
||||
the conversion says so, and correcting it again would break it."""
|
||||
log = LOG.replace("#TZ/UNKNOWN", "#TZ/UTC")
|
||||
|
||||
played, _, _ = submit_scrobbles.parse_log(log, LONDON)
|
||||
|
||||
assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"]
|
||||
|
||||
|
||||
def test_a_scrobbler_log_declaring_unknown_is_corrected():
|
||||
"""Rockbox writes UNKNOWN, meaning local wall clock, so the times are ours
|
||||
to fix."""
|
||||
played, _, _ = submit_scrobbles.parse_log(LOG, LONDON)
|
||||
|
||||
# 1700000100 and 1700000300 are November, when London is on UTC anyway;
|
||||
# the point is that the header did not exempt them from being looked at.
|
||||
assert submit_scrobbles.declares_utc(LOG) is False
|
||||
assert [entry["timestamp"] for entry in played] == ["1700000100", "1700000300"]
|
||||
|
||||
|
||||
def test_the_system_zone_keeps_its_whole_name(monkeypatch, tmp_path):
|
||||
"""/etc/localtime points into the tzdata tree. Taking only the last
|
||||
component gives "London", which no database holds -- and the fallback for
|
||||
an unknown name is a fixed offset, which is wrong for half the year. The
|
||||
region has to survive."""
|
||||
zoneinfo_dir = tmp_path / "share" / "zoneinfo" / "Europe"
|
||||
zoneinfo_dir.mkdir(parents=True)
|
||||
(zoneinfo_dir / "London").write_bytes(b"TZif")
|
||||
monkeypatch.setattr(
|
||||
submit_scrobbles.Path,
|
||||
"read_text",
|
||||
lambda self, **kw: (_ for _ in ()).throw(OSError),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
submit_scrobbles.Path,
|
||||
"resolve",
|
||||
lambda self: zoneinfo_dir / "London",
|
||||
)
|
||||
|
||||
assert submit_scrobbles.system_zone_name() == "Europe/London"
|
||||
|
||||
|
||||
def test_a_named_zone_beats_the_machine_default():
|
||||
"""The player may not be in the same place as the laptop."""
|
||||
assert submit_scrobbles.device_zone("Asia/Tokyo") == ZoneInfo("Asia/Tokyo")
|
||||
|
||||
|
||||
def test_plays_dated_after_now_are_reported():
|
||||
"""A clock never put forward, or the wrong zone, produces plays that have
|
||||
not happened yet. Last.fm cannot tell those from real ones."""
|
||||
now = 1787835600
|
||||
played = [
|
||||
{"timestamp": str(now - 60)},
|
||||
{"timestamp": str(now + 3600)},
|
||||
]
|
||||
|
||||
ahead = submit_scrobbles.future_plays(played, now=now)
|
||||
|
||||
assert [entry["timestamp"] for entry in ahead] == [str(now + 3600)]
|
||||
|
||||
|
||||
def test_a_clock_a_minute_fast_is_not_reported():
|
||||
"""Devices drift. Only an offset large enough to be a zone error matters."""
|
||||
now = 1787835600
|
||||
|
||||
assert submit_scrobbles.future_plays([{"timestamp": str(now + 60)}], now=now) == []
|
||||
|
||||
|
||||
def test_playback_logs_are_found_including_rotations(tmp_path):
|
||||
"""Rockbox rotates the log once it passes half a megabyte."""
|
||||
rockbox = tmp_path / ".rockbox"
|
||||
rockbox.mkdir()
|
||||
for name in ("playback.log", "playback_0001.log", "playback_0002.log"):
|
||||
(rockbox / name).write_text("1700000000:1:1:/Music/a.mp3\n")
|
||||
(rockbox / "empty.log").write_text("")
|
||||
|
||||
found = submit_scrobbles.find_playback_logs(tmp_path)
|
||||
|
||||
assert [path.name for path in found] == [
|
||||
"playback.log", "playback_0001.log", "playback_0002.log"
|
||||
]
|
||||
|
||||
|
||||
def test_without_a_mirror_it_says_what_is_needed(tmp_path, capsys):
|
||||
device = tmp_path / "IPOD"
|
||||
(device / ".rockbox").mkdir(parents=True)
|
||||
(device / ".rockbox" / "playback.log").write_text("1700000000:1:1:/Music/a.mp3\n")
|
||||
|
||||
submit_scrobbles.main([str(device)], transport=fake_transport([]))
|
||||
|
||||
assert "pass --mirror" in capsys.readouterr().err
|
||||
|
||||
|
||||
MIRROR = "/mnt/mirror"
|
||||
|
||||
|
||||
def stub_ffprobe(monkeypatch, **tags):
|
||||
"""Answer for anything under the mirror without invoking ffprobe."""
|
||||
monkeypatch.setattr(submit_scrobbles, "_ffprobe", tags_of(**tags))
|
||||
|
||||
|
||||
def only_mirror_files_exist(monkeypatch, resolvable=None):
|
||||
"""Make the mirror's files appear to exist, and nothing else.
|
||||
|
||||
Patching is_file wholesale makes the device directory look like a log file,
|
||||
which sends main() down the .scrobbler.log path instead.
|
||||
"""
|
||||
real = Path.is_file
|
||||
|
||||
def patched(self):
|
||||
text = str(self)
|
||||
if text.startswith(MIRROR):
|
||||
return resolvable is None or text == resolvable
|
||||
return real(self)
|
||||
|
||||
monkeypatch.setattr(Path, "is_file", patched)
|
||||
|
||||
|
||||
def playback_device(tmp_path, log=PLAYBACK_LOG):
|
||||
device = tmp_path / "IPOD"
|
||||
(device / ".rockbox").mkdir(parents=True)
|
||||
(device / ".rockbox" / "playback.log").write_text(log)
|
||||
return device
|
||||
|
||||
|
||||
def test_a_failed_submission_keeps_every_log(tmp_path, monkeypatch):
|
||||
"""Nothing got through, so nothing may be set aside."""
|
||||
only_mirror_files_exist(monkeypatch)
|
||||
stub_ffprobe(monkeypatch)
|
||||
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
|
||||
device = playback_device(tmp_path)
|
||||
transport = fake_transport([{"error": 29, "message": "Rate limit"}])
|
||||
|
||||
code = submit_scrobbles.main(
|
||||
[str(device), "--mirror", MIRROR, "--api-key", "k", "--api-secret", "s"],
|
||||
transport=transport,
|
||||
)
|
||||
|
||||
assert code == 1
|
||||
assert (device / ".rockbox" / "playback.log").is_file()
|
||||
assert not list((device / ".rockbox").glob("*.submitted"))
|
||||
|
||||
|
||||
def test_an_unmatched_play_is_written_back_not_lost(tmp_path, monkeypatch, capsys):
|
||||
"""A track the mirror does not yet hold is still a play that happened. It
|
||||
is kept so a later run, after the file has been copied, can submit it."""
|
||||
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
|
||||
# Only the first track resolves; the rest are absent from the mirror.
|
||||
only_mirror_files_exist(monkeypatch, f"{MIRROR}/Pendulum/Immersion/01.mp3")
|
||||
stub_ffprobe(monkeypatch)
|
||||
log = (
|
||||
"1700000300:180000:245000:/Music/Pendulum/Immersion/01.mp3\n"
|
||||
"1700000400:180000:245000:/Music/Missing/Album/09.mp3\n"
|
||||
)
|
||||
device = playback_device(tmp_path, log)
|
||||
transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
|
||||
|
||||
code = submit_scrobbles.main(
|
||||
[str(device), "--mirror", MIRROR, "--api-key", "k", "--api-secret", "s"],
|
||||
transport=transport,
|
||||
)
|
||||
|
||||
assert code == 0
|
||||
rockbox = device / ".rockbox"
|
||||
# The original is preserved untouched...
|
||||
assert list(rockbox.glob("playback.log.*.submitted"))
|
||||
# ...and the unmatched play is back in a live log for the next attempt.
|
||||
written = (rockbox / "playback.log").read_text()
|
||||
assert "Missing/Album/09.mp3" in written
|
||||
assert "Pendulum/Immersion/01.mp3" not in written
|
||||
|
||||
|
||||
def test_the_original_is_renamed_rather_than_deleted(tmp_path, monkeypatch):
|
||||
"""If Last.fm quietly dropped something, the evidence stays on the device."""
|
||||
only_mirror_files_exist(monkeypatch)
|
||||
stub_ffprobe(monkeypatch)
|
||||
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
|
||||
device = playback_device(tmp_path)
|
||||
transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
|
||||
|
||||
submit_scrobbles.main(
|
||||
[str(device), "--mirror", MIRROR, "--api-key", "k", "--api-secret", "s"],
|
||||
transport=transport,
|
||||
)
|
||||
|
||||
aside = list((device / ".rockbox").glob("playback.log.*.submitted"))
|
||||
assert len(aside) == 1
|
||||
assert PLAYBACK_LOG.splitlines()[0] in aside[0].read_text()
|
||||
|
||||
|
||||
def test_keep_leaves_everything_alone(tmp_path, monkeypatch):
|
||||
only_mirror_files_exist(monkeypatch)
|
||||
stub_ffprobe(monkeypatch)
|
||||
monkeypatch.setattr(submit_scrobbles, "load_session", lambda: "sk")
|
||||
device = playback_device(tmp_path)
|
||||
transport = fake_transport([{"scrobbles": {"@attr": {"accepted": 1, "ignored": 0}}}])
|
||||
|
||||
submit_scrobbles.main(
|
||||
[str(device), "--mirror", MIRROR, "--keep",
|
||||
"--api-key", "k", "--api-secret", "s"],
|
||||
transport=transport,
|
||||
)
|
||||
|
||||
assert (device / ".rockbox" / "playback.log").read_text() == PLAYBACK_LOG
|
||||
assert not list((device / ".rockbox").glob("*.submitted"))
|
||||
|
||||
|
||||
def test_a_file_ffprobe_cannot_read_does_not_abandon_the_rest(monkeypatch):
|
||||
"""CalledProcessError is not an OSError, so one bad file used to take the
|
||||
whole submission with it."""
|
||||
def broken(path):
|
||||
raise subprocess.CalledProcessError(1, "ffprobe")
|
||||
|
||||
assert submit_scrobbles.read_tags(Path("/mnt/mirror/x.mp3"), runner=broken) == {}
|
||||
|
||||
+1
-181
@@ -5,22 +5,14 @@ protecting against emptying the wrong directory -- a mistake that does not
|
||||
announce itself.
|
||||
"""
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
SCRIPT = Path(__file__).resolve().parent.parent / "tools" / "sync-to-ipod.sh"
|
||||
|
||||
# Skipped rather than failed where the tools are absent: this is a host-side
|
||||
# script, and a machine without rsync is not a machine that would run it.
|
||||
REQUIRED = ("bash", "rsync", "findmnt")
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not all(shutil.which(tool) for tool in REQUIRED),
|
||||
reason=f"needs {', '.join(REQUIRED)} on PATH",
|
||||
)
|
||||
|
||||
|
||||
def run(*arguments):
|
||||
return subprocess.run(
|
||||
@@ -174,175 +166,3 @@ def test_the_help_says_how_to_reach_and_leave_disk_mode():
|
||||
|
||||
assert "Menu+Select" in help_text
|
||||
assert "holding Play" in help_text
|
||||
|
||||
|
||||
def test_counting_is_off_by_default(mirror, tmp_path):
|
||||
"""The counting pass walks and compares both trees exactly as the transfer
|
||||
does. On a FAT card of fifty thousand files that costs more than moving the
|
||||
data, so the percentage has to be asked for."""
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
|
||||
result = run("-f", "-S", "-U", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "files to copy" not in result.stderr
|
||||
assert (destination / "Album" / "track.mp3").is_file()
|
||||
|
||||
|
||||
def test_the_delta_algorithm_is_disabled(mirror, tmp_path):
|
||||
"""It would read every destination file back over USB to checksum it, to
|
||||
avoid resending an MP3 that has changed in its entirety anyway."""
|
||||
script = SCRIPT.read_text()
|
||||
|
||||
assert "--whole-file" in script
|
||||
|
||||
|
||||
def test_directory_timestamps_are_not_set(mirror, tmp_path):
|
||||
"""One setattr round trip per directory, across six thousand albums, to set
|
||||
timestamps nothing reads."""
|
||||
script = SCRIPT.read_text()
|
||||
|
||||
assert "--omit-dir-times" in script
|
||||
|
||||
|
||||
def test_an_interrupt_is_trapped_so_the_filesystem_is_flushed():
|
||||
"""Ctrl-C during a transfer would otherwise skip the sync and the unmount,
|
||||
leaving a journal-less FAT filesystem with dirty buffers -- which is the
|
||||
corruption this script exists to prevent.
|
||||
|
||||
Structural rather than timed: reproducing a mid-transfer signal needs a
|
||||
payload large enough to be slow, and a test that depends on losing a race
|
||||
is a test that fails in CI for no reason.
|
||||
"""
|
||||
script = SCRIPT.read_text()
|
||||
|
||||
assert "trap interrupted INT TERM" in script
|
||||
assert "exit 130" in script
|
||||
|
||||
|
||||
def test_the_flush_and_unmount_happen_on_every_exit_path():
|
||||
script = SCRIPT.read_text()
|
||||
|
||||
# Both the normal path and the interrupt path go through the same function,
|
||||
# so one cannot drift from the other.
|
||||
assert script.count("finish\n") >= 2
|
||||
assert "--partial" not in script, "rsync must delete partial files, not keep them"
|
||||
|
||||
|
||||
def test_the_database_step_is_skipped_without_a_tool(mirror, tmp_path, monkeypatch):
|
||||
"""Opt-in, like the scrobbler: absent configuration is not an error."""
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
monkeypatch.delenv("MUSIC_MIRROR_DATABASE_TOOL", raising=False)
|
||||
|
||||
result = run("-f", "-S", "-U", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "no database tool configured" in result.stderr
|
||||
|
||||
|
||||
def test_a_missing_database_tool_is_refused(mirror, tmp_path, monkeypatch):
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
monkeypatch.setenv("MUSIC_MIRROR_DATABASE_TOOL", str(tmp_path / "nonexistent"))
|
||||
|
||||
result = run("-f", "-S", "-U", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 1
|
||||
assert "not executable" in result.stderr
|
||||
|
||||
|
||||
def test_the_database_step_can_be_skipped(mirror, tmp_path, monkeypatch):
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
monkeypatch.setenv("MUSIC_MIRROR_DATABASE_TOOL", str(tmp_path / "nonexistent"))
|
||||
|
||||
result = run("-f", "-S", "-U", "-B", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "not executable" not in result.stderr
|
||||
|
||||
|
||||
def test_the_scan_reads_from_the_mirror_not_the_device():
|
||||
"""The whole point: tags come off the mirror, only the .tcd files go over
|
||||
USB. Reading 49,600 files through an iPod's USB bridge is the slow path."""
|
||||
script = SCRIPT.read_text()
|
||||
|
||||
assert 'ln -s "$mirror"' in script
|
||||
assert 'cd "$scratch"' in script
|
||||
|
||||
|
||||
def can_bind_mount():
|
||||
"""User namespaces let an unprivileged process bind mount. Not everywhere,
|
||||
notably not inside some containers, so the test that needs it skips."""
|
||||
return (
|
||||
subprocess.run(
|
||||
["unshare", "-Umr", "true"], capture_output=True, check=False
|
||||
).returncode
|
||||
== 0
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not can_bind_mount(), reason="needs unprivileged user namespaces")
|
||||
def test_the_database_lands_at_the_device_root_not_the_music_folder(tmp_path):
|
||||
"""The two tools disagree about where the root is. rsync copies artist
|
||||
folders into <device>/Music; the database tool must run one level up, where
|
||||
.rockbox lives, and must record /Music/... paths while reading the bytes
|
||||
from the mirror. device_prefix is what reconciles them.
|
||||
"""
|
||||
mirror = tmp_path / "mirror" / "Pendulum" / "Immersion"
|
||||
mirror.mkdir(parents=True)
|
||||
(mirror / "01.mp3").write_bytes(b"not really an mp3")
|
||||
card = tmp_path / "card"
|
||||
(card / ".rockbox").mkdir(parents=True)
|
||||
(card / "Music").mkdir()
|
||||
device = tmp_path / "device"
|
||||
device.mkdir()
|
||||
|
||||
tool = tmp_path / "fake-database"
|
||||
# Records where it was run and what it could see, which is the whole
|
||||
# question; producing a real database needs Rockbox's builder.
|
||||
tool.write_text(
|
||||
"#!/bin/sh\n"
|
||||
"printf '%s\\n' \"$PWD\" > .rockbox/where.txt\n"
|
||||
"ls Music/ > .rockbox/saw.txt\n"
|
||||
"echo db > .rockbox/database_0.tcd\n"
|
||||
)
|
||||
tool.chmod(0o755)
|
||||
|
||||
script = (
|
||||
f"mount --bind {card} {device} && "
|
||||
f"XDG_CACHE_HOME={tmp_path / 'cache'} MUSIC_MIRROR_DATABASE_TOOL={tool} "
|
||||
f"bash {SCRIPT} -f -S -U {tmp_path / 'mirror'} {device / 'Music'}"
|
||||
)
|
||||
result = subprocess.run(
|
||||
["unshare", "-Umr", "sh", "-c", script], capture_output=True, text=True
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
# The database lands beside the device root, not inside Music.
|
||||
assert (card / ".rockbox" / "database_0.tcd").is_file()
|
||||
# Only *.tcd is copied across, so the markers stay in the scratch root --
|
||||
# which is itself the point: nothing else is written to the device.
|
||||
scratch = tmp_path / "cache" / "music-mirror" / "database" / ".rockbox"
|
||||
assert not (card / ".rockbox" / "where.txt").exists()
|
||||
|
||||
# It ran in the scratch root, not on the card.
|
||||
where = (scratch / "where.txt").read_text().strip()
|
||||
assert where.endswith("music-mirror/database"), where
|
||||
# ...and could walk into the mirror through a symlink named for the device
|
||||
# prefix, which is how the paths come out as /Music/... while the bytes are
|
||||
# read from somewhere else entirely.
|
||||
assert "Pendulum" in (scratch / "saw.txt").read_text()
|
||||
|
||||
|
||||
def test_counting_can_be_asked_for(mirror, tmp_path):
|
||||
"""When the destination is cheap to traverse, the percentage is worth it."""
|
||||
destination = tmp_path / "dest"
|
||||
destination.mkdir()
|
||||
|
||||
result = run("-f", "-S", "-U", "-P", str(mirror), str(destination))
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "files to copy" in result.stderr
|
||||
|
||||
@@ -1,190 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Render rsync's per-file output as a single updating status line.
|
||||
|
||||
Fed the size and path rsync reports with --out-format='%l %n', one per line.
|
||||
The size is what makes an estimate possible: rsync's own rate is not exposed
|
||||
per file, but bytes completed over time elapsed is the same arithmetic and
|
||||
needs nothing rsync does not already print. Prints one
|
||||
line that rewrites itself, showing how far through the transfer is and which
|
||||
album is currently going across, rather than either scrolling fifty thousand
|
||||
filenames past or -- as rsync does while it builds its file list -- saying
|
||||
nothing at all for several minutes.
|
||||
|
||||
Falls back to periodic plain lines when stderr is not a terminal, so a log does
|
||||
not fill up with carriage returns.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import collections
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
|
||||
# The estimate is taken over a trailing window rather than the whole run, so it
|
||||
# follows a device that slows down instead of averaging the slowdown away.
|
||||
RATE_WINDOW_SECONDS = 30.0
|
||||
|
||||
# Below this the window is too narrow to divide by: the first few files arrive
|
||||
# in microseconds and produce a rate in the gigabytes per second, and an ETA of
|
||||
# nothing at all. Better to show neither until the figure means something.
|
||||
RATE_MINIMUM_SPAN_SECONDS = 2.0
|
||||
|
||||
|
||||
def parse(line):
|
||||
"""Return (bytes, path) for one line of rsync output.
|
||||
|
||||
Tolerates a bare path, in case someone runs this against --out-format='%n'.
|
||||
"""
|
||||
line = line.rstrip("\n")
|
||||
size, separator, path = line.partition(" ")
|
||||
if separator and size.isdigit():
|
||||
return int(size), path
|
||||
return 0, line
|
||||
|
||||
|
||||
def human_bytes(count):
|
||||
size = float(count)
|
||||
for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
|
||||
if size < 1024 or unit == "TiB":
|
||||
return f"{size:.1f} {unit}"
|
||||
size /= 1024
|
||||
|
||||
|
||||
def human_duration(seconds):
|
||||
"""Return a duration nobody has to do arithmetic on."""
|
||||
seconds = int(seconds)
|
||||
if seconds < 60:
|
||||
return f"{seconds}s"
|
||||
if seconds < 3600:
|
||||
return f"{seconds // 60}m{seconds % 60:02d}s"
|
||||
return f"{seconds // 3600}h{(seconds % 3600) // 60:02d}m"
|
||||
|
||||
|
||||
class Rate:
|
||||
"""Bytes per second over a trailing window."""
|
||||
|
||||
def __init__(self, window=RATE_WINDOW_SECONDS):
|
||||
self.window = window
|
||||
self.samples = collections.deque()
|
||||
|
||||
def add(self, when, total_bytes):
|
||||
self.samples.append((when, total_bytes))
|
||||
while len(self.samples) > 2 and when - self.samples[0][0] > self.window:
|
||||
self.samples.popleft()
|
||||
|
||||
def per_second(self):
|
||||
if len(self.samples) < 2:
|
||||
return 0.0
|
||||
(first_time, first_bytes), (last_time, last_bytes) = (
|
||||
self.samples[0],
|
||||
self.samples[-1],
|
||||
)
|
||||
elapsed = last_time - first_time
|
||||
if elapsed < RATE_MINIMUM_SPAN_SECONDS:
|
||||
return 0.0
|
||||
return (last_bytes - first_bytes) / elapsed
|
||||
|
||||
|
||||
def album_of(path):
|
||||
"""Return "Artist / Album" for a mirror-relative path."""
|
||||
parts = [part for part in path.strip("/").split("/") if part]
|
||||
if len(parts) >= 3:
|
||||
return f"{parts[0]} / {parts[1]}"
|
||||
if len(parts) == 2:
|
||||
return parts[0]
|
||||
return ""
|
||||
|
||||
|
||||
def fit(text, width):
|
||||
"""Trim to the terminal, from the left: the album matters more than the artist."""
|
||||
if width <= 1 or len(text) <= width:
|
||||
return text
|
||||
return "…" + text[-(width - 1) :]
|
||||
|
||||
|
||||
def render(done, total, copied, expected, rate, label, width):
|
||||
"""Build the status line, giving whatever room is left to the album."""
|
||||
if total > 0:
|
||||
share = min(100, done * 100 // total)
|
||||
head = f"[{share:>3}%] {done:,}/{total:,}"
|
||||
else:
|
||||
head = f"[{done:,} files]"
|
||||
|
||||
if expected > 0:
|
||||
head += f" {human_bytes(copied)}/{human_bytes(expected)}"
|
||||
if rate > 0:
|
||||
head += f" {human_bytes(rate)}/s"
|
||||
remaining = expected - copied
|
||||
if remaining > 0:
|
||||
head += f" ETA {human_duration(remaining / rate)}"
|
||||
head += " "
|
||||
return head + fit(label, max(0, width - len(head)))
|
||||
|
||||
|
||||
def main(argv=None, stream=None, out=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--total", type=int, default=0, help="files expected")
|
||||
parser.add_argument("--bytes", type=int, default=0, help="bytes expected")
|
||||
parser.add_argument("--interval", type=float, default=0.1, help="seconds between redraws")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
stream = stream or sys.stdin
|
||||
out = out or sys.stderr
|
||||
interactive = out.isatty()
|
||||
width = shutil.get_terminal_size((100, 24)).columns - 1
|
||||
|
||||
done = 0
|
||||
copied = 0
|
||||
rate = Rate()
|
||||
started = time.monotonic()
|
||||
rate.add(started, 0)
|
||||
last_drawn = 0.0
|
||||
label = ""
|
||||
|
||||
for line in stream:
|
||||
size, path = parse(line)
|
||||
# rsync reports directories too, with a trailing slash and an inode
|
||||
# size. Counting them puts the percentage past a hundred and the byte
|
||||
# total well over what will actually be transferred.
|
||||
if not path or path.endswith("/"):
|
||||
continue
|
||||
done += 1
|
||||
copied += size
|
||||
label = album_of(path) or os.path.basename(path)
|
||||
|
||||
now = time.monotonic()
|
||||
rate.add(now, copied)
|
||||
if interactive:
|
||||
if now - last_drawn >= args.interval:
|
||||
out.write(
|
||||
"\r\033[2K"
|
||||
+ render(done, args.total, copied, args.bytes, rate.per_second(),
|
||||
label, width)
|
||||
)
|
||||
out.flush()
|
||||
last_drawn = now
|
||||
elif now - last_drawn >= 30:
|
||||
out.write(
|
||||
render(done, args.total, copied, args.bytes, rate.per_second(), label, width)
|
||||
+ "\n"
|
||||
)
|
||||
out.flush()
|
||||
last_drawn = now
|
||||
|
||||
elapsed = max(1e-9, time.monotonic() - started)
|
||||
if interactive:
|
||||
out.write("\r\033[2K")
|
||||
summary = render(done, args.total, copied, args.bytes, 0, label, width).rstrip()
|
||||
out.write(f"{summary}\n")
|
||||
if copied:
|
||||
out.write(
|
||||
f"copied {human_bytes(copied)} in {human_duration(elapsed)}"
|
||||
f" at {human_bytes(copied / elapsed)}/s\n"
|
||||
)
|
||||
out.flush()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+15
-403
@@ -14,16 +14,12 @@ import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
API_ROOT = "https://ws.audioscrobbler.com/2.0/"
|
||||
|
||||
@@ -35,26 +31,6 @@ BATCH = 50
|
||||
# scrobble.
|
||||
LOG_NAMES = (".scrobbler.log", ".scrobbler-timeless.log")
|
||||
|
||||
# Rockbox core writes this whenever "play log" is on, with no plugin running:
|
||||
# timestamp:elapsed_ms:length_ms:/Music/Artist/Album/Track.mp3
|
||||
# It is rotated once it grows past half a megabyte. Converting it needs tags,
|
||||
# which is why the on-device plugin exists -- reading them back off the player
|
||||
# is slow. Off the mirror it is free, so the plugin can be skipped entirely.
|
||||
PLAYBACK_LOG_NAMES = ("playback.log", "playback_*.log")
|
||||
|
||||
# The plugin counts a track as listened at savepct of its length, defaulting to
|
||||
# fifty. Same rule here, or the two disagree about what a play is.
|
||||
LISTENED_FRACTION = 0.5
|
||||
|
||||
# Below this a timestamp is not a wall-clock time. Without a real-time clock
|
||||
# Rockbox logs ticks in milliseconds instead, which is not a date.
|
||||
EARLIEST_PLAUSIBLE = 1_000_000_000
|
||||
|
||||
# How far ahead of now a converted play may sit before it is reported. Some
|
||||
# slack absorbs a device clock drifting by a minute or two; an hour out means
|
||||
# the zone is wrong or the clock was never put forward.
|
||||
FUTURE_TOLERANCE_SECONDS = 300
|
||||
|
||||
SESSION_FILE = Path(
|
||||
os.getenv("XDG_CONFIG_HOME", Path.home() / ".config")
|
||||
) / "music-mirror" / "lastfm.json"
|
||||
@@ -64,117 +40,14 @@ class LastfmError(Exception):
|
||||
"""A Last.fm request that failed."""
|
||||
|
||||
|
||||
def system_zone_name():
|
||||
"""Return this machine's IANA zone name, or "" if nothing states it.
|
||||
|
||||
The name has to come out whole. /etc/localtime is a symlink into the
|
||||
tzdata tree, so the part after `zoneinfo/` is the name -- taking only the
|
||||
last component yields "London", which no database has, and falls back to a
|
||||
fixed offset that would then be wrong for half the year.
|
||||
"""
|
||||
try:
|
||||
named = Path("/etc/timezone").read_text(encoding="utf-8").strip()
|
||||
if named:
|
||||
return named
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
parts = Path("/etc/localtime").resolve().parts
|
||||
except OSError:
|
||||
return ""
|
||||
if "zoneinfo" in parts:
|
||||
return "/".join(parts[len(parts) - parts[::-1].index("zoneinfo"):])
|
||||
return ""
|
||||
|
||||
|
||||
def device_zone(name=None):
|
||||
"""Return the zone the device's clock is keeping.
|
||||
|
||||
Rockbox has no concept of a timezone, so its clock is set to local wall
|
||||
time and the zone has to be supplied from outside. Defaulting to this
|
||||
machine's zone is right whenever the player and the laptop are in the same
|
||||
place, which for a device synced by cable they are.
|
||||
"""
|
||||
if name:
|
||||
return ZoneInfo(name)
|
||||
for candidate in (os.getenv("TZ"), system_zone_name()):
|
||||
if not candidate:
|
||||
continue
|
||||
try:
|
||||
return ZoneInfo(candidate)
|
||||
except (ZoneInfoNotFoundError, ValueError):
|
||||
continue
|
||||
# Nothing named the zone, so the offset cannot be resolved per play: this
|
||||
# is today's offset applied to every timestamp, which is wrong either side
|
||||
# of a daylight saving change. Still better than pretending it logged UTC.
|
||||
print(
|
||||
"warning: no IANA timezone found for this machine; using its current"
|
||||
" offset for every play. Pass --device-timezone to fix older plays"
|
||||
" across a daylight saving change.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return datetime.now().astimezone().tzinfo
|
||||
|
||||
|
||||
def device_time_to_utc(stamp, zone):
|
||||
"""Return the true UTC epoch of a timestamp Rockbox wrote.
|
||||
|
||||
Rockbox builds its timestamps with `mktime(get_time())`, and its mktime
|
||||
(firmware/libc/mktime.c) is plain calendar arithmetic with no zone applied.
|
||||
Fed the RTC's local fields it yields local-wall-clock-as-if-UTC, so the
|
||||
number is ahead of real UTC by whatever the offset was. Its own scrobbler
|
||||
plugin admits this by writing `#TZ/UNKNOWN`, leaving the correction here.
|
||||
|
||||
Decoding the number back into those fields and reinterpreting them in the
|
||||
device's zone recovers the instant, and does so per play, so a log
|
||||
straddling a daylight-saving change converts each side by its own offset.
|
||||
"""
|
||||
fields = datetime.fromtimestamp(stamp, timezone.utc).replace(tzinfo=zone)
|
||||
return int(fields.timestamp())
|
||||
|
||||
|
||||
def future_plays(played, now=None):
|
||||
"""Return the plays timestamped later than now, which cannot have happened.
|
||||
|
||||
A device clock left on the wrong offset, or never adjusted across a
|
||||
daylight-saving change, shows up here. Last.fm has no way to tell such a
|
||||
scrobble from a real one, so it is worth saying out loud.
|
||||
"""
|
||||
now = time.time() if now is None else now
|
||||
return [
|
||||
entry for entry in played
|
||||
if int(entry["timestamp"]) > now + FUTURE_TOLERANCE_SECONDS
|
||||
]
|
||||
|
||||
|
||||
def declares_utc(text):
|
||||
"""Whether an AUDIOSCROBBLER log says its timestamps are already UTC.
|
||||
|
||||
The format's header carries `#TZ/UTC` or `#TZ/UNKNOWN`, and its spec is
|
||||
explicit that a device may only claim UTC if it converted. Rockbox writes
|
||||
UNKNOWN, meaning the times are local wall clock and want correcting; a log
|
||||
from anything that claims UTC must be left alone.
|
||||
"""
|
||||
for line in text.splitlines():
|
||||
if not line.startswith("#"):
|
||||
break
|
||||
if line.strip().upper().startswith("#TZ/"):
|
||||
return line.strip().upper() == "#TZ/UTC"
|
||||
return False
|
||||
|
||||
|
||||
def parse_log(text, zone=None):
|
||||
def parse_log(text):
|
||||
"""Return the listened tracks in an AUDIOSCROBBLER log, oldest first.
|
||||
|
||||
Fields are artist, album, title, track number, length, rating, timestamp
|
||||
and MusicBrainz id. Rockbox converts any tab inside a field to a space
|
||||
before writing, so splitting on tabs is safe.
|
||||
|
||||
Timestamps are corrected from the device's local wall clock to UTC unless
|
||||
the log's own header claims it did that already.
|
||||
"""
|
||||
played, skipped, timeless = [], 0, 0
|
||||
convert = zone is not None and not declares_utc(text)
|
||||
for line in text.splitlines():
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
@@ -195,8 +68,6 @@ def parse_log(text, zone=None):
|
||||
continue
|
||||
if not artist or not title:
|
||||
continue
|
||||
if convert:
|
||||
when = device_time_to_utc(when, zone)
|
||||
played.append(
|
||||
{
|
||||
"artist": artist,
|
||||
@@ -212,136 +83,6 @@ def parse_log(text, zone=None):
|
||||
return played, skipped, timeless
|
||||
|
||||
|
||||
def parse_playback_log(text):
|
||||
"""Return (timestamp, elapsed_ms, length_ms, path) for each logged play."""
|
||||
plays = []
|
||||
for line in text.splitlines():
|
||||
fields = line.strip().split(":", 3)
|
||||
if len(fields) != 4:
|
||||
continue
|
||||
stamp, elapsed, length, path = fields
|
||||
try:
|
||||
plays.append((int(stamp), int(elapsed), int(length), path))
|
||||
except ValueError:
|
||||
continue
|
||||
return plays
|
||||
|
||||
|
||||
def device_to_local(path, device_prefix, mirror):
|
||||
"""Map a path as the player sees it onto the mirror it was copied from."""
|
||||
prefix = "/" + device_prefix.strip("/")
|
||||
if prefix != "/":
|
||||
if not path.startswith(prefix + "/"):
|
||||
return None
|
||||
path = path[len(prefix) :]
|
||||
return Path(mirror) / path.lstrip("/")
|
||||
|
||||
|
||||
def read_tags(path, runner=None):
|
||||
"""Return the tags of a local file, via ffprobe. Empty if it cannot be read.
|
||||
|
||||
A file ffprobe chokes on is one play left unidentified, not a reason to
|
||||
abandon the rest -- and CalledProcessError is not an OSError, so catching
|
||||
the obvious things is not enough.
|
||||
"""
|
||||
runner = runner or _ffprobe
|
||||
try:
|
||||
payload = json.loads(runner(path))
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
return {}
|
||||
return {
|
||||
key.lower(): value
|
||||
for key, value in (payload.get("format", {}).get("tags") or {}).items()
|
||||
}
|
||||
|
||||
|
||||
def _ffprobe(path):
|
||||
return subprocess.run(
|
||||
["ffprobe", "-v", "error", "-show_entries", "format_tags",
|
||||
"-of", "json", str(path)],
|
||||
capture_output=True, text=True, check=True,
|
||||
).stdout
|
||||
|
||||
|
||||
@dataclass
|
||||
class Conversion:
|
||||
"""What a playback log turned into, and what must not be thrown away.
|
||||
|
||||
`retain` holds the raw lines of plays that were real but could not be
|
||||
submitted -- a track absent from the mirror, usually because the sync had
|
||||
not copied it yet. Those are written back so a later run can try again.
|
||||
Skips and clockless entries are not retained: neither can ever be
|
||||
submitted, and the untouched original is set aside regardless.
|
||||
"""
|
||||
|
||||
played: list
|
||||
skipped: int = 0
|
||||
unresolved: int = 0
|
||||
timeless: int = 0
|
||||
retain: list = None
|
||||
|
||||
def __post_init__(self):
|
||||
if self.retain is None:
|
||||
self.retain = []
|
||||
|
||||
|
||||
def plays_from_playback_log(text, device_prefix, mirror, runner=None, zone=None):
|
||||
"""Return the conversion of a playback log.
|
||||
|
||||
Skips are decided by the same fraction the on-device plugin uses, so the
|
||||
two never disagree about what counted as a play. Timestamps are corrected
|
||||
from the device's wall clock to UTC; the core log has no header to say so,
|
||||
but it is written the same way the plugin's UNKNOWN times are.
|
||||
"""
|
||||
result = Conversion(played=[])
|
||||
for line in text.splitlines():
|
||||
parsed = parse_playback_log(line)
|
||||
if not parsed:
|
||||
continue
|
||||
stamp, elapsed, length, device_path = parsed[0]
|
||||
if stamp < EARLIEST_PLAUSIBLE:
|
||||
result.timeless += 1
|
||||
continue
|
||||
if length > 0 and elapsed < length * LISTENED_FRACTION:
|
||||
result.skipped += 1
|
||||
continue
|
||||
local = device_to_local(device_path, device_prefix, mirror)
|
||||
tags = read_tags(local, runner) if local and local.is_file() else {}
|
||||
artist = tags.get("artist") or tags.get("album_artist") or ""
|
||||
title = tags.get("title") or ""
|
||||
if not artist or not title:
|
||||
# A real play of a track this run could not identify. Kept, so a
|
||||
# later run -- after the file has been copied, or the tags fixed --
|
||||
# can submit it rather than the play being lost.
|
||||
result.unresolved += 1
|
||||
result.retain.append(line)
|
||||
continue
|
||||
result.played.append(
|
||||
{
|
||||
"artist": artist,
|
||||
"track": title,
|
||||
"album": tags.get("album", ""),
|
||||
"trackNumber": (tags.get("track") or "").split("/")[0],
|
||||
"duration": str(length // 1000) if length > 0 else "",
|
||||
"timestamp": str(
|
||||
device_time_to_utc(stamp, zone) if zone is not None else stamp
|
||||
),
|
||||
"mbid": tags.get("musicbrainz_trackid", ""),
|
||||
"line": line,
|
||||
}
|
||||
)
|
||||
result.played.sort(key=lambda entry: int(entry["timestamp"]))
|
||||
return result
|
||||
|
||||
|
||||
def find_playback_logs(device):
|
||||
"""Return every playback log on a device, oldest first."""
|
||||
found = []
|
||||
for pattern in PLAYBACK_LOG_NAMES:
|
||||
found.extend(sorted(Path(device).glob(f".rockbox/{pattern}")))
|
||||
return [path for path in found if path.is_file() and path.stat().st_size]
|
||||
|
||||
|
||||
def sign(params, secret):
|
||||
"""Return Last.fm's method signature for a set of parameters.
|
||||
|
||||
@@ -411,7 +152,7 @@ def batch_params(entries):
|
||||
return params
|
||||
|
||||
|
||||
def submit(entries, key, secret, session, transport, delay=1.0, on_sent=None):
|
||||
def submit(entries, key, secret, session, transport, delay=1.0):
|
||||
"""Submit every entry. Returns how many the service accepted.
|
||||
|
||||
Batches are counted as they succeed rather than at the end, so a failure
|
||||
@@ -427,8 +168,6 @@ def submit(entries, key, secret, session, transport, delay=1.0, on_sent=None):
|
||||
block = payload.get("scrobbles", {})
|
||||
summary = block.get("@attr", block)
|
||||
accepted += int(summary.get("accepted", len(chunk)))
|
||||
if on_sent is not None:
|
||||
on_sent(chunk)
|
||||
ignored = int(summary.get("ignored", 0))
|
||||
if ignored:
|
||||
print(f" {ignored} of {len(chunk)} ignored by Last.fm", file=sys.stderr)
|
||||
@@ -454,26 +193,6 @@ def find_log(device):
|
||||
def main(argv=None, transport=http_post):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("device", help="the mounted device, or a scrobbler log file")
|
||||
parser.add_argument(
|
||||
"--mirror",
|
||||
help="the mirror the device was copied from. Given this, Rockbox's own"
|
||||
" playback.log is converted here rather than needing the on-device"
|
||||
" plugin run first",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--device-prefix",
|
||||
default="/Music",
|
||||
help="where the music sits on the device, stripped when mapping a logged"
|
||||
" path back onto the mirror",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--device-timezone",
|
||||
default=os.getenv("ROCKBOX_TIMEZONE"),
|
||||
help="the zone the player's clock is set to, as an IANA name such as"
|
||||
" Europe/London. Rockbox keeps local wall time and cannot record an"
|
||||
" offset, so its timestamps need this to become the UTC Last.fm wants."
|
||||
" Defaults to this machine's zone",
|
||||
)
|
||||
parser.add_argument("--api-key", default=os.getenv("LASTFM_API_KEY"))
|
||||
parser.add_argument("--api-secret", default=os.getenv("LASTFM_API_SECRET"))
|
||||
parser.add_argument("--dry-run", action="store_true", help="parse and report only")
|
||||
@@ -482,61 +201,14 @@ def main(argv=None, transport=http_post):
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
zone = device_zone(args.device_timezone)
|
||||
except (ZoneInfoNotFoundError, ValueError) as error:
|
||||
print(f"unknown timezone {args.device_timezone!r}: {error}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
target = Path(args.device)
|
||||
log = target if target.is_file() else find_log(target)
|
||||
logs = []
|
||||
|
||||
conversion = None
|
||||
if log is not None:
|
||||
played, skipped, timeless = parse_log(
|
||||
log.read_text(encoding="utf-8", errors="replace"), zone
|
||||
)
|
||||
unresolved = 0
|
||||
logs = [log]
|
||||
print(f"{log}: {len(played)} listened, {skipped} skipped", file=sys.stderr)
|
||||
elif args.mirror:
|
||||
# No plugin has been run, but the core log is there. Tags come off the
|
||||
# mirror, which is the only reason the plugin was needed at all.
|
||||
logs = find_playback_logs(target)
|
||||
if not logs:
|
||||
print("no scrobbler log to submit", file=sys.stderr)
|
||||
return 0
|
||||
text = "\n".join(
|
||||
path.read_text(encoding="utf-8", errors="replace") for path in logs
|
||||
)
|
||||
conversion = plays_from_playback_log(
|
||||
text, args.device_prefix, args.mirror, zone=zone
|
||||
)
|
||||
played = conversion.played
|
||||
skipped, unresolved, timeless = (
|
||||
conversion.skipped,
|
||||
conversion.unresolved,
|
||||
conversion.timeless,
|
||||
)
|
||||
print(
|
||||
f"{len(logs)} playback log(s): {len(played)} listened, {skipped} skipped",
|
||||
file=sys.stderr,
|
||||
)
|
||||
if unresolved:
|
||||
print(
|
||||
f" {unresolved} could not be matched to a file in the mirror."
|
||||
" Those plays are kept for a later run rather than discarded.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
else:
|
||||
print(
|
||||
"no scrobbler log to submit. Rockbox's own playback.log can be used"
|
||||
" instead -- pass --mirror so tags can be read from it.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
if log is None:
|
||||
print("no scrobbler log to submit", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
played, skipped, timeless = parse_log(log.read_text(encoding="utf-8", errors="replace"))
|
||||
print(f"{log}: {len(played)} listened, {skipped} skipped", file=sys.stderr)
|
||||
if timeless:
|
||||
print(
|
||||
f" {timeless} entries have no timestamp, so this target has no clock."
|
||||
@@ -545,24 +217,9 @@ def main(argv=None, transport=http_post):
|
||||
)
|
||||
if not played:
|
||||
return 0
|
||||
|
||||
ahead = future_plays(played)
|
||||
if ahead:
|
||||
newest = int(ahead[-1]["timestamp"]) - int(time.time())
|
||||
print(
|
||||
f" {len(ahead)} plays are timestamped up to {newest // 60} minutes in"
|
||||
f" the future, converting from {zone}. Either the player's clock is"
|
||||
" wrong or that is not the zone it is set to.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
if args.dry_run:
|
||||
for entry in played[:20]:
|
||||
when = datetime.fromtimestamp(int(entry["timestamp"]), zone)
|
||||
print(
|
||||
f"{entry['timestamp']}\t{when:%Y-%m-%d %H:%M %Z}"
|
||||
f"\t{entry['artist']}\t{entry['track']}"
|
||||
)
|
||||
print(f"{entry['timestamp']}\t{entry['artist']}\t{entry['track']}")
|
||||
return 0
|
||||
|
||||
if not args.api_key or not args.api_secret:
|
||||
@@ -578,66 +235,21 @@ def main(argv=None, transport=http_post):
|
||||
session = authorise(args.api_key, args.api_secret, transport)
|
||||
save_session(session)
|
||||
|
||||
# Recorded as each batch is accepted, so a failure partway through knows
|
||||
# exactly what got through and what did not.
|
||||
sent = []
|
||||
try:
|
||||
accepted = submit(
|
||||
played, args.api_key, args.api_secret, session, transport,
|
||||
on_sent=sent.extend,
|
||||
)
|
||||
accepted = submit(played, args.api_key, args.api_secret, session, transport)
|
||||
except LastfmError as error:
|
||||
print(f"submission failed after {len(sent)} scrobbles: {error}", file=sys.stderr)
|
||||
keep_history(logs, played, sent, conversion, args.keep)
|
||||
print(f"submission failed: {error}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"{accepted} scrobbles accepted", file=sys.stderr)
|
||||
keep_history(logs, played, sent, conversion, args.keep)
|
||||
if not args.keep and accepted:
|
||||
# Renamed rather than deleted: if Last.fm quietly dropped something,
|
||||
# the evidence is still on the device.
|
||||
aside = log.with_name(f"{log.name}.{played[-1]['timestamp']}.submitted")
|
||||
log.rename(aside)
|
||||
print(f"log moved to {aside.name}", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
def keep_history(logs, played, sent, conversion, keep):
|
||||
"""Set the logs aside, writing back anything still owed a submission.
|
||||
|
||||
Two separate obligations. The original is preserved untouched, renamed
|
||||
rather than deleted, so a play is never lost to a mistake here. And any
|
||||
play that was not submitted -- unmatched, or in a batch that failed -- is
|
||||
written back into a live log, so the next run tries it again instead of it
|
||||
quietly vanishing with the rest.
|
||||
"""
|
||||
if keep or not logs:
|
||||
if keep:
|
||||
print("logs left in place", file=sys.stderr)
|
||||
return
|
||||
|
||||
submitted = {id(entry) for entry in sent}
|
||||
# A .scrobbler.log was converted by the on-device plugin and carries no
|
||||
# per-line record, so there is nothing to write back for it -- only the
|
||||
# rename below, which loses nothing.
|
||||
pending = list(conversion.retain) if conversion is not None else []
|
||||
pending += [
|
||||
entry["line"] for entry in played
|
||||
if "line" in entry and id(entry) not in submitted
|
||||
]
|
||||
|
||||
if not sent:
|
||||
print("nothing was accepted; logs left untouched", file=sys.stderr)
|
||||
return
|
||||
|
||||
stamp = played[-1]["timestamp"] if played else "0"
|
||||
for path in logs:
|
||||
path.rename(path.with_name(f"{path.name}.{stamp}.submitted"))
|
||||
|
||||
if pending:
|
||||
live = logs[0].with_name("playback.log")
|
||||
live.write_text("\n".join(pending) + "\n", encoding="utf-8")
|
||||
print(
|
||||
f"{len(pending)} plays not submitted were written back to"
|
||||
f" {live.name} for the next run",
|
||||
file=sys.stderr,
|
||||
)
|
||||
print(f"{len(logs)} log(s) set aside as .submitted", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
||||
+17
-159
@@ -22,31 +22,14 @@ usage() {
|
||||
usage: sync-to-ipod.sh [options] <mirror> <destination>
|
||||
|
||||
-n dry run; show what would change and touch nothing
|
||||
-P count what needs copying first, so progress can show a percentage and
|
||||
an estimate. Costs a second full traversal of both trees, which on a
|
||||
FAT card of fifty thousand files is slower than the transfer itself
|
||||
-f copy even if the FAT32 check finds unacceptable paths
|
||||
-S skip submitting the Rockbox scrobbler log to Last.fm
|
||||
-B skip rebuilding the Rockbox database
|
||||
-U leave the destination mounted afterwards
|
||||
|
||||
Rebuilding the database needs MUSIC_MIRROR_DATABASE_TOOL pointing at Rockbox's
|
||||
host-side builder (tools/database, built with ./tools/configure --type=d). It
|
||||
is skipped with a note when unset. The scan reads tags from the mirror rather
|
||||
than from the device, so it costs seconds rather than the hours an on-device
|
||||
commit takes -- and on a large library the on-device commit may not finish at
|
||||
all.
|
||||
|
||||
Submitting scrobbles needs LASTFM_API_KEY and LASTFM_API_SECRET; it is skipped
|
||||
with a note when they are unset. Scrobbling is a write method and needs the
|
||||
secret, unlike the read-only calls elsewhere in these projects.
|
||||
|
||||
Rockbox has no notion of a timezone: its clock holds local wall time and its
|
||||
logs record that, not UTC. Set ROCKBOX_TIMEZONE to the zone the player's clock
|
||||
is keeping (an IANA name, such as Europe/London) if it differs from this
|
||||
machine's, which is otherwise assumed. Getting it wrong shifts every scrobble
|
||||
by the difference.
|
||||
|
||||
The mirror is the directory holding the artist folders. The destination is
|
||||
where those folders should end up on the device -- not the card root, unless
|
||||
that is genuinely where you want them:
|
||||
@@ -62,12 +45,6 @@ The destination must be on a mounted FAT filesystem. That check is also what
|
||||
catches an unmounted device: /media/IPOD/Music then resolves to the host's own
|
||||
root filesystem, and this refuses to empty that.
|
||||
|
||||
Progress is one line that rewrites itself, showing the album currently going
|
||||
across, how far through the transfer is, the rate, and an estimate of what is
|
||||
left. Working the totals out first means a second pass over the tree, which is
|
||||
the price of figures that mean something; rsync's own percentage is computed
|
||||
against a file list it is still building.
|
||||
|
||||
Reach the device with the Apple firmware's disk mode: Menu+Select to reboot,
|
||||
then immediately Select+Play. Power off afterwards by holding Play.
|
||||
USAGE
|
||||
@@ -75,21 +52,17 @@ USAGE
|
||||
}
|
||||
|
||||
dry_run=false
|
||||
counting=false
|
||||
force=false
|
||||
unmount=true
|
||||
scrobble=true
|
||||
database=true
|
||||
for argument in "$@"; do
|
||||
[ "$argument" = "--help" ] && usage help
|
||||
done
|
||||
while getopts ":nPfSBUh" option; do
|
||||
while getopts ":nfSUh" option; do
|
||||
case "$option" in
|
||||
n) dry_run=true ;;
|
||||
P) counting=true ;;
|
||||
f) force=true ;;
|
||||
S) scrobble=false ;;
|
||||
B) database=false ;;
|
||||
U) unmount=false ;;
|
||||
h) usage help ;;
|
||||
*) usage ;;
|
||||
@@ -164,16 +137,7 @@ if $scrobble; then
|
||||
else
|
||||
scrobble_options=()
|
||||
$dry_run && scrobble_options+=(--dry-run)
|
||||
# Rockbox keeps local wall time with no notion of a zone, so its
|
||||
# timestamps are not the UTC Last.fm expects. Naming the zone the
|
||||
# player's clock is set to lets them be corrected.
|
||||
[ -n "${ROCKBOX_TIMEZONE:-}" ] &&
|
||||
scrobble_options+=(--device-timezone "$ROCKBOX_TIMEZONE")
|
||||
# --mirror lets it convert Rockbox's own playback.log, so the on-device
|
||||
# scrobbler plugin never has to be run. The device root, not the music
|
||||
# directory: the logs live in .rockbox.
|
||||
python3 "$here/submit_scrobbles.py" "${scrobble_options[@]}" \
|
||||
--mirror "$mirror" --device-prefix "$device_prefix" "$mounted_on" ||
|
||||
python3 "$here/submit_scrobbles.py" "${scrobble_options[@]}" "$destination" ||
|
||||
die "submitting scrobbles failed; nothing has been copied"
|
||||
fi
|
||||
fi
|
||||
@@ -182,142 +146,36 @@ fi
|
||||
# asking for them produces a screenful of errors and a non-zero exit.
|
||||
# --modify-window=2 because FAT stores mtimes to two-second resolution, without
|
||||
# which every file looks changed and the whole library is copied every time.
|
||||
# --whole-file is already the default when both ends are local paths, and an
|
||||
# SMB or FAT mount counts as one, but stating it documents that the delta
|
||||
# algorithm is deliberately not wanted: it would read every destination file
|
||||
# back over USB to compute a checksum, to save sending an MP3 that has changed
|
||||
# entirely anyway.
|
||||
#
|
||||
# --omit-dir-times drops a setattr round trip per directory. Across six
|
||||
# thousand album folders on a FAT card that is six thousand operations to set
|
||||
# timestamps nothing reads.
|
||||
options=(--recursive --times --delete --modify-window=2 --whole-file --omit-dir-times)
|
||||
# --delete removes tracks whose source has gone, which is the point. It would
|
||||
# also remove everything on the device that the mirror does not contain -- and
|
||||
# if the destination is the card root that means /.rockbox, the Rockbox install
|
||||
# itself. Excluded paths are not deleted unless --delete-excluded is given,
|
||||
# which it never is here.
|
||||
options=(--recursive --times --delete --modify-window=2 --human-readable --info=progress2)
|
||||
for owned in "/.rockbox" "/.scrobbler.log" "/.scrobbler.log.*" "/.playlist_control" \
|
||||
"/System Volume Information" "/.Spotlight-V100" "/.Trashes" "/.fseventsd"; do
|
||||
options+=(--exclude "$owned")
|
||||
done
|
||||
$dry_run && options+=(--dry-run --verbose)
|
||||
|
||||
printf 'sync-to-ipod: %s -> %s\n' "$mirror" "$destination" >&2
|
||||
rsync "${options[@]}" "$mirror/" "$destination/"
|
||||
|
||||
if $dry_run; then
|
||||
rsync "${options[@]}" --dry-run --verbose "$mirror/" "$destination/"
|
||||
printf 'sync-to-ipod: dry run, nothing was written\n' >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# rsync says nothing at all while it builds its file list, which on fifty
|
||||
# thousand files over USB is minutes of apparent hang. Counting first costs a
|
||||
# second pass over the tree but means the transfer can show a real percentage
|
||||
# rather than a number that grows as rsync discovers more work.
|
||||
# Counting is opt-in because it is not cheap. It walks and compares both trees
|
||||
# in full, exactly as the transfer does, and on a FAT card holding fifty
|
||||
# thousand files that traversal costs more than moving the data. Without it the
|
||||
# progress line still shows the running count, the rate and the album in
|
||||
# flight; only the percentage and the estimate are lost, and those were the
|
||||
# least useful part of it.
|
||||
total=0
|
||||
total_bytes=0
|
||||
if $counting; then
|
||||
printf 'sync-to-ipod: working out what needs copying...\n' >&2
|
||||
# %l is the file's size, which is what makes an estimate possible.
|
||||
# Directories are dropped: rsync reports those too, with an inode size that
|
||||
# would inflate the total by several megabytes of nothing.
|
||||
counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
|
||||
awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }')
|
||||
total=${counted% *}
|
||||
total_bytes=${counted#* }
|
||||
printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
|
||||
sync
|
||||
if $unmount; then
|
||||
device=$(findmnt -no SOURCE --target "$destination")
|
||||
printf 'sync-to-ipod: unmounting %s\n' "$device" >&2
|
||||
if command -v udisksctl >/dev/null 2>&1; then
|
||||
udisksctl unmount -b "$device"
|
||||
else
|
||||
umount -- "$destination"
|
||||
fi
|
||||
printf 'sync-to-ipod: safe to disconnect\n' >&2
|
||||
else
|
||||
printf 'sync-to-ipod: still mounted; unmount before disconnecting\n' >&2
|
||||
fi
|
||||
|
||||
# Flushing and unmounting is the whole reason this is a script, so it has to
|
||||
# happen on the way out whichever way that is. Ctrl-C during a transfer would
|
||||
# otherwise leave a FAT filesystem with dirty buffers and no journal, which is
|
||||
# the corruption this exists to avoid.
|
||||
finish() {
|
||||
sync
|
||||
if $unmount; then
|
||||
device=$(findmnt -no SOURCE --target "$destination" 2>/dev/null || true)
|
||||
if [ -n "$device" ]; then
|
||||
printf 'sync-to-ipod: unmounting %s\n' "$device" >&2
|
||||
if command -v udisksctl >/dev/null 2>&1; then
|
||||
udisksctl unmount -b "$device" || umount -- "$destination" || true
|
||||
else
|
||||
umount -- "$destination" || true
|
||||
fi
|
||||
printf 'sync-to-ipod: safe to disconnect\n' >&2
|
||||
fi
|
||||
else
|
||||
printf 'sync-to-ipod: still mounted; unmount before disconnecting\n' >&2
|
||||
fi
|
||||
}
|
||||
|
||||
interrupted() {
|
||||
trap - INT TERM
|
||||
printf '\nsync-to-ipod: interrupted -- rsync leaves no partial files, but the\n' >&2
|
||||
printf 'sync-to-ipod: filesystem still needs flushing before you pull anything\n' >&2
|
||||
finish
|
||||
exit 130
|
||||
}
|
||||
|
||||
trap interrupted INT TERM
|
||||
|
||||
rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" |
|
||||
python3 "$here/rsync_progress.py" --total "$total" --bytes "$total_bytes"
|
||||
status=${PIPESTATUS[0]}
|
||||
[ "$status" -eq 0 ] || die "rsync exited $status"
|
||||
|
||||
# Rockbox reads its database from .tcd files in .rockbox. Building them here
|
||||
# rather than on the device is not just faster: the on-device commit sorts the
|
||||
# whole index in whatever memory it can scrape together, and on a large library
|
||||
# it runs for hours or dies outright.
|
||||
#
|
||||
# The scan reads tags through a scratch root -- a real .rockbox beside a symlink
|
||||
# standing in for where the music lands on the device -- so the paths recorded
|
||||
# match what Rockbox will look up, while the bytes are read from the mirror
|
||||
# instead of over USB. The scratch is kept between runs because the builder is
|
||||
# incremental: a second pass over unchanged files does no work at all.
|
||||
rebuild_database() {
|
||||
local tool=${MUSIC_MIRROR_DATABASE_TOOL:-}
|
||||
if [ -z "$tool" ]; then
|
||||
printf 'sync-to-ipod: no database tool configured, skipping the database\n' >&2
|
||||
return 0
|
||||
fi
|
||||
[ -x "$tool" ] || die "$tool is not executable"
|
||||
|
||||
local device_rockbox="$mounted_on/.rockbox"
|
||||
if [ ! -d "$device_rockbox" ]; then
|
||||
printf 'sync-to-ipod: no .rockbox on the device, skipping the database\n' >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
local scratch="${XDG_CACHE_HOME:-$HOME/.cache}/music-mirror/database"
|
||||
mkdir -p "$scratch/.rockbox"
|
||||
|
||||
# Rebuild the symlink layout each time; the mirror path or the device
|
||||
# prefix may have changed since the last run.
|
||||
find "$scratch" -maxdepth 1 -type l -delete
|
||||
if [ "$device_prefix" = "/" ]; then
|
||||
ln -s "$mirror"/* "$scratch/" 2>/dev/null || true
|
||||
else
|
||||
local under=${device_prefix#/}
|
||||
rm -rf "${scratch:?}/${under%%/*}"
|
||||
mkdir -p "$scratch/$(dirname "$under")"
|
||||
ln -s "$mirror" "$scratch/$under"
|
||||
fi
|
||||
|
||||
printf 'sync-to-ipod: building the database from the mirror...\n' >&2
|
||||
( cd "$scratch" && "$tool" ) >/dev/null || die "the database build failed"
|
||||
|
||||
cp -- "$scratch"/.rockbox/*.tcd "$device_rockbox/" ||
|
||||
die "could not copy the database onto the device"
|
||||
printf 'sync-to-ipod: database copied to %s\n' "$device_rockbox" >&2
|
||||
}
|
||||
|
||||
$database && rebuild_database
|
||||
|
||||
finish
|
||||
|
||||
Reference in New Issue
Block a user