feat: rebuild the Rockbox database during the sync, off the mirror
Build and publish container / build (pull_request) Successful in 6m36s

The on-device database commit does not work at this library size. It sorts the
whole index in whatever memory core_alloc_maximum() can scrape together, and on
fifty thousand tracks it runs for hours or aborts with a data abort -- observed
across several builds including stable.

Rockbox ships a host-side builder for exactly this, and the sync is the moment
the library changes, so it belongs here. MUSIC_MIRROR_DATABASE_TOOL points at
it; the step is skipped with a note when unset, as the scrobbler step is.

The scan runs against a scratch root -- a real .rockbox beside a symlink
standing in for wherever the music lands on the device -- so the paths recorded
are the ones Rockbox will look up, while the bytes are read from the mirror
rather than over USB. Only the dozen .tcd files cross to the card. Verified:
scanning through the symlink records /Music/... paths while reading from
somewhere else entirely.

The scratch root is kept between runs because the builder is incremental. A
second pass over unchanged files performs no metadata reads and finishes in a
fraction of a second, so only the first build pays the full cost.

That cost, measured rather than guessed: about 49 reads and 43 seeks per file,
the parser probing the head for ID3v2 and the tail for ID3v1. Two thousand
files in half a second on local disk. Over SMB the opens and the head/tail
split are real round trips, making a first full scan minutes rather than
seconds -- still preferable to an on-device commit that does not finish. There
is nothing to parallelise: the tool is single-threaded and two instances cannot
produce one database.
This commit is contained in:
Emma Thorpe
2026-08-26 13:20:20 +01:00
parent 9091c4d049
commit c99b423b72
3 changed files with 141 additions and 1 deletions
+38
View File
@@ -185,6 +185,44 @@ is what a percentage and an estimate that mean something cost — and renders th
rest itself. Piped to a log it prints a plain line every thirty seconds
instead, with no carriage returns, and a summary at the end either way.
### The Rockbox database
Point `MUSIC_MIRROR_DATABASE_TOOL` at Rockbox's host-side builder and the sync
rebuilds the database itself, so it never has to happen on the device.
```sh
git clone --depth 1 https://github.com/Rockbox/rockbox.git
cd rockbox && mkdir build-db && cd build-db
../tools/configure --target=ipodvideo --type=d && make -j$(nproc)
```
It needs a native compiler and SDL2 development headers, not the ARM
cross-toolchain, and `tools/configure` detects `__aarch64__` correctly. On a
distribution without `/usr/bin/perl` or `gcc-ar` — NixOS, say — patch the
shebangs in `tools/*.pl` and pass `AR=ar`.
Building it here rather than on the device is not merely faster. The on-device
commit sorts the whole index in whatever memory `core_alloc_maximum()` can
scrape together; on a fifty-thousand-track library it runs for hours or aborts
outright.
The scan runs against a scratch root — a real `.rockbox` beside a symlink
standing in for wherever the music lands on the device — so the paths recorded
are the ones Rockbox will look up, while the **bytes are read from the mirror
rather than over USB**. Only the dozen `.tcd` files cross to the card.
Cost, measured: the parser makes about 49 reads and 43 seeks per file, probing
the head for ID3v2 and the tail for ID3v1. On a local disk that is 2,000 files
in half a second. Over SMB, readahead absorbs most of the reads but the opens
and the head/tail split are real round trips, so a first full scan is minutes
rather than seconds. It is a one-time cost: the builder is incremental, and the
scratch root is kept between runs, so a later pass over unchanged files does no
metadata reads at all.
If minutes is still too many, run the builder where the mirror is local — on
the NAS — and copy the `.tcd` files across. There is nothing to parallelise:
the tool is single-threaded, and two instances cannot produce one database.
### If the sync is interrupted
No partially copied track is ever left under a name Rockbox would play. rsync