feat: rebuild the Rockbox database during the sync, off the mirror
Build and publish container / build (pull_request) Successful in 6m36s

The on-device database commit does not work at this library size. It sorts the
whole index in whatever memory core_alloc_maximum() can scrape together, and on
fifty thousand tracks it runs for hours or aborts with a data abort -- observed
across several builds including stable.

Rockbox ships a host-side builder for exactly this, and the sync is the moment
the library changes, so it belongs here. MUSIC_MIRROR_DATABASE_TOOL points at
it; the step is skipped with a note when unset, as the scrobbler step is.

The scan runs against a scratch root -- a real .rockbox beside a symlink
standing in for wherever the music lands on the device -- so the paths recorded
are the ones Rockbox will look up, while the bytes are read from the mirror
rather than over USB. Only the dozen .tcd files cross to the card. Verified:
scanning through the symlink records /Music/... paths while reading from
somewhere else entirely.

The scratch root is kept between runs because the builder is incremental. A
second pass over unchanged files performs no metadata reads and finishes in a
fraction of a second, so only the first build pays the full cost.

That cost, measured rather than guessed: about 49 reads and 43 seeks per file,
the parser probing the head for ID3v2 and the tail for ID3v1. Two thousand
files in half a second on local disk. Over SMB the opens and the head/tail
split are real round trips, making a first full scan minutes rather than
seconds -- still preferable to an on-device commit that does not finish. There
is nothing to parallelise: the tool is single-threaded and two instances cannot
produce one database.
This commit is contained in:
Emma Thorpe
2026-08-26 13:20:20 +01:00
parent 9091c4d049
commit c99b423b72
3 changed files with 141 additions and 1 deletions
+60 -1
View File
@@ -26,8 +26,16 @@ usage: sync-to-ipod.sh [options] <mirror> <destination>
of the source tree, which over SMB is the expensive part
-f copy even if the FAT32 check finds unacceptable paths
-S skip submitting the Rockbox scrobbler log to Last.fm
-B skip rebuilding the Rockbox database
-U leave the destination mounted afterwards
Rebuilding the database needs MUSIC_MIRROR_DATABASE_TOOL pointing at Rockbox's
host-side builder (tools/database, built with ./tools/configure --type=d). It
is skipped with a note when unset. The scan reads tags from the mirror rather
than from the device, so it costs seconds rather than the hours an on-device
commit takes -- and on a large library the on-device commit may not finish at
all.
Submitting scrobbles needs LASTFM_API_KEY and LASTFM_API_SECRET; it is skipped
with a note when they are unset. Scrobbling is a write method and needs the
secret, unlike the read-only calls elsewhere in these projects.
@@ -64,15 +72,17 @@ quick=false
force=false
unmount=true
scrobble=true
database=true
for argument in "$@"; do
[ "$argument" = "--help" ] && usage help
done
while getopts ":nQfSUh" option; do
while getopts ":nQfSBUh" option; do
case "$option" in
n) dry_run=true ;;
Q) quick=true ;;
f) force=true ;;
S) scrobble=false ;;
B) database=false ;;
U) unmount=false ;;
h) usage help ;;
*) usage ;;
@@ -241,4 +251,53 @@ rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" |
status=${PIPESTATUS[0]}
[ "$status" -eq 0 ] || die "rsync exited $status"
# Rockbox reads its database from .tcd files in .rockbox. Building them here
# rather than on the device is not just faster: the on-device commit sorts the
# whole index in whatever memory it can scrape together, and on a large library
# it runs for hours or dies outright.
#
# The scan reads tags through a scratch root -- a real .rockbox beside a symlink
# standing in for where the music lands on the device -- so the paths recorded
# match what Rockbox will look up, while the bytes are read from the mirror
# instead of over USB. The scratch is kept between runs because the builder is
# incremental: a second pass over unchanged files does no work at all.
rebuild_database() {
local tool=${MUSIC_MIRROR_DATABASE_TOOL:-}
if [ -z "$tool" ]; then
printf 'sync-to-ipod: no database tool configured, skipping the database\n' >&2
return 0
fi
[ -x "$tool" ] || die "$tool is not executable"
local device_rockbox="$mounted_on/.rockbox"
if [ ! -d "$device_rockbox" ]; then
printf 'sync-to-ipod: no .rockbox on the device, skipping the database\n' >&2
return 0
fi
local scratch="${XDG_CACHE_HOME:-$HOME/.cache}/music-mirror/database"
mkdir -p "$scratch/.rockbox"
# Rebuild the symlink layout each time; the mirror path or the device
# prefix may have changed since the last run.
find "$scratch" -maxdepth 1 -type l -delete
if [ "$device_prefix" = "/" ]; then
ln -s "$mirror"/* "$scratch/" 2>/dev/null || true
else
local under=${device_prefix#/}
rm -rf "${scratch:?}/${under%%/*}"
mkdir -p "$scratch/$(dirname "$under")"
ln -s "$mirror" "$scratch/$under"
fi
printf 'sync-to-ipod: building the database from the mirror...\n' >&2
( cd "$scratch" && "$tool" ) >/dev/null || die "the database build failed"
cp -- "$scratch"/.rockbox/*.tcd "$device_rockbox/" ||
die "could not copy the database onto the device"
printf 'sync-to-ipod: database copied to %s\n' "$device_rockbox" >&2
}
$database && rebuild_database
finish