The on-device database commit does not work at this library size. It sorts the
whole index in whatever memory core_alloc_maximum() can scrape together, and on
fifty thousand tracks it runs for hours or aborts with a data abort -- observed
across several builds including stable.
Rockbox ships a host-side builder for exactly this, and the sync is the moment
the library changes, so it belongs here. MUSIC_MIRROR_DATABASE_TOOL points at
it; the step is skipped with a note when unset, as the scrobbler step is.
The scan runs against a scratch root -- a real .rockbox beside a symlink
standing in for wherever the music lands on the device -- so the paths recorded
are the ones Rockbox will look up, while the bytes are read from the mirror
rather than over USB. Only the dozen .tcd files cross to the card. Verified:
scanning through the symlink records /Music/... paths while reading from
somewhere else entirely.
The scratch root is kept between runs because the builder is incremental. A
second pass over unchanged files performs no metadata reads and finishes in a
fraction of a second, so only the first build pays the full cost.
That cost, measured rather than guessed: about 49 reads and 43 seeks per file,
the parser probing the head for ID3v2 and the tail for ID3v1. Two thousand
files in half a second on local disk. Over SMB the opens and the head/tail
split are real round trips, making a first full scan minutes rather than
seconds -- still preferable to an on-device commit that does not finish. There
is nothing to parallelise: the tool is single-threaded and two instances cannot
produce one database.
The README claimed that interrupting left buffers unwritten and therefore
corruption. That is wrong. The kernel flushes dirty pages within
dirty_expire_centisecs, thirty seconds by default, and umount syncs before it
returns, so losing data requires interrupting and pulling the card inside that
window and skipping the unmount.
The trap is still worth having, for a smaller and more honest reason: it
removes a manual step and makes the exit deterministic, so the same "safe to
disconnect" appears whichever way the run ends. What actually loses data is
pulling the card without unmounting at all, which has nothing to do with
whether the transfer was interrupted.
rsync itself is safe under interruption. It writes to a hidden temporary file
and renames it into place only once complete, and by default deletes any
partial file when interrupted -- verified both in the manual and by killing a
transfer and inspecting what was left, which was nothing. --partial is
deliberately absent and there is now a test asserting it stays that way. An
unclean kill can leave a hidden .track.mp3.XXXXXX behind; it is unplayable, it
is not in the source, and the next run's --delete removes it.
The script was not safe. Ctrl-C killed it before the sync and the unmount,
leaving a journal-less FAT filesystem holding dirty buffers -- which is the
exact corruption the script exists to prevent, arrived at by the most likely
route a person would take.
INT and TERM are now trapped. Both the normal path and the interrupt path call
the same finish function, so the flush and the unmount cannot drift apart, and
an interrupted run exits 130 rather than pretending to have succeeded.
The test is structural rather than timed. Reproducing a mid-transfer signal
needs a payload large enough to be slow, and a test that depends on winning a
race is a test that fails in CI for reasons that have nothing to do with the
code. The behaviour was verified by hand: SIGTERM mid-transfer gave exit 130,
the flush ran, and the destination held no short files and no leftover
temporaries.
The transfer is metadata-bound rather than throughput-bound. Fifty thousand
files is fifty thousand round trips, and the counting pass added for the
percentage doubles that.
--whole-file is already implied when both ends are local paths, which an SMB or
FAT mount is, but stating it records that the delta algorithm is deliberately
unwanted here: it would read every destination file back over USB to checksum
it, in order to avoid resending an MP3 that has changed in its entirety anyway.
--omit-dir-times drops one setattr per directory. Across six thousand album
folders on a FAT card that is six thousand operations spent on timestamps
nothing reads.
-Q skips the counting pass. The percentage and the estimate are worth a second
walk of a local tree and frequently are not worth one of a network mount, so
that is now a choice rather than a fixed cost.
The README covers the part that is not an rsync flag at all: SMB defaults to a
one second attribute cache, so nearly every stat goes to the wire, twice. An
actimeo of sixty on the mount does more than any of the above, and closes most
of the gap that would otherwise argue for moving to NFS.
rsync reports each file's size with %l as it completes, which is all an
estimate needs: bytes done over time elapsed is the same arithmetic rsync would
do internally, and requires nothing it does not already print. The scan pass now
sums those sizes as well as counting files, so both a percentage and an estimate
have a real denominator.
The rate is measured over a trailing thirty seconds rather than the whole run,
so it follows a device that slows down instead of averaging the slowdown away --
which for a card reader that thermally throttles, or a USB link that renegotiates
after an hour, is the difference between a useful estimate and a reassuring one.
Below two seconds no rate is reported at all. The first handful of files arrive
microseconds apart, and dividing by that window produces a rate in the gigabytes
per second and an estimate of zero, which is worse than showing nothing.
Directory entries are excluded from the byte total as well as the file count.
rsync reports them with a 4096 inode size, which across six thousand album
directories is several megabytes of transfer that never happens.
The transfer looked hung. rsync prints nothing while it builds its file list,
which on fifty thousand files over USB is several minutes of silence, and
--info=progress2 does not help: with incremental recursion its percentage is
computed against a list rsync has not finished discovering, so it moves
backwards as often as forwards.
The script now counts what needs copying first and says so, then renders its
own single line that rewrites in place, showing the album currently going
across and a percentage against a total that is actually known. Counting costs
a second pass over the tree. That is the price of a percentage meaning
something, and it is cheaper than staring at a blank terminal wondering whether
the thing has died.
Directories are excluded from the count. rsync reports those too, and including
them puts the figure past a hundred per cent.
Piped to a log the line becomes a plain one every thirty seconds, because a log
full of carriage returns and escape codes is not a log anybody reads.
The script required the destination to be its own mount point, while the
documentation and its own usage text both told the user to pass
/media/IPOD/Music. The documented invocation was rejected.
A subdirectory is the better target, so the guard was what was wrong. --delete
is confined to it, and the device path budget is now derived from it -- the
part of the destination below its mount point -- rather than configured, so the
budget cannot disagree with where the files are actually going. The check that
matters is that the destination sits on a FAT filesystem, which is also what
catches an unmounted device: /media/IPOD/Music then resolves to the host's own
root filesystem, and emptying that is the outcome all of these guards exist to
prevent.
Three further faults found by testing the guards rather than reasoning about
them:
Stripping the trailing slash from "/" left an empty string, so the guard
refusing the host root never fired and the user got "destination is not a
directory" instead.
die() printed only its first argument, so the second half of the non-FAT
message -- the half saying to check whether the device is mounted -- was
silently dropped.
--help was not handled at all. Only -h reached the usage text, and it exited 2
to stderr, which is right for misuse and wrong for someone asking a question.
Help now goes to stdout and exits zero, and carries the guidance rather than
leaving it to the README, since the question it answers is asked at a terminal.
The test stage installs bash, rsync and findmnt, none of which are in the base
image, and the tests skip rather than fail where they are absent -- a machine
without rsync is not a machine that would run this script.
Emma Thorpe