rsync itself is safe under interruption. It writes to a hidden temporary file
and renames it into place only once complete, and by default deletes any
partial file when interrupted -- verified both in the manual and by killing a
transfer and inspecting what was left, which was nothing. --partial is
deliberately absent and there is now a test asserting it stays that way. An
unclean kill can leave a hidden .track.mp3.XXXXXX behind; it is unplayable, it
is not in the source, and the next run's --delete removes it.
The script was not safe. Ctrl-C killed it before the sync and the unmount,
leaving a journal-less FAT filesystem holding dirty buffers -- which is the
exact corruption the script exists to prevent, arrived at by the most likely
route a person would take.
INT and TERM are now trapped. Both the normal path and the interrupt path call
the same finish function, so the flush and the unmount cannot drift apart, and
an interrupted run exits 130 rather than pretending to have succeeded.
The test is structural rather than timed. Reproducing a mid-transfer signal
needs a payload large enough to be slow, and a test that depends on winning a
race is a test that fails in CI for reasons that have nothing to do with the
code. The behaviour was verified by hand: SIGTERM mid-transfer gave exit 130,
the flush ran, and the destination held no short files and no leftover
temporaries.
The transfer is metadata-bound rather than throughput-bound. Fifty thousand
files is fifty thousand round trips, and the counting pass added for the
percentage doubles that.
--whole-file is already implied when both ends are local paths, which an SMB or
FAT mount is, but stating it records that the delta algorithm is deliberately
unwanted here: it would read every destination file back over USB to checksum
it, in order to avoid resending an MP3 that has changed in its entirety anyway.
--omit-dir-times drops one setattr per directory. Across six thousand album
folders on a FAT card that is six thousand operations spent on timestamps
nothing reads.
-Q skips the counting pass. The percentage and the estimate are worth a second
walk of a local tree and frequently are not worth one of a network mount, so
that is now a choice rather than a fixed cost.
The README covers the part that is not an rsync flag at all: SMB defaults to a
one second attribute cache, so nearly every stat goes to the wire, twice. An
actimeo of sixty on the mount does more than any of the above, and closes most
of the gap that would otherwise argue for moving to NFS.
rsync reports each file's size with %l as it completes, which is all an
estimate needs: bytes done over time elapsed is the same arithmetic rsync would
do internally, and requires nothing it does not already print. The scan pass now
sums those sizes as well as counting files, so both a percentage and an estimate
have a real denominator.
The rate is measured over a trailing thirty seconds rather than the whole run,
so it follows a device that slows down instead of averaging the slowdown away --
which for a card reader that thermally throttles, or a USB link that renegotiates
after an hour, is the difference between a useful estimate and a reassuring one.
Below two seconds no rate is reported at all. The first handful of files arrive
microseconds apart, and dividing by that window produces a rate in the gigabytes
per second and an estimate of zero, which is worse than showing nothing.
Directory entries are excluded from the byte total as well as the file count.
rsync reports them with a 4096 inode size, which across six thousand album
directories is several megabytes of transfer that never happens.
The transfer looked hung. rsync prints nothing while it builds its file list,
which on fifty thousand files over USB is several minutes of silence, and
--info=progress2 does not help: with incremental recursion its percentage is
computed against a list rsync has not finished discovering, so it moves
backwards as often as forwards.
The script now counts what needs copying first and says so, then renders its
own single line that rewrites in place, showing the album currently going
across and a percentage against a total that is actually known. Counting costs
a second pass over the tree. That is the price of a percentage meaning
something, and it is cheaper than staring at a blank terminal wondering whether
the thing has died.
Directories are excluded from the count. rsync reports those too, and including
them puts the figure past a hundred per cent.
Piped to a log the line becomes a plain one every thirty seconds, because a log
full of carriage returns and escape codes is not a log anybody reads.
The script required the destination to be its own mount point, while the
documentation and its own usage text both told the user to pass
/media/IPOD/Music. The documented invocation was rejected.
A subdirectory is the better target, so the guard was what was wrong. --delete
is confined to it, and the device path budget is now derived from it -- the
part of the destination below its mount point -- rather than configured, so the
budget cannot disagree with where the files are actually going. The check that
matters is that the destination sits on a FAT filesystem, which is also what
catches an unmounted device: /media/IPOD/Music then resolves to the host's own
root filesystem, and emptying that is the outcome all of these guards exist to
prevent.
Three further faults found by testing the guards rather than reasoning about
them:
Stripping the trailing slash from "/" left an empty string, so the guard
refusing the host root never fired and the user got "destination is not a
directory" instead.
die() printed only its first argument, so the second half of the non-FAT
message -- the half saying to check whether the device is mounted -- was
silently dropped.
--help was not handled at all. Only -h reached the usage text, and it exited 2
to stderr, which is right for misuse and wrong for someone asking a question.
Help now goes to stdout and exits zero, and carries the guidance rather than
leaving it to the README, since the question it answers is asked at a terminal.
The test stage installs bash, rsync and findmnt, none of which are in the base
image, and the tests skip rather than fail where they are absent -- a machine
without rsync is not a machine that would run this script.
tools/sync-to-ipod.sh does the whole transfer to a Rockbox device, so the only
manual part left is the disk-mode button sequence.
The guards are the substance rather than decoration. rsync --delete is being
aimed at a whole filesystem, so the destination must exist, be its own mount
point, and be a FAT filesystem; the mirror must be non-empty and must not be
the destination. Emptying the wrong directory is not a mistake that announces
itself.
It also excludes /.rockbox, the scrobbler logs and the usual filesystem
metadata directories. The mirror does not contain them, so a sync to the card
root would otherwise have deleted the Rockbox installation -- which the first
draft of this script would have done.
The unmount is why this is a script at all. FAT32 has no journal, the device is
reached through the Apple firmware's disk mode because Rockbox's own mass
storage is unreliable on an iFlash, and an interrupted write is corruption that
needs fsck.vfat from another machine.
tools/submit_scrobbles.py sends the Rockbox scrobbler log to Last.fm and sets
it aside. Rockbox writes it in AUDIOSCROBBLER 1.1: tab-separated, one line per
track, rated L for listened or S for skipped, and only the listened ones are a
play. It runs before the copy, because the plays already happened and a failed
transfer is no reason to lose them as well.
Two things there differ from every other Last.fm call in these projects.
Scrobbling is a write method, so it needs the API secret and a session key
obtained once through the browser rather than the read-only key. And a target
with no real-time clock gets /.scrobbler-timeless.log with every timestamp set
to zero; those are counted and reported but never sent, since submitting them
would mean inventing when they happened.
Signature generation sorts parameter names by the ASCII table rather than
numerically, so artist[10] precedes artist[1]. Sorting them the obvious way
produces an invalid signature and no other symptom, so there is a test for it.
The log is renamed rather than deleted once accepted, so that if Last.fm
quietly dropped something the evidence is still on the device.
Emma Thorpe