perf: cut round trips over a network mount, and allow skipping the count
Build and publish container / build (pull_request) Successful in 2m39s

The transfer is metadata-bound rather than throughput-bound. Fifty thousand
files is fifty thousand round trips, and the counting pass added for the
percentage doubles that.

--whole-file is already implied when both ends are local paths, which an SMB or
FAT mount is, but stating it records that the delta algorithm is deliberately
unwanted here: it would read every destination file back over USB to checksum
it, in order to avoid resending an MP3 that has changed in its entirety anyway.

--omit-dir-times drops one setattr per directory. Across six thousand album
folders on a FAT card that is six thousand operations spent on timestamps
nothing reads.

-Q skips the counting pass. The percentage and the estimate are worth a second
walk of a local tree and frequently are not worth one of a network mount, so
that is now a choice rather than a fixed cost.

The README covers the part that is not an rsync flag at all: SMB defaults to a
one second attribute cache, so nearly every stat goes to the wire, twice. An
actimeo of sixty on the mount does more than any of the above, and closes most
of the gap that would otherwise argue for moving to NFS.
This commit is contained in:
Emma Thorpe
2026-08-25 12:44:53 +01:00
parent 131c80f5de
commit d6ef70922c
3 changed files with 84 additions and 11 deletions
+30 -11
View File
@@ -22,6 +22,8 @@ usage() {
usage: sync-to-ipod.sh [options] <mirror> <destination>
-n dry run; show what would change and touch nothing
-Q skip the counting pass; no percentage or estimate, but one less walk
of the source tree, which over SMB is the expensive part
-f copy even if the FAT32 check finds unacceptable paths
-S skip submitting the Rockbox scrobbler log to Last.fm
-U leave the destination mounted afterwards
@@ -58,15 +60,17 @@ USAGE
}
dry_run=false
quick=false
force=false
unmount=true
scrobble=true
for argument in "$@"; do
[ "$argument" = "--help" ] && usage help
done
while getopts ":nfSUh" option; do
while getopts ":nQfSUh" option; do
case "$option" in
n) dry_run=true ;;
Q) quick=true ;;
f) force=true ;;
S) scrobble=false ;;
U) unmount=false ;;
@@ -152,7 +156,16 @@ fi
# asking for them produces a screenful of errors and a non-zero exit.
# --modify-window=2 because FAT stores mtimes to two-second resolution, without
# which every file looks changed and the whole library is copied every time.
options=(--recursive --times --delete --modify-window=2)
# --whole-file is already the default when both ends are local paths, and an
# SMB or FAT mount counts as one, but stating it documents that the delta
# algorithm is deliberately not wanted: it would read every destination file
# back over USB to compute a checksum, to save sending an MP3 that has changed
# entirely anyway.
#
# --omit-dir-times drops a setattr round trip per directory. Across six
# thousand album folders on a FAT card that is six thousand operations to set
# timestamps nothing reads.
options=(--recursive --times --delete --modify-window=2 --whole-file --omit-dir-times)
# --delete removes tracks whose source has gone, which is the point. It would
# also remove everything on the device that the mirror does not contain -- and
# if the destination is the card root that means /.rockbox, the Rockbox install
@@ -175,15 +188,21 @@ fi
# thousand files over USB is minutes of apparent hang. Counting first costs a
# second pass over the tree but means the transfer can show a real percentage
# rather than a number that grows as rsync discovers more work.
printf 'sync-to-ipod: working out what needs copying...\n' >&2
# %l is the file's size, which is what makes an estimate possible. Directories
# are dropped: rsync reports those too, with an inode size that would inflate
# the total by several megabytes of nothing.
counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }')
total=${counted% *}
total_bytes=${counted#* }
printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
total=0
total_bytes=0
if $quick; then
printf 'sync-to-ipod: skipping the count; no percentage or estimate\n' >&2
else
printf 'sync-to-ipod: working out what needs copying...\n' >&2
# %l is the file's size, which is what makes an estimate possible.
# Directories are dropped: rsync reports those too, with an inode size that
# would inflate the total by several megabytes of nothing.
counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }')
total=${counted% *}
total_bytes=${counted#* }
printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
fi
rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" |
python3 "$here/rsync_progress.py" --total "$total" --bytes "$total_bytes"