perf: cut round trips over a network mount, and allow skipping the count
Build and publish container / build (pull_request) Successful in 2m39s
Build and publish container / build (pull_request) Successful in 2m39s
The transfer is metadata-bound rather than throughput-bound. Fifty thousand files is fifty thousand round trips, and the counting pass added for the percentage doubles that. --whole-file is already implied when both ends are local paths, which an SMB or FAT mount is, but stating it records that the delta algorithm is deliberately unwanted here: it would read every destination file back over USB to checksum it, in order to avoid resending an MP3 that has changed in its entirety anyway. --omit-dir-times drops one setattr per directory. Across six thousand album folders on a FAT card that is six thousand operations spent on timestamps nothing reads. -Q skips the counting pass. The percentage and the estimate are worth a second walk of a local tree and frequently are not worth one of a network mount, so that is now a choice rather than a fixed cost. The README covers the part that is not an rsync flag at all: SMB defaults to a one second attribute cache, so nearly every stat goes to the wire, twice. An actimeo of sixty on the mount does more than any of the above, and closes most of the gap that would otherwise argue for moving to NFS.
This commit is contained in:
+30
-11
@@ -22,6 +22,8 @@ usage() {
|
||||
usage: sync-to-ipod.sh [options] <mirror> <destination>
|
||||
|
||||
-n dry run; show what would change and touch nothing
|
||||
-Q skip the counting pass; no percentage or estimate, but one less walk
|
||||
of the source tree, which over SMB is the expensive part
|
||||
-f copy even if the FAT32 check finds unacceptable paths
|
||||
-S skip submitting the Rockbox scrobbler log to Last.fm
|
||||
-U leave the destination mounted afterwards
|
||||
@@ -58,15 +60,17 @@ USAGE
|
||||
}
|
||||
|
||||
dry_run=false
|
||||
quick=false
|
||||
force=false
|
||||
unmount=true
|
||||
scrobble=true
|
||||
for argument in "$@"; do
|
||||
[ "$argument" = "--help" ] && usage help
|
||||
done
|
||||
while getopts ":nfSUh" option; do
|
||||
while getopts ":nQfSUh" option; do
|
||||
case "$option" in
|
||||
n) dry_run=true ;;
|
||||
Q) quick=true ;;
|
||||
f) force=true ;;
|
||||
S) scrobble=false ;;
|
||||
U) unmount=false ;;
|
||||
@@ -152,7 +156,16 @@ fi
|
||||
# asking for them produces a screenful of errors and a non-zero exit.
|
||||
# --modify-window=2 because FAT stores mtimes to two-second resolution, without
|
||||
# which every file looks changed and the whole library is copied every time.
|
||||
options=(--recursive --times --delete --modify-window=2)
|
||||
# --whole-file is already the default when both ends are local paths, and an
|
||||
# SMB or FAT mount counts as one, but stating it documents that the delta
|
||||
# algorithm is deliberately not wanted: it would read every destination file
|
||||
# back over USB to compute a checksum, to save sending an MP3 that has changed
|
||||
# entirely anyway.
|
||||
#
|
||||
# --omit-dir-times drops a setattr round trip per directory. Across six
|
||||
# thousand album folders on a FAT card that is six thousand operations to set
|
||||
# timestamps nothing reads.
|
||||
options=(--recursive --times --delete --modify-window=2 --whole-file --omit-dir-times)
|
||||
# --delete removes tracks whose source has gone, which is the point. It would
|
||||
# also remove everything on the device that the mirror does not contain -- and
|
||||
# if the destination is the card root that means /.rockbox, the Rockbox install
|
||||
@@ -175,15 +188,21 @@ fi
|
||||
# thousand files over USB is minutes of apparent hang. Counting first costs a
|
||||
# second pass over the tree but means the transfer can show a real percentage
|
||||
# rather than a number that grows as rsync discovers more work.
|
||||
printf 'sync-to-ipod: working out what needs copying...\n' >&2
|
||||
# %l is the file's size, which is what makes an estimate possible. Directories
|
||||
# are dropped: rsync reports those too, with an inode size that would inflate
|
||||
# the total by several megabytes of nothing.
|
||||
counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
|
||||
awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }')
|
||||
total=${counted% *}
|
||||
total_bytes=${counted#* }
|
||||
printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
|
||||
total=0
|
||||
total_bytes=0
|
||||
if $quick; then
|
||||
printf 'sync-to-ipod: skipping the count; no percentage or estimate\n' >&2
|
||||
else
|
||||
printf 'sync-to-ipod: working out what needs copying...\n' >&2
|
||||
# %l is the file's size, which is what makes an estimate possible.
|
||||
# Directories are dropped: rsync reports those too, with an inode size that
|
||||
# would inflate the total by several megabytes of nothing.
|
||||
counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
|
||||
awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }')
|
||||
total=${counted% *}
|
||||
total_bytes=${counted#* }
|
||||
printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
|
||||
fi
|
||||
|
||||
rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" |
|
||||
python3 "$here/rsync_progress.py" --total "$total" --bytes "$total_bytes"
|
||||
|
||||
Reference in New Issue
Block a user