perf: cut round trips over a network mount, and allow skipping the count

The transfer is metadata-bound rather than throughput-bound. Fifty thousand
files is fifty thousand round trips, and the counting pass added for the
percentage doubles that.

--whole-file is already implied when both ends are local paths, which an SMB or
FAT mount is, but stating it records that the delta algorithm is deliberately
unwanted here: it would read every destination file back over USB to checksum
it, in order to avoid resending an MP3 that has changed in its entirety anyway.

--omit-dir-times drops one setattr per directory. Across six thousand album
folders on a FAT card that is six thousand operations spent on timestamps
nothing reads.

-Q skips the counting pass. The percentage and the estimate are worth a second
walk of a local tree and frequently are not worth one of a network mount, so
that is now a choice rather than a fixed cost.

The README covers the part that is not an rsync flag at all: SMB defaults to a
one second attribute cache, so nearly every stat goes to the wire, twice. An
actimeo of sixty on the mount does more than any of the above, and closes most
of the gap that would otherwise argue for moving to NFS.

README.md

@@ -185,6 +185,30 @@ is what a percentage and an estimate that mean something cost — and renders th
 rest itself. Piped to a log it prints a plain line every thirty seconds
 instead, with no carriage returns, and a summary at the end either way.
 
+### Making it faster over a network mount
+
+The transfer is metadata-bound, not throughput-bound: 49,600 files means 49,600
+round trips, and the counting pass doubles that. In rough order of what it is
+worth doing:
+
+| Lever | Why |
+| ----- | --- |
+| Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. |
+| Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. |
+| `-Q` | Skips the counting pass entirely. Costs the percentage and the estimate, saves a whole walk of the tree. |
+| `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. |
+
+**NFS instead of SMB** is worth trying but is not the big win it looks like.
+Its attribute caching defaults are far more generous than SMB's — `acregmax` of
+sixty seconds against `actimeo=1` — which is precisely the gap that
+`actimeo=60` closes on the mount you already have. Bulk read throughput between
+the two is much of a muchness on a gigabit link. Try the mount option first; it
+is one line and needs no change on the NAS.
+
+And if the destination is the iPod rather than a card reader, none of this
+matters much: the source can feed data faster than USB 2.0 through an iPod will
+take it either way.
+
 The unmount is the point of doing this in a script. FAT32 has no journal and
 the device is reached through disk mode, so an interrupted write is corruption
 that needs `fsck.vfat` from another machine.

tests/test_sync_to_ipod.py

@@ -174,3 +174,33 @@ def test_the_help_says_how_to_reach_and_leave_disk_mode():
 
     assert "Menu+Select" in help_text
     assert "holding Play" in help_text
+
+
+def test_quick_mode_skips_the_counting_pass(mirror, tmp_path):
+    """Over SMB the walk is the expensive part, and doing it twice for a
+    percentage is not always the trade you want."""
+    destination = tmp_path / "dest"
+    destination.mkdir()
+
+    result = run("-f", "-S", "-U", "-Q", str(mirror), str(destination))
+
+    assert result.returncode == 0, result.stderr
+    assert "skipping the count" in result.stderr
+    assert "files to copy" not in result.stderr
+    assert (destination / "Album" / "track.mp3").is_file()
+
+
+def test_the_delta_algorithm_is_disabled(mirror, tmp_path):
+    """It would read every destination file back over USB to checksum it, to
+    avoid resending an MP3 that has changed in its entirety anyway."""
+    script = SCRIPT.read_text()
+
+    assert "--whole-file" in script
+
+
+def test_directory_timestamps_are_not_set(mirror, tmp_path):
+    """One setattr round trip per directory, across six thousand albums, to set
+    timestamps nothing reads."""
+    script = SCRIPT.read_text()
+
+    assert "--omit-dir-times" in script

tools/sync-to-ipod.sh

@@ -22,6 +22,8 @@ usage() {
 usage: sync-to-ipod.sh [options] <mirror> <destination>
 
   -n  dry run; show what would change and touch nothing
+  -Q  skip the counting pass; no percentage or estimate, but one less walk
+      of the source tree, which over SMB is the expensive part
   -f  copy even if the FAT32 check finds unacceptable paths
   -S  skip submitting the Rockbox scrobbler log to Last.fm
   -U  leave the destination mounted afterwards
@@ -58,15 +60,17 @@ USAGE
 }
 
 dry_run=false
+quick=false
 force=false
 unmount=true
 scrobble=true
 for argument in "$@"; do
     [ "$argument" = "--help" ] && usage help
 done
-while getopts ":nfSUh" option; do
+while getopts ":nQfSUh" option; do
     case "$option" in
         n) dry_run=true ;;
+        Q) quick=true ;;
         f) force=true ;;
         S) scrobble=false ;;
         U) unmount=false ;;
@@ -152,7 +156,16 @@ fi
 # asking for them produces a screenful of errors and a non-zero exit.
 # --modify-window=2 because FAT stores mtimes to two-second resolution, without
 # which every file looks changed and the whole library is copied every time.
-options=(--recursive --times --delete --modify-window=2)
+# --whole-file is already the default when both ends are local paths, and an
+# SMB or FAT mount counts as one, but stating it documents that the delta
+# algorithm is deliberately not wanted: it would read every destination file
+# back over USB to compute a checksum, to save sending an MP3 that has changed
+# entirely anyway.
+#
+# --omit-dir-times drops a setattr round trip per directory. Across six
+# thousand album folders on a FAT card that is six thousand operations to set
+# timestamps nothing reads.
+options=(--recursive --times --delete --modify-window=2 --whole-file --omit-dir-times)
 # --delete removes tracks whose source has gone, which is the point. It would
 # also remove everything on the device that the mirror does not contain -- and
 # if the destination is the card root that means /.rockbox, the Rockbox install
@@ -175,15 +188,21 @@ fi
 # thousand files over USB is minutes of apparent hang. Counting first costs a
 # second pass over the tree but means the transfer can show a real percentage
 # rather than a number that grows as rsync discovers more work.
-printf 'sync-to-ipod: working out what needs copying...\n' >&2
-# %l is the file's size, which is what makes an estimate possible. Directories
-# are dropped: rsync reports those too, with an inode size that would inflate
-# the total by several megabytes of nothing.
-counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
+total=0
+total_bytes=0
+if $quick; then
+    printf 'sync-to-ipod: skipping the count; no percentage or estimate\n' >&2
+else
+    printf 'sync-to-ipod: working out what needs copying...\n' >&2
+    # %l is the file's size, which is what makes an estimate possible.
+    # Directories are dropped: rsync reports those too, with an inode size that
+    # would inflate the total by several megabytes of nothing.
+    counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
         awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }')
-total=${counted% *}
-total_bytes=${counted#* }
-printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
+    total=${counted% *}
+    total_bytes=${counted#* }
+    printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
+fi
 
 rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" |
     python3 "$here/rsync_progress.py" --total "$total" --bytes "$total_bytes"