README.md

@@ -185,6 +185,30 @@ is what a percentage and an estimate that mean something cost — and renders th
 rest itself. Piped to a log it prints a plain line every thirty seconds
 instead, with no carriage returns, and a summary at the end either way.
 
+### Making it faster over a network mount
+
+The transfer is metadata-bound, not throughput-bound: 49,600 files means 49,600
+round trips, and the counting pass doubles that. In rough order of what it is
+worth doing:
+
+| Lever | Why |
+| ----- | --- |
+| Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. |
+| Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. |
+| `-Q` | Skips the counting pass entirely. Costs the percentage and the estimate, saves a whole walk of the tree. |
+| `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. |
+
+**NFS instead of SMB** is worth trying but is not the big win it looks like.
+Its attribute caching defaults are far more generous than SMB's — `acregmax` of
+sixty seconds against `actimeo=1` — which is precisely the gap that
+`actimeo=60` closes on the mount you already have. Bulk read throughput between
+the two is much of a muchness on a gigabit link. Try the mount option first; it
+is one line and needs no change on the NAS.
+
+And if the destination is the iPod rather than a card reader, none of this
+matters much: the source can feed data faster than USB 2.0 through an iPod will
+take it either way.
+
 The unmount is the point of doing this in a script. FAT32 has no journal and
 the device is reached through disk mode, so an interrupted write is corruption
 that needs `fsck.vfat` from another machine.

tests/test_sync_to_ipod.py

@@ -174,3 +174,33 @@ def test_the_help_says_how_to_reach_and_leave_disk_mode():
 
     assert "Menu+Select" in help_text
     assert "holding Play" in help_text
+
+
+def test_quick_mode_skips_the_counting_pass(mirror, tmp_path):
+    """Over SMB the walk is the expensive part, and doing it twice for a
+    percentage is not always the trade you want."""
+    destination = tmp_path / "dest"
+    destination.mkdir()
+
+    result = run("-f", "-S", "-U", "-Q", str(mirror), str(destination))
+
+    assert result.returncode == 0, result.stderr
+    assert "skipping the count" in result.stderr
+    assert "files to copy" not in result.stderr
+    assert (destination / "Album" / "track.mp3").is_file()
+
+
+def test_the_delta_algorithm_is_disabled(mirror, tmp_path):
+    """It would read every destination file back over USB to checksum it, to
+    avoid resending an MP3 that has changed in its entirety anyway."""
+    script = SCRIPT.read_text()
+
+    assert "--whole-file" in script
+
+
+def test_directory_timestamps_are_not_set(mirror, tmp_path):
+    """One setattr round trip per directory, across six thousand albums, to set
+    timestamps nothing reads."""
+    script = SCRIPT.read_text()
+
+    assert "--omit-dir-times" in script

tools/sync-to-ipod.sh

@@ -22,6 +22,8 @@ usage() {
 usage: sync-to-ipod.sh [options] <mirror> <destination>
 
   -n  dry run; show what would change and touch nothing
+  -Q  skip the counting pass; no percentage or estimate, but one less walk
+      of the source tree, which over SMB is the expensive part
   -f  copy even if the FAT32 check finds unacceptable paths
   -S  skip submitting the Rockbox scrobbler log to Last.fm
   -U  leave the destination mounted afterwards
@@ -58,15 +60,17 @@ USAGE
 }
 
 dry_run=false
+quick=false
 force=false
 unmount=true
 scrobble=true
 for argument in "$@"; do
     [ "$argument" = "--help" ] && usage help
 done
-while getopts ":nfSUh" option; do
+while getopts ":nQfSUh" option; do
     case "$option" in
         n) dry_run=true ;;
+        Q) quick=true ;;
         f) force=true ;;
         S) scrobble=false ;;
         U) unmount=false ;;
@@ -152,7 +156,16 @@ fi
 # asking for them produces a screenful of errors and a non-zero exit.
 # --modify-window=2 because FAT stores mtimes to two-second resolution, without
 # which every file looks changed and the whole library is copied every time.
-options=(--recursive --times --delete --modify-window=2)
+# --whole-file is already the default when both ends are local paths, and an
+# SMB or FAT mount counts as one, but stating it documents that the delta
+# algorithm is deliberately not wanted: it would read every destination file
+# back over USB to compute a checksum, to save sending an MP3 that has changed
+# entirely anyway.
+#
+# --omit-dir-times drops a setattr round trip per directory. Across six
+# thousand album folders on a FAT card that is six thousand operations to set
+# timestamps nothing reads.
+options=(--recursive --times --delete --modify-window=2 --whole-file --omit-dir-times)
 # --delete removes tracks whose source has gone, which is the point. It would
 # also remove everything on the device that the mirror does not contain -- and
 # if the destination is the card root that means /.rockbox, the Rockbox install
@@ -175,15 +188,21 @@ fi
 # thousand files over USB is minutes of apparent hang. Counting first costs a
 # second pass over the tree but means the transfer can show a real percentage
 # rather than a number that grows as rsync discovers more work.
-printf 'sync-to-ipod: working out what needs copying...\n' >&2
-# %l is the file's size, which is what makes an estimate possible. Directories
-# are dropped: rsync reports those too, with an inode size that would inflate
-# the total by several megabytes of nothing.
-counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
-    awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }')
-total=${counted% *}
-total_bytes=${counted#* }
-printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
+total=0
+total_bytes=0
+if $quick; then
+    printf 'sync-to-ipod: skipping the count; no percentage or estimate\n' >&2
+else
+    printf 'sync-to-ipod: working out what needs copying...\n' >&2
+    # %l is the file's size, which is what makes an estimate possible.
+    # Directories are dropped: rsync reports those too, with an inode size that
+    # would inflate the total by several megabytes of nothing.
+    counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" |
+        awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }')
+    total=${counted% *}
+    total_bytes=${counted#* }
+    printf 'sync-to-ipod: %s files to copy\n' "$total" >&2
+fi
 
 rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" |
     python3 "$here/rsync_progress.py" --total "$total" --bytes "$total_bytes"