diff --git a/README.md b/README.md index 29bde38..e2275e6 100644 --- a/README.md +++ b/README.md @@ -185,6 +185,30 @@ is what a percentage and an estimate that mean something cost — and renders th rest itself. Piped to a log it prints a plain line every thirty seconds instead, with no carriage returns, and a summary at the end either way. +### Making it faster over a network mount + +The transfer is metadata-bound, not throughput-bound: 49,600 files means 49,600 +round trips, and the counting pass doubles that. In rough order of what it is +worth doing: + +| Lever | Why | +| ----- | --- | +| Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. | +| Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. | +| `-Q` | Skips the counting pass entirely. Costs the percentage and the estimate, saves a whole walk of the tree. | +| `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. | + +**NFS instead of SMB** is worth trying but is not the big win it looks like. +Its attribute caching defaults are far more generous than SMB's — `acregmax` of +sixty seconds against `actimeo=1` — which is precisely the gap that +`actimeo=60` closes on the mount you already have. Bulk read throughput between +the two is much of a muchness on a gigabit link. Try the mount option first; it +is one line and needs no change on the NAS. + +And if the destination is the iPod rather than a card reader, none of this +matters much: the source can feed data faster than USB 2.0 through an iPod will +take it either way. + The unmount is the point of doing this in a script. FAT32 has no journal and the device is reached through disk mode, so an interrupted write is corruption that needs `fsck.vfat` from another machine. diff --git a/tests/test_sync_to_ipod.py b/tests/test_sync_to_ipod.py index 2aef186..386599a 100644 --- a/tests/test_sync_to_ipod.py +++ b/tests/test_sync_to_ipod.py @@ -174,3 +174,33 @@ def test_the_help_says_how_to_reach_and_leave_disk_mode(): assert "Menu+Select" in help_text assert "holding Play" in help_text + + +def test_quick_mode_skips_the_counting_pass(mirror, tmp_path): + """Over SMB the walk is the expensive part, and doing it twice for a + percentage is not always the trade you want.""" + destination = tmp_path / "dest" + destination.mkdir() + + result = run("-f", "-S", "-U", "-Q", str(mirror), str(destination)) + + assert result.returncode == 0, result.stderr + assert "skipping the count" in result.stderr + assert "files to copy" not in result.stderr + assert (destination / "Album" / "track.mp3").is_file() + + +def test_the_delta_algorithm_is_disabled(mirror, tmp_path): + """It would read every destination file back over USB to checksum it, to + avoid resending an MP3 that has changed in its entirety anyway.""" + script = SCRIPT.read_text() + + assert "--whole-file" in script + + +def test_directory_timestamps_are_not_set(mirror, tmp_path): + """One setattr round trip per directory, across six thousand albums, to set + timestamps nothing reads.""" + script = SCRIPT.read_text() + + assert "--omit-dir-times" in script diff --git a/tools/sync-to-ipod.sh b/tools/sync-to-ipod.sh index ae7ae5f..71f7c8f 100755 --- a/tools/sync-to-ipod.sh +++ b/tools/sync-to-ipod.sh @@ -22,6 +22,8 @@ usage() { usage: sync-to-ipod.sh [options] -n dry run; show what would change and touch nothing + -Q skip the counting pass; no percentage or estimate, but one less walk + of the source tree, which over SMB is the expensive part -f copy even if the FAT32 check finds unacceptable paths -S skip submitting the Rockbox scrobbler log to Last.fm -U leave the destination mounted afterwards @@ -58,15 +60,17 @@ USAGE } dry_run=false +quick=false force=false unmount=true scrobble=true for argument in "$@"; do [ "$argument" = "--help" ] && usage help done -while getopts ":nfSUh" option; do +while getopts ":nQfSUh" option; do case "$option" in n) dry_run=true ;; + Q) quick=true ;; f) force=true ;; S) scrobble=false ;; U) unmount=false ;; @@ -152,7 +156,16 @@ fi # asking for them produces a screenful of errors and a non-zero exit. # --modify-window=2 because FAT stores mtimes to two-second resolution, without # which every file looks changed and the whole library is copied every time. -options=(--recursive --times --delete --modify-window=2) +# --whole-file is already the default when both ends are local paths, and an +# SMB or FAT mount counts as one, but stating it documents that the delta +# algorithm is deliberately not wanted: it would read every destination file +# back over USB to compute a checksum, to save sending an MP3 that has changed +# entirely anyway. +# +# --omit-dir-times drops a setattr round trip per directory. Across six +# thousand album folders on a FAT card that is six thousand operations to set +# timestamps nothing reads. +options=(--recursive --times --delete --modify-window=2 --whole-file --omit-dir-times) # --delete removes tracks whose source has gone, which is the point. It would # also remove everything on the device that the mirror does not contain -- and # if the destination is the card root that means /.rockbox, the Rockbox install @@ -175,15 +188,21 @@ fi # thousand files over USB is minutes of apparent hang. Counting first costs a # second pass over the tree but means the transfer can show a real percentage # rather than a number that grows as rsync discovers more work. -printf 'sync-to-ipod: working out what needs copying...\n' >&2 -# %l is the file's size, which is what makes an estimate possible. Directories -# are dropped: rsync reports those too, with an inode size that would inflate -# the total by several megabytes of nothing. -counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" | - awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }') -total=${counted% *} -total_bytes=${counted#* } -printf 'sync-to-ipod: %s files to copy\n' "$total" >&2 +total=0 +total_bytes=0 +if $quick; then + printf 'sync-to-ipod: skipping the count; no percentage or estimate\n' >&2 +else + printf 'sync-to-ipod: working out what needs copying...\n' >&2 + # %l is the file's size, which is what makes an estimate possible. + # Directories are dropped: rsync reports those too, with an inode size that + # would inflate the total by several megabytes of nothing. + counted=$(rsync "${options[@]}" --dry-run --out-format='%l %n' "$mirror/" "$destination/" | + awk '!/\/$/ { files++; bytes += $1 } END { print files + 0, bytes + 0 }') + total=${counted% *} + total_bytes=${counted#* } + printf 'sync-to-ipod: %s files to copy\n' "$total" >&2 +fi rsync "${options[@]}" --out-format='%l %n' "$mirror/" "$destination/" | python3 "$here/rsync_progress.py" --total "$total" --bytes "$total_bytes"