perf: cut round trips over a network mount, and allow skipping the count
Build and publish container / build (pull_request) Successful in 2m39s
Build and publish container / build (pull_request) Successful in 2m39s
The transfer is metadata-bound rather than throughput-bound. Fifty thousand files is fifty thousand round trips, and the counting pass added for the percentage doubles that. --whole-file is already implied when both ends are local paths, which an SMB or FAT mount is, but stating it records that the delta algorithm is deliberately unwanted here: it would read every destination file back over USB to checksum it, in order to avoid resending an MP3 that has changed in its entirety anyway. --omit-dir-times drops one setattr per directory. Across six thousand album folders on a FAT card that is six thousand operations spent on timestamps nothing reads. -Q skips the counting pass. The percentage and the estimate are worth a second walk of a local tree and frequently are not worth one of a network mount, so that is now a choice rather than a fixed cost. The README covers the part that is not an rsync flag at all: SMB defaults to a one second attribute cache, so nearly every stat goes to the wire, twice. An actimeo of sixty on the mount does more than any of the above, and closes most of the gap that would otherwise argue for moving to NFS.
This commit is contained in:
@@ -185,6 +185,30 @@ is what a percentage and an estimate that mean something cost — and renders th
|
||||
rest itself. Piped to a log it prints a plain line every thirty seconds
|
||||
instead, with no carriage returns, and a summary at the end either way.
|
||||
|
||||
### Making it faster over a network mount
|
||||
|
||||
The transfer is metadata-bound, not throughput-bound: 49,600 files means 49,600
|
||||
round trips, and the counting pass doubles that. In rough order of what it is
|
||||
worth doing:
|
||||
|
||||
| Lever | Why |
|
||||
| ----- | --- |
|
||||
| Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. |
|
||||
| Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. |
|
||||
| `-Q` | Skips the counting pass entirely. Costs the percentage and the estimate, saves a whole walk of the tree. |
|
||||
| `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. |
|
||||
|
||||
**NFS instead of SMB** is worth trying but is not the big win it looks like.
|
||||
Its attribute caching defaults are far more generous than SMB's — `acregmax` of
|
||||
sixty seconds against `actimeo=1` — which is precisely the gap that
|
||||
`actimeo=60` closes on the mount you already have. Bulk read throughput between
|
||||
the two is much of a muchness on a gigabit link. Try the mount option first; it
|
||||
is one line and needs no change on the NAS.
|
||||
|
||||
And if the destination is the iPod rather than a card reader, none of this
|
||||
matters much: the source can feed data faster than USB 2.0 through an iPod will
|
||||
take it either way.
|
||||
|
||||
The unmount is the point of doing this in a script. FAT32 has no journal and
|
||||
the device is reached through disk mode, so an interrupted write is corruption
|
||||
that needs `fsck.vfat` from another machine.
|
||||
|
||||
Reference in New Issue
Block a user