fix: stop counting by default; the pass costs more than the transfer
Build and publish container / build (pull_request) Successful in 3m32s
Build and publish container / build (pull_request) Successful in 3m32s
The counting pass was added so the progress line could show a percentage and an estimate, and on a real card it turned out to dominate the run. Measured against the device: reading a track from the SMB mirror ran at 35 MB/s and writing to the card at 21 MB/s, while the sync itself managed tens of kilobytes per second. Neither end was slow. The cost was traversing fifty thousand files across six thousand directories on FAT, and the counting pass does that a second time, comparing both trees in full exactly as the transfer does. Counting is now opt-in behind -P. Without it the progress line still shows the running count, the transfer rate and the album in flight; the percentage and the estimate are what needed the extra walk, and they were the least useful part of the display. That the fix for "it looks hung" was itself making it slow is the sort of thing only measuring catches. The line still answers the question it was added for -- whether anything is happening -- without paying for the part that merely made it prettier.
This commit is contained in:
@@ -178,12 +178,20 @@ entirely for the first two seconds, where the window is microseconds wide and
|
||||
would report gigabytes per second.
|
||||
|
||||
rsync says nothing at all while it builds its file list, which on fifty
|
||||
thousand files over USB is minutes of apparent hang, and its own `progress2`
|
||||
percentage is computed against a list it has not finished discovering. So the
|
||||
script counts first — files and bytes both, a second pass over the tree, which
|
||||
is what a percentage and an estimate that mean something cost — and renders the
|
||||
rest itself. Piped to a log it prints a plain line every thirty seconds
|
||||
instead, with no carriage returns, and a summary at the end either way.
|
||||
thousand files is minutes of apparent hang, and its own `progress2` percentage
|
||||
is computed against a list it has not finished discovering. So the script
|
||||
renders its own.
|
||||
|
||||
**The percentage and the estimate are opt-in, via `-P`.** They need a total,
|
||||
the total needs a counting pass, and that pass walks and compares both trees in
|
||||
full exactly as the transfer does. Measured on a real card: read from the
|
||||
source at 35 MB/s and write to the card at 21 MB/s, yet the sync crawled —
|
||||
because the traversal, not the data, was the cost, and it was being paid twice.
|
||||
Without `-P` the line still shows the running count, the rate and the album in
|
||||
flight; only the two figures that needed the second walk are missing.
|
||||
|
||||
Piped to a log it prints a plain line every thirty seconds instead, with no
|
||||
carriage returns, and a summary at the end either way.
|
||||
|
||||
### The Rockbox database
|
||||
|
||||
@@ -260,7 +268,7 @@ worth doing:
|
||||
| ----- | --- |
|
||||
| Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. |
|
||||
| Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. |
|
||||
| `-Q` | Skips the counting pass entirely. Costs the percentage and the estimate, saves a whole walk of the tree. |
|
||||
| Counting is off by default | The percentage costs a second full traversal of both trees. On a FAT card of fifty thousand files that is slower than the transfer. `-P` asks for it. |
|
||||
| `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. |
|
||||
|
||||
**NFS instead of SMB** is worth trying but is not the big win it looks like.
|
||||
|
||||
Reference in New Issue
Block a user