fix: stop counting by default; the pass costs more than the transfer
Build and publish container / build (pull_request) Successful in 3m32s

The counting pass was added so the progress line could show a percentage and an
estimate, and on a real card it turned out to dominate the run. Measured
against the device: reading a track from the SMB mirror ran at 35 MB/s and
writing to the card at 21 MB/s, while the sync itself managed tens of kilobytes
per second. Neither end was slow. The cost was traversing fifty thousand files
across six thousand directories on FAT, and the counting pass does that a
second time, comparing both trees in full exactly as the transfer does.

Counting is now opt-in behind -P. Without it the progress line still shows the
running count, the transfer rate and the album in flight; the percentage and
the estimate are what needed the extra walk, and they were the least useful
part of the display.

That the fix for "it looks hung" was itself making it slow is the sort of thing
only measuring catches. The line still answers the question it was added for --
whether anything is happening -- without paying for the part that merely made
it prettier.
This commit is contained in:
Emma Thorpe
2026-08-26 13:57:49 +01:00
parent 633fbbaf91
commit 19ac9e5d92
3 changed files with 45 additions and 21 deletions
+15 -7
View File
@@ -178,12 +178,20 @@ entirely for the first two seconds, where the window is microseconds wide and
would report gigabytes per second.
rsync says nothing at all while it builds its file list, which on fifty
thousand files over USB is minutes of apparent hang, and its own `progress2`
percentage is computed against a list it has not finished discovering. So the
script counts first — files and bytes both, a second pass over the tree, which
is what a percentage and an estimate that mean something cost — and renders the
rest itself. Piped to a log it prints a plain line every thirty seconds
instead, with no carriage returns, and a summary at the end either way.
thousand files is minutes of apparent hang, and its own `progress2` percentage
is computed against a list it has not finished discovering. So the script
renders its own.
**The percentage and the estimate are opt-in, via `-P`.** They need a total,
the total needs a counting pass, and that pass walks and compares both trees in
full exactly as the transfer does. Measured on a real card: read from the
source at 35 MB/s and write to the card at 21 MB/s, yet the sync crawled —
because the traversal, not the data, was the cost, and it was being paid twice.
Without `-P` the line still shows the running count, the rate and the album in
flight; only the two figures that needed the second walk are missing.
Piped to a log it prints a plain line every thirty seconds instead, with no
carriage returns, and a summary at the end either way.
### The Rockbox database
@@ -260,7 +268,7 @@ worth doing:
| ----- | --- |
| Mount the source with `actimeo=60,cache=loose` | SMB defaults to a **one second** attribute cache, so nearly every `stat` goes to the wire — twice, once per pass. This is the single biggest change and it is a mount option, not an rsync flag. |
| Put the card in a reader for the first load | USB 2.0 through an iPod in disk mode is the floor for the destination. No amount of source tuning gets past it. |
| `-Q` | Skips the counting pass entirely. Costs the percentage and the estimate, saves a whole walk of the tree. |
| Counting is off by default | The percentage costs a second full traversal of both trees. On a FAT card of fifty thousand files that is slower than the transfer. `-P` asks for it. |
| `--whole-file`, `--omit-dir-times` | Already set. The first stops rsync checksumming destination files it is about to overwrite whole; the second drops a setattr per directory, 6,150 of them. |
**NFS instead of SMB** is worth trying but is not the big win it looks like.