8 Commits

Include renames

Author	SHA1	Message	Date
Emma Thorpe	d6ef70922c	perf: cut round trips over a network mount, and allow skipping the count ... The transfer is metadata-bound rather than throughput-bound. Fifty thousand files is fifty thousand round trips, and the counting pass added for the percentage doubles that. --whole-file is already implied when both ends are local paths, which an SMB or FAT mount is, but stating it records that the delta algorithm is deliberately unwanted here: it would read every destination file back over USB to checksum it, in order to avoid resending an MP3 that has changed in its entirety anyway. --omit-dir-times drops one setattr per directory. Across six thousand album folders on a FAT card that is six thousand operations spent on timestamps nothing reads. -Q skips the counting pass. The percentage and the estimate are worth a second walk of a local tree and frequently are not worth one of a network mount, so that is now a choice rather than a fixed cost. The README covers the part that is not an rsync flag at all: SMB defaults to a one second attribute cache, so nearly every stat goes to the wire, twice. An actimeo of sixty on the mount does more than any of the above, and closes most of the gap that would otherwise argue for moving to NFS.	2026-08-25 12:44:53 +01:00
Emma Thorpe	131c80f5de	feat: estimate the time remaining from bytes and observed rate ... rsync reports each file's size with %l as it completes, which is all an estimate needs: bytes done over time elapsed is the same arithmetic rsync would do internally, and requires nothing it does not already print. The scan pass now sums those sizes as well as counting files, so both a percentage and an estimate have a real denominator. The rate is measured over a trailing thirty seconds rather than the whole run, so it follows a device that slows down instead of averaging the slowdown away -- which for a card reader that thermally throttles, or a USB link that renegotiates after an hour, is the difference between a useful estimate and a reassuring one. Below two seconds no rate is reported at all. The first handful of files arrive microseconds apart, and dividing by that window produces a rate in the gigabytes per second and an estimate of zero, which is worse than showing nothing. Directory entries are excluded from the byte total as well as the file count. rsync reports them with a 4096 inode size, which across six thousand album directories is several megabytes of transfer that never happens.	2026-08-25 12:40:32 +01:00
Emma Thorpe	37b841f009	feat: show which album is copying, and how far through ... The transfer looked hung. rsync prints nothing while it builds its file list, which on fifty thousand files over USB is several minutes of silence, and --info=progress2 does not help: with incremental recursion its percentage is computed against a list rsync has not finished discovering, so it moves backwards as often as forwards. The script now counts what needs copying first and says so, then renders its own single line that rewrites in place, showing the album currently going across and a percentage against a total that is actually known. Counting costs a second pass over the tree. That is the price of a percentage meaning something, and it is cheaper than staring at a blank terminal wondering whether the thing has died. Directories are excluded from the count. rsync reports those too, and including them puts the figure past a hundred per cent. Piped to a log the line becomes a plain one every thirty seconds, because a log full of carriage returns and escape codes is not a log anybody reads.	2026-08-25 12:31:13 +01:00
Emma Thorpe	d5dce9c769	fix: accept a destination inside the device, and say so in --help ... The script required the destination to be its own mount point, while the documentation and its own usage text both told the user to pass /media/IPOD/Music. The documented invocation was rejected. A subdirectory is the better target, so the guard was what was wrong. --delete is confined to it, and the device path budget is now derived from it -- the part of the destination below its mount point -- rather than configured, so the budget cannot disagree with where the files are actually going. The check that matters is that the destination sits on a FAT filesystem, which is also what catches an unmounted device: /media/IPOD/Music then resolves to the host's own root filesystem, and emptying that is the outcome all of these guards exist to prevent. Three further faults found by testing the guards rather than reasoning about them: Stripping the trailing slash from "/" left an empty string, so the guard refusing the host root never fired and the user got "destination is not a directory" instead. die() printed only its first argument, so the second half of the non-FAT message -- the half saying to check whether the device is mounted -- was silently dropped. --help was not handled at all. Only -h reached the usage text, and it exited 2 to stderr, which is right for misuse and wrong for someone asking a question. Help now goes to stdout and exits zero, and carries the guidance rather than leaving it to the README, since the question it answers is asked at a terminal. The test stage installs bash, rsync and findmnt, none of which are in the base image, and the tests skip rather than fail where they are absent -- a machine without rsync is not a machine that would run this script.	2026-08-25 12:23:52 +01:00
Emma Thorpe	ece79515c0	fix: cost the device prefix exactly rather than approximately ... The budget subtracted the prefix length plus two, on the assumption of a leading and a trailing slash. That is right for /Music and wrong for an empty prefix, where there is only one slash -- losing a character at the card root, which is exactly where the longest paths sit. Computed from the prefix as it will actually appear instead: /Music/ costs seven characters and gives a mirror-relative budget of 253, the root costs one and gives 259. Worth being exact about because the reverse error is worse. A checker comparing mirror-relative paths against the flat 260 passes everything between 253 and 260, and those are precisely the paths closest to the edge.	2026-08-25 11:52:03 +01:00
Emma Thorpe	d5f67c6de5	feat: shorten paths that exceed the device's limit ... Rockbox's MAX_PATH is 260, defined in firmware/include/fs_defines.h and used to size the directory entry buffer in dir.h. It bounds the path as the device sees it, so the directory the mirror is copied into spends part of the same budget; --device-prefix accounts for that and defaults to /Music. Over-budget paths are shortened from the deepest component outward. The track name carries the least navigational value and the artist directory the most, so the filename is cut first and the artist only if nothing else will serve. A shortened component keeps its extension and gains four hex digits of the original name: two long titles sharing a prefix cut to the same string otherwise, and a silent collision between two tracks is a worse outcome than an ugly filename. The result is stable. The same source always yields the same shortened name, so one pass does not rename what the last one wrote -- an unstable scheme would churn the whole mirror every six hours. A path too deeply nested to fit without reducing every component to nonsense is left alone and reported rather than mangled. Migration now tries more than one previous naming, because there is more than one. A mirror already running with --fat32-safe holds sanitised but unshortened paths, and matching only the original unsanitised name would have re-encoded every one of them instead of moving it. The checker gains the same two options, since it was measuring the mirror-relative path against a limit that applies to the device-absolute one, and so under-reported by the length of the destination directory.	2026-08-25 11:46:26 +01:00
Emma Thorpe	802d91490f	feat: a sync script that submits scrobbles, copies, and unmounts cleanly ... tools/sync-to-ipod.sh does the whole transfer to a Rockbox device, so the only manual part left is the disk-mode button sequence. The guards are the substance rather than decoration. rsync --delete is being aimed at a whole filesystem, so the destination must exist, be its own mount point, and be a FAT filesystem; the mirror must be non-empty and must not be the destination. Emptying the wrong directory is not a mistake that announces itself. It also excludes /.rockbox, the scrobbler logs and the usual filesystem metadata directories. The mirror does not contain them, so a sync to the card root would otherwise have deleted the Rockbox installation -- which the first draft of this script would have done. The unmount is why this is a script at all. FAT32 has no journal, the device is reached through the Apple firmware's disk mode because Rockbox's own mass storage is unreliable on an iFlash, and an interrupted write is corruption that needs fsck.vfat from another machine. tools/submit_scrobbles.py sends the Rockbox scrobbler log to Last.fm and sets it aside. Rockbox writes it in AUDIOSCROBBLER 1.1: tab-separated, one line per track, rated L for listened or S for skipped, and only the listened ones are a play. It runs before the copy, because the plays already happened and a failed transfer is no reason to lose them as well. Two things there differ from every other Last.fm call in these projects. Scrobbling is a write method, so it needs the API secret and a session key obtained once through the browser rather than the read-only key. And a target with no real-time clock gets /.scrobbler-timeless.log with every timestamp set to zero; those are counted and reported but never sent, since submitting them would mean inventing when they happened. Signature generation sorts parameter names by the ASCII table rather than numerically, so artist[10] precedes artist[1]. Sorting them the obvious way produces an invalid signature and no other symptom, so there is a test for it. The log is renamed rather than deleted once accepted, so that if Last.fm quietly dropped something the evidence is still on the device.	2026-08-25 11:05:00 +01:00
Emma Thorpe	3141f7ca87	feat: name the mirror so a FAT32 device will take it, and copy album art ... Two changes for playing the mirror on a Rockbox iPod, where the device is FAT32 and Rockbox reads a plain directory tree rather than a database. --fat32-safe names mirror files acceptably: the reserved characters and control characters become underscores, trailing dots and spaces are stripped because FAT eats them silently and the name then round-trips as a different one, and a component left empty becomes an underscore. Names differing only in case are detected as collisions, since two files here are one file there and the second would silently overwrite the first. "Kick Out the Epic Motherf**ker" is a real example from a real library, and without this it simply never arrives. Off by default. It renames files, and that should be a decision rather than a surprise on somebody's next pass. Turning it on does not re-encode anything. Every track whose name held a reserved character changes path, and encoding those again would be hours of work producing files that already exist byte for byte, so the run moves them instead and logs each one. Prune then finds nothing left behind. Album art is now also copied into the mirror as cover.jpg beside the tracks. Rockbox searches the filesystem for art -- cover.jpg, folder.jpg and the rest, in the track's directory or its parent -- and that search never looks at the picture embedded in the tag, so a mirror that only embeds art displays none of it on the device. Embedding continues for the Apple firmware; both are now satisfied. A cover whose tracks have all been pruned is removed too, or its directory would never look empty and never go. Adds tools/check_fat32.py, which reports unacceptable paths before a copy rather than during one: rsync reports them too, but scattered through fifty thousand files where they are easy to lose. It exits non-zero so it can gate a script. The README documents the rsync invocation, including why --modify-window=2 is required against FAT and why Rhythmbox must be kept out of the transfer -- rb_ipod_helpers_is_ipod() reads access-protocols from media-player-info and returns true on the USB id alone, without looking at the filesystem, so removing iPod_Control changes nothing.	2026-08-25 10:29:55 +01:00