17 Commits

Include renames

Author	SHA1	Message	Date
Emma Thorpe	8644184caa	fix: remove playlists left behind by the switch to .m3u8 ... Every playlist appeared twice on the device. Pruning is driven by a record of what was written last time, deliberately, so that a playlist put in that directory by hand is never touched. But the record cannot reach back before it existed: the .m3u files were written by a version that kept none, so when the extension changed there was nothing to prune them by, and rsync carried both copies across. A file with the same name as one being written now, under an extension this tool used to write, is removed as well. That is narrow enough to match only its own leavings -- a hand-made playlist has a name this tool never produces, so it survives whatever its extension, and there is a test for that alongside the one for the duplicate.	2026-08-26 18:06:47 +01:00
Emma Thorpe	35d5e98642	fix: write playlists as .m3u8 so Rockbox reads them as UTF-8 ... Rockbox's is_m3u8_name() treats every playlist extension as UTF-8 except an explicit ".m3u", which it instead decodes through the user's configured codepage: /* Default to M3U8 unless explicitly told otherwise. */ return (!dot || strcasecmp(dot, ".m3u") != 0); The one extension being used was therefore the only one that mangles accented filenames, and this library holds Motley Crue, Beyonce and Sigur Ros. Renaming the output is the whole fix. No byte order mark is written. One would promote a .m3u file to UTF-8 as well, but it is unnecessary at this extension and upsets players that do not expect to find one. Kept with the pruning change rather than raised separately, because the rename depends on it: without pruning, seventeen dead .m3u files would sit on the device for ever, and every one of them full of paths that still resolve.	2026-08-25 10:43:43 +01:00
Emma Thorpe	9fdd61b648	feat: report which albums have never been played ... Stage four, read-only. An album is cold when it has files, not one of its tracks has ever been scrobbled across the whole nine-year history, and its newest file landed over a year ago. The age floor is measured from the file rather than the release date: what matters is how long a record has been available to play, not how old it is. Album-level rather than track-level. A record with two played tracks is a record that gets played, and picking the other ten off it leaves gaps in an album rather than reclaiming anything worth having. Track file sizes are now indexed so the report can rank by the disk actually recoverable, which is the point of the exercise. That needed a column on a table that already exists, so the store gained an in-place column migration; a full rebuild would mean re-downloading sixty thousand scrobbles. The report refuses to produce anything when the backfill is unfinished, when any artist failed to index, when any artist has no albums, or when there is no history to judge against. Each of those makes played music look unplayed, which is the one failure that costs a library, and they are checked rather than trusted because the report they gate is the one that ends in deletion. Nothing is written to Lidarr. Every call there remains a GET, and unmonitoring waits until the list this produces has been looked at -- no flag protects against a list that is wrong. Also prunes playlists no longer produced, which #12 made necessary by renaming moods: high-energy-rock would otherwise have stayed on the device for ever. It is driven by a record of what was written last time rather than by deleting every M3U that is not currently ours, so a playlist put in that directory by hand is left alone.	2026-08-24 19:21:57 +01:00
Emma Thorpe	a7d16ca0b2	fix: give playlists the mirror's ownership rather than root's ... Playlists were being written owned by root. The image runs as root by default, deliberately, so that a bind-mounted dataset of any ownership stays writable -- but everything it writes then comes out root-owned, and a root-owned playlist inside a mirror owned by the apps account is unreadable to whatever serves it. The group-read bit does not help when the group is also root. Copy the ownership of the mirror the playlist is being written into. That needs no configuration, cannot drift from whatever the mirror actually is, and does nothing at all when the two already agree -- which is the case whenever the container is run with an explicit user. Applied to the temporary file before the rename, so a playlist is never briefly visible owned by the wrong account, and to the _playlists directory when this is the run that creates it. A chown that is refused is ignored rather than fatal: that only happens when not running as root, which is precisely the case where the ownership was already right.	2026-08-24 18:45:25 +01:00
Emma Thorpe	8c4da6e14e	feat: let a mood exclude tags, and widen the eighties one to the canon ... The eighties mood selected on synth-specific tags only, on the grounds that the bare `80s` tag drags in Def Leppard and Bon Jovi. Checking the canon against live Last.fm pages shows what that costs. Eurythmics is tagged 80s, new wave, pop, female vocalists, synth pop -- four of those five are tags no mood may use. Frankie Goes to Hollywood is tagged 80s, new wave, pop, british, dance, and survives only on new wave. An act tagged purely 80s, pop and dance, which is a great deal of eighties pop, was missed outright. Two spellings also matter more than the census suggested. Of Depeche Mode, Duran Duran, Eurythmics and Frankie Goes to Hollywood, three carry "synth pop" with a space and only one carries "synthpop" without. The census could not show this because it only sees artists already in the library; the spelling that was kept was the rarer one. Add an `exclude` list to a mood. An excluded tag drops the artist outright rather than docking their score, which is the only construction that expresses "the eighties, but not the stadium rock" -- weighting cannot do it, because the tag it would weight is the one both share. The synth acts do not carry hard rock or hair metal, and that is the whole of the difference. A mood may not both select on and exclude the same tag; that is rejected when the definitions are read rather than silently producing nothing.	2026-08-24 18:39:59 +01:00
Emma Thorpe	41f6b290d8	feat: rebuild the moods from the library's measured tag distribution ... The first set of moods was guessed at, and a tag census of the real library shows how badly. Four tags in the dance mood -- edm, big room, hardstyle, electronic dance music -- carry nothing whatsoever, while techno, electro, big beat, rave and progressive house together carry over seven thousand plays and were all absent. The drum and bass mood missed four of the six spellings the taggers actually use, plus techstep, darkstep and hospital records. The eighties mood was the worst of it. It leaned on the bare `80s` tag, which is the eleventh most-played tag here and sits on Def Leppard, Bon Jovi and AC/DC rather than on anything with a synthesiser in it. Removing it and adding a hair-metal mood -- glam rock, hair metal, glam metal, arena rock, AOR, windowed to 1975-1994 -- puts roughly eight thousand plays somewhere sensible and stops the synth playlist being a stadium rock playlist. Whole clusters had no mood at all: metal at around twenty-five thousand plays across heavy, thrash, speed, power, death and NWOBHM, and pop punk and emo at a similar figure. Both now exist, along with bass, nu-metal and indie. Three kinds of tag are excluded on principle, with a test enforcing it, because each produces a playlist that looks reasonable and is not. Nationality describes a passport rather than a sound, and `american` alone spans 241 artists. `rock` and `electronic` span 340 and 275, which is most of the collection. And Last.fm's most popular tag for an artist is frequently the artist's own name, so selecting on `green day` yields a Green Day playlist wearing a genre's clothes.	2026-08-24 18:34:34 +01:00
Emma Thorpe	40dfce8a4c	fix: look tags up by artist name rather than by MusicBrainz id ... Tag lookups asked by MusicBrainz id whenever Lidarr had one, which is always. The reasoning was that an id cannot be ambiguous the way a name can. In practice Last.fm's mbid index is stale and partial, and it answered "the artist you supplied could not be found" for Devo, Escape the Fate, Blasterjaxx, Frank Carter & the Rattlesnakes and a long tail of others -- artists whose Last.fm pages plainly exist and carry precisely the tags the moods are built from. Devo is tagged new wave, post-punk and 80s; Escape the Fate is tagged post-hardcore, screamo and emocore. Both were skipped. Ask by name first, which is the index Last.fm's own site runs on, and keep the id only as a fallback for a name Lidarr spells differently. Failures were also being dropped without recording the attempt, so every one of those artists was re-queried on every subsequent pass, indefinitely. They are now distinguished: an artist that neither key resolves is recorded as fetched with no tags and not asked about again, while a genuine failure -- a rate limit, a bad key -- is deliberately left unrecorded so the next pass retries it. Telling the two apart needed the service's own error number, so LastfmError now carries it.	2026-08-24 18:13:23 +01:00
Emma Thorpe	15e5ee5aea	feat: mood playlists from Last.fm crowd tags ... A second set of playlists selecting by genre and mood rather than by play history: eighties synths, high energy rock, screamo, drum and bass, dance, classic rock. The tags come from Last.fm rather than MusicBrainz. MusicBrainz genres arrive free with the Lidarr index, which makes them the obvious choice and the wrong one: they are sparse and formal, and will not tell you a record is screamo or synthwave. Crowd tags will, because people typed them. One request per artist, by MusicBrainz id where Lidarr has one, refreshed every ninety days. An artist Last.fm has never heard of is recorded as fetched with no tags rather than left unmarked, so it is not asked about again on every pass forever. The tag table is keyed on the normalised artist name, not the Lidarr id, so it survives an artist being removed and re-added there. Weights are taken from the response's count where it has one. The documented sample carries only a name and a URL, a live response also carries a 0-100 count, and depending on either alone would be a guess -- so the count is used when present and the documented ordering by popularity stands in when it is not. An artist qualifies for a mood when their weights inside it sum to at least thirty. A single low-weight tag is not a genre, it is somebody's stray opinion. A mood may also restrict release years, which is what separates eighties synth records from everything else a synthpop tag drags in. The built-in set is chosen for this library rather than as a taxonomy, and --vibes replaces it wholesale with a JSON file so a new mood does not need a new release. Names are validated when that file is read: an invalid one would otherwise only surface as a playlist written somewhere unintended.	2026-08-24 17:56:35 +01:00
Emma Thorpe	cda3d8463b	feat: write playlists into the mirror from the listening history ... Six rules over the scrobble history and the library index, written as extended M3U into <mirror>/_playlists/ and rebuilt every pass. Three look at what gets played -- heavy rotation, all-time, and things played heavily once and silent for a year. Three look at what does not: album tracks skipped on records otherwise played constantly, unplayed tracks by the artists played most, and unplayed tracks from anywhere. Four fifths of the library has never been played, so both halves are needed. Ninety days was the obvious window for "recent" and is the wrong one. On a real history it holds a few hundred plays spread across a twenty-thousand track rotation, and nothing ranks meaningfully. Twelve months does. The two rotating playlists are shuffled by week rather than by pass. A pass runs every few hours; a playlist that reorders itself each time is one that has to be re-imported each time, because the Music app imports a snapshot of a file rather than tracking it. Lidarr knows where the lossless source is, and the playlists have to point at the MP3s music-mirror produced from it. The library root is derived from the common parent of the indexed artist folders, so it agrees with Lidarr by construction instead of being kept in step by hand, and can be overridden. Entries are relative to the playlist file, so one file works from the NAS, from a Mac over SMB and from Linux. A track is listed only once its mirror file has been confirmed to exist: Lidarr holding the FLAC says nothing about whether the MP3 has been encoded. When many are missing the run says so, because that is what a misconfigured root looks like -- every path mapping to nothing -- and silence there would present an empty playlist as a correct one.	2026-08-24 17:50:58 +01:00
Emma Thorpe	61ce751ac5	fix: distinguish a title collision from an attribution miss, and index for it ... Two faults in the split added last change, both visible in the first real run. It reported 700 pairs as attribution disagreements on the strength of the library holding the same title under a different artist. The examples show what that actually caught: "Everyday" matched Def Leppard, "Kaleidoscope" matched Chappell Roan, "Fight for Your Right" matched Motley Crue. Different songs that happen to share a name. Across fifty thousand tracks that is not an edge case, it is the common case, and presenting it as a matcher failure argues for exactly the title-only matching tier that would produce this rubbish on purpose. The signal for a real attribution miss is narrower: the library's own title credits the artist the play is filed under, as in "Voodoo People (Pendulum Remix)" against a scrobble credited to Pendulum. The normalised title has that suffix stripped -- which is what let the two meet in the first place -- so the raw title is searched for the name. Collisions are now counted and named separately, as what they are. The same query also took forty-seven seconds. lidarr_track was indexed on (norm_artist, norm_title), which a lookup by title alone cannot use because its leading column is the artist, so every unmatched key scanned all eighty-four thousand tracks. Add the index on the title by itself; the query plan changes from an automatic partial index to a covering one.	2026-08-24 17:35:32 +01:00
Emma Thorpe	2886c02a2e	feat: split unmatched listening by whether the library holds the title ... "Unmatched by an artist the library holds" was presented as the matcher's misses. On real data it is not: owning one album by an artist says nothing about owning a particular single of theirs, and most of that figure turned out to be drum and bass tracks streamed but never bought. Split it in two. A title the library holds under some other artist is an attribution disagreement -- a remixer credited as the artist, a guest billed as one -- and is a genuine miss worth fixing; the report now names the artist the library files it under, which is the information needed to judge it. A title the library does not hold at all under any artist was never bought, and no improvement to matching will conjure it. The distinction matters beyond presentation. That figure is the gate on the cull, and a gate computed from a number that overstates the failure rate blocks work that is actually safe to do.	2026-08-24 17:15:56 +01:00
Emma Thorpe	7588fee302	fix: match bracketed guest credits and hyphenated version suffixes ... Two normalisation faults, both found by running a real coverage report's unmatched list back through the normaliser. Between them they account for five of the fifteen worst misses by play count. The guest-credit pattern required whitespace immediately before the word, so it caught "Yellowcard feat. Tay Jardine" but missed "Self vs Self (feat. In Flames)" -- and the bracketed form is the more common of the two. An opening bracket is now allowed in that position. The trailing-version pattern matched the suffix as a run of non-hyphens, which cannot cross a hyphen inside the suffix itself: "Gold Dust - Shy FX Re-Edit" and "Back To Your Roots - Friction & K-Tee Remix" both survived untouched. Matched lazily instead. Four version words are added for how drum and bass marks its variants: vip, bootleg, rework, extended. They only apply inside a bracket or after a trailing dash, so the exposure is small, and ordinary titles carrying those words -- Editors, Mixed Emotions, Radio Ga Ga, Live and Let Die -- are pinned as tests against exactly that. The report also gains the figures that explain why the MBID tier contributes so little. Two thirds of scrobbles carry a recording id, and only a twentieth of them join on one: MusicBrainz holds a separate recording per release, and the two sides rarely choose the same one. Counting the pairs that carried an id and matched on name anyway measures that disagreement directly, and settles that the weakness is not a bug in the join.	2026-08-24 16:56:13 +01:00
Emma Thorpe	edebecc8ea	fix: hold one connection to Lidarr open, and retry what deserves retrying ... Indexing makes two requests per artist, and more when the album fallback fires. urllib opens a new TCP connection and performs a new DNS lookup for every one of them, so a large library becomes thousands of lookups inside a few minutes. That is enough to exhaust a container's resolver, and the result is "[Errno -3] Try again" on every artist at once -- a failure caused entirely by how the requests were made rather than by anything wrong with Lidarr. Add a transport that keeps one connection open per host, so the name is resolved once and the socket is reused. It retries once on a connection the server has already closed, since a stale keep-alive announces itself only on use. Retries were previously declined on the grounds that Lidarr is on the same LAN. That is not a safe assumption -- it may sit behind a public hostname and a reverse proxy -- and a transient failure currently costs an artist their entire entry for that pass. Transient failures are now retried with a backoff. HTTP 500 is deliberately excluded: it is an exception inside Lidarr's serialisation, not a busy server, and three attempts only delay finding that out. The same distinction gates the album probe added alongside this. Naming the offending album costs one request per album of that artist, which is worth it for a deterministic fault and actively harmful during a network-wide one, where every artist fails and probing each of them multiplies the load responsible. The keep-alive transport is tested against a real local HTTP server rather than a fake, because connection reuse and status mapping are exactly the properties a fake would assume rather than demonstrate.	2026-08-24 14:40:00 +01:00
Emma Thorpe	997627f4fe	fix: survive an album with two monitored releases ... Fetching every album in one unfiltered request avoided Lidarr's unguarded per-artist path, but not the exception underneath it. Every album endpoint maps through AlbumResource.ToResource, which selects the release with SingleOrDefault(x => x.Monitored). An album with two monitored releases makes that throw -- "Sequence contains more than one element" -- and the bulk call loses the entire library to one bad row. Keep the unfiltered call as the first attempt, since it is a single request and is still the only path that skips albums whose artist metadata is missing. When it fails, fall back to one request per artist. That cannot dodge the exception either, but it confines the loss to whichever artist owns the offending album and names them, which is the only practical way to find it in a large library. Album failures are counted separately from artist failures because they do not mean the same thing. Tracks come from a different endpoint with a different mapper, so an artist whose albums cannot be fetched still gets indexed and still matches; it is the cull that cannot run. The report distinguishes the two rather than lumping them into one warning that overstates the damage.	2026-08-24 14:24:49 +01:00
Emma Thorpe	3e78f8ebd4	fix: index albums through the endpoint Lidarr does not throw from ... Indexing fetched albums one artist at a time, and `GET /api/v1/album?artistId=` is Lidarr's unguarded path. It maps straight from the album service with no hydration: the mapper then dereferences model.Images and model.SecondaryTypes without a null check, follows model.Artist?.Value where only the first link is guarded, and selects the monitored release with SingleOrDefault, which throws outright when an album has two of them. Any of those is a 500 that aborts the whole index. The unfiltered `GET /api/v1/album` builds its own artist and release lookups and skips an album whose metadata is missing rather than dereferencing it. Use that instead, once, and group by artistId locally. It is the defensive path and it costs N fewer requests. Tracks and files have no unfiltered endpoint -- Lidarr rejects a call with no filter at all -- so those stay per artist. A failure on one artist now skips that artist rather than ending the run, but the count is recorded in the store and the coverage report leads with it: a missing artist makes their played music look unplayed, which is precisely the error that costs music later, so an incomplete index must not be culled against. Errors now carry the request URL and whatever the server put in the body. The original report of this failure was "album: HTTP 500", which points at the URL and the credentials -- neither of which was at fault.	2026-08-24 14:05:21 +01:00
Emma Thorpe	5c4797ef38	feat: index the library from Lidarr and match it against the scrobbles ... Stage two. The scrobble history says what was played by name; Lidarr says what is owned, and where the files are. Neither is useful for curation until the two are tied together, and the quality of that join is what decides whether the later cull can be trusted at all. The index is a wholesale rebuild of every artist, album and track Lidarr holds, including file paths and the date each file landed -- the latter for the age floor a cull will need. It is rebuilt rather than reconciled because Lidarr is the authority and a deletion there has to disappear here, not linger as a library entry with no file behind it. Every call is a GET; nothing is written back. Matching runs at the level of the distinct artist/track pair rather than the individual play, because a verdict is a property of the name pair and there are three plays for every one of them. Two tiers: a MusicBrainz recording id, which Last.fm supplies per scrobble and Lidarr exposes as ForeignRecordingId, gives an exact join; everything else falls to a normalised name comparison. There is deliberately no third tier. A near-miss guess is worse than an admitted one, since the entire purpose of the resulting number is to state how far the matching can be relied on. Normalisation folds the ways the two sides habitually disagree: case, accents, guest credits that Last.fm puts in the artist field, trailing version suffixes, ampersands, and a leading article. Punctuation needs two opposing rules and both are load-bearing -- apostrophes are deleted so "Don't" meets "Dont", while every other mark becomes a space so "AC/DC", "AC-DC" and "AC DC" meet as well. It errs towards collapsing too much: a false match makes a track look played, a missed match makes it look abandoned, and only the second one loses music. The coverage report deliberately does not lead with matched versus unmatched. Most unmatched listening is music that was never in the library and says nothing about the matcher. The figure that matters is unmatched listening by an artist the library does hold: a track that was played, sitting next to a file it should have matched. The worst fifteen are listed by play count. The schema gains its tables additively and migrates a version 1 store in place, because rebuilding a nine-year history costs several thousand API requests.	2026-08-24 13:31:08 +01:00
Emma Thorpe	18f05d3d55	feat: ingest a Last.fm scrobble history into a local store ... First stage of a curation tool for the music library that music-mirror mirrors. Before anything can build playlists or decide what has gone cold, there has to be a local, queryable record of what is actually played; an API call per question does not scale to a library-sized analysis. Ingest is in two halves. A catch-up fetches everything scrobbled since the newest scrobble held, and a backfill walks the history backwards until it runs out. Both take their bounds from the database rather than from a saved cursor, so an interrupted run resumes from what it actually has, and both windows are bounded at each end so paging cannot shift under the fetch while new scrobbles arrive mid-run. Scrobbles carry no identifier, so the primary key is timestamp, artist and track. Two plays of one track in the same second collapse into a single row: they are indistinguishable in the data, and a surrogate key would make re-ingest non-idempotent, which is the worse trade. Three API behaviours are handled explicitly because each fails silently: the currently-playing track arrives with no timestamp and would be re-ingested on every pass; a lone result is returned as a bare object rather than a one-item list; and MBIDs are empty strings rather than absent when unknown, which would later look like a usable join key. Retries cover the rate limit and the transient backend errors with an exponential backoff. An invalid or suspended key fails immediately. The report exists to surface one number before the next stage is built: the share of scrobbles carrying a MusicBrainz recording id. Lidarr exposes the same identifier per track, so those can be joined exactly and the rest must go through name matching. That percentage bounds how far the matcher can be trusted. No runtime dependencies, and the tests run against a fake transport that reproduces the service's paging and response shapes, so they need neither network nor credentials.	2026-08-24 12:03:08 +01:00