12 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Emma Thorpe	40dfce8a4c	fix: look tags up by artist name rather than by MusicBrainz id ... Tag lookups asked by MusicBrainz id whenever Lidarr had one, which is always. The reasoning was that an id cannot be ambiguous the way a name can. In practice Last.fm's mbid index is stale and partial, and it answered "the artist you supplied could not be found" for Devo, Escape the Fate, Blasterjaxx, Frank Carter & the Rattlesnakes and a long tail of others -- artists whose Last.fm pages plainly exist and carry precisely the tags the moods are built from. Devo is tagged new wave, post-punk and 80s; Escape the Fate is tagged post-hardcore, screamo and emocore. Both were skipped. Ask by name first, which is the index Last.fm's own site runs on, and keep the id only as a fallback for a name Lidarr spells differently. Failures were also being dropped without recording the attempt, so every one of those artists was re-queried on every subsequent pass, indefinitely. They are now distinguished: an artist that neither key resolves is recorded as fetched with no tags and not asked about again, while a genuine failure -- a rate limit, a bad key -- is deliberately left unrecorded so the next pass retries it. Telling the two apart needed the service's own error number, so LastfmError now carries it.	2026-08-24 18:13:23 +01:00
Emma Thorpe	15e5ee5aea	feat: mood playlists from Last.fm crowd tags ... A second set of playlists selecting by genre and mood rather than by play history: eighties synths, high energy rock, screamo, drum and bass, dance, classic rock. The tags come from Last.fm rather than MusicBrainz. MusicBrainz genres arrive free with the Lidarr index, which makes them the obvious choice and the wrong one: they are sparse and formal, and will not tell you a record is screamo or synthwave. Crowd tags will, because people typed them. One request per artist, by MusicBrainz id where Lidarr has one, refreshed every ninety days. An artist Last.fm has never heard of is recorded as fetched with no tags rather than left unmarked, so it is not asked about again on every pass forever. The tag table is keyed on the normalised artist name, not the Lidarr id, so it survives an artist being removed and re-added there. Weights are taken from the response's count where it has one. The documented sample carries only a name and a URL, a live response also carries a 0-100 count, and depending on either alone would be a guess -- so the count is used when present and the documented ordering by popularity stands in when it is not. An artist qualifies for a mood when their weights inside it sum to at least thirty. A single low-weight tag is not a genre, it is somebody's stray opinion. A mood may also restrict release years, which is what separates eighties synth records from everything else a synthpop tag drags in. The built-in set is chosen for this library rather than as a taxonomy, and --vibes replaces it wholesale with a JSON file so a new mood does not need a new release. Names are validated when that file is read: an invalid one would otherwise only surface as a playlist written somewhere unintended.	2026-08-24 17:56:35 +01:00
Emma Thorpe	cda3d8463b	feat: write playlists into the mirror from the listening history ... Six rules over the scrobble history and the library index, written as extended M3U into <mirror>/_playlists/ and rebuilt every pass. Three look at what gets played -- heavy rotation, all-time, and things played heavily once and silent for a year. Three look at what does not: album tracks skipped on records otherwise played constantly, unplayed tracks by the artists played most, and unplayed tracks from anywhere. Four fifths of the library has never been played, so both halves are needed. Ninety days was the obvious window for "recent" and is the wrong one. On a real history it holds a few hundred plays spread across a twenty-thousand track rotation, and nothing ranks meaningfully. Twelve months does. The two rotating playlists are shuffled by week rather than by pass. A pass runs every few hours; a playlist that reorders itself each time is one that has to be re-imported each time, because the Music app imports a snapshot of a file rather than tracking it. Lidarr knows where the lossless source is, and the playlists have to point at the MP3s music-mirror produced from it. The library root is derived from the common parent of the indexed artist folders, so it agrees with Lidarr by construction instead of being kept in step by hand, and can be overridden. Entries are relative to the playlist file, so one file works from the NAS, from a Mac over SMB and from Linux. A track is listed only once its mirror file has been confirmed to exist: Lidarr holding the FLAC says nothing about whether the MP3 has been encoded. When many are missing the run says so, because that is what a misconfigured root looks like -- every path mapping to nothing -- and silence there would present an empty playlist as a correct one.	2026-08-24 17:50:58 +01:00
Emma Thorpe	61ce751ac5	fix: distinguish a title collision from an attribution miss, and index for it ... Two faults in the split added last change, both visible in the first real run. It reported 700 pairs as attribution disagreements on the strength of the library holding the same title under a different artist. The examples show what that actually caught: "Everyday" matched Def Leppard, "Kaleidoscope" matched Chappell Roan, "Fight for Your Right" matched Motley Crue. Different songs that happen to share a name. Across fifty thousand tracks that is not an edge case, it is the common case, and presenting it as a matcher failure argues for exactly the title-only matching tier that would produce this rubbish on purpose. The signal for a real attribution miss is narrower: the library's own title credits the artist the play is filed under, as in "Voodoo People (Pendulum Remix)" against a scrobble credited to Pendulum. The normalised title has that suffix stripped -- which is what let the two meet in the first place -- so the raw title is searched for the name. Collisions are now counted and named separately, as what they are. The same query also took forty-seven seconds. lidarr_track was indexed on (norm_artist, norm_title), which a lookup by title alone cannot use because its leading column is the artist, so every unmatched key scanned all eighty-four thousand tracks. Add the index on the title by itself; the query plan changes from an automatic partial index to a covering one.	2026-08-24 17:35:32 +01:00
Emma Thorpe	45d99ff039	ci: build the image once instead of twice ... A pull request took roughly eleven minutes to go green, and the log shows one CACHED line in the whole run. The image was being built twice, in full. The test stage is built by the runner's docker daemon. The runtime stage was then built by docker/build-push-action, which runs under a buildx builder that setup-buildx-action creates in its own container with its own cache. The two share nothing, so the second build pulled the base image again, ran pip install again, and exported the layers again -- about four and a half minutes, plus another thirty-five seconds to boot buildkit. The comment above the test step claimed those layers were shared, which is what made this look reasonable. buildx earns that overhead when producing several architectures. This produces linux/amd64 only, by an explicit decision recorded in the workflow, so it earns nothing here. Use plain docker build against the same daemon that ran the tests, and push with docker push. The runtime stage is a strict prefix of the test stage, so every layer is a cache hit: measured at 1.3 seconds locally against roughly four and a half minutes in CI. The remaining time is the runner itself, which is slow in absolute terms -- pytest takes four seconds locally and a hundred and two in CI. That is not something the workflow can fix.	2026-08-24 17:24:56 +01:00
Emma Thorpe	2886c02a2e	feat: split unmatched listening by whether the library holds the title ... "Unmatched by an artist the library holds" was presented as the matcher's misses. On real data it is not: owning one album by an artist says nothing about owning a particular single of theirs, and most of that figure turned out to be drum and bass tracks streamed but never bought. Split it in two. A title the library holds under some other artist is an attribution disagreement -- a remixer credited as the artist, a guest billed as one -- and is a genuine miss worth fixing; the report now names the artist the library files it under, which is the information needed to judge it. A title the library does not hold at all under any artist was never bought, and no improvement to matching will conjure it. The distinction matters beyond presentation. That figure is the gate on the cull, and a gate computed from a number that overstates the failure rate blocks work that is actually safe to do.	2026-08-24 17:15:56 +01:00
Emma Thorpe	7588fee302	fix: match bracketed guest credits and hyphenated version suffixes ... Two normalisation faults, both found by running a real coverage report's unmatched list back through the normaliser. Between them they account for five of the fifteen worst misses by play count. The guest-credit pattern required whitespace immediately before the word, so it caught "Yellowcard feat. Tay Jardine" but missed "Self vs Self (feat. In Flames)" -- and the bracketed form is the more common of the two. An opening bracket is now allowed in that position. The trailing-version pattern matched the suffix as a run of non-hyphens, which cannot cross a hyphen inside the suffix itself: "Gold Dust - Shy FX Re-Edit" and "Back To Your Roots - Friction & K-Tee Remix" both survived untouched. Matched lazily instead. Four version words are added for how drum and bass marks its variants: vip, bootleg, rework, extended. They only apply inside a bracket or after a trailing dash, so the exposure is small, and ordinary titles carrying those words -- Editors, Mixed Emotions, Radio Ga Ga, Live and Let Die -- are pinned as tests against exactly that. The report also gains the figures that explain why the MBID tier contributes so little. Two thirds of scrobbles carry a recording id, and only a twentieth of them join on one: MusicBrainz holds a separate recording per release, and the two sides rarely choose the same one. Counting the pairs that carried an id and matched on name anyway measures that disagreement directly, and settles that the weakness is not a bug in the join.	2026-08-24 16:56:13 +01:00
Emma Thorpe	edebecc8ea	fix: hold one connection to Lidarr open, and retry what deserves retrying ... Indexing makes two requests per artist, and more when the album fallback fires. urllib opens a new TCP connection and performs a new DNS lookup for every one of them, so a large library becomes thousands of lookups inside a few minutes. That is enough to exhaust a container's resolver, and the result is "[Errno -3] Try again" on every artist at once -- a failure caused entirely by how the requests were made rather than by anything wrong with Lidarr. Add a transport that keeps one connection open per host, so the name is resolved once and the socket is reused. It retries once on a connection the server has already closed, since a stale keep-alive announces itself only on use. Retries were previously declined on the grounds that Lidarr is on the same LAN. That is not a safe assumption -- it may sit behind a public hostname and a reverse proxy -- and a transient failure currently costs an artist their entire entry for that pass. Transient failures are now retried with a backoff. HTTP 500 is deliberately excluded: it is an exception inside Lidarr's serialisation, not a busy server, and three attempts only delay finding that out. The same distinction gates the album probe added alongside this. Naming the offending album costs one request per album of that artist, which is worth it for a deterministic fault and actively harmful during a network-wide one, where every artist fails and probing each of them multiplies the load responsible. The keep-alive transport is tested against a real local HTTP server rather than a fake, because connection reuse and status mapping are exactly the properties a fake would assume rather than demonstrate.	2026-08-24 14:40:00 +01:00
Emma Thorpe	997627f4fe	fix: survive an album with two monitored releases ... Fetching every album in one unfiltered request avoided Lidarr's unguarded per-artist path, but not the exception underneath it. Every album endpoint maps through AlbumResource.ToResource, which selects the release with SingleOrDefault(x => x.Monitored). An album with two monitored releases makes that throw -- "Sequence contains more than one element" -- and the bulk call loses the entire library to one bad row. Keep the unfiltered call as the first attempt, since it is a single request and is still the only path that skips albums whose artist metadata is missing. When it fails, fall back to one request per artist. That cannot dodge the exception either, but it confines the loss to whichever artist owns the offending album and names them, which is the only practical way to find it in a large library. Album failures are counted separately from artist failures because they do not mean the same thing. Tracks come from a different endpoint with a different mapper, so an artist whose albums cannot be fetched still gets indexed and still matches; it is the cull that cannot run. The report distinguishes the two rather than lumping them into one warning that overstates the damage.	2026-08-24 14:24:49 +01:00
Emma Thorpe	3e78f8ebd4	fix: index albums through the endpoint Lidarr does not throw from ... Indexing fetched albums one artist at a time, and `GET /api/v1/album?artistId=` is Lidarr's unguarded path. It maps straight from the album service with no hydration: the mapper then dereferences model.Images and model.SecondaryTypes without a null check, follows model.Artist?.Value where only the first link is guarded, and selects the monitored release with SingleOrDefault, which throws outright when an album has two of them. Any of those is a 500 that aborts the whole index. The unfiltered `GET /api/v1/album` builds its own artist and release lookups and skips an album whose metadata is missing rather than dereferencing it. Use that instead, once, and group by artistId locally. It is the defensive path and it costs N fewer requests. Tracks and files have no unfiltered endpoint -- Lidarr rejects a call with no filter at all -- so those stay per artist. A failure on one artist now skips that artist rather than ending the run, but the count is recorded in the store and the coverage report leads with it: a missing artist makes their played music look unplayed, which is precisely the error that costs music later, so an incomplete index must not be culled against. Errors now carry the request URL and whatever the server put in the body. The original report of this failure was "album: HTTP 500", which points at the URL and the credentials -- neither of which was at fault.	2026-08-24 14:05:21 +01:00
Emma Thorpe	5c4797ef38	feat: index the library from Lidarr and match it against the scrobbles ... Stage two. The scrobble history says what was played by name; Lidarr says what is owned, and where the files are. Neither is useful for curation until the two are tied together, and the quality of that join is what decides whether the later cull can be trusted at all. The index is a wholesale rebuild of every artist, album and track Lidarr holds, including file paths and the date each file landed -- the latter for the age floor a cull will need. It is rebuilt rather than reconciled because Lidarr is the authority and a deletion there has to disappear here, not linger as a library entry with no file behind it. Every call is a GET; nothing is written back. Matching runs at the level of the distinct artist/track pair rather than the individual play, because a verdict is a property of the name pair and there are three plays for every one of them. Two tiers: a MusicBrainz recording id, which Last.fm supplies per scrobble and Lidarr exposes as ForeignRecordingId, gives an exact join; everything else falls to a normalised name comparison. There is deliberately no third tier. A near-miss guess is worse than an admitted one, since the entire purpose of the resulting number is to state how far the matching can be relied on. Normalisation folds the ways the two sides habitually disagree: case, accents, guest credits that Last.fm puts in the artist field, trailing version suffixes, ampersands, and a leading article. Punctuation needs two opposing rules and both are load-bearing -- apostrophes are deleted so "Don't" meets "Dont", while every other mark becomes a space so "AC/DC", "AC-DC" and "AC DC" meet as well. It errs towards collapsing too much: a false match makes a track look played, a missed match makes it look abandoned, and only the second one loses music. The coverage report deliberately does not lead with matched versus unmatched. Most unmatched listening is music that was never in the library and says nothing about the matcher. The figure that matters is unmatched listening by an artist the library does hold: a track that was played, sitting next to a file it should have matched. The worst fifteen are listed by play count. The schema gains its tables additively and migrates a version 1 store in place, because rebuilding a nine-year history costs several thousand API requests.	2026-08-24 13:31:08 +01:00
Emma Thorpe	18f05d3d55	feat: ingest a Last.fm scrobble history into a local store ... First stage of a curation tool for the music library that music-mirror mirrors. Before anything can build playlists or decide what has gone cold, there has to be a local, queryable record of what is actually played; an API call per question does not scale to a library-sized analysis. Ingest is in two halves. A catch-up fetches everything scrobbled since the newest scrobble held, and a backfill walks the history backwards until it runs out. Both take their bounds from the database rather than from a saved cursor, so an interrupted run resumes from what it actually has, and both windows are bounded at each end so paging cannot shift under the fetch while new scrobbles arrive mid-run. Scrobbles carry no identifier, so the primary key is timestamp, artist and track. Two plays of one track in the same second collapse into a single row: they are indistinguishable in the data, and a surrogate key would make re-ingest non-idempotent, which is the worse trade. Three API behaviours are handled explicitly because each fails silently: the currently-playing track arrives with no timestamp and would be re-ingested on every pass; a lone result is returned as a bare object rather than a one-item list; and MBIDs are empty strings rather than absent when unknown, which would later look like a usable join key. Retries cover the rate limit and the transient backend errors with an exponential backoff. An invalid or suspended key fails immediately. The report exists to surface one number before the next stage is built: the share of scrobbles carrying a MusicBrainz recording id. Lidarr exposes the same identifier per track, so those can be joined exactly and the rest must go through name matching. That percentage bounds how far the matcher can be trusted. No runtime dependencies, and the tests run against a fake transport that reproduces the service's paging and response shapes, so they need neither network nor credentials.	2026-08-24 12:03:08 +01:00