7 Commits

Include renames

Author	SHA1	Message	Date
Emma Thorpe	2886c02a2e	feat: split unmatched listening by whether the library holds the title ... "Unmatched by an artist the library holds" was presented as the matcher's misses. On real data it is not: owning one album by an artist says nothing about owning a particular single of theirs, and most of that figure turned out to be drum and bass tracks streamed but never bought. Split it in two. A title the library holds under some other artist is an attribution disagreement -- a remixer credited as the artist, a guest billed as one -- and is a genuine miss worth fixing; the report now names the artist the library files it under, which is the information needed to judge it. A title the library does not hold at all under any artist was never bought, and no improvement to matching will conjure it. The distinction matters beyond presentation. That figure is the gate on the cull, and a gate computed from a number that overstates the failure rate blocks work that is actually safe to do.	2026-08-24 17:15:56 +01:00
Emma Thorpe	7588fee302	fix: match bracketed guest credits and hyphenated version suffixes ... Two normalisation faults, both found by running a real coverage report's unmatched list back through the normaliser. Between them they account for five of the fifteen worst misses by play count. The guest-credit pattern required whitespace immediately before the word, so it caught "Yellowcard feat. Tay Jardine" but missed "Self vs Self (feat. In Flames)" -- and the bracketed form is the more common of the two. An opening bracket is now allowed in that position. The trailing-version pattern matched the suffix as a run of non-hyphens, which cannot cross a hyphen inside the suffix itself: "Gold Dust - Shy FX Re-Edit" and "Back To Your Roots - Friction & K-Tee Remix" both survived untouched. Matched lazily instead. Four version words are added for how drum and bass marks its variants: vip, bootleg, rework, extended. They only apply inside a bracket or after a trailing dash, so the exposure is small, and ordinary titles carrying those words -- Editors, Mixed Emotions, Radio Ga Ga, Live and Let Die -- are pinned as tests against exactly that. The report also gains the figures that explain why the MBID tier contributes so little. Two thirds of scrobbles carry a recording id, and only a twentieth of them join on one: MusicBrainz holds a separate recording per release, and the two sides rarely choose the same one. Counting the pairs that carried an id and matched on name anyway measures that disagreement directly, and settles that the weakness is not a bug in the join.	2026-08-24 16:56:13 +01:00
Emma Thorpe	edebecc8ea	fix: hold one connection to Lidarr open, and retry what deserves retrying ... Indexing makes two requests per artist, and more when the album fallback fires. urllib opens a new TCP connection and performs a new DNS lookup for every one of them, so a large library becomes thousands of lookups inside a few minutes. That is enough to exhaust a container's resolver, and the result is "[Errno -3] Try again" on every artist at once -- a failure caused entirely by how the requests were made rather than by anything wrong with Lidarr. Add a transport that keeps one connection open per host, so the name is resolved once and the socket is reused. It retries once on a connection the server has already closed, since a stale keep-alive announces itself only on use. Retries were previously declined on the grounds that Lidarr is on the same LAN. That is not a safe assumption -- it may sit behind a public hostname and a reverse proxy -- and a transient failure currently costs an artist their entire entry for that pass. Transient failures are now retried with a backoff. HTTP 500 is deliberately excluded: it is an exception inside Lidarr's serialisation, not a busy server, and three attempts only delay finding that out. The same distinction gates the album probe added alongside this. Naming the offending album costs one request per album of that artist, which is worth it for a deterministic fault and actively harmful during a network-wide one, where every artist fails and probing each of them multiplies the load responsible. The keep-alive transport is tested against a real local HTTP server rather than a fake, because connection reuse and status mapping are exactly the properties a fake would assume rather than demonstrate.	2026-08-24 14:40:00 +01:00
Emma Thorpe	997627f4fe	fix: survive an album with two monitored releases ... Fetching every album in one unfiltered request avoided Lidarr's unguarded per-artist path, but not the exception underneath it. Every album endpoint maps through AlbumResource.ToResource, which selects the release with SingleOrDefault(x => x.Monitored). An album with two monitored releases makes that throw -- "Sequence contains more than one element" -- and the bulk call loses the entire library to one bad row. Keep the unfiltered call as the first attempt, since it is a single request and is still the only path that skips albums whose artist metadata is missing. When it fails, fall back to one request per artist. That cannot dodge the exception either, but it confines the loss to whichever artist owns the offending album and names them, which is the only practical way to find it in a large library. Album failures are counted separately from artist failures because they do not mean the same thing. Tracks come from a different endpoint with a different mapper, so an artist whose albums cannot be fetched still gets indexed and still matches; it is the cull that cannot run. The report distinguishes the two rather than lumping them into one warning that overstates the damage.	2026-08-24 14:24:49 +01:00
Emma Thorpe	3e78f8ebd4	fix: index albums through the endpoint Lidarr does not throw from ... Indexing fetched albums one artist at a time, and `GET /api/v1/album?artistId=` is Lidarr's unguarded path. It maps straight from the album service with no hydration: the mapper then dereferences model.Images and model.SecondaryTypes without a null check, follows model.Artist?.Value where only the first link is guarded, and selects the monitored release with SingleOrDefault, which throws outright when an album has two of them. Any of those is a 500 that aborts the whole index. The unfiltered `GET /api/v1/album` builds its own artist and release lookups and skips an album whose metadata is missing rather than dereferencing it. Use that instead, once, and group by artistId locally. It is the defensive path and it costs N fewer requests. Tracks and files have no unfiltered endpoint -- Lidarr rejects a call with no filter at all -- so those stay per artist. A failure on one artist now skips that artist rather than ending the run, but the count is recorded in the store and the coverage report leads with it: a missing artist makes their played music look unplayed, which is precisely the error that costs music later, so an incomplete index must not be culled against. Errors now carry the request URL and whatever the server put in the body. The original report of this failure was "album: HTTP 500", which points at the URL and the credentials -- neither of which was at fault.	2026-08-24 14:05:21 +01:00
Emma Thorpe	5c4797ef38	feat: index the library from Lidarr and match it against the scrobbles ... Stage two. The scrobble history says what was played by name; Lidarr says what is owned, and where the files are. Neither is useful for curation until the two are tied together, and the quality of that join is what decides whether the later cull can be trusted at all. The index is a wholesale rebuild of every artist, album and track Lidarr holds, including file paths and the date each file landed -- the latter for the age floor a cull will need. It is rebuilt rather than reconciled because Lidarr is the authority and a deletion there has to disappear here, not linger as a library entry with no file behind it. Every call is a GET; nothing is written back. Matching runs at the level of the distinct artist/track pair rather than the individual play, because a verdict is a property of the name pair and there are three plays for every one of them. Two tiers: a MusicBrainz recording id, which Last.fm supplies per scrobble and Lidarr exposes as ForeignRecordingId, gives an exact join; everything else falls to a normalised name comparison. There is deliberately no third tier. A near-miss guess is worse than an admitted one, since the entire purpose of the resulting number is to state how far the matching can be relied on. Normalisation folds the ways the two sides habitually disagree: case, accents, guest credits that Last.fm puts in the artist field, trailing version suffixes, ampersands, and a leading article. Punctuation needs two opposing rules and both are load-bearing -- apostrophes are deleted so "Don't" meets "Dont", while every other mark becomes a space so "AC/DC", "AC-DC" and "AC DC" meet as well. It errs towards collapsing too much: a false match makes a track look played, a missed match makes it look abandoned, and only the second one loses music. The coverage report deliberately does not lead with matched versus unmatched. Most unmatched listening is music that was never in the library and says nothing about the matcher. The figure that matters is unmatched listening by an artist the library does hold: a track that was played, sitting next to a file it should have matched. The worst fifteen are listed by play count. The schema gains its tables additively and migrates a version 1 store in place, because rebuilding a nine-year history costs several thousand API requests.	2026-08-24 13:31:08 +01:00
Emma Thorpe	18f05d3d55	feat: ingest a Last.fm scrobble history into a local store ... First stage of a curation tool for the music library that music-mirror mirrors. Before anything can build playlists or decide what has gone cold, there has to be a local, queryable record of what is actually played; an API call per question does not scale to a library-sized analysis. Ingest is in two halves. A catch-up fetches everything scrobbled since the newest scrobble held, and a backfill walks the history backwards until it runs out. Both take their bounds from the database rather than from a saved cursor, so an interrupted run resumes from what it actually has, and both windows are bounded at each end so paging cannot shift under the fetch while new scrobbles arrive mid-run. Scrobbles carry no identifier, so the primary key is timestamp, artist and track. Two plays of one track in the same second collapse into a single row: they are indistinguishable in the data, and a surrogate key would make re-ingest non-idempotent, which is the worse trade. Three API behaviours are handled explicitly because each fails silently: the currently-playing track arrives with no timestamp and would be re-ingested on every pass; a lone result is returned as a bare object rather than a one-item list; and MBIDs are empty strings rather than absent when unknown, which would later look like a usable join key. Retries cover the rate limit and the transient backend errors with an exponential backoff. An invalid or suspended key fails immediately. The report exists to surface one number before the next stage is built: the share of scrobbles carrying a MusicBrainz recording id. Lidarr exposes the same identifier per track, so those can be joined exactly and the rest must go through name matching. That percentage bounds how far the matcher can be trusted. No runtime dependencies, and the tests run against a fake transport that reproduces the service's paging and response shapes, so they need neither network nor credentials.	2026-08-24 12:03:08 +01:00