feat: index the library from Lidarr and match it against the scrobbles #1

2026-08-24T13:32:35+01:00

lyrathorpe commented

2026-08-24 13:32:35 +01:00

What this adds

The scrobble history says what was played, by name. Lidarr says what is owned,
and where. Neither is useful for curation until the two are joined, and the
quality of that join decides whether the cull in stage four can be trusted.

Library index. Every artist, album and track Lidarr holds, with file
paths and the date each file landed -- the latter for the age floor a cull
will need. Rebuilt wholesale each pass rather than reconciled: Lidarr is the
authority, and a deletion there has to disappear here rather than linger as a
library entry with no file behind it. Every call is a GET.
Matcher. Two tiers, at the granularity of the distinct artist/track pair
rather than the individual play. A MusicBrainz recording id gives an exact
join; everything else falls to a normalised name comparison. No third tier --
a near-miss guess is worse than an admitted one when the whole point of the
output is to say how far the matching can be relied on.
Coverage report. Deliberately does not lead with matched versus
unmatched.

The normalisation

Folds the ways the two sides habitually disagree: case, accents, guest credits
Last.fm puts in the artist field, trailing version suffixes, ampersands, a
leading article. Punctuation needs two opposing rules, both load-bearing:
apostrophes are deleted so Don't meets Dont, every other mark becomes a
space so AC/DC, AC-DC and AC DC meet too. One rule doing both jobs
was the first implementation, and the tests caught it.

It errs towards collapsing too much. A false match makes a track look played; a
missed match makes it look abandoned. Only the second one loses music.

Reading the report

unmatched by an artist the library holds: N pairs, M plays is the line that
matters. Most unmatched listening is music never in the library and says
nothing about the matcher. That line is a track that was played, sitting next
to a file it should have matched. The worst fifteen are listed.

Testing

39 tests, green locally and in docker build --target test. The Lidarr fake
serves a canned library split across the same four endpoints the indexer calls,
so the stitching is exercised rather than stubbed. Real cases from the live
report are pinned as tests: Yellowcard feat. Tay Jardine, BABYMETAL Feat. F.Hero, AC/DC, Breathe (Remastered).

The schema migrates a version 1 store in place, so an existing history is not
re-downloaded.

Known limits

Indexing costs 1 + 3N requests for N artists, every pass. Lidarr is local,
so this should be seconds; --skip-index matches against the index already
held if it is not.
Field names come from Lidarr's resource definitions rather than from a live
call. The first real run is the actual test.

## What this adds The scrobble history says what was played, by name. Lidarr says what is owned, and where. Neither is useful for curation until the two are joined, and the quality of that join decides whether the cull in stage four can be trusted. - **Library index.** Every artist, album and track Lidarr holds, with file paths and the date each file landed -- the latter for the age floor a cull will need. Rebuilt wholesale each pass rather than reconciled: Lidarr is the authority, and a deletion there has to disappear here rather than linger as a library entry with no file behind it. Every call is a `GET`. - **Matcher.** Two tiers, at the granularity of the distinct artist/track pair rather than the individual play. A MusicBrainz recording id gives an exact join; everything else falls to a normalised name comparison. No third tier -- a near-miss guess is worse than an admitted one when the whole point of the output is to say how far the matching can be relied on. - **Coverage report.** Deliberately does not lead with matched versus unmatched. ## The normalisation Folds the ways the two sides habitually disagree: case, accents, guest credits Last.fm puts in the artist field, trailing version suffixes, ampersands, a leading article. Punctuation needs two opposing rules, both load-bearing: apostrophes are **deleted** so `Don't` meets `Dont`, every other mark becomes a **space** so `AC/DC`, `AC-DC` and `AC DC` meet too. One rule doing both jobs was the first implementation, and the tests caught it. It errs towards collapsing too much. A false match makes a track look played; a missed match makes it look abandoned. Only the second one loses music. ## Reading the report `unmatched by an artist the library holds: N pairs, M plays` is the line that matters. Most unmatched listening is music never in the library and says nothing about the matcher. That line is a track that was played, sitting next to a file it should have matched. The worst fifteen are listed. ## Testing 39 tests, green locally and in `docker build --target test`. The Lidarr fake serves a canned library split across the same four endpoints the indexer calls, so the stitching is exercised rather than stubbed. Real cases from the live report are pinned as tests: `Yellowcard feat. Tay Jardine`, `BABYMETAL Feat. F.Hero`, `AC/DC`, `Breathe (Remastered)`. The schema migrates a version 1 store in place, so an existing history is not re-downloaded. ## Known limits - Indexing costs `1 + 3N` requests for N artists, every pass. Lidarr is local, so this should be seconds; `--skip-index` matches against the index already held if it is not. - Field names come from Lidarr's resource definitions rather than from a live call. The first real run is the actual test.

lyrathorpe added 1 commit 2026-08-24 13:32:35 +01:00

feat: index the library from Lidarr and match it against the scrobbles

Build and publish container / build (pull_request) Successful in 10m2s

Details

5c4797ef38

Stage two. The scrobble history says what was played by name; Lidarr says what
is owned, and where the files are. Neither is useful for curation until the two
are tied together, and the quality of that join is what decides whether the
later cull can be trusted at all.

The index is a wholesale rebuild of every artist, album and track Lidarr holds,
including file paths and the date each file landed -- the latter for the age
floor a cull will need. It is rebuilt rather than reconciled because Lidarr is
the authority and a deletion there has to disappear here, not linger as a
library entry with no file behind it. Every call is a GET; nothing is written
back.

Matching runs at the level of the distinct artist/track pair rather than the
individual play, because a verdict is a property of the name pair and there are
three plays for every one of them. Two tiers: a MusicBrainz recording id, which
Last.fm supplies per scrobble and Lidarr exposes as ForeignRecordingId, gives an
exact join; everything else falls to a normalised name comparison. There is
deliberately no third tier. A near-miss guess is worse than an admitted one,
since the entire purpose of the resulting number is to state how far the
matching can be relied on.

Normalisation folds the ways the two sides habitually disagree: case, accents,
guest credits that Last.fm puts in the artist field, trailing version suffixes,
ampersands, and a leading article. Punctuation needs two opposing rules and both
are load-bearing -- apostrophes are deleted so "Don't" meets "Dont", while every
other mark becomes a space so "AC/DC", "AC-DC" and "AC DC" meet as well. It errs
towards collapsing too much: a false match makes a track look played, a missed
match makes it look abandoned, and only the second one loses music.

The coverage report deliberately does not lead with matched versus unmatched.
Most unmatched listening is music that was never in the library and says nothing
about the matcher. The figure that matters is unmatched listening by an artist
the library does hold: a track that was played, sitting next to a file it should
have matched. The worst fifteen are listed by play count.

The schema gains its tables additively and migrates a version 1 store in place,
because rebuilding a nine-year history costs several thousand API requests.

lyrathorpe merged commit f7769af835 into main

2026-08-24 13:49:57 +01:00

lyrathorpe deleted branch feat/lidarr-index-and-matcher

2026-08-24 13:49:59 +01:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lyrathorpe/music-curator#1