Six rules over the scrobble history and the library index, written as extended
M3U into <mirror>/_playlists/ and rebuilt every pass. Three look at what gets
played -- heavy rotation, all-time, and things played heavily once and silent
for a year. Three look at what does not: album tracks skipped on records
otherwise played constantly, unplayed tracks by the artists played most, and
unplayed tracks from anywhere. Four fifths of the library has never been played,
so both halves are needed.
Ninety days was the obvious window for "recent" and is the wrong one. On a real
history it holds a few hundred plays spread across a twenty-thousand track
rotation, and nothing ranks meaningfully. Twelve months does.
The two rotating playlists are shuffled by week rather than by pass. A pass runs
every few hours; a playlist that reorders itself each time is one that has to be
re-imported each time, because the Music app imports a snapshot of a file rather
than tracking it.
Lidarr knows where the lossless source is, and the playlists have to point at
the MP3s music-mirror produced from it. The library root is derived from the
common parent of the indexed artist folders, so it agrees with Lidarr by
construction instead of being kept in step by hand, and can be overridden.
Entries are relative to the playlist file, so one file works from the NAS, from
a Mac over SMB and from Linux.
A track is listed only once its mirror file has been confirmed to exist: Lidarr
holding the FLAC says nothing about whether the MP3 has been encoded. When many
are missing the run says so, because that is what a misconfigured root looks
like -- every path mapping to nothing -- and silence there would present an
empty playlist as a correct one.
Two faults in the split added last change, both visible in the first real run.
It reported 700 pairs as attribution disagreements on the strength of the
library holding the same title under a different artist. The examples show what
that actually caught: "Everyday" matched Def Leppard, "Kaleidoscope" matched
Chappell Roan, "Fight for Your Right" matched Motley Crue. Different songs that
happen to share a name. Across fifty thousand tracks that is not an edge case,
it is the common case, and presenting it as a matcher failure argues for exactly
the title-only matching tier that would produce this rubbish on purpose.
The signal for a real attribution miss is narrower: the library's own title
credits the artist the play is filed under, as in "Voodoo People (Pendulum
Remix)" against a scrobble credited to Pendulum. The normalised title has that
suffix stripped -- which is what let the two meet in the first place -- so the
raw title is searched for the name. Collisions are now counted and named
separately, as what they are.
The same query also took forty-seven seconds. lidarr_track was indexed on
(norm_artist, norm_title), which a lookup by title alone cannot use because its
leading column is the artist, so every unmatched key scanned all eighty-four
thousand tracks. Add the index on the title by itself; the query plan changes
from an automatic partial index to a covering one.
A pull request took roughly eleven minutes to go green, and the log shows one
CACHED line in the whole run. The image was being built twice, in full.
The test stage is built by the runner's docker daemon. The runtime stage was
then built by docker/build-push-action, which runs under a buildx builder that
setup-buildx-action creates in its own container with its own cache. The two
share nothing, so the second build pulled the base image again, ran pip install
again, and exported the layers again -- about four and a half minutes, plus
another thirty-five seconds to boot buildkit. The comment above the test step
claimed those layers were shared, which is what made this look reasonable.
buildx earns that overhead when producing several architectures. This produces
linux/amd64 only, by an explicit decision recorded in the workflow, so it earns
nothing here. Use plain docker build against the same daemon that ran the
tests, and push with docker push. The runtime stage is a strict prefix of the
test stage, so every layer is a cache hit: measured at 1.3 seconds locally
against roughly four and a half minutes in CI.
The remaining time is the runner itself, which is slow in absolute terms --
pytest takes four seconds locally and a hundred and two in CI. That is not
something the workflow can fix.
"Unmatched by an artist the library holds" was presented as the matcher's
misses. On real data it is not: owning one album by an artist says nothing about
owning a particular single of theirs, and most of that figure turned out to be
drum and bass tracks streamed but never bought.
Split it in two. A title the library holds under some other artist is an
attribution disagreement -- a remixer credited as the artist, a guest billed as
one -- and is a genuine miss worth fixing; the report now names the artist the
library files it under, which is the information needed to judge it. A title the
library does not hold at all under any artist was never bought, and no
improvement to matching will conjure it.
The distinction matters beyond presentation. That figure is the gate on the
cull, and a gate computed from a number that overstates the failure rate blocks
work that is actually safe to do.
Two normalisation faults, both found by running a real coverage report's
unmatched list back through the normaliser. Between them they account for five
of the fifteen worst misses by play count.
The guest-credit pattern required whitespace immediately before the word, so it
caught "Yellowcard feat. Tay Jardine" but missed "Self vs Self (feat. In
Flames)" -- and the bracketed form is the more common of the two. An opening
bracket is now allowed in that position.
The trailing-version pattern matched the suffix as a run of non-hyphens, which
cannot cross a hyphen inside the suffix itself: "Gold Dust - Shy FX Re-Edit" and
"Back To Your Roots - Friction & K-Tee Remix" both survived untouched. Matched
lazily instead.
Four version words are added for how drum and bass marks its variants: vip,
bootleg, rework, extended. They only apply inside a bracket or after a trailing
dash, so the exposure is small, and ordinary titles carrying those words --
Editors, Mixed Emotions, Radio Ga Ga, Live and Let Die -- are pinned as tests
against exactly that.
The report also gains the figures that explain why the MBID tier contributes so
little. Two thirds of scrobbles carry a recording id, and only a twentieth of
them join on one: MusicBrainz holds a separate recording per release, and the
two sides rarely choose the same one. Counting the pairs that carried an id and
matched on name anyway measures that disagreement directly, and settles that the
weakness is not a bug in the join.
Indexing makes two requests per artist, and more when the album fallback fires.
urllib opens a new TCP connection and performs a new DNS lookup for every one of
them, so a large library becomes thousands of lookups inside a few minutes. That
is enough to exhaust a container's resolver, and the result is
"[Errno -3] Try again" on every artist at once -- a failure caused entirely by
how the requests were made rather than by anything wrong with Lidarr.
Add a transport that keeps one connection open per host, so the name is resolved
once and the socket is reused. It retries once on a connection the server has
already closed, since a stale keep-alive announces itself only on use.
Retries were previously declined on the grounds that Lidarr is on the same LAN.
That is not a safe assumption -- it may sit behind a public hostname and a
reverse proxy -- and a transient failure currently costs an artist their entire
entry for that pass. Transient failures are now retried with a backoff. HTTP 500
is deliberately excluded: it is an exception inside Lidarr's serialisation, not
a busy server, and three attempts only delay finding that out.
The same distinction gates the album probe added alongside this. Naming the
offending album costs one request per album of that artist, which is worth it
for a deterministic fault and actively harmful during a network-wide one, where
every artist fails and probing each of them multiplies the load responsible.
The keep-alive transport is tested against a real local HTTP server rather than
a fake, because connection reuse and status mapping are exactly the properties a
fake would assume rather than demonstrate.
Fetching every album in one unfiltered request avoided Lidarr's unguarded
per-artist path, but not the exception underneath it. Every album endpoint maps
through AlbumResource.ToResource, which selects the release with
SingleOrDefault(x => x.Monitored). An album with two monitored releases makes
that throw -- "Sequence contains more than one element" -- and the bulk call
loses the entire library to one bad row.
Keep the unfiltered call as the first attempt, since it is a single request and
is still the only path that skips albums whose artist metadata is missing. When
it fails, fall back to one request per artist. That cannot dodge the exception
either, but it confines the loss to whichever artist owns the offending album
and names them, which is the only practical way to find it in a large library.
Album failures are counted separately from artist failures because they do not
mean the same thing. Tracks come from a different endpoint with a different
mapper, so an artist whose albums cannot be fetched still gets indexed and still
matches; it is the cull that cannot run. The report distinguishes the two rather
than lumping them into one warning that overstates the damage.
Indexing fetched albums one artist at a time, and `GET /api/v1/album?artistId=`
is Lidarr's unguarded path. It maps straight from the album service with no
hydration: the mapper then dereferences model.Images and model.SecondaryTypes
without a null check, follows model.Artist?.Value where only the first link is
guarded, and selects the monitored release with SingleOrDefault, which throws
outright when an album has two of them. Any of those is a 500 that aborts the
whole index.
The unfiltered `GET /api/v1/album` builds its own artist and release lookups and
skips an album whose metadata is missing rather than dereferencing it. Use that
instead, once, and group by artistId locally. It is the defensive path and it
costs N fewer requests.
Tracks and files have no unfiltered endpoint -- Lidarr rejects a call with no
filter at all -- so those stay per artist. A failure on one artist now skips
that artist rather than ending the run, but the count is recorded in the store
and the coverage report leads with it: a missing artist makes their played music
look unplayed, which is precisely the error that costs music later, so an
incomplete index must not be culled against.
Errors now carry the request URL and whatever the server put in the body. The
original report of this failure was "album: HTTP 500", which points at the URL
and the credentials -- neither of which was at fault.
Stage two. The scrobble history says what was played by name; Lidarr says what
is owned, and where the files are. Neither is useful for curation until the two
are tied together, and the quality of that join is what decides whether the
later cull can be trusted at all.
The index is a wholesale rebuild of every artist, album and track Lidarr holds,
including file paths and the date each file landed -- the latter for the age
floor a cull will need. It is rebuilt rather than reconciled because Lidarr is
the authority and a deletion there has to disappear here, not linger as a
library entry with no file behind it. Every call is a GET; nothing is written
back.
Matching runs at the level of the distinct artist/track pair rather than the
individual play, because a verdict is a property of the name pair and there are
three plays for every one of them. Two tiers: a MusicBrainz recording id, which
Last.fm supplies per scrobble and Lidarr exposes as ForeignRecordingId, gives an
exact join; everything else falls to a normalised name comparison. There is
deliberately no third tier. A near-miss guess is worse than an admitted one,
since the entire purpose of the resulting number is to state how far the
matching can be relied on.
Normalisation folds the ways the two sides habitually disagree: case, accents,
guest credits that Last.fm puts in the artist field, trailing version suffixes,
ampersands, and a leading article. Punctuation needs two opposing rules and both
are load-bearing -- apostrophes are deleted so "Don't" meets "Dont", while every
other mark becomes a space so "AC/DC", "AC-DC" and "AC DC" meet as well. It errs
towards collapsing too much: a false match makes a track look played, a missed
match makes it look abandoned, and only the second one loses music.
The coverage report deliberately does not lead with matched versus unmatched.
Most unmatched listening is music that was never in the library and says nothing
about the matcher. The figure that matters is unmatched listening by an artist
the library does hold: a track that was played, sitting next to a file it should
have matched. The worst fifteen are listed by play count.
The schema gains its tables additively and migrates a version 1 store in place,
because rebuilding a nine-year history costs several thousand API requests.
First stage of a curation tool for the music library that music-mirror
mirrors. Before anything can build playlists or decide what has gone cold,
there has to be a local, queryable record of what is actually played; an API
call per question does not scale to a library-sized analysis.
Ingest is in two halves. A catch-up fetches everything scrobbled since the
newest scrobble held, and a backfill walks the history backwards until it runs
out. Both take their bounds from the database rather than from a saved cursor,
so an interrupted run resumes from what it actually has, and both windows are
bounded at each end so paging cannot shift under the fetch while new scrobbles
arrive mid-run.
Scrobbles carry no identifier, so the primary key is timestamp, artist and
track. Two plays of one track in the same second collapse into a single row:
they are indistinguishable in the data, and a surrogate key would make
re-ingest non-idempotent, which is the worse trade.
Three API behaviours are handled explicitly because each fails silently:
the currently-playing track arrives with no timestamp and would be re-ingested
on every pass; a lone result is returned as a bare object rather than a
one-item list; and MBIDs are empty strings rather than absent when unknown,
which would later look like a usable join key.
Retries cover the rate limit and the transient backend errors with an
exponential backoff. An invalid or suspended key fails immediately.
The report exists to surface one number before the next stage is built: the
share of scrobbles carrying a MusicBrainz recording id. Lidarr exposes the same
identifier per track, so those can be joined exactly and the rest must go
through name matching. That percentage bounds how far the matcher can be
trusted.
No runtime dependencies, and the tests run against a fake transport that
reproduces the service's paging and response shapes, so they need neither
network nor credentials.
lyrathorpe
lyrathorpe