music-curator/README.md

music-curator

Build a local record of what actually gets listened to, from Last.fm.

Companion to music-mirror, which keeps an MP3 copy of a lossless library for an iPod. This one answers the question that mirror cannot: which of it is worth carrying, and which of it has not been played in years.

This is stage three. It ingests the scrobble history, indexes the library from Lidarr, matches one to the other, and writes playlists into the mirror — by listening history and by mood. Nothing is written back to Lidarr — every call there is a GET. See "Where this is going" below.

What it does today

• Pulls the full Last.fm scrobble history into SQLite, then keeps it current.
• Tracks loved tracks separately, as the protected set for later stages.
• Indexes every artist, album and track Lidarr knows about, with file paths and the date each file landed.
• Ties the two together and reports how well it managed.
• Writes M3U playlists into the mirror, from the listening history.

Matching

Two tiers, and no third.

Tier	Key	Notes
mbid	MusicBrainz recording id	Exact. Last.fm's per-scrobble mbid against Lidarr's ForeignRecordingId
name	Normalised artist and title	Everything the first tier could not carry
none	—	Recorded as a miss, never guessed at

The name tier does almost all of the work. A recording MBID is exact when it lands, but MusicBrainz holds a separate recording per release, and Last.fm and Lidarr rarely pick the same one: on a real library, two thirds of scrobbles carry a recording id and barely a twentieth of them join on it. The report counts how many carried an id and matched on name anyway, which is the measure of that disagreement.

So the normalisation is the load-bearing part, because the two sides disagree in predictable ways. It folds case and accents, drops guest credits (Yellowcard feat. Tay Jardine against a tag of Yellowcard), strips a trailing version suffix ((Remastered 2011), - Live), expands &, and removes a leading The. Punctuation gets two different rules that pull against each other and are both required: apostrophes are deleted, so Don't meets Dont, while every other mark becomes a space, so AC/DC, AC-DC and AC DC all meet as well.

It leans towards collapsing too much. A false match makes something look played; a missed match makes something look abandoned. Only one of those deletes music.

Indexing quirks

Albums are fetched from the unfiltered GET /api/v1/album first: one request, and the only path that skips albums whose artist metadata is missing rather than dereferencing it.

That is not enough on its own. Every album endpoint maps through a resource that picks the release with SingleOrDefault(x => x.Monitored), which throws for an album with two monitored releases and takes the whole response with it:

HTTP 500: Sequence contains more than one element

When the bulk call dies that way, the indexer falls back to one request per artist. It cannot avoid the exception, but it confines it to whichever artist owns the offending album and names them in the log — which is the only practical way to find it in a large library. Open that artist in Lidarr and check the Releases tab of each album: exactly one release may be monitored.

Losing an artist's albums does not cost their tracks, which come from a different endpoint with a different mapper, so matching is unaffected. A cull would not be, and the report says so.

Talking to Lidarr

Indexing is two requests per artist, and more when the album fallback fires. On a large library that is thousands of requests in a few minutes. urllib opens a new TCP connection and performs a new DNS lookup for every one of them, which is enough to exhaust a container's resolver and produce [Errno -3] Try again on everything at once. The client therefore holds one connection open per host and resolves once.

Transient failures — a dropped connection, a resolver hiccup, 429, 502, 503, 504 — are retried with a backoff. An HTTP 500 is not: it is an unhandled exception inside Lidarr's own serialisation and will be raised again identically. That distinction also decides whether a failure is worth investigating; a library-wide outage is not probed artist by artist, because doing so multiplies the load that caused it.

A local address is preferable to a public hostname here. It removes DNS, the reverse proxy and its timeouts from a path that needs none of them.

Tracks and files have no unfiltered endpoint — Lidarr rejects a call with no filter — so they stay per artist. If one artist cannot be served, that artist is skipped and the run continues, but the count is recorded and the coverage report says so loudly. A missing artist makes their played music look cold, so an incomplete index must never be culled against.

Reading the coverage report

Matched against unmatched is the wrong comparison — most unmatched listening is music that was never in the library, which says nothing at all about the matcher. The line to watch is:

unmatched by an artist the library holds: N pairs, M plays

That is a track that was played by an artist the library holds. It is then split three ways, because owning an artist is a weak proxy for owning a track and a shared title is a weak proxy for a shared song:

• the library's own title credits the scrobbled artist — Voodoo People (Pendulum Remix) against a play credited to Pendulum. Same song, filed under the original artist. These are the genuine misses.
• the same title under an unrelated artist — a collision, not a miss. Across fifty thousand tracks these are constant: Everyday is Rusko and also Def Leppard, Kaleidoscope is Delta Heavy and also Chappell Roan. Matching on title alone would be far worse than missing them, which is why there is no such tier.
• the title is nowhere in the library — never bought.

Only the first is worth chasing. Counting all three as matcher failures overstates the problem and would over-block the cull.

Playlists

Written into <mirror>/_playlists/ as extended M3U, rebuilt every pass. Six rules, capped at --playlist-limit tracks each:

Playlist	Rule
heavy-rotation	Most played over the last twelve months
all-time	Most played ever
neglected	Played heavily once, silent for twelve months
deep-cuts	Never played, from albums whose other tracks you play constantly
unheard-favourites	Never played, by the artists you play most
unheard	Never played, anywhere in the library

Ninety days was the obvious window for "recent" and is the wrong one: on a real history it holds a few hundred plays spread thinly across a twenty-thousand track rotation, so nothing ranks meaningfully. Twelve months does.

The two unheard playlists rotate weekly, not per pass. A pass runs every few hours, and a playlist that reorders itself each time is one that has to be re-imported each time — the Music app imports a snapshot of a file, it does not track it.

Moods

A second set of playlists selects by Last.fm's crowd tags rather than by listening history. MusicBrainz genres arrive free with the Lidarr index and are no use for this: they are sparse and formal, and will not tell you a record is screamo or synthwave. People typing tags will.

Tags are fetched once per artist — one artist.getTopTags call each — and refreshed every ninety days. --tag-limit spreads the first sweep over several passes.

Artists are looked up by name, not by MusicBrainz id, despite Lidarr having an id for every one of them. Last.fm's mbid index is stale and partial: it answers "the artist you supplied could not be found" for Devo, Escape the Fate, Blasterjaxx and a few hundred others whose pages plainly exist and carry exactly the tags wanted. Its name index is the one its own site runs on. The id is kept only as a fallback, for a name Lidarr spells differently.

An artist neither key resolves is recorded as fetched with no tags, so the next pass does not spend a request on it again. A genuine failure — a rate limit, a bad key — is not recorded, so that one is retried.

The built-in moods are built from the tag distribution of this library, measured, rather than from a general taxonomy:

Mood	Selected on
drum-and-bass	drum and bass and its six spellings, liquid funk, neurofunk, jungle, techstep, hospital records
bass	dubstep, brostep, grime, trip-hop, big beat
dance	house and its variants, trance, techno, electro, rave
pop-punk	pop punk, punk, emo, emocore, easycore, power pop
screamo	screamo, post-hardcore, metalcore, melodic hardcore, trancecore
heavy-metal	heavy metal, thrash, speed, power, death, prog, NWOBHM
hair-metal	hair metal, glam metal, glam rock, arena rock, AOR — 1975-1994
nu-metal	nu metal, alternative metal, rapcore, industrial
classic-rock	classic rock, prog, psychedelic, blues rock, 70s, 60s
80s-synths	80s, new wave, synth pop, electropop, post-punk — 1975-1992, rock excluded
indie	indie, indie rock, indie pop, britpop, singer-songwriter

Measuring first mattered. synthwave, edm, big room and hardstyle are plausible tags that carry nothing at all here, while techstep, easycore and hospital records carry real weight. Guessing produces the first list.

Three kinds of tag are never used, and there is a test enforcing it:

• Nationality — american alone spans 241 artists. A passport is not a sound.
• rock and electronic — 340 and 275 artists, most of the library. A mood that matches everything is not a mood.
• Artist names — Last.fm's most popular tag for an artist is frequently their own name. green day, paramore and queen are single-artist playlists waiting to happen.

Exclusions

A mood may also list exclude. An excluded tag drops the artist outright rather than docking their score, and it exists because 80s-synths cannot be written any other way.

80s is the eleventh most-played tag here, and it sits on Def Leppard and Bon Jovi exactly as heavily as on Eurythmics. Weighting cannot separate them, because the tag it would weight is the one they share. What does separate them is that the stadium rock also carries hard rock and hair metal, and the synth acts do not.

Checked against live Last.fm pages, since the tag census only sees artists already in the library:

Artist	Tags
Eurythmics	80s, new wave, pop, female vocalists, synth pop
Frankie Goes to Hollywood	80s, new wave, pop, british, dance
Depeche Mode	electronic, synthpop, new wave, 80s, synth pop
Duran Duran	new wave, 80s, pop, synth pop, rock

Four of Eurythmics' five tags are ones no mood may use. Three of the four artists spell it synth pop with a space; only one spells it synthpop. Guessing one spelling would have missed most of the canon.

An artist qualifies when their tag weights inside a mood sum to at least 30 out of Last.fm's 0-100 scale. One low-weight tag is not a genre, it is somebody's stray opinion.

years filters on the album's release date, which is what separates eighties synth records from everything else a synthpop tag drags in.

--vibes replaces the whole set with a JSON file of the same shape, so a new mood does not need a new release:

[{ "name": "shoegaze", "tags": ["shoegaze", "dream pop"], "min_score": 40 }]

Names are validated when the file is read, not when the file is written. A bad one would otherwise surface as a playlist created somewhere unintended.

Paths

Lidarr knows where the lossless source is; the playlists have to point at the MP3s music-mirror made from it. The mapping strips a library root from Lidarr's track paths and re-roots them under the mirror, with the suffix changed.

--library-root is derived from the common parent of the indexed artist folders when unset, so it agrees with Lidarr by construction rather than by being kept in step by hand. Override it if that guess is wrong.

Entries are written relative to the playlist file, so one playlist works from the NAS, from a Mac over SMB, and from Linux, without rewriting.

Each playlist is given the owner and group of the mirror it is written into. The image runs as root by default so that a bind mount of any ownership stays writable, and the cost of that is output owned by root — which the account serving the share cannot read, group bit or no group bit, because the group is also root. Copying the mirror's own ownership avoids having to be told what it should be, and does nothing when the two already agree.

A track is only listed once its mirror file has been confirmed to exist. Lidarr holding the FLAC says nothing about whether the MP3 has been encoded yet. If a large number are missing, the run says so — that is what a wrong --library-root or --mirror looks like, since the paths then map to nothing at all.

_playlists/ survives music-mirror's prune: it only deletes *.mp3, and its empty-directory sweep skips a directory holding M3Us.

How the ingest works

Two halves, both taking their bounds from the database rather than from a saved cursor, so an interrupted run resumes from what it actually has.

Half	Window	Purpose
Catch-up	newest held → now	New scrobbles since the last pass
Backfill	start → oldest held	Walks towards the beginning of the history

The backfill asks repeatedly for the newest page of everything at or before a cursor, and moves the cursor to the oldest scrobble that came back. When a whole page shares a single second the cursor cannot move without stepping over the rest of that second, so it takes the next page of the same window instead.

Both windows are bounded at both ends, so paging cannot shift under the fetch while new scrobbles arrive mid-run.

Three details of the API that are easy to get wrong, all handled:

• The currently-playing track is prepended to the first page with no timestamp at all. Stored once, it would come back on every pass forever.
• A lone result is returned as a bare object, not a one-item list.
• MBIDs are empty strings rather than absent when unknown, and an empty string looks like a usable join key right up until it silently matches everything.

Scrobbles have no identifier, so the primary key is timestamp plus artist plus track. Two plays of the same track in the same second collapse into one; they are genuinely indistinguishable, and a surrogate key would make re-ingest non-idempotent, which is a far worse trade.

Retries cover error 29 (rate limit) and the backend failures, 8, 11 and 16, with an exponential backoff. An invalid or suspended key fails immediately rather than retrying four more times to reach the same conclusion.

Usage

music-curator                 # one pass, then report
music-curator --interval 6h   # keep running
music-curator --report-only   # report on the store, fetch nothing

Option	Environment variable	Default	Meaning
--user	MUSIC_CURATOR_LASTFM_USER	—	Last.fm username to read
--api-key	MUSIC_CURATOR_LASTFM_API_KEY	—	Last.fm API key
--db	MUSIC_CURATOR_DB	/data/curator.db	Path to the SQLite store
--interval	MUSIC_CURATOR_INTERVAL	unset	Repeat forever, e.g. 45m, 6h, 1d
--request-delay	MUSIC_CURATOR_REQUEST_DELAY	0.25	Seconds between API requests
--backfill-limit	MUSIC_CURATOR_BACKFILL_LIMIT	0	Cap backfill requests per pass; 0 for no cap
--lidarr-url	MUSIC_CURATOR_LIDARR_URL	unset	Lidarr base URL, e.g. http://lidarr:8686
--lidarr-api-key	MUSIC_CURATOR_LIDARR_API_KEY	unset	Lidarr API key
--mirror	MUSIC_CURATOR_MIRROR	unset	Root of the MP3 mirror; playlists go here
--library-root	MUSIC_CURATOR_LIBRARY_ROOT	derived	Prefix to strip from Lidarr's paths
--playlist-limit	MUSIC_CURATOR_PLAYLIST_LIMIT	100	Most tracks in any one playlist
--vibes	MUSIC_CURATOR_VIBES	built-in	JSON file of mood definitions
--tag-limit	MUSIC_CURATOR_TAG_LIMIT	0	Cap artist tag lookups per pass
--skip-index	—	off	Match against the index already held
--report-only	—	off	Report without fetching

A Last.fm API key is all that is needed. None of the endpoints used here authenticate a user, so there is no shared secret, no session key and no signing.

The first pass over a long history is thousands of requests at 200 scrobbles each. --backfill-limit spreads that over several passes if you would rather not do it in one.

Running it on TrueNAS Scale

compose.yaml is a Custom App definition. Adjust the host path and the user: to match your pool, set the API key through the TrueNAS UI rather than in the file, then add it as a custom app. The image is published to this Gitea's registry on every release:

code.emmathe.dev/lyrathorpe/music-curator:latest

Give the store its own dataset. It is derived data and can be rebuilt from Last.fm, but rebuilding means downloading the whole history again.

Tests

docker build --target test .   # what CI runs
pytest                         # needs pytest on PATH

The suite runs against fake transports for both services. The Last.fm one reproduces its paging, its from/to semantics and its awkward response shapes; the Lidarr one serves a canned library split across the same four endpoints the indexer calls, so the stitching is exercised rather than stubbed. No network, no credentials, no rate limit. On a Nix machine:

nix shell nixpkgs#python3Packages.pytest -c pytest

Tools

Host-side scripts under tools/. Not part of the container image; run them wherever they are needed.

find_missing_tracks.py

Reports tracks in an Apple Music library whose files are no longer on disk — which happens whenever Lidarr renames an artist or album folder and music-mirror prunes the old path.

# Music: File > Library > Export Library... then, with the share mounted:
python3 tools/find_missing_tracks.py Library.xml --root /Volumes/music-mp3

Reading the exported XML rather than asking Music itself is deliberate. A broken track makes AppleScript's location raise instead of returning a value, so a bulk query dies on the first one with -1728 and a per-track loop costs an Apple event apiece.

It checks against a single directory walk, not a test per file. On a 50,000-track library that is ~7,000 directory reads instead of 50,000 stat calls, and over SMB every one of those stats is a network round trip.

Two comparisons that have to be loosened, or most of the library reads as missing:

• Unicode. macOS stores filenames decomposed; the share composes them. Mötley Crüe is two different byte strings depending on which side wrote it. Both sides are normalised to NFC.
• Case. The share is very likely case-insensitive. A file is not missing because someone capitalised it differently.

Pass --root if the library holds anything outside the mirror: the root it otherwise derives is the common parent of every track, which can be /.

Where this is going

Stage	Status
Last.fm ingest and store	done
Lidarr index and the scrobble-to-track matcher	done
M3U playlists from the listening history	done
Genre and mood playlists from Last.fm tags	done
Cold-music report, unmonitoring what is not played	last

The cull will unmonitor cold albums in Lidarr and tag their artists. It will never delete files: Lidarr's AlbumResource has no tags at all, so tagging only works per artist, and an artist is only tagged when every one of their albums qualifies. It will also refuse to run at all if the matcher's coverage is poor, because an unmatched track is not an unplayed track.