Emma Thorpe
15e5ee5aea
Build and publish container / build (pull_request) Successful in 3m50s
A second set of playlists selecting by genre and mood rather than by play history: eighties synths, high energy rock, screamo, drum and bass, dance, classic rock. The tags come from Last.fm rather than MusicBrainz. MusicBrainz genres arrive free with the Lidarr index, which makes them the obvious choice and the wrong one: they are sparse and formal, and will not tell you a record is screamo or synthwave. Crowd tags will, because people typed them. One request per artist, by MusicBrainz id where Lidarr has one, refreshed every ninety days. An artist Last.fm has never heard of is recorded as fetched with no tags rather than left unmarked, so it is not asked about again on every pass forever. The tag table is keyed on the normalised artist name, not the Lidarr id, so it survives an artist being removed and re-added there. Weights are taken from the response's count where it has one. The documented sample carries only a name and a URL, a live response also carries a 0-100 count, and depending on either alone would be a guess -- so the count is used when present and the documented ordering by popularity stands in when it is not. An artist qualifies for a mood when their weights inside it sum to at least thirty. A single low-weight tag is not a genre, it is somebody's stray opinion. A mood may also restrict release years, which is what separates eighties synth records from everything else a synthpop tag drags in. The built-in set is chosen for this library rather than as a taxonomy, and --vibes replaces it wholesale with a JSON file so a new mood does not need a new release. Names are validated when that file is read: an invalid one would otherwise only surface as a playlist written somewhere unintended.
334 lines
16 KiB
Markdown
334 lines
16 KiB
Markdown
# music-curator
|
|
|
|
Build a local record of what actually gets listened to, from Last.fm.
|
|
|
|
Companion to [music-mirror](https://code.emmathe.dev/lyrathorpe/music-mirror),
|
|
which keeps an MP3 copy of a lossless library for an iPod. This one answers the
|
|
question that mirror cannot: which of it is worth carrying, and which of it has
|
|
not been played in years.
|
|
|
|
**This is stage three.** It ingests the scrobble history, indexes the library
|
|
from Lidarr, matches one to the other, and writes playlists into the mirror —
|
|
by listening history and by mood.
|
|
Nothing is written back to Lidarr — every call there is a `GET`. See "Where this
|
|
is going" below.
|
|
|
|
## What it does today
|
|
|
|
- Pulls the full Last.fm scrobble history into SQLite, then keeps it current.
|
|
- Tracks loved tracks separately, as the protected set for later stages.
|
|
- Indexes every artist, album and track Lidarr knows about, with file paths and
|
|
the date each file landed.
|
|
- Ties the two together and reports how well it managed.
|
|
- Writes M3U playlists into the mirror, from the listening history.
|
|
|
|
## Matching
|
|
|
|
Two tiers, and no third.
|
|
|
|
| Tier | Key | Notes |
|
|
| ------ | --------------------------- | ----------------------------------------- |
|
|
| `mbid` | MusicBrainz recording id | Exact. Last.fm's per-scrobble `mbid` against Lidarr's `ForeignRecordingId` |
|
|
| `name` | Normalised artist and title | Everything the first tier could not carry |
|
|
| `none` | — | Recorded as a miss, never guessed at |
|
|
|
|
The name tier does almost all of the work. A recording MBID is exact when it
|
|
lands, but MusicBrainz holds a separate recording per release, and Last.fm and
|
|
Lidarr rarely pick the same one: on a real library, two thirds of scrobbles
|
|
carry a recording id and barely a twentieth of them join on it. The report
|
|
counts how many carried an id and matched on name anyway, which is the measure
|
|
of that disagreement.
|
|
|
|
So the normalisation is the load-bearing part, because the two sides disagree in
|
|
predictable ways. It folds case and accents, drops guest credits (`Yellowcard
|
|
feat. Tay Jardine` against a tag of `Yellowcard`), strips a trailing
|
|
version suffix (`(Remastered 2011)`, `- Live`), expands `&`, and removes a
|
|
leading `The`. Punctuation gets two different rules that pull against each
|
|
other and are both required: apostrophes are **deleted**, so `Don't` meets
|
|
`Dont`, while every other mark becomes a **space**, so `AC/DC`, `AC-DC` and
|
|
`AC DC` all meet as well.
|
|
|
|
It leans towards collapsing too much. A false match makes something look
|
|
played; a missed match makes something look abandoned. Only one of those
|
|
deletes music.
|
|
|
|
### Indexing quirks
|
|
|
|
Albums are fetched from the **unfiltered** `GET /api/v1/album` first: one
|
|
request, and the only path that skips albums whose artist metadata is missing
|
|
rather than dereferencing it.
|
|
|
|
That is not enough on its own. Every album endpoint maps through a resource
|
|
that picks the release with `SingleOrDefault(x => x.Monitored)`, which throws
|
|
for an album with **two monitored releases** and takes the whole response with
|
|
it:
|
|
|
|
```
|
|
HTTP 500: Sequence contains more than one element
|
|
```
|
|
|
|
When the bulk call dies that way, the indexer falls back to one request per
|
|
artist. It cannot avoid the exception, but it confines it to whichever artist
|
|
owns the offending album and names them in the log — which is the only
|
|
practical way to find it in a large library. Open that artist in Lidarr and
|
|
check the Releases tab of each album: exactly one release may be monitored.
|
|
|
|
Losing an artist's albums does not cost their tracks, which come from a
|
|
different endpoint with a different mapper, so matching is unaffected. A cull
|
|
would not be, and the report says so.
|
|
|
|
### Talking to Lidarr
|
|
|
|
Indexing is two requests per artist, and more when the album fallback fires. On
|
|
a large library that is thousands of requests in a few minutes. `urllib` opens a
|
|
new TCP connection and performs a new DNS lookup for every one of them, which is
|
|
enough to exhaust a container's resolver and produce `[Errno -3] Try again` on
|
|
everything at once. The client therefore holds one connection open per host and
|
|
resolves once.
|
|
|
|
Transient failures — a dropped connection, a resolver hiccup, `429`, `502`,
|
|
`503`, `504` — are retried with a backoff. An HTTP `500` is not: it is an
|
|
unhandled exception inside Lidarr's own serialisation and will be raised again
|
|
identically. That distinction also decides whether a failure is worth
|
|
investigating; a library-wide outage is not probed artist by artist, because
|
|
doing so multiplies the load that caused it.
|
|
|
|
A local address is preferable to a public hostname here. It removes DNS, the
|
|
reverse proxy and its timeouts from a path that needs none of them.
|
|
|
|
Tracks and files have no unfiltered endpoint — Lidarr rejects a call with no
|
|
filter — so they stay per artist. If one artist cannot be served, that artist is
|
|
skipped and the run continues, but the count is recorded and the coverage report
|
|
says so loudly. A missing artist makes their played music look cold, so an
|
|
incomplete index must never be culled against.
|
|
|
|
### Reading the coverage report
|
|
|
|
Matched against unmatched is the wrong comparison — most unmatched listening is
|
|
music that was never in the library, which says nothing at all about the
|
|
matcher. The line to watch is:
|
|
|
|
```
|
|
unmatched by an artist the library holds: N pairs, M plays
|
|
```
|
|
|
|
That is a track that was played by an artist the library holds. It is then
|
|
split three ways, because owning an artist is a weak proxy for owning a track
|
|
and a shared title is a weak proxy for a shared song:
|
|
|
|
- **the library's own title credits the scrobbled artist** — `Voodoo People
|
|
(Pendulum Remix)` against a play credited to Pendulum. Same song, filed
|
|
under the original artist. These are the genuine misses.
|
|
- **the same title under an unrelated artist** — a collision, not a miss.
|
|
Across fifty thousand tracks these are constant: `Everyday` is Rusko and
|
|
also Def Leppard, `Kaleidoscope` is Delta Heavy and also Chappell Roan.
|
|
Matching on title alone would be far worse than missing them, which is why
|
|
there is no such tier.
|
|
- **the title is nowhere in the library** — never bought.
|
|
|
|
Only the first is worth chasing. Counting all three as matcher failures
|
|
overstates the problem and would over-block the cull.
|
|
|
|
## Playlists
|
|
|
|
Written into `<mirror>/_playlists/` as extended M3U, rebuilt every pass. Six
|
|
rules, capped at `--playlist-limit` tracks each:
|
|
|
|
| Playlist | Rule |
|
|
| -------------------- | --------------------------------------------------------- |
|
|
| `heavy-rotation` | Most played over the last twelve months |
|
|
| `all-time` | Most played ever |
|
|
| `neglected` | Played heavily once, silent for twelve months |
|
|
| `deep-cuts` | Never played, from albums whose other tracks you play constantly |
|
|
| `unheard-favourites` | Never played, by the artists you play most |
|
|
| `unheard` | Never played, anywhere in the library |
|
|
|
|
Ninety days was the obvious window for "recent" and is the wrong one: on a real
|
|
history it holds a few hundred plays spread thinly across a twenty-thousand
|
|
track rotation, so nothing ranks meaningfully. Twelve months does.
|
|
|
|
The two `unheard` playlists rotate **weekly**, not per pass. A pass runs every
|
|
few hours, and a playlist that reorders itself each time is one that has to be
|
|
re-imported each time — the Music app imports a snapshot of a file, it does not
|
|
track it.
|
|
|
|
### Moods
|
|
|
|
A second set of playlists selects by **Last.fm's crowd tags** rather than by
|
|
listening history. MusicBrainz genres arrive free with the Lidarr index and are
|
|
no use for this: they are sparse and formal, and will not tell you a record is
|
|
screamo or synthwave. People typing tags will.
|
|
|
|
Tags are fetched once per artist — one `artist.getTopTags` call each, by
|
|
MusicBrainz id where Lidarr has one — and refreshed every ninety days. An
|
|
artist Last.fm has never heard of is recorded as fetched with no tags, so it is
|
|
not asked about again on every pass. `--tag-limit` spreads the first sweep over
|
|
several passes.
|
|
|
|
The built-in moods are chosen for this library rather than as a general
|
|
taxonomy:
|
|
|
|
| Mood | Selected on |
|
|
| ------------------ | ------------------------------------------------- |
|
|
| `80s-synths` | synthpop, new wave, synthwave — released 1975-1992 |
|
|
| `high-energy-rock` | hard rock, punk, pop punk, alternative |
|
|
| `screamo` | screamo, post-hardcore, metalcore, emo |
|
|
| `drum-and-bass` | drum and bass, liquid funk, neurofunk, jungle |
|
|
| `dance` | house, big room, hardstyle, trance, dubstep |
|
|
| `classic-rock` | classic rock, prog, psychedelic, blues rock |
|
|
|
|
An artist qualifies when their tag weights inside a mood sum to at least 30 out
|
|
of Last.fm's 0-100 scale. One low-weight tag is not a genre, it is somebody's
|
|
stray opinion.
|
|
|
|
`years` filters on the album's release date, which is what separates eighties
|
|
synth records from everything else a synthpop tag drags in.
|
|
|
|
`--vibes` replaces the whole set with a JSON file of the same shape, so a new
|
|
mood does not need a new release:
|
|
|
|
```json
|
|
[{ "name": "shoegaze", "tags": ["shoegaze", "dream pop"], "min_score": 40 }]
|
|
```
|
|
|
|
Names are validated when the file is read, not when the file is written. A bad
|
|
one would otherwise surface as a playlist created somewhere unintended.
|
|
|
|
### Paths
|
|
|
|
Lidarr knows where the lossless source is; the playlists have to point at the
|
|
MP3s music-mirror made from it. The mapping strips a library root from Lidarr's
|
|
track paths and re-roots them under the mirror, with the suffix changed.
|
|
|
|
`--library-root` is derived from the common parent of the indexed artist folders
|
|
when unset, so it agrees with Lidarr by construction rather than by being kept
|
|
in step by hand. Override it if that guess is wrong.
|
|
|
|
Entries are written **relative to the playlist file**, so one playlist works
|
|
from the NAS, from a Mac over SMB, and from Linux, without rewriting.
|
|
|
|
A track is only listed once its mirror file has been confirmed to exist. Lidarr
|
|
holding the FLAC says nothing about whether the MP3 has been encoded yet. If a
|
|
large number are missing, the run says so — that is what a wrong `--library-root`
|
|
or `--mirror` looks like, since the paths then map to nothing at all.
|
|
|
|
`_playlists/` survives music-mirror's prune: it only deletes `*.mp3`, and its
|
|
empty-directory sweep skips a directory holding M3Us.
|
|
|
|
## How the ingest works
|
|
|
|
Two halves, both taking their bounds from the database rather than from a saved
|
|
cursor, so an interrupted run resumes from what it actually has.
|
|
|
|
| Half | Window | Purpose |
|
|
| --------- | ---------------------- | ---------------------------------------- |
|
|
| Catch-up | `newest held` → `now` | New scrobbles since the last pass |
|
|
| Backfill | start → `oldest held` | Walks towards the beginning of the history |
|
|
|
|
The backfill asks repeatedly for the newest page of everything at or before a
|
|
cursor, and moves the cursor to the oldest scrobble that came back. When a whole
|
|
page shares a single second the cursor cannot move without stepping over the
|
|
rest of that second, so it takes the next page of the same window instead.
|
|
|
|
Both windows are bounded at both ends, so paging cannot shift under the fetch
|
|
while new scrobbles arrive mid-run.
|
|
|
|
Three details of the API that are easy to get wrong, all handled:
|
|
|
|
- The **currently-playing** track is prepended to the first page with no
|
|
timestamp at all. Stored once, it would come back on every pass forever.
|
|
- A **lone result** is returned as a bare object, not a one-item list.
|
|
- **MBIDs are empty strings** rather than absent when unknown, and an empty
|
|
string looks like a usable join key right up until it silently matches
|
|
everything.
|
|
|
|
Scrobbles have no identifier, so the primary key is timestamp plus artist plus
|
|
track. Two plays of the same track in the same second collapse into one; they
|
|
are genuinely indistinguishable, and a surrogate key would make re-ingest
|
|
non-idempotent, which is a far worse trade.
|
|
|
|
Retries cover error 29 (rate limit) and the backend failures, 8, 11 and 16,
|
|
with an exponential backoff. An invalid or suspended key fails immediately
|
|
rather than retrying four more times to reach the same conclusion.
|
|
|
|
## Usage
|
|
|
|
```sh
|
|
music-curator # one pass, then report
|
|
music-curator --interval 6h # keep running
|
|
music-curator --report-only # report on the store, fetch nothing
|
|
```
|
|
|
|
| Option | Environment variable | Default | Meaning |
|
|
| ------------------ | ----------------------------- | ------------------ | ------------------------------------------- |
|
|
| `--user` | `MUSIC_CURATOR_LASTFM_USER` | — | Last.fm username to read |
|
|
| `--api-key` | `MUSIC_CURATOR_LASTFM_API_KEY` | — | Last.fm API key |
|
|
| `--db` | `MUSIC_CURATOR_DB` | `/data/curator.db` | Path to the SQLite store |
|
|
| `--interval` | `MUSIC_CURATOR_INTERVAL` | unset | Repeat forever, e.g. `45m`, `6h`, `1d` |
|
|
| `--request-delay` | `MUSIC_CURATOR_REQUEST_DELAY` | `0.25` | Seconds between API requests |
|
|
| `--backfill-limit` | `MUSIC_CURATOR_BACKFILL_LIMIT` | `0` | Cap backfill requests per pass; 0 for no cap |
|
|
| `--lidarr-url` | `MUSIC_CURATOR_LIDARR_URL` | unset | Lidarr base URL, e.g. `http://lidarr:8686` |
|
|
| `--lidarr-api-key` | `MUSIC_CURATOR_LIDARR_API_KEY` | unset | Lidarr API key |
|
|
| `--mirror` | `MUSIC_CURATOR_MIRROR` | unset | Root of the MP3 mirror; playlists go here |
|
|
| `--library-root` | `MUSIC_CURATOR_LIBRARY_ROOT` | derived | Prefix to strip from Lidarr's paths |
|
|
| `--playlist-limit` | `MUSIC_CURATOR_PLAYLIST_LIMIT` | `100` | Most tracks in any one playlist |
|
|
| `--vibes` | `MUSIC_CURATOR_VIBES` | built-in | JSON file of mood definitions |
|
|
| `--tag-limit` | `MUSIC_CURATOR_TAG_LIMIT` | `0` | Cap artist tag lookups per pass |
|
|
| `--skip-index` | — | off | Match against the index already held |
|
|
| `--report-only` | — | off | Report without fetching |
|
|
|
|
A [Last.fm API key](https://www.last.fm/api/account/create) is all that is
|
|
needed. None of the endpoints used here authenticate a user, so there is no
|
|
shared secret, no session key and no signing.
|
|
|
|
The first pass over a long history is thousands of requests at 200 scrobbles
|
|
each. `--backfill-limit` spreads that over several passes if you would rather
|
|
not do it in one.
|
|
|
|
## Running it on TrueNAS Scale
|
|
|
|
`compose.yaml` is a Custom App definition. Adjust the host path and the `user:`
|
|
to match your pool, set the API key through the TrueNAS UI rather than in the
|
|
file, then add it as a custom app. The image is published to this Gitea's
|
|
registry on every release:
|
|
|
|
```
|
|
code.emmathe.dev/lyrathorpe/music-curator:latest
|
|
```
|
|
|
|
Give the store its own dataset. It is derived data and can be rebuilt from
|
|
Last.fm, but rebuilding means downloading the whole history again.
|
|
|
|
## Tests
|
|
|
|
```sh
|
|
docker build --target test . # what CI runs
|
|
pytest # needs pytest on PATH
|
|
```
|
|
|
|
The suite runs against fake transports for both services. The Last.fm one
|
|
reproduces its paging, its `from`/`to` semantics and its awkward response
|
|
shapes; the Lidarr one serves a canned library split across the same four
|
|
endpoints the indexer calls, so the stitching is exercised rather than
|
|
stubbed. No network, no credentials, no rate limit. On a Nix machine:
|
|
|
|
```sh
|
|
nix shell nixpkgs#python3Packages.pytest -c pytest
|
|
```
|
|
|
|
## Where this is going
|
|
|
|
| Stage | Status |
|
|
| ------------------------------------------------ | ------------ |
|
|
| Last.fm ingest and store | done |
|
|
| Lidarr index and the scrobble-to-track matcher | done |
|
|
| M3U playlists from the listening history | done |
|
|
| Genre and mood playlists from Last.fm tags | done |
|
|
| Cold-music report, unmonitoring what is not played | last |
|
|
|
|
The cull will unmonitor cold albums in Lidarr and tag their artists. It will
|
|
never delete files: Lidarr's `AlbumResource` has no tags at all, so tagging
|
|
only works per artist, and an artist is only tagged when every one of their
|
|
albums qualifies. It will also refuse to run at all if the matcher's coverage
|
|
is poor, because an unmatched track is not an unplayed track.
|