chore(release): v0.2.3

Reviewed-on: #4

README.md

@@ -7,22 +7,105 @@ which keeps an MP3 copy of a lossless library for an iPod. This one answers the
 question that mirror cannot: which of it is worth carrying, and which of it has
 not been played in years.
 
-**This is stage one.** It ingests the scrobble history and nothing else. There
-are no playlists yet, and nothing touches Lidarr or the music library. See
-"Where this is going" below.
+**This is stage two.** It ingests the scrobble history, indexes the library from
+Lidarr, and matches one to the other. There are no playlists yet, and it writes
+nothing back — every Lidarr call is a `GET`. See "Where this is going" below.
 
 ## What it does today
 
 - Pulls the full Last.fm scrobble history into SQLite, then keeps it current.
 - Tracks loved tracks separately, as the protected set for later stages.
-- Reports what it holds, including the number that matters most: how many
-  scrobbles carry a MusicBrainz recording id.
+- Indexes every artist, album and track Lidarr knows about, with file paths and
+  the date each file landed.
+- Ties the two together and reports how well it managed.
 
-That last figure decides the next stage. Lidarr exposes a `ForeignRecordingId`
-on every track, which is the same identifier, so scrobbles carrying one can be
-joined to the library exactly. The rest have to go through name matching, which
-is where a curation tool goes wrong and starts recommending the deletion of
-music you love. Measure the join rate before trusting the verdict.
+## Matching
+
+Two tiers, and no third.
+
+| Tier | Key | Notes |
+| ------ | --------------------------- | ----------------------------------------- |
+| `mbid` | MusicBrainz recording id | Exact. Last.fm's per-scrobble `mbid` against Lidarr's `ForeignRecordingId` |
+| `name` | Normalised artist and title | Everything the first tier could not carry |
+| `none` | — | Recorded as a miss, never guessed at |
+
+The normalisation is the load-bearing part, because the two sides disagree in
+predictable ways. It folds case and accents, drops guest credits (`Yellowcard
+feat. Tay Jardine` against a tag of `Yellowcard`), strips a trailing
+version suffix (`(Remastered 2011)`, `- Live`), expands `&`, and removes a
+leading `The`. Punctuation gets two different rules that pull against each
+other and are both required: apostrophes are **deleted**, so `Don't` meets
+`Dont`, while every other mark becomes a **space**, so `AC/DC`, `AC-DC` and
+`AC DC` all meet as well.
+
+It leans towards collapsing too much. A false match makes something look
+played; a missed match makes something look abandoned. Only one of those
+deletes music.
+
+### Indexing quirks
+
+Albums are fetched from the **unfiltered** `GET /api/v1/album` first: one
+request, and the only path that skips albums whose artist metadata is missing
+rather than dereferencing it.
+
+That is not enough on its own. Every album endpoint maps through a resource
+that picks the release with `SingleOrDefault(x => x.Monitored)`, which throws
+for an album with **two monitored releases** and takes the whole response with
+it:
+
+```
+HTTP 500: Sequence contains more than one element
+```
+
+When the bulk call dies that way, the indexer falls back to one request per
+artist. It cannot avoid the exception, but it confines it to whichever artist
+owns the offending album and names them in the log — which is the only
+practical way to find it in a large library. Open that artist in Lidarr and
+check the Releases tab of each album: exactly one release may be monitored.
+
+Losing an artist's albums does not cost their tracks, which come from a
+different endpoint with a different mapper, so matching is unaffected. A cull
+would not be, and the report says so.
+
+### Talking to Lidarr
+
+Indexing is two requests per artist, and more when the album fallback fires. On
+a large library that is thousands of requests in a few minutes. `urllib` opens a
+new TCP connection and performs a new DNS lookup for every one of them, which is
+enough to exhaust a container's resolver and produce `[Errno -3] Try again` on
+everything at once. The client therefore holds one connection open per host and
+resolves once.
+
+Transient failures — a dropped connection, a resolver hiccup, `429`, `502`,
+`503`, `504` — are retried with a backoff. An HTTP `500` is not: it is an
+unhandled exception inside Lidarr's own serialisation and will be raised again
+identically. That distinction also decides whether a failure is worth
+investigating; a library-wide outage is not probed artist by artist, because
+doing so multiplies the load that caused it.
+
+A local address is preferable to a public hostname here. It removes DNS, the
+reverse proxy and its timeouts from a path that needs none of them.
+
+Tracks and files have no unfiltered endpoint — Lidarr rejects a call with no
+filter — so they stay per artist. If one artist cannot be served, that artist is
+skipped and the run continues, but the count is recorded and the coverage report
+says so loudly. A missing artist makes their played music look cold, so an
+incomplete index must never be culled against.
+
+### Reading the coverage report
+
+Matched against unmatched is the wrong comparison — most unmatched listening is
+music that was never in the library, which says nothing at all about the
+matcher. The line to watch is:
+
+```
+unmatched by an artist the library holds: N pairs, M plays
+```
+
+That is a track that was played, sitting beside a file it should have matched.
+Those are the matcher's real misses, and every one is a candidate for being
+wrongly called cold in stage four. The report lists the worst fifteen by play
+count so they can be eyeballed.
 
 ## How the ingest works
 
@@ -76,6 +159,9 @@ music-curator --report-only   # report on the store, fetch nothing
 | `--interval` | `MUSIC_CURATOR_INTERVAL` | unset | Repeat forever, e.g. `45m`, `6h`, `1d` |
 | `--request-delay` | `MUSIC_CURATOR_REQUEST_DELAY` | `0.25` | Seconds between API requests |
 | `--backfill-limit` | `MUSIC_CURATOR_BACKFILL_LIMIT` | `0` | Cap backfill requests per pass; 0 for no cap |
+| `--lidarr-url` | `MUSIC_CURATOR_LIDARR_URL` | unset | Lidarr base URL, e.g. `http://lidarr:8686` |
+| `--lidarr-api-key` | `MUSIC_CURATOR_LIDARR_API_KEY` | unset | Lidarr API key |
+| `--skip-index` | — | off | Match against the index already held |
 | `--report-only` | — | off | Report without fetching |
 
 A [Last.fm API key](https://www.last.fm/api/account/create) is all that is
@@ -107,9 +193,11 @@ docker build --target test .   # what CI runs
 pytest                         # needs pytest on PATH
 ```
 
-The suite runs against a fake transport that reproduces the real service's
-paging, its `from`/`to` semantics and its awkward response shapes. No network,
-no credentials, no rate limit. On a Nix machine:
+The suite runs against fake transports for both services. The Last.fm one
+reproduces its paging, its `from`/`to` semantics and its awkward response
+shapes; the Lidarr one serves a canned library split across the same four
+endpoints the indexer calls, so the stitching is exercised rather than
+stubbed. No network, no credentials, no rate limit. On a Nix machine:
 
 ```sh
 nix shell nixpkgs#python3Packages.pytest -c pytest
@@ -120,8 +208,8 @@ nix shell nixpkgs#python3Packages.pytest -c pytest
 | Stage | Status |
 | ------------------------------------------------ | ------------ |
 | Last.fm ingest and store | done |
-| Lidarr index and the scrobble-to-track matcher | next |
-| M3U playlists written into the mirror | after that |
+| Lidarr index and the scrobble-to-track matcher | done |
+| M3U playlists written into the mirror | next |
 | Cold-music report, unmonitoring what is not played | last |
 
 The cull will unmonitor cold albums in Lidarr and tag their artists. It will

compose.yaml

@@ -17,6 +17,10 @@ services:
       # enough, none of the endpoints used here authenticate a user.
       MUSIC_CURATOR_LASTFM_API_KEY: set-me-in-the-truenas-ui
       MUSIC_CURATOR_DB: /data/curator.db
+      # Lidarr, for indexing the library. Read-only: every call is a GET.
+      # Leave unset to ingest scrobbles and nothing else.
+      MUSIC_CURATOR_LIDARR_URL: http://lidarr:8686
+      MUSIC_CURATOR_LIDARR_API_KEY: set-me-in-the-truenas-ui
       # How long to wait between passes. Each one catches up on new scrobbles
       # and continues the backfill if it has not finished.
       MUSIC_CURATOR_INTERVAL: 6h

music_curator.py

@@ -17,6 +17,8 @@ cursor, so an interrupted run resumes from what it actually has.
 
 import argparse
 import fcntl
+import http.client
+import io
 import json
 import logging
 import os
@@ -25,6 +27,7 @@ import signal
 import sqlite3
 import sys
 import time
+import unicodedata
 import urllib.error
 import urllib.parse
 import urllib.request
@@ -47,6 +50,11 @@ PAGE_SIZE = 200
 RETRYABLE_ERRORS = {8, 11, 16, 29}
 RETRYABLE_STATUS = {429, 500, 502, 503, 504}
 
+# Lidarr's own list, and 500 is deliberately absent. A 500 from Lidarr is an
+# unhandled exception inside its serialisation, not a busy server; it will be
+# raised again identically, and retrying only delays finding that out.
+LIDARR_RETRYABLE_STATUS = {429, 502, 503, 504}
+
 # Last.fm asks for no more than five requests a second averaged over five
 # minutes. A full backfill is thousands of requests, so it is worth staying
 # well inside that rather than discovering error 29 halfway through.
@@ -54,7 +62,12 @@ REQUEST_DELAY_SECONDS = 0.25
 BACKOFF_SECONDS = 2.0
 BACKOFF_CEILING_SECONDS = 60.0
 
-SCHEMA_VERSION = "1"
+SCHEMA_VERSION = "2"
+
+# Versions this build upgrades in place. Everything added since version 1 is a
+# new table, and the schema script only ever creates what is missing, so running
+# it is the whole migration -- an existing history is not re-downloaded.
+MIGRATABLE_FROM = {"1"}
 
 SCHEMA = """
 -- One row per scrobble. The primary key collapses two plays of the same track
@@ -89,13 +102,160 @@ CREATE TABLE IF NOT EXISTS state (
     key   TEXT PRIMARY KEY,
     value TEXT NOT NULL
 );
+
+-- The library as Lidarr sees it. Rebuilt from the API rather than by walking
+-- the files, because Lidarr is the only thing holding the MusicBrainz ids that
+-- make an exact join possible.
+CREATE TABLE IF NOT EXISTS lidarr_artist (
+    id        INTEGER PRIMARY KEY,
+    mbid      TEXT,
+    name      TEXT NOT NULL,
+    norm_name TEXT NOT NULL,
+    path      TEXT,
+    monitored INTEGER NOT NULL DEFAULT 1
+);
+CREATE INDEX IF NOT EXISTS lidarr_artist_norm ON lidarr_artist (norm_name);
+
+CREATE TABLE IF NOT EXISTS lidarr_album (
+    id           INTEGER PRIMARY KEY,
+    artist_id    INTEGER NOT NULL,
+    mbid         TEXT,
+    title        TEXT NOT NULL,
+    monitored    INTEGER NOT NULL DEFAULT 1,
+    release_date TEXT
+);
+CREATE INDEX IF NOT EXISTS lidarr_album_artist ON lidarr_album (artist_id);
+
+CREATE TABLE IF NOT EXISTS lidarr_track (
+    id             INTEGER PRIMARY KEY,
+    artist_id      INTEGER NOT NULL,
+    album_id       INTEGER NOT NULL,
+    recording_mbid TEXT,
+    title          TEXT NOT NULL,
+    norm_artist    TEXT NOT NULL,
+    norm_title     TEXT NOT NULL,
+    has_file       INTEGER NOT NULL DEFAULT 0,
+    path           TEXT,
+    added          INTEGER,
+    duration       INTEGER
+);
+CREATE INDEX IF NOT EXISTS lidarr_track_recording ON lidarr_track (recording_mbid);
+CREATE INDEX IF NOT EXISTS lidarr_track_norm ON lidarr_track (norm_artist, norm_title);
+CREATE INDEX IF NOT EXISTS lidarr_track_album ON lidarr_track (album_id);
+
+-- One row per distinct thing listened to, with the verdict on whether it could
+-- be tied to the library. Matching belongs at this granularity rather than per
+-- play: it is a property of the name pair, and there are three plays for every
+-- pair.
+CREATE TABLE IF NOT EXISTS scrobble_key (
+    artist      TEXT NOT NULL,
+    track       TEXT NOT NULL,
+    norm_artist TEXT NOT NULL,
+    norm_track  TEXT NOT NULL,
+    track_mbid  TEXT,
+    plays       INTEGER NOT NULL,
+    first_uts   INTEGER NOT NULL,
+    last_uts    INTEGER NOT NULL,
+    track_id    INTEGER,
+    method      TEXT NOT NULL DEFAULT 'none',
+    PRIMARY KEY (artist, track)
+);
+CREATE INDEX IF NOT EXISTS scrobble_key_track ON scrobble_key (track_id);
 """
 
 
+# Suffixes that a file tag carries and a scrobble does not, or the reverse.
+# Stripped only from a trailing bracket or after a trailing dash, so a title
+# like "(Don't Fear) The Reaper" keeps its opening parenthetical.
+VERSION_WORDS = (
+    "remaster",
+    "remastered",
+    "live",
+    "mono",
+    "stereo",
+    "version",
+    "edit",
+    "mix",
+    "remix",
+    "deluxe",
+    "bonus",
+    "explicit",
+    "acoustic",
+    "demo",
+    "radio",
+    "single",
+    "anniversary",
+    "reissue",
+    "instrumental",
+)
+_VERSIONS = "|".join(VERSION_WORDS)
+BRACKETED_VERSION = re.compile(
+    rf"\s*[\(\[][^\)\]]*\b(?:{_VERSIONS})\b[^\)\]]*[\)\]]\s*$", re.IGNORECASE
+)
+TRAILING_VERSION = re.compile(rf"\s+-\s+[^-]*\b(?:{_VERSIONS})\b.*$", re.IGNORECASE)
+
+# Last.fm routinely carries the guest credit in the artist field where the file
+# tag holds only the primary artist -- "Yellowcard feat. Tay Jardine" against a
+# tag of "Yellowcard". `with` is deliberately absent: it appears in far too many
+# real titles to cut on sight.
+GUEST_CREDIT = re.compile(r"\s+(?:feat|ft|featuring)\b.*$", re.IGNORECASE)
+
+LEADING_ARTICLE = re.compile(r"^the\s+")
+# Deleted rather than spaced, so "Don't" and "Dont" agree. Every other mark
+# becomes a space instead, so "AC/DC", "AC-DC" and "AC DC" all agree too; the
+# two rules pull opposite ways and both are needed.
+APOSTROPHES = re.compile(r"['‘’ʼ`´]")
+NOT_ALPHANUMERIC = re.compile(r"[^0-9a-z ]+")
+WHITESPACE = re.compile(r"\s+")
+
+
 class LastfmError(Exception):
     """A Last.fm request that failed in a way retrying will not fix."""
 
 
+class LidarrError(Exception):
+    """A Lidarr request that failed.
+
+    `transient` separates "the network or the server had a moment" from "this
+    request will fail identically forever". The distinction matters twice: only
+    the first is worth retrying, and only the second is worth investigating,
+    since probing a library-wide outage artist by artist multiplies the load
+    that caused it.
+    """
+
+    def __init__(self, message, transient=False):
+        super().__init__(message)
+        self.transient = transient
+
+
+def normalise(text):
+    """Return a comparison key for an artist or a track title.
+
+    Scrobbles and file tags disagree in predictable ways, and every one of them
+    has to be flattened or a track that was played reads as one that was not.
+    That is the direction that costs music, so this leans towards collapsing too
+    much rather than too little: a false match makes something look played, a
+    missed match makes something look abandoned.
+    """
+    if not text:
+        return ""
+    text = unicodedata.normalize("NFKD", str(text)).casefold()
+    text = "".join(character for character in text if not unicodedata.combining(character))
+    text = GUEST_CREDIT.sub("", text)
+    # A title can carry more than one, e.g. "Song (Live) (Remastered)".
+    while True:
+        stripped = BRACKETED_VERSION.sub("", text)
+        if stripped == text:
+            break
+        text = stripped
+    text = TRAILING_VERSION.sub("", text)
+    text = text.replace("&", " and ")
+    text = APOSTROPHES.sub("", text)
+    text = NOT_ALPHANUMERIC.sub(" ", text)
+    text = WHITESPACE.sub(" ", text).strip()
+    return LEADING_ARTICLE.sub("", text)
+
+
 @dataclass(frozen=True)
 class Scrobble:
     """One play, as Last.fm recorded it."""
@@ -176,9 +336,10 @@ def parse_scrobble(entry):
     )
 
 
-def http_get(url, timeout=30):
+def http_get(url, timeout=30, headers=None):
     """Fetch a URL and return its body. Replaced in tests."""
-    with urllib.request.urlopen(url, timeout=timeout) as response:  # noqa: S310 - fixed https root
+    request = urllib.request.Request(url, headers=headers or {})
+    with urllib.request.urlopen(request, timeout=timeout) as response:  # noqa: S310
         return response.read().decode("utf-8")
 
 
@@ -267,12 +428,17 @@ class Store:
         # the tail of it.
         self.connection.execute("PRAGMA journal_mode = WAL")
         self.connection.execute("PRAGMA synchronous = NORMAL")
+        # Registered so matching runs in SQL, rather than pulling twenty
+        # thousand name pairs into Python to compare them one at a time.
+        self.connection.create_function("normalise", 1, normalise, deterministic=True)
         self.connection.executescript(SCHEMA)
         self._check_version()
 
     def _check_version(self):
         held = self.get_state("schema_version")
-        if held is None:
+        if held is None or held in MIGRATABLE_FROM:
+            if held is not None:
+                logger.info("store migrated from schema version %s to %s", held, SCHEMA_VERSION)
             self.set_state("schema_version", SCHEMA_VERSION)
         elif held != SCHEMA_VERSION:
             raise RuntimeError(
@@ -330,6 +496,58 @@ class Store:
                 rows,
             )
 
+    def replace_library(self, artists, albums, tracks):
+        """Swap in a freshly indexed library, in one transaction."""
+        with self.connection:
+            for table in ("lidarr_track", "lidarr_album", "lidarr_artist"):
+                self.connection.execute(f"DELETE FROM {table}")
+            self.connection.executemany(
+                "INSERT INTO lidarr_artist (id, mbid, name, norm_name, path, monitored)"
+                " VALUES (?, ?, ?, ?, ?, ?)",
+                artists,
+            )
+            self.connection.executemany(
+                "INSERT INTO lidarr_album"
+                " (id, artist_id, mbid, title, monitored, release_date)"
+                " VALUES (?, ?, ?, ?, ?, ?)",
+                albums,
+            )
+            self.connection.executemany(
+                "INSERT INTO lidarr_track"
+                " (id, artist_id, album_id, recording_mbid, title, norm_artist, norm_title,"
+                "  has_file, path, added, duration)"
+                " VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+                tracks,
+            )
+
+    def rebuild_keys(self):
+        """Collapse the scrobble history into one row per distinct track.
+
+        Rows whose plays have not changed keep their existing verdict until the
+        matcher overwrites it; rows for tracks no longer in the history are
+        dropped, so an edited history does not leave orphans behind.
+        """
+        with self.connection:
+            self.connection.execute(
+                "DELETE FROM scrobble_key WHERE (artist, track) NOT IN"
+                " (SELECT artist, track FROM scrobble)"
+            )
+            self.connection.execute(
+                "INSERT INTO scrobble_key"
+                " (artist, track, norm_artist, norm_track, track_mbid, plays,"
+                "  first_uts, last_uts)"
+                " SELECT artist, track, normalise(artist), normalise(track),"
+                "        MAX(track_mbid), COUNT(*), MIN(uts), MAX(uts)"
+                " FROM scrobble GROUP BY artist, track"
+                " ON CONFLICT (artist, track) DO UPDATE SET"
+                "   norm_artist = excluded.norm_artist,"
+                "   norm_track  = excluded.norm_track,"
+                "   track_mbid  = excluded.track_mbid,"
+                "   plays       = excluded.plays,"
+                "   first_uts   = excluded.first_uts,"
+                "   last_uts    = excluded.last_uts"
+            )
+
     def scalar(self, sql):
         return self.connection.execute(sql).fetchone()[0]
 
@@ -445,6 +663,367 @@ def sync_loved(client, store, user):
     return len(rows)
 
 
+class KeepAlive:
+    """A transport that holds one connection open per host.
+
+    urllib opens a fresh TCP connection -- and performs a fresh DNS lookup --
+    for every request it makes. Indexing a library is two requests per artist,
+    which on a large collection is thousands of lookups inside a few minutes.
+    That is enough to exhaust a container's resolver, and the failure it
+    produces is `[Errno -3] Try again` on everything at once. Resolving once and
+    reusing the socket removes the cause rather than papering over it, and is
+    considerably faster besides.
+    """
+
+    def __init__(self, timeout=60):
+        self.timeout = timeout
+        self._connections = {}
+
+    def __call__(self, url, timeout=None, headers=None):
+        parsed = urllib.parse.urlparse(url)
+        key = (parsed.scheme, parsed.hostname, parsed.port)
+        target = parsed.path + (f"?{parsed.query}" if parsed.query else "")
+        request_headers = {**(headers or {}), "Accept": "application/json"}
+
+        # Two attempts, because a kept-alive connection the server has since
+        # closed fails on use rather than announcing itself. The second attempt
+        # is on a fresh socket.
+        for attempt in (1, 2):
+            connection = self._connections.get(key)
+            if connection is None:
+                connection = self._connect(parsed, timeout or self.timeout)
+                self._connections[key] = connection
+            try:
+                connection.request("GET", target, headers=request_headers)
+                response = connection.getresponse()
+                body = response.read()
+            except (http.client.HTTPException, OSError) as error:
+                self.close(key)
+                if attempt == 2:
+                    raise urllib.error.URLError(error) from error
+                continue
+
+            if response.status >= 300:
+                # Includes redirects: this client does not follow them, and one
+                # here means the URL is pointing somewhere unintended.
+                raise urllib.error.HTTPError(
+                    url, response.status, response.reason, response.headers, io.BytesIO(body)
+                )
+            return body.decode("utf-8")
+        raise urllib.error.URLError("unreachable")
+
+    @staticmethod
+    def _connect(parsed, timeout):
+        if parsed.scheme == "https":
+            return http.client.HTTPSConnection(parsed.hostname, parsed.port, timeout=timeout)
+        return http.client.HTTPConnection(parsed.hostname, parsed.port, timeout=timeout)
+
+    def close(self, key=None):
+        for handle in [self._connections.pop(key, None)] if key else self._connections.values():
+            if handle is not None:
+                handle.close()
+        if key is None:
+            self._connections.clear()
+
+
+class Lidarr:
+    """Minimal read-only Lidarr client.
+
+    Retries only what is worth retrying. A dropped connection or a resolver
+    hiccup is transient; an HTTP 500 out of Lidarr is an exception in its own
+    serialisation and will be thrown again identically, so spending three
+    attempts on it only slows down finding out.
+    """
+
+    def __init__(self, url, api_key, timeout=60, attempts=3, backoff=1.0, transport=None):
+        self.root = url.rstrip("/")
+        self.api_key = api_key
+        self.timeout = timeout
+        self.attempts = attempts
+        self.backoff = backoff
+        self.transport = transport or KeepAlive(timeout)
+
+    def get(self, path, params=None):
+        """Return the decoded response for one API path."""
+        query = urllib.parse.urlencode(params or {})
+        url = f"{self.root}/api/v1/{path}" + (f"?{query}" if query else "")
+
+        for attempt in range(1, self.attempts + 1):
+            try:
+                body = self.transport(
+                    url, timeout=self.timeout, headers={"X-Api-Key": self.api_key}
+                )
+            except urllib.error.HTTPError as error:
+                detail = error_detail(error)
+                message = f"GET {url}: HTTP {error.code}{': ' + detail if detail else ''}"
+                if error.code not in LIDARR_RETRYABLE_STATUS:
+                    raise LidarrError(message)
+                if attempt >= self.attempts:
+                    raise LidarrError(message, transient=True)
+            except (urllib.error.URLError, TimeoutError) as error:
+                message = f"GET {url}: {error}"
+                if attempt >= self.attempts:
+                    raise LidarrError(message, transient=True) from error
+            else:
+                try:
+                    return json.loads(body)
+                except json.JSONDecodeError as error:
+                    raise LidarrError(f"GET {url}: malformed response") from error
+
+            pause = min(self.backoff * 2 ** (attempt - 1), BACKOFF_CEILING_SECONDS)
+            logger.warning("%s; retrying in %.0fs", message, pause)
+            time.sleep(pause)
+
+        raise LidarrError(f"GET {url}: gave up after {self.attempts} attempts", transient=True)
+
+
+def error_detail(error, limit=300):
+    """Return whatever the server said about a failure, for the log.
+
+    A bare status code sends you looking in the wrong place; Lidarr puts the
+    actual exception in the body.
+    """
+    try:
+        body = error.read().decode("utf-8", "replace").strip()
+    except Exception:  # noqa: BLE001 - diagnostics must never raise
+        return ""
+    if not body:
+        return ""
+    try:
+        payload = json.loads(body)
+    except json.JSONDecodeError:
+        return body[:limit]
+    for key in ("message", "description", "error"):
+        if payload.get(key):
+            return str(payload[key])[:limit]
+    return body[:limit]
+
+
+def parse_added(value):
+    """Return a unix timestamp for a Lidarr ISO-8601 date, or None."""
+    if not value:
+        return None
+    try:
+        return int(datetime.fromisoformat(str(value).replace("Z", "+00:00")).timestamp())
+    except ValueError:
+        return None
+
+
+def find_bad_albums(client, artist_id, artist_name):
+    """Name the specific albums Lidarr cannot serialise for one artist.
+
+    Runs only once that artist's album fetch has already failed, so the extra
+    requests are spent on a problem that already exists. Tracks come from an
+    endpoint that still works, and their album ids give a list to probe one at a
+    time; the ones that throw are the culprits. A track title from each is
+    enough to recognise the album in the UI, which the id alone is not.
+    """
+    try:
+        tracks = client.get("track", {"artistId": artist_id})
+    except LidarrError as error:
+        logger.warning("could not probe %s for the offending album: %s", artist_name, error)
+        return []
+
+    sample = {}
+    for track in tracks:
+        sample.setdefault(track.get("albumId"), track.get("title") or "")
+
+    bad = []
+    for album_id, title in sorted(sample.items(), key=lambda item: item[0] or 0):
+        if not album_id:
+            continue
+        try:
+            client.get("album", {"albumIds": album_id})
+        except LidarrError:
+            bad.append((album_id, title))
+    return bad
+
+
+def fetch_albums(client, artists):
+    """Return albums grouped by artist id, plus the artists whose albums failed.
+
+    Every album endpoint maps through a resource that picks the release with
+    `SingleOrDefault(x => x.Monitored)`, which throws for an album with two
+    monitored releases -- "Sequence contains more than one element" -- and takes
+    the entire response down with it.
+
+    The unfiltered endpoint is one request and is the only one that also skips
+    albums whose artist metadata is missing, so it is tried first. When it dies,
+    asking per artist confines the loss to whichever artist owns the offending
+    album, and names them, which is the only way to find it.
+    """
+    try:
+        grouped = {}
+        for album in client.get("album"):
+            grouped.setdefault(album.get("artistId"), []).append(album)
+        return grouped, []
+    except LidarrError as error:
+        logger.warning("fetching all albums failed (%s)", error)
+        logger.warning("falling back to one request per artist to isolate the bad album")
+
+    grouped, failed = {}, []
+    for artist in artists:
+        artist_id = artist["id"]
+        try:
+            grouped[artist_id] = client.get("album", {"artistId": artist_id})
+        except LidarrError as error:
+            name = artist.get("artistName") or str(artist_id)
+            logger.warning("could not fetch albums for %s: %s", name, error)
+            failed.append(name)
+            # Only worth probing a deterministic failure. When the network or
+            # the resolver is the problem, every artist fails, and probing each
+            # of them album by album multiplies the load that caused it.
+            if not error.transient:
+                for album_id, sample in find_bad_albums(client, artist_id, name):
+                    logger.warning(
+                        "  album id %d is the one Lidarr cannot serialise (it holds the"
+                        " track %r). Open it in Lidarr and leave exactly one release"
+                        " monitored.",
+                        album_id,
+                        sample,
+                    )
+    return grouped, failed
+
+
+def index_library(client, store):
+    """Rebuild the library index from Lidarr. Returns (artists, tracks).
+
+    Rebuilt wholesale rather than reconciled: Lidarr is the authority, the index
+    is small, and a delete in Lidarr has to disappear here or it lingers as a
+    library entry with no file behind it.
+    """
+    artists = client.get("artist")
+    artist_rows, album_rows, track_rows = [], [], []
+    skipped = []
+
+    albums_by_artist, album_failures = fetch_albums(client, artists)
+
+    for artist in artists:
+        artist_id = artist["id"]
+        name = artist.get("artistName") or ""
+        artist_rows.append(
+            (
+                artist_id,
+                mbid_of(artist.get("foreignArtistId")),
+                name,
+                normalise(name),
+                artist.get("path"),
+                1 if artist.get("monitored") else 0,
+            )
+        )
+
+        for album in albums_by_artist.get(artist_id, []):
+            album_rows.append(
+                (
+                    album["id"],
+                    artist_id,
+                    mbid_of(album.get("foreignAlbumId")),
+                    album.get("title") or "",
+                    1 if album.get("monitored") else 0,
+                    (album.get("releaseDate") or "")[:10] or None,
+                )
+            )
+
+        # Tracks and files have no unfiltered endpoint -- Lidarr rejects a call
+        # with no filter at all -- so these stay per artist. One artist it
+        # cannot serve must not cost the whole index, but it cannot pass
+        # silently either: their tracks end up absent, and every scrobble of
+        # theirs then reads as unmatched.
+        try:
+            # Paths and the date a track landed live on the file, not the track.
+            files = {
+                handle["id"]: handle
+                for handle in client.get("trackfile", {"artistId": artist_id})
+            }
+            tracks = client.get("track", {"artistId": artist_id})
+        except LidarrError as error:
+            logger.warning("could not index %s: %s", name or artist_id, error)
+            skipped.append(name or str(artist_id))
+            continue
+
+        for track in tracks:
+            handle = files.get(track.get("trackFileId") or 0) or {}
+            title = track.get("title") or ""
+            track_rows.append(
+                (
+                    track["id"],
+                    artist_id,
+                    track.get("albumId") or 0,
+                    mbid_of(track.get("foreignRecordingId")),
+                    title,
+                    normalise(name),
+                    normalise(title),
+                    1 if track.get("hasFile") else 0,
+                    handle.get("path"),
+                    parse_added(handle.get("dateAdded")),
+                    track.get("duration"),
+                )
+            )
+
+    store.replace_library(artist_rows, album_rows, track_rows)
+    store.set_state("index_skipped", str(len(skipped)))
+    store.set_state("index_albums_skipped", str(len(album_failures)))
+    logger.info(
+        "library indexed: %d artists, %d albums, %d tracks (%d with files)",
+        len(artist_rows),
+        len(album_rows),
+        len(track_rows),
+        sum(row[7] for row in track_rows),
+    )
+    if skipped:
+        logger.warning(
+            "%d artists could not be indexed, so their tracks are missing: %s",
+            len(skipped),
+            ", ".join(sorted(skipped)[:10]),
+        )
+    if album_failures:
+        logger.warning(
+            "no albums indexed for %d artists: %s."
+            " In Lidarr, open each one and check the Releases tab of their albums:"
+            " exactly one release per album may be monitored.",
+            len(album_failures),
+            ", ".join(sorted(album_failures)[:10]),
+        )
+    return len(artist_rows), len(track_rows)
+
+
+def match_library(store):
+    """Tie every distinct thing listened to back to a library track.
+
+    Two tiers. A MusicBrainz recording id is an exact join and is tried first;
+    everything else falls to the normalised name pair. There is no third tier on
+    purpose -- a guess that is nearly right is worse here than an admitted miss,
+    because the whole point of the number this produces is to say how far the
+    matching can be trusted.
+    """
+    store.rebuild_keys()
+    with store.connection:
+        store.connection.execute("UPDATE scrobble_key SET track_id = NULL, method = 'none'")
+        store.connection.execute(
+            "UPDATE scrobble_key SET"
+            "   track_id = (SELECT t.id FROM lidarr_track t"
+            "               WHERE t.recording_mbid = scrobble_key.track_mbid"
+            "               ORDER BY t.has_file DESC, t.id LIMIT 1),"
+            "   method = 'mbid'"
+            " WHERE track_mbid IS NOT NULL"
+            "   AND EXISTS (SELECT 1 FROM lidarr_track t"
+            "               WHERE t.recording_mbid = scrobble_key.track_mbid)"
+        )
+        store.connection.execute(
+            "UPDATE scrobble_key SET"
+            "   track_id = (SELECT t.id FROM lidarr_track t"
+            "               WHERE t.norm_artist = scrobble_key.norm_artist"
+            "                 AND t.norm_title = scrobble_key.norm_track"
+            "               ORDER BY t.has_file DESC, t.id LIMIT 1),"
+            "   method = 'name'"
+            " WHERE track_id IS NULL"
+            "   AND norm_artist <> '' AND norm_track <> ''"
+            "   AND EXISTS (SELECT 1 FROM lidarr_track t"
+            "               WHERE t.norm_artist = scrobble_key.norm_artist"
+            "                 AND t.norm_title = scrobble_key.norm_track)"
+        )
+
+
 def format_time(uts):
     """Return a UTC timestamp as a readable date."""
     if uts is None:
@@ -452,6 +1031,88 @@ def format_time(uts):
     return datetime.fromtimestamp(uts, tz=timezone.utc).strftime("%Y-%m-%d %H:%M")
 
 
+def coverage_report(store):
+    """Log how much of the listening history could be tied to the library.
+
+    The split that matters is not matched against unmatched. Most unmatched
+    listening is music that was never in the library at all, which says nothing
+    about the matcher. The number to watch is unmatched listening by an artist
+    the library *does* hold: that is a track that was played, sitting next to a
+    file it should have matched, and every one of those is a candidate for being
+    wrongly called cold later on.
+    """
+    tracks = store.scalar("SELECT COUNT(*) FROM lidarr_track")
+    if not tracks:
+        return
+
+    pairs = store.scalar("SELECT COUNT(*) FROM scrobble_key")
+    if not pairs:
+        return
+
+    logger.info("--- match coverage ---")
+    skipped = int(store.get_state("index_skipped") or 0)
+    if skipped:
+        logger.warning(
+            "the index is incomplete: %d artists are missing their tracks, so every"
+            " figure below is a floor. Do not cull against it.",
+            skipped,
+        )
+    albums_skipped = int(store.get_state("index_albums_skipped") or 0)
+    if albums_skipped:
+        logger.warning(
+            "%d artists have no albums indexed. Matching is unaffected -- it runs off"
+            " tracks -- but a cull cannot be run until this is fixed.",
+            albums_skipped,
+        )
+    for method in ("mbid", "name", "none"):
+        row = store.connection.execute(
+            "SELECT COUNT(*) AS pairs, COALESCE(SUM(plays), 0) AS plays"
+            " FROM scrobble_key WHERE method = ?",
+            (method,),
+        ).fetchone()
+        logger.info(
+            "%-5s: %6d pairs (%4.1f%%), %7d plays",
+            method,
+            row["pairs"],
+            100 * row["pairs"] / pairs,
+            row["plays"],
+        )
+
+    suspect = store.connection.execute(
+        "SELECT COUNT(*) AS pairs, COALESCE(SUM(plays), 0) AS plays FROM scrobble_key k"
+        " WHERE k.track_id IS NULL"
+        "   AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)"
+    ).fetchone()
+    logger.info(
+        "unmatched by an artist the library holds: %d pairs, %d plays -- these are the"
+        " matcher's misses, not music you do not own",
+        suspect["pairs"],
+        suspect["plays"],
+    )
+
+    with_files = store.scalar("SELECT COUNT(*) FROM lidarr_track WHERE has_file = 1")
+    played = store.scalar(
+        "SELECT COUNT(DISTINCT k.track_id) FROM scrobble_key k"
+        " JOIN lidarr_track t ON t.id = k.track_id WHERE t.has_file = 1"
+    )
+    logger.info(
+        "library tracks with a file: %d, of which played at least once: %d (%.1f%%)",
+        with_files,
+        played,
+        100 * played / with_files if with_files else 0,
+    )
+
+    worst = store.connection.execute(
+        "SELECT k.artist, k.track, k.plays FROM scrobble_key k"
+        " WHERE k.track_id IS NULL"
+        "   AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)"
+        " ORDER BY k.plays DESC, k.artist LIMIT 15"
+    ).fetchall()
+    for position, row in enumerate(worst, start=1):
+        label = f"{row['artist']} - {row['track']}"
+        logger.info("  unmatched %2d: %-60s %d plays", position, label[:60], row["plays"])
+
+
 def report(store, now):
     """Log what the store holds.
 
@@ -485,6 +1146,8 @@ def report(store, now):
     for position, row in enumerate(top, start=1):
         logger.info("  top artist %2d: %-40s %d", position, row["artist"][:40], row["plays"])
 
+    coverage_report(store)
+
     recent = store.connection.execute(
         "SELECT artist, track, COUNT(*) AS plays FROM scrobble WHERE uts >= ?"
         " GROUP BY artist, track ORDER BY plays DESC, artist LIMIT 10",
@@ -495,12 +1158,20 @@ def report(store, now):
         logger.info("  top track 90d %2d: %-50s %d", position, label[:50], row["plays"])
 
 
-def run_once(client, store, user, now, backfill_limit):
-    """Run a single ingest pass. Returns the number of scrobbles added."""
+def run_once(client, store, user, now, backfill_limit, lidarr=None):
+    """Run a single pass. Returns the number of scrobbles added."""
     started = time.monotonic()
     added = catch_up(client, store, user, now)
     added += backfill(client, store, user, now, limit=backfill_limit)
     loved = sync_loved(client, store, user)
+
+    if lidarr is not None:
+        index_library(lidarr, store)
+    # Matching is worth redoing even without a fresh index: new scrobbles have
+    # arrived, and they need a verdict too.
+    if store.scalar("SELECT COUNT(*) FROM lidarr_track"):
+        match_library(store)
+
     logger.info(
         "pass complete in %.1fs: %d scrobbles added, %d loved",
         time.monotonic() - started,
@@ -562,6 +1233,21 @@ def build_parser():
         help="cap the backfill at this many requests per pass; 0 for no cap"
         " (env MUSIC_CURATOR_BACKFILL_LIMIT)",
     )
+    parser.add_argument(
+        "--lidarr-url",
+        default=os.getenv("MUSIC_CURATOR_LIDARR_URL"),
+        help="base URL of Lidarr, e.g. http://lidarr:8686 (env MUSIC_CURATOR_LIDARR_URL)",
+    )
+    parser.add_argument(
+        "--lidarr-api-key",
+        default=os.getenv("MUSIC_CURATOR_LIDARR_API_KEY"),
+        help="Lidarr API key (env MUSIC_CURATOR_LIDARR_API_KEY)",
+    )
+    parser.add_argument(
+        "--skip-index",
+        action="store_true",
+        help="do not re-index the library this pass; match against the index already held",
+    )
     parser.add_argument(
         "--report-only",
         action="store_true",
@@ -601,6 +1287,12 @@ def main(argv=None, clock=time.time):
 
     client = Lastfm(args.api_key, delay=args.request_delay)
 
+    lidarr = None
+    if args.lidarr_url and args.lidarr_api_key and not args.skip_index:
+        lidarr = Lidarr(args.lidarr_url, args.lidarr_api_key)
+    elif not args.lidarr_url:
+        logger.info("no Lidarr URL configured; ingesting scrobbles only")
+
     stopping = False
 
     def stop(signum, _frame):
@@ -615,8 +1307,8 @@ def main(argv=None, clock=time.time):
         while True:
             now = int(clock())
             try:
-                run_once(client, store, args.user, now, args.backfill_limit)
-            except LastfmError as error:
+                run_once(client, store, args.user, now, args.backfill_limit, lidarr)
+            except (LastfmError, LidarrError) as error:
                 logger.error("%s", error)
                 if interval is None:
                     return 1

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "music-curator"
-version = "0.1.0"
+version = "0.2.3"
 description = "Ingest a Last.fm listening history and curate a music library from it"
 readme = "README.md"
 requires-python = ">=3.11"

tests/conftest.py

@@ -1,6 +1,10 @@
+import http.server
+import io
 import json
 import os
 import sys
+import threading
+import urllib.error
 import urllib.parse
 
 import pytest
@@ -120,6 +124,115 @@ class FakeLastfm:
         }
 
 
+class FakeLidarr:
+    """A transport serving a canned library over the Lidarr v1 API surface.
+
+    Built from a nested description -- artist, album, tracks -- and split back
+    out across the four endpoints the indexer actually calls, so the tests
+    exercise the same stitching the real thing does.
+    """
+
+    def __init__(self, artists=(), fail=()):
+        # (path, artistId) pairs the fake refuses to serve, standing in for the
+        # 500s Lidarr returns on data it cannot hydrate.
+        self.fail = set(fail)
+        self.artists, self.albums, self.tracks, self.files = [], [], [], []
+        for artist_index, entry in enumerate(artists, start=1):
+            artist_id = artist_index
+            self.artists.append(
+                {
+                    "id": artist_id,
+                    "artistName": entry["name"],
+                    "foreignArtistId": entry.get("mbid", f"artist-mbid-{artist_id}"),
+                    "path": f"/music/{entry['name']}",
+                    "monitored": entry.get("monitored", True),
+                }
+            )
+            for album_index, album in enumerate(entry.get("albums", []), start=1):
+                album_id = artist_id * 100 + album_index
+                self.albums.append(
+                    {
+                        "id": album_id,
+                        "artistId": artist_id,
+                        "title": album["title"],
+                        "foreignAlbumId": f"album-mbid-{album_id}",
+                        "monitored": album.get("monitored", True),
+                        "releaseDate": album.get("release_date", "2019-01-01T00:00:00Z"),
+                    }
+                )
+                for track_index, track in enumerate(album.get("tracks", []), start=1):
+                    track_id = album_id * 100 + track_index
+                    has_file = track.get("has_file", True)
+                    self.tracks.append(
+                        {
+                            "id": track_id,
+                            "artistId": artist_id,
+                            "albumId": album_id,
+                            "title": track["title"],
+                            "foreignRecordingId": track.get("recording_mbid", ""),
+                            "trackFileId": track_id if has_file else 0,
+                            "hasFile": has_file,
+                            "duration": 210000,
+                        }
+                    )
+                    if has_file:
+                        self.files.append(
+                            {
+                                "id": track_id,
+                                "artistId": artist_id,
+                                "albumId": album_id,
+                                "path": f"/music/{entry['name']}/{album['title']}/"
+                                f"{track['title']}.flac",
+                                "dateAdded": track.get("added", "2020-05-01T12:00:00Z"),
+                            }
+                        )
+        self.calls = []
+
+    def __call__(self, url, timeout=None, headers=None):
+        parsed = urllib.parse.urlparse(url)
+        assert (headers or {}).get("X-Api-Key"), "Lidarr requires the API key header"
+        path = parsed.path.rsplit("/", 1)[-1]
+        query = {
+            key: value[0] for key, value in urllib.parse.parse_qs(parsed.query).items()
+        }
+        self.calls.append((path, query))
+
+        artist_id = int(query.get("artistId", 0))
+        if (path, artist_id) in self.fail:
+            raise urllib.error.HTTPError(
+                url, 500, "Internal Server Error", {}, io.BytesIO(b'{"message": "boom"}')
+            )
+
+        album_ids = query.get("albumIds")
+        if path == "album" and album_ids:
+            album_id = int(album_ids)
+            if ("albumid", album_id) in self.fail:
+                raise urllib.error.HTTPError(
+                    url,
+                    500,
+                    "Internal Server Error",
+                    {},
+                    io.BytesIO(b'{"message": "Sequence contains more than one element"}'),
+                )
+            return json.dumps([row for row in self.albums if row["id"] == album_id])
+
+        if path == "artist":
+            return json.dumps(self.artists)
+        source = {"album": self.albums, "track": self.tracks, "trackfile": self.files}[path]
+        if path == "album" and not artist_id:
+            # Lidarr's unfiltered album endpoint returns the lot.
+            return json.dumps(source)
+        if not artist_id:
+            raise urllib.error.HTTPError(
+                url,
+                400,
+                "Bad Request",
+                {},
+                io.BytesIO(b'{"message": "artistId must be provided"}'),
+            )
+        return json.dumps([row for row in source if row["artistId"] == artist_id])
+
+
 @pytest.fixture
 def now_playing():
     """The entry Last.fm prepends for a track in progress: no `date` at all."""
@@ -130,3 +243,53 @@ def now_playing():
         "mbid": "",
         "@attr": {"nowplaying": "true"},
     }
+
+
+class _CountingServer(http.server.ThreadingHTTPServer):
+    """Counts accepted connections, which is what connection reuse is about."""
+
+    daemon_threads = True
+
+    def __init__(self, *args, **kwargs):
+        self.connections = 0
+        super().__init__(*args, **kwargs)
+
+    def process_request(self, request, client_address):
+        self.connections += 1
+        super().process_request(request, client_address)
+
+
+class _Handler(http.server.BaseHTTPRequestHandler):
+    # Without HTTP/1.1 the server closes after every response and no client
+    # could reuse anything, which would make the test prove nothing.
+    protocol_version = "HTTP/1.1"
+
+    def do_GET(self):
+        if self.path.startswith("/boom"):
+            body, status = b'{"message": "boom"}', 500
+        else:
+            body = json.dumps(
+                {"path": self.path, "key": self.headers.get("X-Api-Key")}
+            ).encode()
+            status = 200
+        self.send_response(status)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args):
+        pass
+
+
+@pytest.fixture
+def http_server():
+    """A real local HTTP server, for the one component that talks sockets."""
+    server = _CountingServer(("127.0.0.1", 0), _Handler)
+    thread = threading.Thread(target=server.serve_forever, daemon=True)
+    thread.start()
+    try:
+        yield server, f"http://127.0.0.1:{server.server_port}"
+    finally:
+        server.shutdown()
+        server.server_close()

tests/test_music_curator.py

@@ -1,7 +1,8 @@
+import json
 import urllib.error
 
 import pytest
-from conftest import FakeLastfm, make_loved, make_tracks
+from conftest import FakeLastfm, FakeLidarr, make_loved, make_tracks
 
 import music_curator
 
@@ -23,6 +24,62 @@ def ingest(store, api, backfill_limit=0):
     return music_curator.run_once(client, store, "lyra", NOW, backfill_limit)
 
 
+def scrobble_of(artist, track, uts=NOW - 3600, mbid=""):
+    """One recent-tracks entry, in the shape Last.fm actually sends."""
+    return {
+        "artist": {"#text": artist, "mbid": ""},
+        "album": {"#text": "Some Album", "mbid": ""},
+        "name": track,
+        "mbid": mbid,
+        "date": {"uts": str(uts)},
+    }
+
+
+LIBRARY = [
+    {
+        "name": "Yellowcard",
+        "albums": [
+            {
+                "title": "Lift a Sail",
+                "tracks": [
+                    {"title": "Here I Am Alive", "recording_mbid": "rec-alive"},
+                    {"title": "Transmission Home"},
+                ],
+            }
+        ],
+    },
+    {
+        "name": "AC/DC",
+        "albums": [{"title": "Back in Black", "tracks": [{"title": "Hells Bells"}]}],
+    },
+    {
+        "name": "The Prodigy",
+        "albums": [
+            {"title": "The Fat of the Land", "tracks": [{"title": "Breathe (Remastered)"}]}
+        ],
+    },
+]
+
+
+def indexed(tmp_path, scrobbles):
+    """Ingest scrobbles, index the canned library, and match the two."""
+    store = store_at(tmp_path)
+    ingest(store, FakeLastfm(scrobbles))
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=FakeLidarr(LIBRARY)), store
+    )
+    music_curator.match_library(store)
+    return store
+
+
+def verdict(store, artist, track):
+    row = store.connection.execute(
+        "SELECT method, track_id FROM scrobble_key WHERE artist = ? AND track = ?",
+        (artist, track),
+    ).fetchone()
+    return (row["method"], row["track_id"]) if row else (None, None)
+
+
 def test_parse_interval_units():
     assert music_curator.parse_interval("90") == 90
     assert music_curator.parse_interval("30m") == 1800
@@ -241,3 +298,392 @@ def test_main_ingests_through_the_module_level_fetcher(tmp_path, monkeypatch):
 
     assert code == 0
     assert store_at(tmp_path).count() == 20
+
+
+def test_normalise_flattens_the_ways_tags_and_scrobbles_disagree():
+    assert music_curator.normalise("Yellowcard feat. Tay Jardine") == "yellowcard"
+    assert music_curator.normalise("BABYMETAL Feat. F.Hero") == "babymetal"
+    assert music_curator.normalise("AC/DC") == "ac dc"
+    assert music_curator.normalise("The Prodigy") == "prodigy"
+    assert music_curator.normalise("Beyoncé") == "beyonce"
+    assert music_curator.normalise("Simon & Garfunkel") == "simon and garfunkel"
+    assert music_curator.normalise("Breathe (Remastered 2011)") == "breathe"
+    assert music_curator.normalise("Breathe (Live) (Remastered)") == "breathe"
+    assert music_curator.normalise("Vice Grip - Live") == "vice grip"
+
+
+def test_normalise_keeps_a_leading_parenthetical():
+    """Stripping every bracket would destroy real titles."""
+    assert music_curator.normalise("(Don't Fear) The Reaper") == "dont fear the reaper"
+
+
+def test_normalise_treats_apostrophes_and_other_punctuation_differently():
+    """The two rules pull opposite ways: one deletes, the other separates."""
+    assert music_curator.normalise("Don't") == music_curator.normalise("Dont")
+    assert music_curator.normalise("Don’t") == music_curator.normalise("Dont")
+    assert music_curator.normalise("AC/DC") == music_curator.normalise("AC-DC")
+    assert music_curator.normalise("AC/DC") == music_curator.normalise("AC DC")
+    # ...and they must not collide with each other.
+    assert music_curator.normalise("AC/DC") != music_curator.normalise("ACDC")
+
+
+def test_normalise_leaves_with_alone():
+    """`with` appears in too many genuine titles to cut on sight."""
+    assert music_curator.normalise("Dancing with Myself") == "dancing with myself"
+
+
+def test_a_recording_mbid_matches_exactly(tmp_path):
+    store = indexed(tmp_path, [scrobble_of("Yellowcard", "Here I Am Alive", mbid="rec-alive")])
+
+    assert verdict(store, "Yellowcard", "Here I Am Alive")[0] == "mbid"
+
+
+def test_a_guest_credit_still_matches_by_name(tmp_path):
+    """Last.fm puts the guest in the artist field; the file tag does not."""
+    store = indexed(
+        tmp_path, [scrobble_of("Yellowcard feat. Tay Jardine", "Here I Am Alive")]
+    )
+
+    method, track_id = verdict(store, "Yellowcard feat. Tay Jardine", "Here I Am Alive")
+    assert method == "name"
+    assert track_id is not None
+
+
+def test_a_remaster_suffix_on_the_library_side_still_matches(tmp_path):
+    store = indexed(tmp_path, [scrobble_of("The Prodigy", "Breathe")])
+
+    assert verdict(store, "The Prodigy", "Breathe")[0] == "name"
+
+
+def test_music_that_is_not_in_the_library_stays_unmatched(tmp_path):
+    store = indexed(tmp_path, [scrobble_of("Some Band", "Some Song")])
+
+    assert verdict(store, "Some Band", "Some Song") == ("none", None)
+
+
+def test_the_mbid_tier_is_preferred_over_the_name_tier(tmp_path):
+    store = indexed(
+        tmp_path,
+        [scrobble_of("Yellowcard", "Here I Am Alive", mbid="rec-alive")],
+    )
+    method, track_id = verdict(store, "Yellowcard", "Here I Am Alive")
+
+    expected = store.connection.execute(
+        "SELECT id FROM lidarr_track WHERE recording_mbid = 'rec-alive'"
+    ).fetchone()["id"]
+    assert (method, track_id) == ("mbid", expected)
+
+
+def test_keys_carry_the_play_count_and_the_span(tmp_path):
+    store = indexed(
+        tmp_path,
+        [
+            scrobble_of("AC/DC", "Hells Bells", uts=NOW - 7200),
+            scrobble_of("AC/DC", "Hells Bells", uts=NOW - 3600),
+        ],
+    )
+
+    row = store.connection.execute(
+        "SELECT plays, first_uts, last_uts FROM scrobble_key WHERE artist = 'AC/DC'"
+    ).fetchone()
+    assert row["plays"] == 2
+    assert row["last_uts"] - row["first_uts"] == 3600
+
+
+def test_re_indexing_drops_what_lidarr_no_longer_has(tmp_path):
+    """The index is Lidarr's mirror, not an accumulation of everything ever seen."""
+    store = indexed(tmp_path, [scrobble_of("AC/DC", "Hells Bells")])
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_artist") == 3
+
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=FakeLidarr(LIBRARY[:1])), store
+    )
+
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_artist") == 1
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_track WHERE artist_id NOT IN"
+                        " (SELECT id FROM lidarr_artist)") == 0
+
+
+def test_the_index_records_paths_and_when_a_file_landed(tmp_path):
+    store = indexed(tmp_path, [])
+
+    row = store.connection.execute(
+        "SELECT path, added, has_file FROM lidarr_track WHERE title = 'Hells Bells'"
+    ).fetchone()
+    assert row["path"].endswith("Hells Bells.flac")
+    assert row["has_file"] == 1
+    assert row["added"] == music_curator.parse_added("2020-05-01T12:00:00Z")
+
+
+def test_a_track_without_a_file_has_no_path(tmp_path):
+    library = [
+        {
+            "name": "Ghost Artist",
+            "albums": [{"title": "Unowned", "tracks": [{"title": "Missing", "has_file": False}]}],
+        }
+    ]
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=FakeLidarr(library)), store
+    )
+
+    row = store.connection.execute("SELECT path, has_file FROM lidarr_track").fetchone()
+    assert row["has_file"] == 0
+    assert row["path"] is None
+
+
+def test_lidarr_requires_the_api_key_header(tmp_path):
+    """The fake asserts on it, which is the point: a missing header is a 401."""
+    client = music_curator.Lidarr("http://lidarr", "key", transport=FakeLidarr(LIBRARY))
+
+    assert client.get("artist")
+
+
+def test_a_lidarr_failure_is_reported_not_swallowed():
+    def broken(url, timeout=None, headers=None):
+        raise urllib.error.HTTPError(url, 401, "unauthorised", {}, None)
+
+    client = music_curator.Lidarr("http://lidarr", "wrong", transport=broken)
+
+    with pytest.raises(music_curator.LidarrError, match="HTTP 401"):
+        client.get("artist")
+
+
+def test_an_existing_version_one_store_is_migrated_not_discarded(tmp_path):
+    """A rebuilt history is thousands of API requests; migration is additive."""
+    store = store_at(tmp_path)
+    store.add_scrobbles([music_curator.Scrobble(uts=NOW, artist="A", track="B")])
+    store.set_state("schema_version", "1")
+    store.close()
+
+    reopened = store_at(tmp_path)
+
+    assert reopened.count() == 1
+    assert reopened.get_state("schema_version") == music_curator.SCHEMA_VERSION
+
+
+def test_an_unmatched_track_by_a_known_artist_is_a_matcher_miss(tmp_path):
+    """The split that matters: music you own and played, that failed to match."""
+    store = indexed(
+        tmp_path,
+        [
+            scrobble_of("AC/DC", "A Track Lidarr Has Never Heard Of"),
+            scrobble_of("Some Band", "Some Song"),
+        ],
+    )
+
+    misses = store.connection.execute(
+        "SELECT k.artist FROM scrobble_key k WHERE k.track_id IS NULL"
+        "   AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)"
+    ).fetchall()
+
+    # Only the AC/DC one: "Some Band" is not in the library, so it says nothing
+    # about the matcher.
+    assert [row["artist"] for row in misses] == ["AC/DC"]
+
+
+def test_the_report_runs_over_a_matched_store(tmp_path):
+    store = indexed(tmp_path, [scrobble_of("Yellowcard", "Here I Am Alive", mbid="rec-alive")])
+
+    music_curator.report(store, NOW)
+
+
+def test_matching_is_redone_when_new_scrobbles_arrive(tmp_path):
+    """A pass with no re-index still has to give the new plays a verdict."""
+    store = indexed(tmp_path, [scrobble_of("AC/DC", "Hells Bells")])
+    api = FakeLastfm(
+        [
+            scrobble_of("AC/DC", "Hells Bells"),
+            scrobble_of("Yellowcard", "Transmission Home", uts=NOW - 60),
+        ]
+    )
+
+    music_curator.run_once(client_for(api), store, "lyra", NOW, 0)
+
+    assert verdict(store, "Yellowcard", "Transmission Home")[0] == "name"
+
+
+def test_albums_come_from_the_unfiltered_endpoint(tmp_path):
+    """One request, and the only path that skips albums it cannot hydrate."""
+    api = FakeLidarr(LIBRARY)
+    store = store_at(tmp_path)
+
+    music_curator.index_library(music_curator.Lidarr("http://lidarr", "key", transport=api), store)
+
+    album_calls = [query for path, query in api.calls if path == "album"]
+    assert album_calls == [{}]
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_album") == 3
+
+
+def test_a_bad_album_falls_back_to_asking_per_artist(tmp_path):
+    """An album with two monitored releases throws in the resource mapper, so
+    the bulk call dies wholesale. Per artist, only its owner is lost."""
+    # The bulk call fails because artist 2 owns the offending album.
+    api = FakeLidarr(LIBRARY, fail=[("album", 0), ("album", 2)])
+    store = store_at(tmp_path)
+
+    music_curator.index_library(music_curator.Lidarr("http://lidarr", "key", transport=api), store)
+
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_album WHERE artist_id = 2") == 0
+    # The other two artists keep their albums.
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_album") == 2
+    assert store.get_state("index_albums_skipped") == "1"
+
+
+def test_a_bad_album_does_not_cost_that_artist_their_tracks(tmp_path):
+    """Tracks come from a different endpoint with a different mapper, so
+    matching survives an album Lidarr cannot serialise."""
+    api = FakeLidarr(LIBRARY, fail=[("album", 0), ("album", 2)])
+    store = store_at(tmp_path)
+
+    music_curator.index_library(music_curator.Lidarr("http://lidarr", "key", transport=api), store)
+
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_track WHERE artist_id = 2") == 1
+    assert store.get_state("index_skipped") == "0"
+
+
+def test_an_artist_lidarr_cannot_serve_does_not_kill_the_index(tmp_path):
+    # Artist 2 is AC/DC in LIBRARY; its track lookup fails.
+    api = FakeLidarr(LIBRARY, fail=[("track", 2)])
+    store = store_at(tmp_path)
+
+    music_curator.index_library(music_curator.Lidarr("http://lidarr", "key", transport=api), store)
+
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_artist") == 3
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_track WHERE artist_id = 2") == 0
+    # The other two artists are indexed in full.
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_track") == 3
+    assert store.get_state("index_skipped") == "1"
+
+
+def test_a_skipped_artist_is_recorded_so_the_report_can_disown_the_numbers(tmp_path):
+    """An incomplete index makes played music look cold. It has to be loud."""
+    api = FakeLidarr(LIBRARY, fail=[("trackfile", 1)])
+    store = store_at(tmp_path)
+
+    music_curator.index_library(music_curator.Lidarr("http://lidarr", "key", transport=api), store)
+    music_curator.match_library(store)
+
+    assert store.get_state("index_skipped") == "1"
+
+
+def test_a_clean_index_records_no_skips(tmp_path):
+    store = indexed(tmp_path, [])
+
+    assert store.get_state("index_skipped") == "0"
+
+
+def test_a_lidarr_error_carries_the_url_and_what_the_server_said():
+    """A bare status code sends you looking at the wrong thing entirely."""
+    api = FakeLidarr(LIBRARY, fail=[("album", 0)])
+    client = music_curator.Lidarr("http://lidarr:8686", "key", transport=api)
+
+    with pytest.raises(music_curator.LidarrError) as raised:
+        client.get("album")
+
+    message = str(raised.value)
+    assert "http://lidarr:8686/api/v1/album" in message
+    assert "HTTP 500" in message
+    assert "boom" in message
+
+
+def test_the_offending_album_is_named_not_just_its_artist(tmp_path, caplog):
+    """An artist's whole discography is too much to click through by hand."""
+    # AC/DC is artist 2; its only album is id 201, holding "Hells Bells".
+    api = FakeLidarr(LIBRARY, fail=[("album", 0), ("album", 2), ("albumid", 201)])
+    store = store_at(tmp_path)
+
+    with caplog.at_level("WARNING"):
+        music_curator.index_library(
+            music_curator.Lidarr("http://lidarr", "key", transport=api), store
+        )
+
+    assert "album id 201" in caplog.text
+    assert "Hells Bells" in caplog.text
+
+
+def test_a_transient_failure_is_not_probed_album_by_album(tmp_path, caplog):
+    """When the resolver is the problem every artist fails, and probing each of
+    them multiplies the load that caused it."""
+
+    def unresolvable(url, timeout=None, headers=None):
+        if "artistId" in url:
+            raise urllib.error.URLError("[Errno -3] Try again")
+        return json.dumps(FakeLidarr(LIBRARY).artists) if url.endswith("artist") else "[]"
+
+    store = store_at(tmp_path)
+    client = music_curator.Lidarr("http://lidarr", "key", transport=unresolvable, backoff=0)
+
+    with caplog.at_level("WARNING"):
+        music_curator.index_library(client, store)
+
+    assert "Try again" in caplog.text
+    assert "cannot serialise" not in caplog.text
+
+
+def test_a_transient_failure_is_retried():
+    attempts = []
+
+    def flaky(url, timeout=None, headers=None):
+        attempts.append(url)
+        if len(attempts) < 3:
+            raise urllib.error.URLError("[Errno -3] Try again")
+        return "[]"
+
+    client = music_curator.Lidarr("http://lidarr", "key", transport=flaky, backoff=0)
+
+    assert client.get("artist") == []
+    assert len(attempts) == 3
+
+
+def test_a_lidarr_500_is_not_retried():
+    """It is an exception inside Lidarr's serialisation, not a busy server."""
+    api = FakeLidarr(LIBRARY, fail=[("album", 0)])
+    client = music_curator.Lidarr("http://lidarr", "key", transport=api, backoff=0)
+
+    with pytest.raises(music_curator.LidarrError) as raised:
+        client.get("album")
+
+    assert raised.value.transient is False
+    assert len(api.calls) == 1
+
+
+def test_keep_alive_uses_one_connection_for_many_requests(http_server):
+    """The point of the whole class: one DNS lookup and one socket, not N."""
+    server, base = http_server
+    transport = music_curator.KeepAlive()
+
+    try:
+        for index in range(5):
+            body = transport(f"{base}/api/v1/artist?n={index}", headers={"X-Api-Key": "key"})
+            assert json.loads(body)["key"] == "key"
+    finally:
+        transport.close()
+
+    assert server.connections == 1
+
+
+def test_keep_alive_maps_an_error_status_onto_httperror(http_server):
+    _, base = http_server
+    transport = music_curator.KeepAlive()
+
+    try:
+        with pytest.raises(urllib.error.HTTPError) as raised:
+            transport(f"{base}/boom", headers={"X-Api-Key": "key"})
+        assert raised.value.code == 500
+        assert music_curator.error_detail(raised.value) == "boom"
+    finally:
+        transport.close()
+
+
+def test_lidarr_talks_to_a_real_server_through_keep_alive(http_server):
+    server, base = http_server
+    client = music_curator.Lidarr(base, "secret")
+
+    try:
+        assert client.get("artist", {"x": 1})["key"] == "secret"
+        assert client.get("album")["path"] == "/api/v1/album"
+    finally:
+        client.transport.close()
+
+    assert server.connections == 1