chore: add a tool for finding Music tracks whose files have gone

Lidarr renames an artist or album folder, music-mirror prunes the old path and
encodes the new one, and every entry in Apple Music pointing at the old path
dies. Music offers no view of those, and the exclamation mark only appears once
a track is touched.

Reads the XML from Music's own Export Library rather than asking Music itself.
A broken track makes AppleScript's `location` raise instead of returning a
value, so a bulk query dies on the first one with error -1728 and a per-track
loop costs an Apple event apiece.

Checked against a single directory walk rather than a test per file. On a
fifty-thousand-track library that is around seven thousand directory reads
instead of fifty thousand stat calls, and over SMB each of those stats is a
network round trip -- which is the difference between seconds and minutes.

Two comparisons have to be loosened or most of the library reads as missing.
macOS stores filenames decomposed while the share composes them, so the umlauts
in Motley Crue are two different byte strings depending on which side wrote the
name; both are normalised to NFC. And the share is very likely
case-insensitive, so a file is not missing because someone capitalised it
differently.

The test stage now copies tools/ as well, since the suite covers this and the
runtime image deliberately does not carry it.

.gitea/workflows/build-and-publish.yaml

@@ -45,7 +45,8 @@ jobs:
 
       # The suite runs inside the image, against the interpreter that ships,
       # rather than against whatever the runner happens to provide. A failing
-      # test fails the build. Layers are shared with the push build below.
+      # test fails the build. The runtime stage below is built from the same
+      # daemon afterwards, so its layers are already in cache.
       - name: Run the test suite inside the image
         run: docker build --target test -t music-curator:test .
 
@@ -124,9 +125,6 @@ jobs:
           echo "release=${release}" >> "$GITHUB_OUTPUT"
           echo "Computed bump=${bump}, release=${release}, base=${base}"
 
-      - name: Set up Buildx
-        uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4
-
       - name: Log in to the Gitea container registry
         if: github.event_name != 'pull_request'
         uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4
@@ -135,21 +133,33 @@ jobs:
           username: ${{ github.repository_owner }}
           password: ${{ secrets.PACKAGES_TOKEN }}
 
-      - name: Build and push
-        uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7
-        with:
-          context: .
-          # Without this the last stage in the Dockerfile -- the test stage --
-          # would be what gets published.
-          target: runtime
-          # The NAS is the only host this runs on. Building arm64 as well would
-          # mean emulating it under QEMU for no consumer.
-          platforms: linux/amd64
-          push: ${{ github.event_name != 'pull_request' }}
-          tags: ${{ steps.version.outputs.tags }}
-          labels: |
-            org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}
-            org.opencontainers.image.revision=${{ github.sha }}
+      # Plain `docker build` rather than buildx. buildx boots its own buildkit
+      # in a container with a cache of its own, so it shared nothing with the
+      # test build above and rebuilt the image from the base image up -- two
+      # full builds per run. It earns that cost when building for several
+      # platforms; this only ever targets the amd64 NAS, so it does not.
+      #
+      # `--target runtime` is a strict prefix of the test stage, so every layer
+      # is already in the daemon's cache and this resolves in seconds.
+      - name: Build the runtime image
+        run: |
+          set -euo pipefail
+          tags=()
+          while IFS= read -r tag; do
+            [ -n "$tag" ] && tags+=(-t "$tag")
+          done <<< "${{ steps.version.outputs.tags }}"
+          docker build --target runtime \
+            --label "org.opencontainers.image.source=${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}" \
+            --label "org.opencontainers.image.revision=${GITHUB_SHA}" \
+            "${tags[@]}" .
+
+      - name: Push
+        if: github.event_name != 'pull_request'
+        run: |
+          set -euo pipefail
+          while IFS= read -r tag; do
+            [ -n "$tag" ] && docker push "$tag"
+          done <<< "${{ steps.version.outputs.tags }}"
 
       # Record the release: write the computed version into pyproject.toml, then
       # commit and tag it, so the packaging metadata always matches the release

Dockerfile

@@ -28,5 +28,8 @@ FROM runtime AS test
 
 RUN pip install --no-cache-dir pytest
 COPY pytest.ini ./
+# The host-side tools are not part of the runtime image, but their tests are
+# part of the suite, so they have to be present for it.
+COPY tools ./tools
 COPY tests ./tests
 RUN python -m pytest

README.md

@@ -7,22 +7,274 @@ which keeps an MP3 copy of a lossless library for an iPod. This one answers the
 question that mirror cannot: which of it is worth carrying, and which of it has
 not been played in years.
 
-**This is stage one.** It ingests the scrobble history and nothing else. There
-are no playlists yet, and nothing touches Lidarr or the music library. See
-"Where this is going" below.
+**This is stage three.** It ingests the scrobble history, indexes the library
+from Lidarr, matches one to the other, and writes playlists into the mirror —
+by listening history and by mood.
+Nothing is written back to Lidarr — every call there is a `GET`. See "Where this
+is going" below.
 
 ## What it does today
 
 - Pulls the full Last.fm scrobble history into SQLite, then keeps it current.
 - Tracks loved tracks separately, as the protected set for later stages.
-- Reports what it holds, including the number that matters most: how many
-  scrobbles carry a MusicBrainz recording id.
+- Indexes every artist, album and track Lidarr knows about, with file paths and
+  the date each file landed.
+- Ties the two together and reports how well it managed.
+- Writes M3U playlists into the mirror, from the listening history.
 
-That last figure decides the next stage. Lidarr exposes a `ForeignRecordingId`
-on every track, which is the same identifier, so scrobbles carrying one can be
-joined to the library exactly. The rest have to go through name matching, which
-is where a curation tool goes wrong and starts recommending the deletion of
-music you love. Measure the join rate before trusting the verdict.
+## Matching
+
+Two tiers, and no third.
+
+| Tier | Key | Notes |
+| ------ | --------------------------- | ----------------------------------------- |
+| `mbid` | MusicBrainz recording id | Exact. Last.fm's per-scrobble `mbid` against Lidarr's `ForeignRecordingId` |
+| `name` | Normalised artist and title | Everything the first tier could not carry |
+| `none` | — | Recorded as a miss, never guessed at |
+
+The name tier does almost all of the work. A recording MBID is exact when it
+lands, but MusicBrainz holds a separate recording per release, and Last.fm and
+Lidarr rarely pick the same one: on a real library, two thirds of scrobbles
+carry a recording id and barely a twentieth of them join on it. The report
+counts how many carried an id and matched on name anyway, which is the measure
+of that disagreement.
+
+So the normalisation is the load-bearing part, because the two sides disagree in
+predictable ways. It folds case and accents, drops guest credits (`Yellowcard
+feat. Tay Jardine` against a tag of `Yellowcard`), strips a trailing
+version suffix (`(Remastered 2011)`, `- Live`), expands `&`, and removes a
+leading `The`. Punctuation gets two different rules that pull against each
+other and are both required: apostrophes are **deleted**, so `Don't` meets
+`Dont`, while every other mark becomes a **space**, so `AC/DC`, `AC-DC` and
+`AC DC` all meet as well.
+
+It leans towards collapsing too much. A false match makes something look
+played; a missed match makes something look abandoned. Only one of those
+deletes music.
+
+### Indexing quirks
+
+Albums are fetched from the **unfiltered** `GET /api/v1/album` first: one
+request, and the only path that skips albums whose artist metadata is missing
+rather than dereferencing it.
+
+That is not enough on its own. Every album endpoint maps through a resource
+that picks the release with `SingleOrDefault(x => x.Monitored)`, which throws
+for an album with **two monitored releases** and takes the whole response with
+it:
+
+```
+HTTP 500: Sequence contains more than one element
+```
+
+When the bulk call dies that way, the indexer falls back to one request per
+artist. It cannot avoid the exception, but it confines it to whichever artist
+owns the offending album and names them in the log — which is the only
+practical way to find it in a large library. Open that artist in Lidarr and
+check the Releases tab of each album: exactly one release may be monitored.
+
+Losing an artist's albums does not cost their tracks, which come from a
+different endpoint with a different mapper, so matching is unaffected. A cull
+would not be, and the report says so.
+
+### Talking to Lidarr
+
+Indexing is two requests per artist, and more when the album fallback fires. On
+a large library that is thousands of requests in a few minutes. `urllib` opens a
+new TCP connection and performs a new DNS lookup for every one of them, which is
+enough to exhaust a container's resolver and produce `[Errno -3] Try again` on
+everything at once. The client therefore holds one connection open per host and
+resolves once.
+
+Transient failures — a dropped connection, a resolver hiccup, `429`, `502`,
+`503`, `504` — are retried with a backoff. An HTTP `500` is not: it is an
+unhandled exception inside Lidarr's own serialisation and will be raised again
+identically. That distinction also decides whether a failure is worth
+investigating; a library-wide outage is not probed artist by artist, because
+doing so multiplies the load that caused it.
+
+A local address is preferable to a public hostname here. It removes DNS, the
+reverse proxy and its timeouts from a path that needs none of them.
+
+Tracks and files have no unfiltered endpoint — Lidarr rejects a call with no
+filter — so they stay per artist. If one artist cannot be served, that artist is
+skipped and the run continues, but the count is recorded and the coverage report
+says so loudly. A missing artist makes their played music look cold, so an
+incomplete index must never be culled against.
+
+### Reading the coverage report
+
+Matched against unmatched is the wrong comparison — most unmatched listening is
+music that was never in the library, which says nothing at all about the
+matcher. The line to watch is:
+
+```
+unmatched by an artist the library holds: N pairs, M plays
+```
+
+That is a track that was played by an artist the library holds. It is then
+split three ways, because owning an artist is a weak proxy for owning a track
+and a shared title is a weak proxy for a shared song:
+
+- **the library's own title credits the scrobbled artist** — `Voodoo People
+  (Pendulum Remix)` against a play credited to Pendulum. Same song, filed
+  under the original artist. These are the genuine misses.
+- **the same title under an unrelated artist** — a collision, not a miss.
+  Across fifty thousand tracks these are constant: `Everyday` is Rusko and
+  also Def Leppard, `Kaleidoscope` is Delta Heavy and also Chappell Roan.
+  Matching on title alone would be far worse than missing them, which is why
+  there is no such tier.
+- **the title is nowhere in the library** — never bought.
+
+Only the first is worth chasing. Counting all three as matcher failures
+overstates the problem and would over-block the cull.
+
+## Playlists
+
+Written into `<mirror>/_playlists/` as extended M3U, rebuilt every pass. Six
+rules, capped at `--playlist-limit` tracks each:
+
+| Playlist | Rule |
+| -------------------- | --------------------------------------------------------- |
+| `heavy-rotation` | Most played over the last twelve months |
+| `all-time` | Most played ever |
+| `neglected` | Played heavily once, silent for twelve months |
+| `deep-cuts` | Never played, from albums whose other tracks you play constantly |
+| `unheard-favourites` | Never played, by the artists you play most |
+| `unheard` | Never played, anywhere in the library |
+
+Ninety days was the obvious window for "recent" and is the wrong one: on a real
+history it holds a few hundred plays spread thinly across a twenty-thousand
+track rotation, so nothing ranks meaningfully. Twelve months does.
+
+The two `unheard` playlists rotate **weekly**, not per pass. A pass runs every
+few hours, and a playlist that reorders itself each time is one that has to be
+re-imported each time — the Music app imports a snapshot of a file, it does not
+track it.
+
+### Moods
+
+A second set of playlists selects by **Last.fm's crowd tags** rather than by
+listening history. MusicBrainz genres arrive free with the Lidarr index and are
+no use for this: they are sparse and formal, and will not tell you a record is
+screamo or synthwave. People typing tags will.
+
+Tags are fetched once per artist — one `artist.getTopTags` call each — and
+refreshed every ninety days. `--tag-limit` spreads the first sweep over several
+passes.
+
+Artists are looked up **by name**, not by MusicBrainz id, despite Lidarr having
+an id for every one of them. Last.fm's mbid index is stale and partial: it
+answers "the artist you supplied could not be found" for Devo, Escape the Fate,
+Blasterjaxx and a few hundred others whose pages plainly exist and carry exactly
+the tags wanted. Its name index is the one its own site runs on. The id is kept
+only as a fallback, for a name Lidarr spells differently.
+
+An artist neither key resolves is recorded as fetched with no tags, so the next
+pass does not spend a request on it again. A genuine failure — a rate limit, a
+bad key — is *not* recorded, so that one is retried.
+
+The built-in moods are built from the **tag distribution of this library**,
+measured, rather than from a general taxonomy:
+
+| Mood | Selected on |
+| --------------- | -------------------------------------------------------- |
+| `drum-and-bass` | drum and bass and its six spellings, liquid funk, neurofunk, jungle, techstep, hospital records |
+| `bass` | dubstep, brostep, grime, trip-hop, big beat |
+| `dance` | house and its variants, trance, techno, electro, rave |
+| `pop-punk` | pop punk, punk, emo, emocore, easycore, power pop |
+| `screamo` | screamo, post-hardcore, metalcore, melodic hardcore, trancecore |
+| `heavy-metal` | heavy metal, thrash, speed, power, death, prog, NWOBHM |
+| `hair-metal` | hair metal, glam metal, glam rock, arena rock, AOR — 1975-1994 |
+| `nu-metal` | nu metal, alternative metal, rapcore, industrial |
+| `classic-rock` | classic rock, prog, psychedelic, blues rock, 70s, 60s |
+| `80s-synths` | 80s, new wave, synth pop, electropop, post-punk — 1975-1992, rock excluded |
+| `indie` | indie, indie rock, indie pop, britpop, singer-songwriter |
+
+Measuring first mattered. `synthwave`, `edm`, `big room` and `hardstyle` are
+plausible tags that carry **nothing at all** here, while `techstep`, `easycore`
+and `hospital records` carry real weight. Guessing produces the first list.
+
+Three kinds of tag are never used, and there is a test enforcing it:
+
+- **Nationality** — `american` alone spans 241 artists. A passport is not a
+  sound.
+- **`rock` and `electronic`** — 340 and 275 artists, most of the library. A
+  mood that matches everything is not a mood.
+- **Artist names** — Last.fm's most popular tag for an artist is frequently
+  their own name. `green day`, `paramore` and `queen` are single-artist
+  playlists waiting to happen.
+
+### Exclusions
+
+A mood may also list `exclude`. An excluded tag drops the artist outright rather
+than docking their score, and it exists because `80s-synths` cannot be written
+any other way.
+
+`80s` is the eleventh most-played tag here, and it sits on Def Leppard and Bon
+Jovi exactly as heavily as on Eurythmics. Weighting cannot separate them,
+because the tag it would weight is the one they share. What does separate them
+is that the stadium rock also carries `hard rock` and `hair metal`, and the
+synth acts do not.
+
+Checked against live Last.fm pages, since the tag census only sees artists
+already in the library:
+
+| Artist | Tags |
+| --- | --- |
+| Eurythmics | `80s`, `new wave`, `pop`, `female vocalists`, `synth pop` |
+| Frankie Goes to Hollywood | `80s`, `new wave`, `pop`, `british`, `dance` |
+| Depeche Mode | `electronic`, `synthpop`, `new wave`, `80s`, `synth pop` |
+| Duran Duran | `new wave`, `80s`, `pop`, `synth pop`, `rock` |
+
+Four of Eurythmics' five tags are ones no mood may use. Three of the four
+artists spell it **`synth pop`** with a space; only one spells it `synthpop`.
+Guessing one spelling would have missed most of the canon.
+
+An artist qualifies when their tag weights inside a mood sum to at least 30 out
+of Last.fm's 0-100 scale. One low-weight tag is not a genre, it is somebody's
+stray opinion.
+
+`years` filters on the album's release date, which is what separates eighties
+synth records from everything else a synthpop tag drags in.
+
+`--vibes` replaces the whole set with a JSON file of the same shape, so a new
+mood does not need a new release:
+
+```json
+[{ "name": "shoegaze", "tags": ["shoegaze", "dream pop"], "min_score": 40 }]
+```
+
+Names are validated when the file is read, not when the file is written. A bad
+one would otherwise surface as a playlist created somewhere unintended.
+
+### Paths
+
+Lidarr knows where the lossless source is; the playlists have to point at the
+MP3s music-mirror made from it. The mapping strips a library root from Lidarr's
+track paths and re-roots them under the mirror, with the suffix changed.
+
+`--library-root` is derived from the common parent of the indexed artist folders
+when unset, so it agrees with Lidarr by construction rather than by being kept
+in step by hand. Override it if that guess is wrong.
+
+Entries are written **relative to the playlist file**, so one playlist works
+from the NAS, from a Mac over SMB, and from Linux, without rewriting.
+
+Each playlist is given the **owner and group of the mirror** it is written
+into. The image runs as root by default so that a bind mount of any ownership
+stays writable, and the cost of that is output owned by root — which the account
+serving the share cannot read, group bit or no group bit, because the group is
+also root. Copying the mirror's own ownership avoids having to be told what it
+should be, and does nothing when the two already agree.
+
+A track is only listed once its mirror file has been confirmed to exist. Lidarr
+holding the FLAC says nothing about whether the MP3 has been encoded yet. If a
+large number are missing, the run says so — that is what a wrong `--library-root`
+or `--mirror` looks like, since the paths then map to nothing at all.
+
+`_playlists/` survives music-mirror's prune: it only deletes `*.mp3`, and its
+empty-directory sweep skips a directory holding M3Us.
 
 ## How the ingest works
 
@@ -76,6 +328,14 @@ music-curator --report-only   # report on the store, fetch nothing
 | `--interval` | `MUSIC_CURATOR_INTERVAL` | unset | Repeat forever, e.g. `45m`, `6h`, `1d` |
 | `--request-delay` | `MUSIC_CURATOR_REQUEST_DELAY` | `0.25` | Seconds between API requests |
 | `--backfill-limit` | `MUSIC_CURATOR_BACKFILL_LIMIT` | `0` | Cap backfill requests per pass; 0 for no cap |
+| `--lidarr-url` | `MUSIC_CURATOR_LIDARR_URL` | unset | Lidarr base URL, e.g. `http://lidarr:8686` |
+| `--lidarr-api-key` | `MUSIC_CURATOR_LIDARR_API_KEY` | unset | Lidarr API key |
+| `--mirror` | `MUSIC_CURATOR_MIRROR` | unset | Root of the MP3 mirror; playlists go here |
+| `--library-root` | `MUSIC_CURATOR_LIBRARY_ROOT` | derived | Prefix to strip from Lidarr's paths |
+| `--playlist-limit` | `MUSIC_CURATOR_PLAYLIST_LIMIT` | `100` | Most tracks in any one playlist |
+| `--vibes` | `MUSIC_CURATOR_VIBES` | built-in | JSON file of mood definitions |
+| `--tag-limit` | `MUSIC_CURATOR_TAG_LIMIT` | `0` | Cap artist tag lookups per pass |
+| `--skip-index` | — | off | Match against the index already held |
 | `--report-only` | — | off | Report without fetching |
 
 A [Last.fm API key](https://www.last.fm/api/account/create) is all that is
@@ -107,21 +367,61 @@ docker build --target test .   # what CI runs
 pytest                         # needs pytest on PATH
 ```
 
-The suite runs against a fake transport that reproduces the real service's
-paging, its `from`/`to` semantics and its awkward response shapes. No network,
-no credentials, no rate limit. On a Nix machine:
+The suite runs against fake transports for both services. The Last.fm one
+reproduces its paging, its `from`/`to` semantics and its awkward response
+shapes; the Lidarr one serves a canned library split across the same four
+endpoints the indexer calls, so the stitching is exercised rather than
+stubbed. No network, no credentials, no rate limit. On a Nix machine:
 
 ```sh
 nix shell nixpkgs#python3Packages.pytest -c pytest
 ```
 
+## Tools
+
+Host-side scripts under `tools/`. Not part of the container image; run them
+wherever they are needed.
+
+### `find_missing_tracks.py`
+
+Reports tracks in an Apple Music library whose files are no longer on disk —
+which happens whenever Lidarr renames an artist or album folder and
+music-mirror prunes the old path.
+
+```sh
+# Music: File > Library > Export Library... then, with the share mounted:
+python3 tools/find_missing_tracks.py Library.xml --root /Volumes/music-mp3
+```
+
+Reading the exported XML rather than asking Music itself is deliberate. A
+broken track makes AppleScript's `location` raise instead of returning a value,
+so a bulk query dies on the first one with `-1728` and a per-track loop costs an
+Apple event apiece.
+
+It checks against a **single directory walk**, not a test per file. On a
+50,000-track library that is ~7,000 directory reads instead of 50,000 stat
+calls, and over SMB every one of those stats is a network round trip.
+
+Two comparisons that have to be loosened, or most of the library reads as
+missing:
+
+- **Unicode.** macOS stores filenames decomposed; the share composes them.
+  `Mötley Crüe` is two different byte strings depending on which side wrote it.
+  Both sides are normalised to NFC.
+- **Case.** The share is very likely case-insensitive. A file is not missing
+  because someone capitalised it differently.
+
+Pass `--root` if the library holds anything outside the mirror: the root it
+otherwise derives is the common parent of every track, which can be `/`.
+
 ## Where this is going
 
 | Stage | Status |
 | ------------------------------------------------ | ------------ |
 | Last.fm ingest and store | done |
-| Lidarr index and the scrobble-to-track matcher | next |
-| M3U playlists written into the mirror | after that |
+| Lidarr index and the scrobble-to-track matcher | done |
+| M3U playlists from the listening history | done |
+| Genre and mood playlists from Last.fm tags | done |
 | Cold-music report, unmonitoring what is not played | last |
 
 The cull will unmonitor cold albums in Lidarr and tag their artists. It will

compose.yaml

@@ -17,6 +17,10 @@ services:
       # enough, none of the endpoints used here authenticate a user.
       MUSIC_CURATOR_LASTFM_API_KEY: set-me-in-the-truenas-ui
       MUSIC_CURATOR_DB: /data/curator.db
+      # Lidarr, for indexing the library. Read-only: every call is a GET.
+      # Leave unset to ingest scrobbles and nothing else.
+      MUSIC_CURATOR_LIDARR_URL: http://lidarr:8686
+      MUSIC_CURATOR_LIDARR_API_KEY: set-me-in-the-truenas-ui
       # How long to wait between passes. Each one catches up on new scrobbles
       # and continues the backfill if it has not finished.
       MUSIC_CURATOR_INTERVAL: 6h
@@ -26,5 +30,11 @@ services:
       # Cap the backfill at this many requests per pass. Unlimited by default,
       # which finishes a long history in one go.
       # MUSIC_CURATOR_BACKFILL_LIMIT: "0"
+      # The MP3 mirror music-mirror maintains. Playlists are written into
+      # _playlists/ inside it; leave unset to skip them.
+      MUSIC_CURATOR_MIRROR: /mirror
+      # Most tracks in any one playlist.
+      # MUSIC_CURATOR_PLAYLIST_LIMIT: "100"
     volumes:
       - /mnt/tank/apps/music-curator:/data
+      - /mnt/tank/media/music-mp3:/mirror

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "music-curator"
-version = "0.1.0"
+version = "0.6.0"
 description = "Ingest a Last.fm listening history and curate a music library from it"
 readme = "README.md"
 requires-python = ">=3.11"

tests/conftest.py

@@ -1,6 +1,10 @@
+import http.server
+import io
 import json
 import os
 import sys
+import threading
+import urllib.error
 import urllib.parse
 
 import pytest
@@ -52,7 +56,9 @@ class FakeLastfm:
     is prepended to the first page with no `date`.
     """
 
-    def __init__(self, tracks=(), loved=(), nowplaying=None, outcomes=()):
+    def __init__(self, tracks=(), loved=(), nowplaying=None, outcomes=(), tags=None):
+        # Keyed by mbid or by artist name, whichever the caller asked with.
+        self.tags = tags or {}
         self.tracks = sorted(tracks, key=lambda track: int(track["date"]["uts"]), reverse=True)
         self.loved = list(loved)
         self.nowplaying = nowplaying
@@ -78,6 +84,15 @@ class FakeLastfm:
             return json.dumps(self._recent(query))
         if method == "user.getlovedtracks":
             return json.dumps(self._loved(query))
+        if method == "artist.gettoptags":
+            key = query.get("mbid") or query.get("artist", "")
+            if key not in self.tags:
+                # What the real service says for a key it cannot resolve, which
+                # for mbids is a great many artists whose pages plainly exist.
+                return json.dumps(
+                    {"error": 6, "message": "The artist you supplied could not be found"}
+                )
+            return json.dumps({"toptags": {"tag": self.tags[key]}})
         raise AssertionError(f"unexpected method {method}")
 
     def _recent(self, query):
@@ -120,6 +135,115 @@ class FakeLastfm:
         }
 
 
+class FakeLidarr:
+    """A transport serving a canned library over the Lidarr v1 API surface.
+
+    Built from a nested description -- artist, album, tracks -- and split back
+    out across the four endpoints the indexer actually calls, so the tests
+    exercise the same stitching the real thing does.
+    """
+
+    def __init__(self, artists=(), fail=()):
+        # (path, artistId) pairs the fake refuses to serve, standing in for the
+        # 500s Lidarr returns on data it cannot hydrate.
+        self.fail = set(fail)
+        self.artists, self.albums, self.tracks, self.files = [], [], [], []
+        for artist_index, entry in enumerate(artists, start=1):
+            artist_id = artist_index
+            self.artists.append(
+                {
+                    "id": artist_id,
+                    "artistName": entry["name"],
+                    "foreignArtistId": entry.get("mbid", f"artist-mbid-{artist_id}"),
+                    "path": f"/music/{entry['name']}",
+                    "monitored": entry.get("monitored", True),
+                }
+            )
+            for album_index, album in enumerate(entry.get("albums", []), start=1):
+                album_id = artist_id * 100 + album_index
+                self.albums.append(
+                    {
+                        "id": album_id,
+                        "artistId": artist_id,
+                        "title": album["title"],
+                        "foreignAlbumId": f"album-mbid-{album_id}",
+                        "monitored": album.get("monitored", True),
+                        "releaseDate": album.get("release_date", "2019-01-01T00:00:00Z"),
+                    }
+                )
+                for track_index, track in enumerate(album.get("tracks", []), start=1):
+                    track_id = album_id * 100 + track_index
+                    has_file = track.get("has_file", True)
+                    self.tracks.append(
+                        {
+                            "id": track_id,
+                            "artistId": artist_id,
+                            "albumId": album_id,
+                            "title": track["title"],
+                            "foreignRecordingId": track.get("recording_mbid", ""),
+                            "trackFileId": track_id if has_file else 0,
+                            "hasFile": has_file,
+                            "duration": 210000,
+                        }
+                    )
+                    if has_file:
+                        self.files.append(
+                            {
+                                "id": track_id,
+                                "artistId": artist_id,
+                                "albumId": album_id,
+                                "path": f"/music/{entry['name']}/{album['title']}/"
+                                f"{track['title']}.flac",
+                                "dateAdded": track.get("added", "2020-05-01T12:00:00Z"),
+                            }
+                        )
+        self.calls = []
+
+    def __call__(self, url, timeout=None, headers=None):
+        parsed = urllib.parse.urlparse(url)
+        assert (headers or {}).get("X-Api-Key"), "Lidarr requires the API key header"
+        path = parsed.path.rsplit("/", 1)[-1]
+        query = {
+            key: value[0] for key, value in urllib.parse.parse_qs(parsed.query).items()
+        }
+        self.calls.append((path, query))
+
+        artist_id = int(query.get("artistId", 0))
+        if (path, artist_id) in self.fail:
+            raise urllib.error.HTTPError(
+                url, 500, "Internal Server Error", {}, io.BytesIO(b'{"message": "boom"}')
+            )
+
+        album_ids = query.get("albumIds")
+        if path == "album" and album_ids:
+            album_id = int(album_ids)
+            if ("albumid", album_id) in self.fail:
+                raise urllib.error.HTTPError(
+                    url,
+                    500,
+                    "Internal Server Error",
+                    {},
+                    io.BytesIO(b'{"message": "Sequence contains more than one element"}'),
+                )
+            return json.dumps([row for row in self.albums if row["id"] == album_id])
+
+        if path == "artist":
+            return json.dumps(self.artists)
+        source = {"album": self.albums, "track": self.tracks, "trackfile": self.files}[path]
+        if path == "album" and not artist_id:
+            # Lidarr's unfiltered album endpoint returns the lot.
+            return json.dumps(source)
+        if not artist_id:
+            raise urllib.error.HTTPError(
+                url,
+                400,
+                "Bad Request",
+                {},
+                io.BytesIO(b'{"message": "artistId must be provided"}'),
+            )
+        return json.dumps([row for row in source if row["artistId"] == artist_id])
+
+
 @pytest.fixture
 def now_playing():
     """The entry Last.fm prepends for a track in progress: no `date` at all."""
@@ -130,3 +254,53 @@ def now_playing():
         "mbid": "",
         "@attr": {"nowplaying": "true"},
     }
+
+
+class _CountingServer(http.server.ThreadingHTTPServer):
+    """Counts accepted connections, which is what connection reuse is about."""
+
+    daemon_threads = True
+
+    def __init__(self, *args, **kwargs):
+        self.connections = 0
+        super().__init__(*args, **kwargs)
+
+    def process_request(self, request, client_address):
+        self.connections += 1
+        super().process_request(request, client_address)
+
+
+class _Handler(http.server.BaseHTTPRequestHandler):
+    # Without HTTP/1.1 the server closes after every response and no client
+    # could reuse anything, which would make the test prove nothing.
+    protocol_version = "HTTP/1.1"
+
+    def do_GET(self):
+        if self.path.startswith("/boom"):
+            body, status = b'{"message": "boom"}', 500
+        else:
+            body = json.dumps(
+                {"path": self.path, "key": self.headers.get("X-Api-Key")}
+            ).encode()
+            status = 200
+        self.send_response(status)
+        self.send_header("Content-Type", "application/json")
+        self.send_header("Content-Length", str(len(body)))
+        self.end_headers()
+        self.wfile.write(body)
+
+    def log_message(self, *args):
+        pass
+
+
+@pytest.fixture
+def http_server():
+    """A real local HTTP server, for the one component that talks sockets."""
+    server = _CountingServer(("127.0.0.1", 0), _Handler)
+    thread = threading.Thread(target=server.serve_forever, daemon=True)
+    thread.start()
+    try:
+        yield server, f"http://127.0.0.1:{server.server_port}"
+    finally:
+        server.shutdown()
+        server.server_close()

tests/test_find_missing_tracks.py

@@ -0,0 +1,168 @@
+import os
+import plistlib
+import sys
+import unicodedata
+import urllib.parse
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
+
+import find_missing_tracks  # noqa: E402
+
+
+def write_library(path, tracks):
+    """Write a Library.xml in the shape Music exports."""
+    path.write_bytes(plistlib.dumps({"Tracks": {str(i): t for i, t in enumerate(tracks)}}))
+    return path
+
+
+def track(root, name, filename=None):
+    """A local track entry. Omit filename for a streaming entry with no file."""
+    entry = {"Name": name, "Artist": "An Artist", "Album": "An Album"}
+    if filename is not None:
+        entry["Location"] = "file://" + urllib.parse.quote(str(root / filename))
+    return entry
+
+
+def missing_from(library, roots):
+    with open(library, "rb") as handle:
+        loaded = plistlib.load(handle)
+    present = find_missing_tracks.existing_under(roots)
+    gone = []
+    for entry in loaded["Tracks"].values():
+        path = find_missing_tracks.location_of(entry)
+        if path is not None and find_missing_tracks.key_for(path) not in present:
+            gone.append(entry["Name"])
+    return sorted(gone)
+
+
+def test_a_deleted_file_is_reported(tmp_path):
+    (tmp_path / "kept.mp3").write_bytes(b"x")
+    library = write_library(
+        tmp_path / "Library.xml",
+        [track(tmp_path, "Kept", "kept.mp3"), track(tmp_path, "Gone", "gone.mp3")],
+    )
+
+    assert missing_from(library, [tmp_path]) == ["Gone"]
+
+
+def test_a_decomposed_filename_is_not_reported_missing(tmp_path):
+    """macOS stores filenames decomposed and most everything else composes them,
+    so the umlauts in Motley Crue are two different byte strings depending on
+    which side wrote the name."""
+    composed = unicodedata.normalize("NFC", "Mötley Crüe.mp3")
+    (tmp_path / composed).write_bytes(b"x")
+    library = write_library(
+        tmp_path / "Library.xml",
+        [track(tmp_path, "Umlauts", unicodedata.normalize("NFD", "Mötley Crüe.mp3"))],
+    )
+
+    assert missing_from(library, [tmp_path]) == []
+
+
+def test_a_case_difference_is_not_reported_missing(tmp_path):
+    """The share is very likely case-insensitive, and a file is not missing
+    because someone capitalised it differently."""
+    (tmp_path / "Hells Bells.mp3").write_bytes(b"x")
+    library = write_library(
+        tmp_path / "Library.xml", [track(tmp_path, "Bells", "hells bells.MP3")]
+    )
+
+    assert missing_from(library, [tmp_path]) == []
+
+
+def test_a_track_with_no_file_is_not_a_missing_file(tmp_path):
+    """A streaming entry has never had a file to lose."""
+    library = write_library(tmp_path / "Library.xml", [track(tmp_path, "Streamed", None)])
+
+    assert missing_from(library, [tmp_path]) == []
+
+
+def test_the_walk_finds_files_nested_below_the_root(tmp_path):
+    nested = tmp_path / "Artist" / "Album"
+    nested.mkdir(parents=True)
+    (nested / "deep.mp3").write_bytes(b"x")
+
+    found = find_missing_tracks.existing_under([tmp_path])
+
+    assert find_missing_tracks.key_for(nested / "deep.mp3") in found
+
+
+def test_the_root_is_derived_from_the_tracks(tmp_path):
+    paths = [tmp_path / "a" / "one.mp3", tmp_path / "b" / "two.mp3"]
+
+    assert find_missing_tracks.roots_of(paths, None) == [tmp_path]
+
+
+def test_an_explicit_root_overrides_the_derived_one(tmp_path):
+    """Worth using: tracks spread beyond the mirror can derive a common parent
+    of / and send the walk across the whole disk."""
+    paths = [tmp_path / "a" / "one.mp3"]
+
+    assert find_missing_tracks.roots_of(paths, ["/Volumes/music-mp3"]) == [
+        Path("/Volumes/music-mp3")
+    ]
+
+
+def test_the_whole_library_missing_is_called_out(tmp_path, capsys):
+    """Almost always an unmounted share rather than an empty library."""
+    library = write_library(
+        tmp_path / "Library.xml", [track(tmp_path / "elsewhere", "Gone", "gone.mp3")]
+    )
+
+    find_missing_tracks.main([str(library), "--root", str(tmp_path)])
+
+    assert "not mounted" in capsys.readouterr().err
+
+
+def test_a_healthy_library_says_nothing_alarming(tmp_path, capsys):
+    (tmp_path / "kept.mp3").write_bytes(b"x")
+    library = write_library(tmp_path / "Library.xml", [track(tmp_path, "Kept", "kept.mp3")])
+
+    find_missing_tracks.main([str(library), "--root", str(tmp_path)])
+
+    captured = capsys.readouterr()
+    assert "0 of 1 local tracks are missing" in captured.err
+    assert "not mounted" not in captured.err
+    assert captured.out == ""
+
+
+@pytest.mark.parametrize(
+    ("location", "expected"),
+    [
+        ("file:///music/a%20b.mp3", Path("/music/a b.mp3")),
+        ("file:///music/plain.mp3", Path("/music/plain.mp3")),
+        ("https://example.invalid/stream", None),
+        ("", None),
+    ],
+)
+def test_locations_are_decoded(location, expected):
+    assert find_missing_tracks.location_of({"Location": location} if location else {}) == expected
+
+
+def test_the_walk_costs_one_read_per_directory_not_one_per_file(tmp_path):
+    """The whole point. Over SMB a per-file check is a round trip per track."""
+    for album in range(20):
+        directory = tmp_path / f"Album {album}"
+        directory.mkdir()
+        for index in range(25):
+            (directory / f"{index}.mp3").write_bytes(b"x")
+
+    reads = {"n": 0}
+    real_walk = os.walk
+
+    def counting_walk(*args, **kwargs):
+        for entry in real_walk(*args, **kwargs):
+            reads["n"] += 1
+            yield entry
+
+    find_missing_tracks.os.walk = counting_walk
+    try:
+        found = find_missing_tracks.existing_under([tmp_path])
+    finally:
+        find_missing_tracks.os.walk = real_walk
+
+    assert len(found) == 500
+    assert reads["n"] == 21  # the root and its twenty albums, not 500 files

tools/find_missing_tracks.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+"""Report tracks in a Music library whose files are no longer on disk.
+
+Reads the XML from Music's File > Library > Export Library. Asking Music itself
+does not work: a broken track makes AppleScript's `location` raise rather than
+return a value, so a bulk query dies on the first one with error -1728, and a
+per-track loop costs an Apple event apiece.
+
+The library is checked against a single directory walk rather than by testing
+each file. Over SMB a per-file test is one network round trip per track --
+fifty thousand of them -- where a walk reads each directory once and gets every
+name in it back at once.
+"""
+
+import argparse
+import os
+import plistlib
+import sys
+import unicodedata
+import urllib.parse
+from pathlib import Path
+
+
+def location_of(track):
+    """Return the filesystem path a track points at, or None if it has none."""
+    location = track.get("Location")
+    if not location or not location.startswith("file://"):
+        return None
+    return Path(urllib.parse.unquote(urllib.parse.urlparse(location).path))
+
+
+def key_for(path):
+    """Return a comparison key for a path.
+
+    Normalised to NFC because macOS stores filenames decomposed and most
+    everything else composes them, so "Motley Crue" with its umlauts is two
+    different byte strings depending on which side wrote it. Casefolded because
+    the share is very likely case-insensitive and a file is not missing merely
+    because someone capitalised it differently.
+    """
+    return unicodedata.normalize("NFC", str(path)).casefold()
+
+
+def existing_under(roots):
+    """Return every file below the given roots, keyed for comparison."""
+    found = set()
+    for root in roots:
+        for base, _, names in os.walk(root):
+            for name in names:
+                found.add(key_for(os.path.join(base, name)))
+    return found
+
+
+def roots_of(paths, given):
+    """Return the directories worth walking."""
+    if given:
+        return [Path(root) for root in given]
+    try:
+        return [Path(os.path.commonpath([str(path) for path in paths]))]
+    except ValueError:
+        # Tracks spread across separate volumes have no common parent.
+        return sorted({path.parents[-2] for path in paths if len(path.parents) > 1})
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("library", help="Library.xml exported from Music")
+    parser.add_argument("--root", action="append", help="directory to scan; repeatable")
+    parser.add_argument("--limit", type=int, default=0, help="show at most this many")
+    args = parser.parse_args(argv)
+
+    with open(args.library, "rb") as handle:
+        library = plistlib.load(handle)
+
+    tracks = [
+        (track, location_of(track)) for track in library.get("Tracks", {}).values()
+    ]
+    local = [(track, path) for track, path in tracks if path is not None]
+    if not local:
+        print("no local files in this library", file=sys.stderr)
+        return 0
+
+    roots = roots_of([path for _, path in local], args.root)
+    print(f"scanning {', '.join(str(root) for root in roots)}", file=sys.stderr)
+    present = existing_under(roots)
+
+    gone = [(track, path) for track, path in local if key_for(path) not in present]
+    for track, path in gone[: args.limit or None]:
+        print(
+            "\t".join(
+                (
+                    track.get("Artist", ""),
+                    track.get("Album", ""),
+                    track.get("Name", ""),
+                    str(path),
+                )
+            )
+        )
+
+    print(
+        f"\n{len(gone)} of {len(local)} local tracks are missing their file"
+        f" ({len(tracks) - len(local)} have no file at all)",
+        file=sys.stderr,
+    )
+    if gone and len(gone) == len(local):
+        print(
+            "Every single one is missing, which almost certainly means the share"
+            " is not mounted rather than that the library is empty.",
+            file=sys.stderr,
+        )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())