chore: add a tool for finding Music tracks whose files have gone

Lidarr renames an artist or album folder, music-mirror prunes the old path and
encodes the new one, and every entry in Apple Music pointing at the old path
dies. Music offers no view of those, and the exclamation mark only appears once
a track is touched.

Reads the XML from Music's own Export Library rather than asking Music itself.
A broken track makes AppleScript's `location` raise instead of returning a
value, so a bulk query dies on the first one with error -1728 and a per-track
loop costs an Apple event apiece.

Checked against a single directory walk rather than a test per file. On a
fifty-thousand-track library that is around seven thousand directory reads
instead of fifty thousand stat calls, and over SMB each of those stats is a
network round trip -- which is the difference between seconds and minutes.

Two comparisons have to be loosened or most of the library reads as missing.
macOS stores filenames decomposed while the share composes them, so the umlauts
in Motley Crue are two different byte strings depending on which side wrote the
name; both are normalised to NFC. And the share is very likely
case-insensitive, so a file is not missing because someone capitalised it
differently.

The test stage now copies tools/ as well, since the suite covers this and the
runtime image deliberately does not carry it.

Dockerfile

@@ -28,5 +28,8 @@ FROM runtime AS test
 
 RUN pip install --no-cache-dir pytest
 COPY pytest.ini ./
+# The host-side tools are not part of the runtime image, but their tests are
+# part of the suite, so they have to be present for it.
+COPY tools ./tools
 COPY tests ./tests
 RUN python -m pytest

README.md

@@ -159,23 +159,77 @@ listening history. MusicBrainz genres arrive free with the Lidarr index and are
 no use for this: they are sparse and formal, and will not tell you a record is
 screamo or synthwave. People typing tags will.
 
-Tags are fetched once per artist — one `artist.getTopTags` call each, by
-MusicBrainz id where Lidarr has one — and refreshed every ninety days. An
-artist Last.fm has never heard of is recorded as fetched with no tags, so it is
-not asked about again on every pass. `--tag-limit` spreads the first sweep over
-several passes.
+Tags are fetched once per artist — one `artist.getTopTags` call each — and
+refreshed every ninety days. `--tag-limit` spreads the first sweep over several
+passes.
 
-The built-in moods are chosen for this library rather than as a general
-taxonomy:
+Artists are looked up **by name**, not by MusicBrainz id, despite Lidarr having
+an id for every one of them. Last.fm's mbid index is stale and partial: it
+answers "the artist you supplied could not be found" for Devo, Escape the Fate,
+Blasterjaxx and a few hundred others whose pages plainly exist and carry exactly
+the tags wanted. Its name index is the one its own site runs on. The id is kept
+only as a fallback, for a name Lidarr spells differently.
+
+An artist neither key resolves is recorded as fetched with no tags, so the next
+pass does not spend a request on it again. A genuine failure — a rate limit, a
+bad key — is *not* recorded, so that one is retried.
+
+The built-in moods are built from the **tag distribution of this library**,
+measured, rather than from a general taxonomy:
 
 | Mood | Selected on |
-| ------------------ | ------------------------------------------------- |
-| `80s-synths` | synthpop, new wave, synthwave — released 1975-1992 |
-| `high-energy-rock` | hard rock, punk, pop punk, alternative |
-| `screamo` | screamo, post-hardcore, metalcore, emo |
-| `drum-and-bass` | drum and bass, liquid funk, neurofunk, jungle |
-| `dance` | house, big room, hardstyle, trance, dubstep |
-| `classic-rock` | classic rock, prog, psychedelic, blues rock |
+| --------------- | -------------------------------------------------------- |
+| `drum-and-bass` | drum and bass and its six spellings, liquid funk, neurofunk, jungle, techstep, hospital records |
+| `bass` | dubstep, brostep, grime, trip-hop, big beat |
+| `dance` | house and its variants, trance, techno, electro, rave |
+| `pop-punk` | pop punk, punk, emo, emocore, easycore, power pop |
+| `screamo` | screamo, post-hardcore, metalcore, melodic hardcore, trancecore |
+| `heavy-metal` | heavy metal, thrash, speed, power, death, prog, NWOBHM |
+| `hair-metal` | hair metal, glam metal, glam rock, arena rock, AOR — 1975-1994 |
+| `nu-metal` | nu metal, alternative metal, rapcore, industrial |
+| `classic-rock` | classic rock, prog, psychedelic, blues rock, 70s, 60s |
+| `80s-synths` | 80s, new wave, synth pop, electropop, post-punk — 1975-1992, rock excluded |
+| `indie` | indie, indie rock, indie pop, britpop, singer-songwriter |
+
+Measuring first mattered. `synthwave`, `edm`, `big room` and `hardstyle` are
+plausible tags that carry **nothing at all** here, while `techstep`, `easycore`
+and `hospital records` carry real weight. Guessing produces the first list.
+
+Three kinds of tag are never used, and there is a test enforcing it:
+
+- **Nationality** — `american` alone spans 241 artists. A passport is not a
+  sound.
+- **`rock` and `electronic`** — 340 and 275 artists, most of the library. A
+  mood that matches everything is not a mood.
+- **Artist names** — Last.fm's most popular tag for an artist is frequently
+  their own name. `green day`, `paramore` and `queen` are single-artist
+  playlists waiting to happen.
+
+### Exclusions
+
+A mood may also list `exclude`. An excluded tag drops the artist outright rather
+than docking their score, and it exists because `80s-synths` cannot be written
+any other way.
+
+`80s` is the eleventh most-played tag here, and it sits on Def Leppard and Bon
+Jovi exactly as heavily as on Eurythmics. Weighting cannot separate them,
+because the tag it would weight is the one they share. What does separate them
+is that the stadium rock also carries `hard rock` and `hair metal`, and the
+synth acts do not.
+
+Checked against live Last.fm pages, since the tag census only sees artists
+already in the library:
+
+| Artist | Tags |
+| --- | --- |
+| Eurythmics | `80s`, `new wave`, `pop`, `female vocalists`, `synth pop` |
+| Frankie Goes to Hollywood | `80s`, `new wave`, `pop`, `british`, `dance` |
+| Depeche Mode | `electronic`, `synthpop`, `new wave`, `80s`, `synth pop` |
+| Duran Duran | `new wave`, `80s`, `pop`, `synth pop`, `rock` |
+
+Four of Eurythmics' five tags are ones no mood may use. Three of the four
+artists spell it **`synth pop`** with a space; only one spells it `synthpop`.
+Guessing one spelling would have missed most of the canon.
 
 An artist qualifies when their tag weights inside a mood sum to at least 30 out
 of Last.fm's 0-100 scale. One low-weight tag is not a genre, it is somebody's
@@ -207,6 +261,13 @@ in step by hand. Override it if that guess is wrong.
 Entries are written **relative to the playlist file**, so one playlist works
 from the NAS, from a Mac over SMB, and from Linux, without rewriting.
 
+Each playlist is given the **owner and group of the mirror** it is written
+into. The image runs as root by default so that a bind mount of any ownership
+stays writable, and the cost of that is output owned by root — which the account
+serving the share cannot read, group bit or no group bit, because the group is
+also root. Copying the mirror's own ownership avoids having to be told what it
+should be, and does nothing when the two already agree.
+
 A track is only listed once its mirror file has been confirmed to exist. Lidarr
 holding the FLAC says nothing about whether the MP3 has been encoded yet. If a
 large number are missing, the run says so — that is what a wrong `--library-root`
@@ -316,6 +377,43 @@ stubbed. No network, no credentials, no rate limit. On a Nix machine:
 nix shell nixpkgs#python3Packages.pytest -c pytest
 ```
 
+## Tools
+
+Host-side scripts under `tools/`. Not part of the container image; run them
+wherever they are needed.
+
+### `find_missing_tracks.py`
+
+Reports tracks in an Apple Music library whose files are no longer on disk —
+which happens whenever Lidarr renames an artist or album folder and
+music-mirror prunes the old path.
+
+```sh
+# Music: File > Library > Export Library... then, with the share mounted:
+python3 tools/find_missing_tracks.py Library.xml --root /Volumes/music-mp3
+```
+
+Reading the exported XML rather than asking Music itself is deliberate. A
+broken track makes AppleScript's `location` raise instead of returning a value,
+so a bulk query dies on the first one with `-1728` and a per-track loop costs an
+Apple event apiece.
+
+It checks against a **single directory walk**, not a test per file. On a
+50,000-track library that is ~7,000 directory reads instead of 50,000 stat
+calls, and over SMB every one of those stats is a network round trip.
+
+Two comparisons that have to be loosened, or most of the library reads as
+missing:
+
+- **Unicode.** macOS stores filenames decomposed; the share composes them.
+  `Mötley Crüe` is two different byte strings depending on which side wrote it.
+  Both sides are normalised to NFC.
+- **Case.** The share is very likely case-insensitive. A file is not missing
+  because someone capitalised it differently.
+
+Pass `--root` if the library holds anything outside the mirror: the root it
+otherwise derives is the common parent of every track, which can be `/`.
+
 ## Where this is going
 
 | Stage | Status |

music_curator.py

@@ -266,7 +266,16 @@ WHITESPACE = re.compile(r"\s+")
 
 
 class LastfmError(Exception):
-    """A Last.fm request that failed in a way retrying will not fix."""
+    """A Last.fm request that failed in a way retrying will not fix.
+
+    `code` is the service's own error number where the failure came from the
+    API rather than the transport. Callers need it to tell "this thing does not
+    exist", which is final, from "something went wrong", which is not.
+    """
+
+    def __init__(self, message, code=None):
+        super().__init__(message)
+        self.code = code
 
 
 class LidarrError(Exception):
@@ -447,7 +456,7 @@ class Lastfm:
                 return payload
             detail = f"error {code}: {payload.get('message', '')}".strip()
             if code not in RETRYABLE_ERRORS:
-                raise LastfmError(f"{method}: {detail}")
+                raise LastfmError(f"{method}: {detail}", code=code)
             self._retry_or_raise(method, attempt, detail, None)
 
         raise LastfmError(f"{method}: gave up after {self.attempts} attempts")
@@ -1233,49 +1242,108 @@ PLAYLISTS = (
 # `years` filters on the album's release date, which is what separates eighties
 # synth records from everything a synthpop tag would otherwise drag in.
 DEFAULT_VIBES = (
+    # Built from the tag distribution of this library rather than from a general
+    # taxonomy, which is why some obvious-looking tags are absent and some
+    # unobvious ones are here. `synthwave`, `edm`, `big room` and `hardstyle`
+    # carry nothing at all; `techstep`, `easycore` and `hospital records` carry
+    # real weight.
+    #
+    # Three kinds of tag are deliberately never used. Nationality -- american,
+    # british, swedish -- describes a passport, not a sound, and `american`
+    # alone spans 241 artists. `rock` and `electronic` span 340 and 275, which
+    # is most of the library and therefore no mood at all. And Last.fm's most
+    # popular tag for an artist is frequently their own name, so `green day`,
+    # `paramore` and `queen` are single-artist playlists waiting to happen.
     {
-        "name": "80s-synths",
+        "name": "drum-and-bass",
         "tags": [
-            "synthpop", "synth pop", "synth-pop", "new wave", "synthwave",
-            "new romantic", "electropop", "80s", "1980s",
+            "drum and bass", "dnb", "drum n bass", "drum'n'bass", "drum & bass",
+            "drum 'n' bass", "liquid funk", "neurofunk", "jungle", "techstep",
+            "darkstep", "drumstep", "breakbeat", "hospital records",
         ],
-        "years": [1975, 1992],
     },
     {
-        "name": "high-energy-rock",
+        "name": "bass",
+        "tags": ["dubstep", "brostep", "grime", "trip-hop", "big beat"],
+    },
+    {
+        "name": "dance",
         "tags": [
-            "hard rock", "punk rock", "pop punk", "punk", "alternative rock",
-            "rock", "garage rock", "skate punk",
+            "house", "electro house", "progressive house", "tech house", "trance",
+            "techno", "electro", "rave", "dance", "minimal",
+        ],
+    },
+    {
+        "name": "pop-punk",
+        "tags": [
+            "pop punk", "pop-punk", "punk rock", "punk", "skate punk", "emo",
+            "emocore", "easycore", "powerpop", "power pop", "post-grunge",
         ],
     },
     {
         "name": "screamo",
         "tags": [
-            "screamo", "post-hardcore", "metalcore", "emo", "hardcore",
-            "melodic hardcore", "emocore",
+            "screamo", "post-hardcore", "metalcore", "melodic metalcore",
+            "melodic hardcore", "hardcore", "trancecore", "deathcore",
         ],
     },
     {
-        "name": "drum-and-bass",
+        "name": "heavy-metal",
         "tags": [
-            "drum and bass", "drum n bass", "dnb", "liquid funk", "neurofunk",
-            "jungle", "breakbeat",
+            "heavy metal", "metal", "thrash metal", "thrash", "speed metal",
+            "power metal", "death metal", "progressive metal", "nwobhm",
+            "classic metal",
         ],
     },
     {
-        "name": "dance",
+        "name": "hair-metal",
         "tags": [
-            "electro house", "house", "big room", "electronic dance music",
-            "edm", "hardstyle", "trance", "dubstep", "electro",
+            "hair metal", "glam metal", "glam rock", "arena rock", "aor",
+            "rock and roll", "rock n roll",
+        ],
+        "years": [1975, 1994],
+    },
+    {
+        "name": "nu-metal",
+        "tags": [
+            "nu metal", "nu-metal", "alternative metal", "rapcore",
+            "industrial metal", "industrial rock", "industrial",
         ],
     },
     {
         "name": "classic-rock",
         "tags": [
-            "classic rock", "progressive rock", "psychedelic rock",
-            "blues rock", "70s", "60s",
+            "classic rock", "progressive rock", "psychedelic rock", "psychedelic",
+            "blues rock", "blues", "southern rock", "art rock", "space rock",
+            "british invasion", "folk rock", "70s", "60s",
         ],
     },
+    {
+        "name": "80s-synths",
+        # `80s` is included, which on its own would drag in Def Leppard and Bon
+        # Jovi -- they carry it as heavily as Eurythmics does. The exclusion is
+        # what separates them: the stadium rock is also tagged hard rock and
+        # hair metal, and the synth acts are not. Weighting cannot do this,
+        # because the tag it would weight is the one they share.
+        #
+        # Both spellings of synth pop are listed. Of Depeche Mode, Duran Duran,
+        # Eurythmics and Frankie Goes to Hollywood, three carry "synth pop" with
+        # a space and only one carries "synthpop" without.
+        "tags": [
+            "80s", "new wave", "synth pop", "synthpop", "synth-pop", "synthwave",
+            "electropop", "new romantic", "post-punk", "post-punk revival",
+        ],
+        "exclude": [
+            "hard rock", "hair metal", "glam metal", "glam rock", "heavy metal",
+            "metal", "arena rock", "aor", "nwobhm", "thrash metal",
+            "classic rock", "southern rock", "blues rock",
+        ],
+        "years": [1975, 1992],
+    },
+    {
+        "name": "indie",
+        "tags": ["indie", "indie rock", "indie pop", "britpop", "singer-songwriter"],
+    },
 )
 
 
@@ -1301,6 +1369,13 @@ def load_vibes(path):
             raise ValueError(f"{path}: {name!r} is not a usable vibe name (a-z, 0-9, -)")
         if not vibe.get("tags"):
             raise ValueError(f"{path}: vibe {name!r} lists no tags")
+        overlap = {str(tag).casefold() for tag in vibe["tags"]} & {
+            str(tag).casefold() for tag in vibe.get("exclude", [])
+        }
+        if overlap:
+            raise ValueError(
+                f"{path}: vibe {name!r} both selects on and excludes {sorted(overlap)}"
+            )
     return tuple(loaded)
 
 
@@ -1323,6 +1398,34 @@ def parse_tags(payload):
     return pairs
 
 
+def fetch_tags(client, name, mbid):
+    """Return an artist's tags, asking by name first.
+
+    By name, not by MusicBrainz id, despite Lidarr having an id for everything.
+    Last.fm's mbid index is stale and partial -- it answers "the artist you
+    supplied could not be found" for Devo, Escape the Fate and a few hundred
+    others whose pages plainly exist -- while the name index is the one its own
+    site runs on. The id is kept only as a fallback for a name Lidarr spells
+    differently.
+
+    Returns an empty list when the artist is genuinely unknown, which the caller
+    records so it is not asked again.
+    """
+    attempts = [{"artist": name}] if name else []
+    if mbid:
+        attempts.append({"mbid": mbid})
+
+    for query in attempts:
+        try:
+            return parse_tags(client.call("artist.getTopTags", {**query, "autocorrect": 1}))
+        except LastfmError as error:
+            # Error 6 here means "no such artist", which the next key may still
+            # answer. Anything else is a real failure and belongs to the caller.
+            if error.code != 6:
+                raise
+    return []
+
+
 def sync_tags(client, store, now, limit=0):
     """Fetch crowd tags for library artists that have none, or stale ones.
 
@@ -1348,20 +1451,23 @@ def sync_tags(client, store, now, limit=0):
     else:
         logger.info("fetching tags for %d artists", len(stale))
 
-    tagged = 0
+    resolved = 0
+    unknown = 0
     for artist in stale:
-        query = {"mbid": artist["mbid"]} if artist["mbid"] else {"artist": artist["name"]}
         try:
-            payload = client.call("artist.getTopTags", {**query, "autocorrect": 1})
+            pairs = fetch_tags(client, artist["name"], artist["mbid"])
         except LastfmError as error:
-            # One artist Last.fm cannot answer for is not worth losing the pass.
+            # Something went wrong rather than the artist not existing. Left
+            # unrecorded on purpose, so the next pass tries again.
             logger.warning("no tags for %s: %s", artist["name"], error)
             continue
-        store.replace_tags(artist["norm_name"], parse_tags(payload), now)
-        tagged += 1
+        store.replace_tags(artist["norm_name"], pairs, now)
+        resolved += 1
+        if not pairs:
+            unknown += 1
 
-    logger.info("tagged %d artists", tagged)
-    return tagged
+    logger.info("tagged %d artists (%d with nothing to say about them)", resolved, unknown)
+    return resolved
 
 
 def build_vibe_playlists(store, vibes, mirror_root, library_root, limit, now):
@@ -1376,8 +1482,24 @@ def build_vibe_playlists(store, vibes, mirror_root, library_root, limit, now):
     for vibe in vibes:
         tags = [str(tag).strip().casefold() for tag in vibe["tags"]]
         placeholders = ",".join("?" * len(tags))
-        years = vibe.get("years")
         parameters = [*tags, vibe.get("min_score", VIBE_MIN_SCORE)]
+
+        # An exclusion drops the artist outright rather than docking their
+        # score. It is the only way to write "the eighties, but not the stadium
+        # rock": those artists carry `80s` as heavily as the synth acts do, so
+        # no amount of weighting separates them -- but they also carry `hard
+        # rock`, and the synth acts do not.
+        excluded = [str(tag).strip().casefold() for tag in vibe.get("exclude", [])]
+        exclude_clause = ""
+        if excluded:
+            exclude_clause = (
+                " AND NOT EXISTS (SELECT 1 FROM artist_tag x"
+                "                  WHERE x.norm_artist = a.norm_name"
+                f"                   AND x.tag IN ({','.join('?' * len(excluded))}))"
+            )
+            parameters += excluded
+
+        years = vibe.get("years")
         year_clause = ""
         if years:
             year_clause = (
@@ -1400,7 +1522,7 @@ def build_vibe_playlists(store, vibes, mirror_root, library_root, limit, now):
               JOIN lidarr_artist a ON a.id = t.artist_id
               JOIN vibe v ON v.norm_artist = a.norm_name
               LEFT JOIN lidarr_album al ON al.id = t.album_id
-             WHERE {PLAYABLE}{year_clause}
+             WHERE {PLAYABLE}{exclude_clause}{year_clause}
              ORDER BY ((t.id * {SHUFFLE_MULTIPLIER}) + ?) % {SHUFFLE_MODULUS}
              LIMIT ?
         """
@@ -1412,7 +1534,7 @@ def build_vibe_playlists(store, vibes, mirror_root, library_root, limit, now):
                 continue
             entries.append({**dict(row), "mirror": mirror})
 
-        write_playlist(directory / f"{vibe['name']}.m3u", entries)
+        write_playlist(directory / f"{vibe['name']}.m3u", entries, Path(mirror_root))
         total += len(entries)
         logger.info("playlist %-20s %4d tracks -- by tag", vibe["name"], len(entries))
 
@@ -1450,7 +1572,40 @@ def mirror_path_for(source, library_root, mirror_root):
     return (Path(mirror_root) / relative).with_suffix(MIRROR_SUFFIX)
 
 
-def write_playlist(path, entries):
+def set_ownership(path, uid, gid):
+    """Give a path an owner and group. Returns whether anything changed."""
+    try:
+        current = path.stat()
+        if (current.st_uid, current.st_gid) == (uid, gid):
+            return False
+        os.chown(path, uid, gid)
+    except OSError:
+        # Not permitted unless running as root, which is the case where the
+        # ownership is already whatever the caller runs as.
+        return False
+    return True
+
+
+def match_ownership(path, reference):
+    """Give a path the owner and group of the tree it is joining.
+
+    The image runs as root by default, so that a bind mount of any ownership
+    stays writable. The cost is that everything it writes comes out root-owned,
+    and a root-owned playlist inside a mirror owned by the apps account is
+    unreadable to the thing that serves it -- the group bit does not help when
+    the group is root.
+
+    Copying the mirror's own ownership avoids having to be told what it should
+    be, and is a no-op when the two already agree.
+    """
+    try:
+        wanted = reference.stat()
+    except OSError:
+        return False
+    return set_ownership(path, wanted.st_uid, wanted.st_gid)
+
+
+def write_playlist(path, entries, reference=None):
     """Write one extended M3U, atomically.
 
     Paths are relative to the playlist file, so the same playlist works from the
@@ -1462,7 +1617,11 @@ def write_playlist(path, entries):
         lines.append(f"#EXTINF:{seconds},{entry['artist']} - {entry['title']}")
         lines.append(os.path.relpath(entry["mirror"], path.parent))
 
+    fresh = not path.parent.exists()
     path.parent.mkdir(parents=True, exist_ok=True)
+    if fresh and reference is not None:
+        match_ownership(path.parent, reference)
+
     handle, temporary = tempfile.mkstemp(dir=path.parent, suffix=".m3u.part")
     os.close(handle)
     temporary = Path(temporary)
@@ -1473,6 +1632,10 @@ def write_playlist(path, entries):
         mode = temporary.stat().st_mode
         if not mode & GROUP_READ:
             temporary.chmod(mode | GROUP_READ)
+        # Before the rename, so the playlist is never briefly visible owned by
+        # the wrong account.
+        if reference is not None:
+            match_ownership(temporary, reference)
         os.replace(temporary, path)
     finally:
         temporary.unlink(missing_ok=True)
@@ -1503,7 +1666,7 @@ def build_playlists(store, mirror_root, library_root, limit, now):
                 missing += 1
                 continue
             entries.append({**dict(row), "mirror": mirror})
-        write_playlist(directory / f"{name}.m3u", entries)
+        write_playlist(directory / f"{name}.m3u", entries, Path(mirror_root))
         total += len(entries)
         logger.info("playlist %-20s %4d tracks -- %s", name, len(entries), description)
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "music-curator"
-version = "0.4.0"
+version = "0.6.0"
 description = "Ingest a Last.fm listening history and curate a music library from it"
 readme = "README.md"
 requires-python = ">=3.11"

tests/conftest.py

@@ -86,7 +86,13 @@ class FakeLastfm:
             return json.dumps(self._loved(query))
         if method == "artist.gettoptags":
             key = query.get("mbid") or query.get("artist", "")
-            return json.dumps({"toptags": {"tag": self.tags.get(key, [])}})
+            if key not in self.tags:
+                # What the real service says for a key it cannot resolve, which
+                # for mbids is a great many artists whose pages plainly exist.
+                return json.dumps(
+                    {"error": 6, "message": "The artist you supplied could not be found"}
+                )
+            return json.dumps({"toptags": {"tag": self.tags[key]}})
         raise AssertionError(f"unexpected method {method}")
 
     def _recent(self, query):

tests/test_find_missing_tracks.py

@@ -0,0 +1,168 @@
+import os
+import plistlib
+import sys
+import unicodedata
+import urllib.parse
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools"))
+
+import find_missing_tracks  # noqa: E402
+
+
+def write_library(path, tracks):
+    """Write a Library.xml in the shape Music exports."""
+    path.write_bytes(plistlib.dumps({"Tracks": {str(i): t for i, t in enumerate(tracks)}}))
+    return path
+
+
+def track(root, name, filename=None):
+    """A local track entry. Omit filename for a streaming entry with no file."""
+    entry = {"Name": name, "Artist": "An Artist", "Album": "An Album"}
+    if filename is not None:
+        entry["Location"] = "file://" + urllib.parse.quote(str(root / filename))
+    return entry
+
+
+def missing_from(library, roots):
+    with open(library, "rb") as handle:
+        loaded = plistlib.load(handle)
+    present = find_missing_tracks.existing_under(roots)
+    gone = []
+    for entry in loaded["Tracks"].values():
+        path = find_missing_tracks.location_of(entry)
+        if path is not None and find_missing_tracks.key_for(path) not in present:
+            gone.append(entry["Name"])
+    return sorted(gone)
+
+
+def test_a_deleted_file_is_reported(tmp_path):
+    (tmp_path / "kept.mp3").write_bytes(b"x")
+    library = write_library(
+        tmp_path / "Library.xml",
+        [track(tmp_path, "Kept", "kept.mp3"), track(tmp_path, "Gone", "gone.mp3")],
+    )
+
+    assert missing_from(library, [tmp_path]) == ["Gone"]
+
+
+def test_a_decomposed_filename_is_not_reported_missing(tmp_path):
+    """macOS stores filenames decomposed and most everything else composes them,
+    so the umlauts in Motley Crue are two different byte strings depending on
+    which side wrote the name."""
+    composed = unicodedata.normalize("NFC", "Mötley Crüe.mp3")
+    (tmp_path / composed).write_bytes(b"x")
+    library = write_library(
+        tmp_path / "Library.xml",
+        [track(tmp_path, "Umlauts", unicodedata.normalize("NFD", "Mötley Crüe.mp3"))],
+    )
+
+    assert missing_from(library, [tmp_path]) == []
+
+
+def test_a_case_difference_is_not_reported_missing(tmp_path):
+    """The share is very likely case-insensitive, and a file is not missing
+    because someone capitalised it differently."""
+    (tmp_path / "Hells Bells.mp3").write_bytes(b"x")
+    library = write_library(
+        tmp_path / "Library.xml", [track(tmp_path, "Bells", "hells bells.MP3")]
+    )
+
+    assert missing_from(library, [tmp_path]) == []
+
+
+def test_a_track_with_no_file_is_not_a_missing_file(tmp_path):
+    """A streaming entry has never had a file to lose."""
+    library = write_library(tmp_path / "Library.xml", [track(tmp_path, "Streamed", None)])
+
+    assert missing_from(library, [tmp_path]) == []
+
+
+def test_the_walk_finds_files_nested_below_the_root(tmp_path):
+    nested = tmp_path / "Artist" / "Album"
+    nested.mkdir(parents=True)
+    (nested / "deep.mp3").write_bytes(b"x")
+
+    found = find_missing_tracks.existing_under([tmp_path])
+
+    assert find_missing_tracks.key_for(nested / "deep.mp3") in found
+
+
+def test_the_root_is_derived_from_the_tracks(tmp_path):
+    paths = [tmp_path / "a" / "one.mp3", tmp_path / "b" / "two.mp3"]
+
+    assert find_missing_tracks.roots_of(paths, None) == [tmp_path]
+
+
+def test_an_explicit_root_overrides_the_derived_one(tmp_path):
+    """Worth using: tracks spread beyond the mirror can derive a common parent
+    of / and send the walk across the whole disk."""
+    paths = [tmp_path / "a" / "one.mp3"]
+
+    assert find_missing_tracks.roots_of(paths, ["/Volumes/music-mp3"]) == [
+        Path("/Volumes/music-mp3")
+    ]
+
+
+def test_the_whole_library_missing_is_called_out(tmp_path, capsys):
+    """Almost always an unmounted share rather than an empty library."""
+    library = write_library(
+        tmp_path / "Library.xml", [track(tmp_path / "elsewhere", "Gone", "gone.mp3")]
+    )
+
+    find_missing_tracks.main([str(library), "--root", str(tmp_path)])
+
+    assert "not mounted" in capsys.readouterr().err
+
+
+def test_a_healthy_library_says_nothing_alarming(tmp_path, capsys):
+    (tmp_path / "kept.mp3").write_bytes(b"x")
+    library = write_library(tmp_path / "Library.xml", [track(tmp_path, "Kept", "kept.mp3")])
+
+    find_missing_tracks.main([str(library), "--root", str(tmp_path)])
+
+    captured = capsys.readouterr()
+    assert "0 of 1 local tracks are missing" in captured.err
+    assert "not mounted" not in captured.err
+    assert captured.out == ""
+
+
+@pytest.mark.parametrize(
+    ("location", "expected"),
+    [
+        ("file:///music/a%20b.mp3", Path("/music/a b.mp3")),
+        ("file:///music/plain.mp3", Path("/music/plain.mp3")),
+        ("https://example.invalid/stream", None),
+        ("", None),
+    ],
+)
+def test_locations_are_decoded(location, expected):
+    assert find_missing_tracks.location_of({"Location": location} if location else {}) == expected
+
+
+def test_the_walk_costs_one_read_per_directory_not_one_per_file(tmp_path):
+    """The whole point. Over SMB a per-file check is a round trip per track."""
+    for album in range(20):
+        directory = tmp_path / f"Album {album}"
+        directory.mkdir()
+        for index in range(25):
+            (directory / f"{index}.mp3").write_bytes(b"x")
+
+    reads = {"n": 0}
+    real_walk = os.walk
+
+    def counting_walk(*args, **kwargs):
+        for entry in real_walk(*args, **kwargs):
+            reads["n"] += 1
+            yield entry
+
+    find_missing_tracks.os.walk = counting_walk
+    try:
+        found = find_missing_tracks.existing_under([tmp_path])
+    finally:
+        find_missing_tracks.os.walk = real_walk
+
+    assert len(found) == 500
+    assert reads["n"] == 21  # the root and its twenty albums, not 500 files

tests/test_music_curator.py

@@ -1,4 +1,5 @@
 import json
+import os
 import stat
 import urllib.error
 from pathlib import Path
@@ -1050,8 +1051,8 @@ def test_a_vibe_selects_by_tag(tmp_path):
     store, source, mirror = tagged_store(
         tmp_path,
         {
-            "artist-mbid-1": [{"name": "screamo", "count": 100}],
-            "artist-mbid-2": [{"name": "classic rock", "count": 100}],
+            "Played Band": [{"name": "screamo", "count": 100}],
+            "Silent Band": [{"name": "classic rock", "count": 100}],
         },
     )
     vibes = [{"name": "screamo", "tags": ["screamo"]}]
@@ -1066,7 +1067,7 @@ def test_a_vibe_selects_by_tag(tmp_path):
 def test_a_weakly_tagged_artist_is_below_the_threshold(tmp_path):
     """A single low-weight tag is not a genre, it is somebody's stray opinion."""
     store, source, mirror = tagged_store(
-        tmp_path, {"artist-mbid-1": [{"name": "screamo", "count": 3}]}
+        tmp_path, {"Played Band": [{"name": "screamo", "count": 3}]}
     )
     vibes = [{"name": "screamo", "tags": ["screamo"]}]
 
@@ -1079,7 +1080,7 @@ def test_a_vibe_can_be_restricted_by_release_year(tmp_path):
     """What separates eighties synth records from everything else a synthpop
     tag drags in."""
     store, source, mirror = tagged_store(
-        tmp_path, {"artist-mbid-1": [{"name": "synthpop", "count": 100}]}
+        tmp_path, {"Played Band": [{"name": "synthpop", "count": 100}]}
     )
     inside = [{"name": "eighties", "tags": ["synthpop"], "years": [1975, 1992]}]
     outside = [{"name": "nineties", "tags": ["synthpop"], "years": [1993, 1999]}]
@@ -1127,3 +1128,217 @@ def test_a_bad_vibes_file_is_refused_up_front(tmp_path, content):
 
     with pytest.raises(ValueError):
         music_curator.load_vibes(str(path))
+
+
+def test_tags_are_looked_up_by_name_not_by_mbid(tmp_path):
+    """Last.fm's mbid index is stale: it cannot find Devo or Escape the Fate by
+    one, though their pages plainly exist. Its name index can."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(tags={"Played Band": [{"name": "screamo", "count": 90}]})
+
+    music_curator.sync_tags(client_for(lastfm), store, NOW)
+
+    asked = [call for call in lastfm.calls if call.get("method") == "artist.getTopTags"]
+    # The name is always tried first; the mbid only appears as a fallback for
+    # the artist that the name could not resolve.
+    assert "artist" in asked[0]
+    assert [call for call in asked if call.get("artist") == "Played Band"]
+    assert not [call for call in asked if call.get("mbid") == "artist-mbid-1"]
+    assert store.scalar("SELECT COUNT(*) FROM artist_tag WHERE tag = 'screamo'") == 1
+
+
+def test_the_mbid_is_tried_when_the_name_is_not_found(tmp_path):
+    """Kept only for a name Lidarr spells differently to Last.fm."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(tags={"artist-mbid-1": [{"name": "dnb", "count": 80}]})
+
+    music_curator.sync_tags(client_for(lastfm), store, NOW)
+
+    assert store.scalar("SELECT COUNT(*) FROM artist_tag WHERE tag = 'dnb'") == 1
+    asked = [call for call in lastfm.calls if call.get("method") == "artist.getTopTags"]
+    assert any("mbid" in call for call in asked)
+
+
+def test_an_artist_neither_key_resolves_is_recorded_and_not_retried(tmp_path):
+    """Otherwise every pass spends a request on it again, for ever."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(tags={})
+
+    assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 2
+    spent = len(lastfm.calls)
+
+    assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 0
+    assert len(lastfm.calls) == spent
+
+
+def test_a_real_failure_is_not_recorded_so_the_next_pass_retries(tmp_path):
+    """A rate limit is not the same as an artist not existing."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(
+        tags={"Played Band": [], "Silent Band": []},
+        outcomes=[{"error": 10, "message": "Invalid API key"}],
+    )
+
+    music_curator.sync_tags(client_for(lastfm), store, NOW)
+
+    # One artist failed hard and must still be pending.
+    assert store.scalar("SELECT COUNT(*) FROM artist_tag_fetched") == 1
+
+
+# Tags that carry real weight in a real library, against tags that describe a
+# passport, span most of the collection, or are somebody's artist name.
+USELESS_TAGS = {
+    "american", "british", "australian", "canadian", "swedish", "dutch", "german",
+    "scottish", "english", "uk", "usa", "canada",
+    "rock", "electronic", "pop", "alternative", "metal ", "all", "heavy",
+    "female vocalists", "male vocalists", "female vocalist",
+    "my top songs", "cover", "covers", "not emo",
+    "green day", "paramore", "queen", "bon jovi", "shinedown", "aerosmith",
+    "journey", "fleetwood mac",
+}
+
+
+def test_no_mood_selects_on_a_useless_tag():
+    """Nationality is not a sound; `rock` and `electronic` span most of the
+    library; and Last.fm's top tag for an artist is often their own name."""
+    for vibe in music_curator.DEFAULT_VIBES:
+        overlap = {tag.casefold() for tag in vibe["tags"]} & USELESS_TAGS
+        assert not overlap, f"{vibe['name']} selects on {overlap}"
+
+
+def test_mood_names_are_unique():
+    names = [vibe["name"] for vibe in music_curator.DEFAULT_VIBES]
+    assert len(names) == len(set(names))
+
+
+ROCK_TAGS = {
+    "classic rock", "hard rock", "blues rock", "southern rock", "arena rock",
+    "glam rock", "hair metal", "glam metal", "heavy metal", "metal", "art rock",
+    "psychedelic rock", "progressive rock", "rock and roll", "rock n roll",
+}
+
+
+def test_a_decade_tag_in_a_non_rock_mood_must_exclude_the_rock():
+    """`80s` sits on Def Leppard and Bon Jovi as heavily as on Eurythmics, so a
+    mood that reaches for a decade without wanting rock has to say so. A mood
+    that does want it -- classic-rock reaching for 70s -- is exempt."""
+    for vibe in music_curator.DEFAULT_VIBES:
+        tags = {tag.casefold() for tag in vibe["tags"]}
+        decades = {"60s", "70s", "80s", "90s"} & tags
+        if not decades or tags & ROCK_TAGS:
+            continue
+        excluded = {tag.casefold() for tag in vibe.get("exclude", [])}
+        missing = {"hard rock", "hair metal"} - excluded
+        assert not missing, f"{vibe['name']} selects on {decades} without excluding {missing}"
+
+
+def test_the_eighties_mood_covers_the_canon():
+    """Depeche Mode, Duran Duran, Eurythmics and Frankie Goes to Hollywood --
+    checked against their live Last.fm tags. Three of the four carry "synth pop"
+    with a space; only one carries it without."""
+    synths = next(v for v in music_curator.DEFAULT_VIBES if v["name"] == "80s-synths")
+    tags = {t.casefold() for t in synths["tags"]}
+    for artist_tags in (
+        {"80s", "new wave", "pop", "female vocalists", "synth pop"},   # Eurythmics
+        {"80s", "new wave", "pop", "british", "dance"},                # Frankie
+        {"electronic", "synthpop", "new wave", "80s", "synth pop"},    # Depeche Mode
+        {"new wave", "80s", "pop", "synth pop", "rock"},               # Duran Duran
+    ):
+        assert tags & artist_tags, artist_tags
+    assert synths["years"] == [1975, 1992]
+
+
+def test_an_excluded_tag_drops_the_artist(tmp_path):
+    """Weighting cannot separate the eighties synth acts from the eighties
+    stadium rock, because the tag they would be weighted on is the one they
+    share."""
+    store, source, mirror = tagged_store(
+        tmp_path,
+        {
+            "Played Band": [{"name": "80s", "count": 100}, {"name": "hard rock", "count": 90}],
+            "Silent Band": [{"name": "80s", "count": 100}, {"name": "synth pop", "count": 90}],
+        },
+    )
+    vibes = [{"name": "eighties", "tags": ["80s", "synth pop"], "exclude": ["hard rock"]}]
+
+    music_curator.build_vibe_playlists(store, vibes, mirror, str(source), 100, NOW)
+
+    written = (mirror / "_playlists" / "eighties.m3u").read_text()
+    assert "Silent Band" in written
+    assert "Played Band" not in written
+
+
+def test_a_vibe_cannot_both_select_and_exclude_a_tag(tmp_path):
+    path = tmp_path / "vibes.json"
+    path.write_text(json.dumps([{"name": "x", "tags": ["80s"], "exclude": ["80s"]}]))
+
+    with pytest.raises(ValueError, match="selects on and excludes"):
+        music_curator.load_vibes(str(path))
+
+
+def test_matching_ownership_is_a_no_op_when_it_already_agrees(tmp_path):
+    target = tmp_path / "file"
+    target.write_text("x")
+
+    assert music_curator.match_ownership(target, tmp_path) is False
+
+
+def test_ownership_failure_is_tolerated(tmp_path):
+    """Not permitted unless running as root -- which is exactly the case where
+    the ownership is already whatever the caller runs as."""
+    target = tmp_path / "file"
+    target.write_text("x")
+
+    # uid 0 from a non-root test process: refused, and must not raise.
+    assert music_curator.set_ownership(target, 0, 0) is False
+
+
+def test_a_playlist_is_chowned_to_match_the_mirror(tmp_path, monkeypatch):
+    """The image runs as root, so its output is root-owned, and a root-owned
+    playlist in an apps-owned mirror is unreadable to whatever serves it."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    music_curator.match_library(store)
+
+    attempted = []
+    real_stat = music_curator.Path.stat
+
+    def pretend_mirror_is_owned_by_568(self, *args, **kwargs):
+        info = real_stat(self, *args, **kwargs)
+        if self == mirror:
+            return os.stat_result(
+                (info.st_mode, info.st_ino, info.st_dev, info.st_nlink, 568, 568,
+                 info.st_size, int(info.st_atime), int(info.st_mtime), int(info.st_ctime))
+            )
+        return info
+
+    monkeypatch.setattr(music_curator.Path, "stat", pretend_mirror_is_owned_by_568)
+    monkeypatch.setattr(
+        music_curator.os, "chown", lambda p, u, g: attempted.append((str(p), u, g))
+    )
+
+    music_curator.build_playlists(store, mirror, str(source), 100, NOW)
+
+    assert attempted, "no ownership was applied"
+    assert all(tuple(owner) == (568, 568) for _, *owner in attempted)
+    # The temporary file, before the rename, never the finished playlist.
+    assert all(path.endswith(".part") or path.endswith("_playlists") for path, *_ in attempted)

tools/find_missing_tracks.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+"""Report tracks in a Music library whose files are no longer on disk.
+
+Reads the XML from Music's File > Library > Export Library. Asking Music itself
+does not work: a broken track makes AppleScript's `location` raise rather than
+return a value, so a bulk query dies on the first one with error -1728, and a
+per-track loop costs an Apple event apiece.
+
+The library is checked against a single directory walk rather than by testing
+each file. Over SMB a per-file test is one network round trip per track --
+fifty thousand of them -- where a walk reads each directory once and gets every
+name in it back at once.
+"""
+
+import argparse
+import os
+import plistlib
+import sys
+import unicodedata
+import urllib.parse
+from pathlib import Path
+
+
+def location_of(track):
+    """Return the filesystem path a track points at, or None if it has none."""
+    location = track.get("Location")
+    if not location or not location.startswith("file://"):
+        return None
+    return Path(urllib.parse.unquote(urllib.parse.urlparse(location).path))
+
+
+def key_for(path):
+    """Return a comparison key for a path.
+
+    Normalised to NFC because macOS stores filenames decomposed and most
+    everything else composes them, so "Motley Crue" with its umlauts is two
+    different byte strings depending on which side wrote it. Casefolded because
+    the share is very likely case-insensitive and a file is not missing merely
+    because someone capitalised it differently.
+    """
+    return unicodedata.normalize("NFC", str(path)).casefold()
+
+
+def existing_under(roots):
+    """Return every file below the given roots, keyed for comparison."""
+    found = set()
+    for root in roots:
+        for base, _, names in os.walk(root):
+            for name in names:
+                found.add(key_for(os.path.join(base, name)))
+    return found
+
+
+def roots_of(paths, given):
+    """Return the directories worth walking."""
+    if given:
+        return [Path(root) for root in given]
+    try:
+        return [Path(os.path.commonpath([str(path) for path in paths]))]
+    except ValueError:
+        # Tracks spread across separate volumes have no common parent.
+        return sorted({path.parents[-2] for path in paths if len(path.parents) > 1})
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("library", help="Library.xml exported from Music")
+    parser.add_argument("--root", action="append", help="directory to scan; repeatable")
+    parser.add_argument("--limit", type=int, default=0, help="show at most this many")
+    args = parser.parse_args(argv)
+
+    with open(args.library, "rb") as handle:
+        library = plistlib.load(handle)
+
+    tracks = [
+        (track, location_of(track)) for track in library.get("Tracks", {}).values()
+    ]
+    local = [(track, path) for track, path in tracks if path is not None]
+    if not local:
+        print("no local files in this library", file=sys.stderr)
+        return 0
+
+    roots = roots_of([path for _, path in local], args.root)
+    print(f"scanning {', '.join(str(root) for root in roots)}", file=sys.stderr)
+    present = existing_under(roots)
+
+    gone = [(track, path) for track, path in local if key_for(path) not in present]
+    for track, path in gone[: args.limit or None]:
+        print(
+            "\t".join(
+                (
+                    track.get("Artist", ""),
+                    track.get("Album", ""),
+                    track.get("Name", ""),
+                    str(path),
+                )
+            )
+        )
+
+    print(
+        f"\n{len(gone)} of {len(local)} local tracks are missing their file"
+        f" ({len(tracks) - len(local)} have no file at all)",
+        file=sys.stderr,
+    )
+    if gone and len(gone) == len(local):
+        print(
+            "Every single one is missing, which almost certainly means the share"
+            " is not mounted rather than that the library is empty.",
+            file=sys.stderr,
+        )
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())