chore(release): v0.7.0

Reviewed-on: #13

README.md

@@ -7,8 +7,9 @@ which keeps an MP3 copy of a lossless library for an iPod. This one answers the
 question that mirror cannot: which of it is worth carrying, and which of it has
 not been played in years.
 
-**This is stage three.** It ingests the scrobble history, indexes the library
+**This is stage four, read-only.** It ingests the scrobble history, indexes the library
-from Lidarr, matches one to the other, and writes playlists into the mirror.
+from Lidarr, matches one to the other, and writes playlists into the mirror —
+by listening history and by mood.
 Nothing is written back to Lidarr — every call there is a `GET`. See "Where this
 is going" below.
 
@@ -19,7 +20,8 @@ is going" below.
 - Indexes every artist, album and track Lidarr knows about, with file paths and
   the date each file landed.
 - Ties the two together and reports how well it managed.
-- Writes M3U playlists into the mirror, from the listening history.
+- Writes M3U playlists into the mirror, from the listening history and by mood.
+- Reports which albums have never been played. It does not act on that.
 
 ## Matching
 
@@ -130,7 +132,13 @@ overstates the problem and would over-block the cull.
 
 ## Playlists
 
-Written into `<mirror>/_playlists/` as extended M3U, rebuilt every pass. Six
+Written into `<mirror>/_playlists/` as extended M3U, rebuilt every pass.
+
+The extension is **`.m3u8`**, not `.m3u`. Rockbox's `is_m3u8_name()` treats
+every extension as UTF-8 *except* an explicit `.m3u`, which it decodes through
+the user's configured codepage instead — so a plain `.m3u` mangles every
+accented filename. No byte order mark is written: Rockbox does not need one at
+this extension, and a BOM upsets players that do not expect it. Six
 rules, capped at `--playlist-limit` tracks each:
 
 | Playlist | Rule |
@@ -151,6 +159,102 @@ few hours, and a playlist that reorders itself each time is one that has to be
 re-imported each time — the Music app imports a snapshot of a file, it does not
 track it.
 
+### Moods
+
+A second set of playlists selects by **Last.fm's crowd tags** rather than by
+listening history. MusicBrainz genres arrive free with the Lidarr index and are
+no use for this: they are sparse and formal, and will not tell you a record is
+screamo or synthwave. People typing tags will.
+
+Tags are fetched once per artist — one `artist.getTopTags` call each — and
+refreshed every ninety days. `--tag-limit` spreads the first sweep over several
+passes.
+
+Artists are looked up **by name**, not by MusicBrainz id, despite Lidarr having
+an id for every one of them. Last.fm's mbid index is stale and partial: it
+answers "the artist you supplied could not be found" for Devo, Escape the Fate,
+Blasterjaxx and a few hundred others whose pages plainly exist and carry exactly
+the tags wanted. Its name index is the one its own site runs on. The id is kept
+only as a fallback, for a name Lidarr spells differently.
+
+An artist neither key resolves is recorded as fetched with no tags, so the next
+pass does not spend a request on it again. A genuine failure — a rate limit, a
+bad key — is *not* recorded, so that one is retried.
+
+The built-in moods are built from the **tag distribution of this library**,
+measured, rather than from a general taxonomy:
+
+| Mood | Selected on |
+| --------------- | -------------------------------------------------------- |
+| `drum-and-bass` | drum and bass and its six spellings, liquid funk, neurofunk, jungle, techstep, hospital records |
+| `bass` | dubstep, brostep, grime, trip-hop, big beat |
+| `dance` | house and its variants, trance, techno, electro, rave |
+| `pop-punk` | pop punk, punk, emo, emocore, easycore, power pop |
+| `screamo` | screamo, post-hardcore, metalcore, melodic hardcore, trancecore |
+| `heavy-metal` | heavy metal, thrash, speed, power, death, prog, NWOBHM |
+| `hair-metal` | hair metal, glam metal, glam rock, arena rock, AOR — 1975-1994 |
+| `nu-metal` | nu metal, alternative metal, rapcore, industrial |
+| `classic-rock` | classic rock, prog, psychedelic, blues rock, 70s, 60s |
+| `80s-synths` | 80s, new wave, synth pop, electropop, post-punk — 1975-1992, rock excluded |
+| `indie` | indie, indie rock, indie pop, britpop, singer-songwriter |
+
+Measuring first mattered. `synthwave`, `edm`, `big room` and `hardstyle` are
+plausible tags that carry **nothing at all** here, while `techstep`, `easycore`
+and `hospital records` carry real weight. Guessing produces the first list.
+
+Three kinds of tag are never used, and there is a test enforcing it:
+
+- **Nationality** — `american` alone spans 241 artists. A passport is not a
+  sound.
+- **`rock` and `electronic`** — 340 and 275 artists, most of the library. A
+  mood that matches everything is not a mood.
+- **Artist names** — Last.fm's most popular tag for an artist is frequently
+  their own name. `green day`, `paramore` and `queen` are single-artist
+  playlists waiting to happen.
+
+### Exclusions
+
+A mood may also list `exclude`. An excluded tag drops the artist outright rather
+than docking their score, and it exists because `80s-synths` cannot be written
+any other way.
+
+`80s` is the eleventh most-played tag here, and it sits on Def Leppard and Bon
+Jovi exactly as heavily as on Eurythmics. Weighting cannot separate them,
+because the tag it would weight is the one they share. What does separate them
+is that the stadium rock also carries `hard rock` and `hair metal`, and the
+synth acts do not.
+
+Checked against live Last.fm pages, since the tag census only sees artists
+already in the library:
+
+| Artist | Tags |
+| --- | --- |
+| Eurythmics | `80s`, `new wave`, `pop`, `female vocalists`, `synth pop` |
+| Frankie Goes to Hollywood | `80s`, `new wave`, `pop`, `british`, `dance` |
+| Depeche Mode | `electronic`, `synthpop`, `new wave`, `80s`, `synth pop` |
+| Duran Duran | `new wave`, `80s`, `pop`, `synth pop`, `rock` |
+
+Four of Eurythmics' five tags are ones no mood may use. Three of the four
+artists spell it **`synth pop`** with a space; only one spells it `synthpop`.
+Guessing one spelling would have missed most of the canon.
+
+An artist qualifies when their tag weights inside a mood sum to at least 30 out
+of Last.fm's 0-100 scale. One low-weight tag is not a genre, it is somebody's
+stray opinion.
+
+`years` filters on the album's release date, which is what separates eighties
+synth records from everything else a synthpop tag drags in.
+
+`--vibes` replaces the whole set with a JSON file of the same shape, so a new
+mood does not need a new release:
+
+```json
+[{ "name": "shoegaze", "tags": ["shoegaze", "dream pop"], "min_score": 40 }]
+```
+
+Names are validated when the file is read, not when the file is written. A bad
+one would otherwise surface as a playlist created somewhere unintended.
+
 ### Paths
 
 Lidarr knows where the lossless source is; the playlists have to point at the
@@ -164,6 +268,13 @@ in step by hand. Override it if that guess is wrong.
 Entries are written **relative to the playlist file**, so one playlist works
 from the NAS, from a Mac over SMB, and from Linux, without rewriting.
 
+Each playlist is given the **owner and group of the mirror** it is written
+into. The image runs as root by default so that a bind mount of any ownership
+stays writable, and the cost of that is output owned by root — which the account
+serving the share cannot read, group bit or no group bit, because the group is
+also root. Copying the mirror's own ownership avoids having to be told what it
+should be, and does nothing when the two already agree.
+
 A track is only listed once its mirror file has been confirmed to exist. Lidarr
 holding the FLAC says nothing about whether the MP3 has been encoded yet. If a
 large number are missing, the run says so — that is what a wrong `--library-root`
@@ -172,6 +283,40 @@ or `--mirror` looks like, since the paths then map to nothing at all.
 `_playlists/` survives music-mirror's prune: it only deletes `*.mp3`, and its
 empty-directory sweep skips a directory holding M3Us.
 
+## The cold report
+
+Which albums have files, have never had a single track played in the whole
+history, and have sat there long enough to have had the chance. Ranked by the
+disk they occupy, because that is the point of the exercise.
+
+**Album-level, not track-level.** A record with two played tracks is a record
+that gets played; picking the other ten off it leaves gaps rather than
+reclaiming anything worth having.
+
+The age floor is measured from the newest file in the album, not from the
+release date — what matters is how long it has been available to play, not how
+old the record is. `--cold-after` sets it, defaulting to a year.
+
+Artists whose *every* album is cold are counted separately. That is a different
+proposition from one cold record by somebody otherwise played, and Lidarr can
+only tag at artist level anyway.
+
+### What stops it
+
+The report refuses to produce anything at all when:
+
+- the scrobble backfill is unfinished
+- any artist failed to index, or any artist has no albums indexed
+- the library has not been indexed, or there is no history to judge against
+
+Each of those makes played music look unplayed, which is the single failure that
+costs a library. They are checked rather than trusted, because the report they
+gate is the one that ends in deletion.
+
+**Nothing is written to Lidarr.** Every call there is still a `GET`. Unmonitoring
+comes once the list has been looked at, because no flag protects against a list
+that is wrong.
+
 ## How the ingest works
 
 Two halves, both taking their bounds from the database rather than from a saved
@@ -229,6 +374,9 @@ music-curator --report-only   # report on the store, fetch nothing
 | `--mirror` | `MUSIC_CURATOR_MIRROR` | unset | Root of the MP3 mirror; playlists go here |
 | `--library-root` | `MUSIC_CURATOR_LIBRARY_ROOT` | derived | Prefix to strip from Lidarr's paths |
 | `--playlist-limit` | `MUSIC_CURATOR_PLAYLIST_LIMIT` | `100` | Most tracks in any one playlist |
+| `--vibes` | `MUSIC_CURATOR_VIBES` | built-in | JSON file of mood definitions |
+| `--tag-limit` | `MUSIC_CURATOR_TAG_LIMIT` | `0` | Cap artist tag lookups per pass |
+| `--cold-after` | `MUSIC_CURATOR_COLD_AFTER` | `365` | Days a file must sit unplayed to count as cold |
 | `--skip-index` | — | off | Match against the index already held |
 | `--report-only` | — | off | Report without fetching |
 
@@ -278,8 +426,9 @@ nix shell nixpkgs#python3Packages.pytest -c pytest
 | Last.fm ingest and store | done |
 | Lidarr index and the scrobble-to-track matcher | done |
 | M3U playlists from the listening history | done |
-| Genre and mood playlists from Last.fm tags | next |
+| Genre and mood playlists from Last.fm tags | done |
-| Cold-music report, unmonitoring what is not played | last |
+| Cold-music report | done |
+| Unmonitoring what is not played | last |
 
 The cull will unmonitor cold albums in Lidarr and tag their artists. It will
 never delete files: Lidarr's `AlbumResource` has no tags at all, so tagging

music_curator.py

@@ -69,16 +69,39 @@ BACKOFF_CEILING_SECONDS = 60.0
 MIRROR_SUFFIX = ".mp3"
 PLAYLIST_DIRECTORY = "_playlists"
 
+# .m3u8, not .m3u. Rockbox's is_m3u8_name() treats every extension as UTF-8
+# except an explicit ".m3u", which it decodes through the user's configured
+# codepage instead -- so a plain .m3u mangles every accented filename, and this
+# library holds Mötley Crüe, Beyoncé and Sigur Rós.
+PLAYLIST_SUFFIX = ".m3u8"
+
+# Tags are re-fetched this often. They move slowly, and the first pass over a
+# library already costs one request per artist.
+TAG_REFRESH_SECONDS = 90 * 86400
+
+# How much of an artist's tag weight has to fall inside a vibe before their
+# tracks are eligible for it. Weights are Last.fm's own 0-100 popularity, summed
+# across whichever of the vibe's tags the artist carries.
+VIBE_MIN_SCORE = 30
+
+# A vibe writes a file named after itself, so the name has to be a filename.
+SAFE_VIBE_NAME = re.compile(r"^[a-z0-9][a-z0-9-]*$")
+
 # Group-readable, for the same reason music-mirror sets it: the mirror is read
 # back by whatever serves it, and a playlist nobody can read is not a playlist.
 GROUP_READ = 0o040
 
-SCHEMA_VERSION = "2"
+SCHEMA_VERSION = "3"
 
 # Versions this build upgrades in place. Everything added since version 1 is a
-# new table, and the schema script only ever creates what is missing, so running
+# new table or a new column, both of which are applied in place, so an existing
-# it is the whole migration -- an existing history is not re-downloaded.
+# history is never re-downloaded -- a full backfill is thousands of requests.
-MIGRATABLE_FROM = {"1"}
+MIGRATABLE_FROM = {"1", "2"}
+
+# Columns added to existing tables after the fact. CREATE TABLE IF NOT EXISTS
+# will not add a column to a table that already exists, so these are applied
+# separately and only when missing.
+ADDED_COLUMNS = (("lidarr_track", "size", "INTEGER"),)
 
 SCHEMA = """
 -- One row per scrobble. The primary key collapses two plays of the same track
@@ -148,7 +171,8 @@ CREATE TABLE IF NOT EXISTS lidarr_track (
     has_file       INTEGER NOT NULL DEFAULT 0,
     path           TEXT,
     added          INTEGER,
-    duration       INTEGER
+    duration       INTEGER,
+    size           INTEGER
 );
 CREATE INDEX IF NOT EXISTS lidarr_track_recording ON lidarr_track (recording_mbid);
 CREATE INDEX IF NOT EXISTS lidarr_track_norm ON lidarr_track (norm_artist, norm_title);
@@ -177,6 +201,24 @@ CREATE TABLE IF NOT EXISTS scrobble_key (
     PRIMARY KEY (artist, track)
 );
 CREATE INDEX IF NOT EXISTS scrobble_key_track ON scrobble_key (track_id);
+
+-- Last.fm's crowd tags for each library artist. MusicBrainz genres come free
+-- with the Lidarr index and are useless for this: they are sparse and formal,
+-- and will not tell you a record is screamo or synthwave. People typing tags
+-- will. Keyed on the normalised name rather than the Lidarr id so it survives
+-- an artist being removed and re-added there.
+CREATE TABLE IF NOT EXISTS artist_tag (
+    norm_artist TEXT    NOT NULL,
+    tag         TEXT    NOT NULL,
+    weight      INTEGER NOT NULL,
+    PRIMARY KEY (norm_artist, tag)
+);
+CREATE INDEX IF NOT EXISTS artist_tag_tag ON artist_tag (tag);
+
+CREATE TABLE IF NOT EXISTS artist_tag_fetched (
+    norm_artist TEXT PRIMARY KEY,
+    fetched_at  INTEGER NOT NULL
+);
 """
 
 
@@ -236,7 +278,16 @@ WHITESPACE = re.compile(r"\s+")
 
 
 class LastfmError(Exception):
-    """A Last.fm request that failed in a way retrying will not fix."""
+    """A Last.fm request that failed in a way retrying will not fix.
+
+    `code` is the service's own error number where the failure came from the
+    API rather than the transport. Callers need it to tell "this thing does not
+    exist", which is final, from "something went wrong", which is not.
+    """
+
+    def __init__(self, message, code=None):
+        super().__init__(message)
+        self.code = code
 
 
 class LidarrError(Exception):
@@ -417,7 +468,7 @@ class Lastfm:
                 return payload
             detail = f"error {code}: {payload.get('message', '')}".strip()
             if code not in RETRYABLE_ERRORS:
-                raise LastfmError(f"{method}: {detail}")
+                raise LastfmError(f"{method}: {detail}", code=code)
             self._retry_or_raise(method, attempt, detail, None)
 
         raise LastfmError(f"{method}: gave up after {self.attempts} attempts")
@@ -458,8 +509,18 @@ class Store:
         # thousand name pairs into Python to compare them one at a time.
         self.connection.create_function("normalise", 1, normalise, deterministic=True)
         self.connection.executescript(SCHEMA)
+        self._add_missing_columns()
         self._check_version()
 
+    def _add_missing_columns(self):
+        """Apply column additions to tables that predate them."""
+        for table, column, kind in ADDED_COLUMNS:
+            held = {row[1] for row in self.connection.execute(f"PRAGMA table_info({table})")}
+            if column not in held:
+                logger.info("adding %s.%s to the store", table, column)
+                with self.connection:
+                    self.connection.execute(f"ALTER TABLE {table} ADD COLUMN {column} {kind}")
+
     def _check_version(self):
         held = self.get_state("schema_version")
         if held is None or held in MIGRATABLE_FROM:
@@ -541,11 +602,29 @@ class Store:
             self.connection.executemany(
                 "INSERT INTO lidarr_track"
                 " (id, artist_id, album_id, recording_mbid, title, norm_artist, norm_title,"
-                "  has_file, path, added, duration)"
+                "  has_file, path, added, duration, size)"
-                " VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+                " VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
                 tracks,
             )
 
+    def replace_tags(self, norm_artist, pairs, now):
+        """Replace one artist's tags, and record that they were fetched.
+
+        The fetch is recorded even when nothing came back, so an artist Last.fm
+        has never heard of is not asked about again on every pass.
+        """
+        with self.connection:
+            self.connection.execute("DELETE FROM artist_tag WHERE norm_artist = ?", (norm_artist,))
+            self.connection.executemany(
+                "INSERT OR IGNORE INTO artist_tag (norm_artist, tag, weight) VALUES (?, ?, ?)",
+                [(norm_artist, tag, weight) for tag, weight in pairs],
+            )
+            self.connection.execute(
+                "INSERT INTO artist_tag_fetched (norm_artist, fetched_at) VALUES (?, ?)"
+                " ON CONFLICT (norm_artist) DO UPDATE SET fetched_at = excluded.fetched_at",
+                (norm_artist, now),
+            )
+
     def rebuild_keys(self):
         """Collapse the scrobble history into one row per distinct track.
 
@@ -983,6 +1062,7 @@ def index_library(client, store):
                     handle.get("path"),
                     parse_added(handle.get("dateAdded")),
                     track.get("duration"),
+                    handle.get("size"),
                 )
             )
 
@@ -1177,6 +1257,317 @@ PLAYLISTS = (
 )
 
 
+# Moods, as sets of Last.fm tags. Chosen for this library -- drum and bass,
+# punk and its descendants, big-room dance, classic rock -- rather than as a
+# general taxonomy. `--vibes` replaces the lot with a JSON file of the same
+# shape, so a new one does not need a new release.
+#
+# `years` filters on the album's release date, which is what separates eighties
+# synth records from everything a synthpop tag would otherwise drag in.
+DEFAULT_VIBES = (
+    # Built from the tag distribution of this library rather than from a general
+    # taxonomy, which is why some obvious-looking tags are absent and some
+    # unobvious ones are here. `synthwave`, `edm`, `big room` and `hardstyle`
+    # carry nothing at all; `techstep`, `easycore` and `hospital records` carry
+    # real weight.
+    #
+    # Three kinds of tag are deliberately never used. Nationality -- american,
+    # british, swedish -- describes a passport, not a sound, and `american`
+    # alone spans 241 artists. `rock` and `electronic` span 340 and 275, which
+    # is most of the library and therefore no mood at all. And Last.fm's most
+    # popular tag for an artist is frequently their own name, so `green day`,
+    # `paramore` and `queen` are single-artist playlists waiting to happen.
+    {
+        "name": "drum-and-bass",
+        "tags": [
+            "drum and bass", "dnb", "drum n bass", "drum'n'bass", "drum & bass",
+            "drum 'n' bass", "liquid funk", "neurofunk", "jungle", "techstep",
+            "darkstep", "drumstep", "breakbeat", "hospital records",
+        ],
+    },
+    {
+        "name": "bass",
+        "tags": ["dubstep", "brostep", "grime", "trip-hop", "big beat"],
+    },
+    {
+        "name": "dance",
+        "tags": [
+            "house", "electro house", "progressive house", "tech house", "trance",
+            "techno", "electro", "rave", "dance", "minimal",
+        ],
+    },
+    {
+        "name": "pop-punk",
+        "tags": [
+            "pop punk", "pop-punk", "punk rock", "punk", "skate punk", "emo",
+            "emocore", "easycore", "powerpop", "power pop", "post-grunge",
+        ],
+    },
+    {
+        "name": "screamo",
+        "tags": [
+            "screamo", "post-hardcore", "metalcore", "melodic metalcore",
+            "melodic hardcore", "hardcore", "trancecore", "deathcore",
+        ],
+    },
+    {
+        "name": "heavy-metal",
+        "tags": [
+            "heavy metal", "metal", "thrash metal", "thrash", "speed metal",
+            "power metal", "death metal", "progressive metal", "nwobhm",
+            "classic metal",
+        ],
+    },
+    {
+        "name": "hair-metal",
+        "tags": [
+            "hair metal", "glam metal", "glam rock", "arena rock", "aor",
+            "rock and roll", "rock n roll",
+        ],
+        "years": [1975, 1994],
+    },
+    {
+        "name": "nu-metal",
+        "tags": [
+            "nu metal", "nu-metal", "alternative metal", "rapcore",
+            "industrial metal", "industrial rock", "industrial",
+        ],
+    },
+    {
+        "name": "classic-rock",
+        "tags": [
+            "classic rock", "progressive rock", "psychedelic rock", "psychedelic",
+            "blues rock", "blues", "southern rock", "art rock", "space rock",
+            "british invasion", "folk rock", "70s", "60s",
+        ],
+    },
+    {
+        "name": "80s-synths",
+        # `80s` is included, which on its own would drag in Def Leppard and Bon
+        # Jovi -- they carry it as heavily as Eurythmics does. The exclusion is
+        # what separates them: the stadium rock is also tagged hard rock and
+        # hair metal, and the synth acts are not. Weighting cannot do this,
+        # because the tag it would weight is the one they share.
+        #
+        # Both spellings of synth pop are listed. Of Depeche Mode, Duran Duran,
+        # Eurythmics and Frankie Goes to Hollywood, three carry "synth pop" with
+        # a space and only one carries "synthpop" without.
+        "tags": [
+            "80s", "new wave", "synth pop", "synthpop", "synth-pop", "synthwave",
+            "electropop", "new romantic", "post-punk", "post-punk revival",
+        ],
+        "exclude": [
+            "hard rock", "hair metal", "glam metal", "glam rock", "heavy metal",
+            "metal", "arena rock", "aor", "nwobhm", "thrash metal",
+            "classic rock", "southern rock", "blues rock",
+        ],
+        "years": [1975, 1992],
+    },
+    {
+        "name": "indie",
+        "tags": ["indie", "indie rock", "indie pop", "britpop", "singer-songwriter"],
+    },
+)
+
+
+def load_vibes(path):
+    """Return the vibe definitions, from a file when one is given.
+
+    Validated up front rather than at write time: a bad name would otherwise
+    surface as a file created somewhere unintended, which is a poor way to find
+    out about a typo.
+    """
+    if not path:
+        return DEFAULT_VIBES
+    try:
+        loaded = json.loads(Path(path).read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError) as error:
+        raise ValueError(f"cannot read vibes from {path}: {error}") from error
+
+    if not isinstance(loaded, list):
+        raise ValueError(f"{path}: expected a list of vibes")
+    for vibe in loaded:
+        name = vibe.get("name") if isinstance(vibe, dict) else None
+        if not name or not SAFE_VIBE_NAME.fullmatch(str(name)):
+            raise ValueError(f"{path}: {name!r} is not a usable vibe name (a-z, 0-9, -)")
+        if not vibe.get("tags"):
+            raise ValueError(f"{path}: vibe {name!r} lists no tags")
+        overlap = {str(tag).casefold() for tag in vibe["tags"]} & {
+            str(tag).casefold() for tag in vibe.get("exclude", [])
+        }
+        if overlap:
+            raise ValueError(
+                f"{path}: vibe {name!r} both selects on and excludes {sorted(overlap)}"
+            )
+    return tuple(loaded)
+
+
+def parse_tags(payload):
+    """Return (tag, weight) pairs from an artist.getTopTags response.
+
+    The documented sample carries only a name and a URL per tag; a live response
+    also carries a 0-100 `count`. Rather than depend on which, the count is used
+    when present and the documented ordering -- by popularity -- stands in for it
+    when it is not.
+    """
+    block = payload.get("toptags") or {}
+    pairs = []
+    for position, entry in enumerate(as_list(block.get("tag"))):
+        tag = (entry.get("name") or "").strip().casefold()
+        if not tag:
+            continue
+        weight = int(entry.get("count") or 0) or max(1, 100 - position * 5)
+        pairs.append((tag, weight))
+    return pairs
+
+
+def fetch_tags(client, name, mbid):
+    """Return an artist's tags, asking by name first.
+
+    By name, not by MusicBrainz id, despite Lidarr having an id for everything.
+    Last.fm's mbid index is stale and partial -- it answers "the artist you
+    supplied could not be found" for Devo, Escape the Fate and a few hundred
+    others whose pages plainly exist -- while the name index is the one its own
+    site runs on. The id is kept only as a fallback for a name Lidarr spells
+    differently.
+
+    Returns an empty list when the artist is genuinely unknown, which the caller
+    records so it is not asked again.
+    """
+    attempts = [{"artist": name}] if name else []
+    if mbid:
+        attempts.append({"mbid": mbid})
+
+    for query in attempts:
+        try:
+            return parse_tags(client.call("artist.getTopTags", {**query, "autocorrect": 1}))
+        except LastfmError as error:
+            # Error 6 here means "no such artist", which the next key may still
+            # answer. Anything else is a real failure and belongs to the caller.
+            if error.code != 6:
+                raise
+    return []
+
+
+def sync_tags(client, store, now, limit=0):
+    """Fetch crowd tags for library artists that have none, or stale ones.
+
+    One request per artist, once, and then only for artists newly added. An
+    artist Last.fm has never heard of is recorded as fetched with no tags, so it
+    is not asked about again every pass.
+    """
+    stale = store.connection.execute(
+        "SELECT a.norm_name AS norm_name, a.name AS name, a.mbid AS mbid"
+        "  FROM lidarr_artist a"
+        "  LEFT JOIN artist_tag_fetched f ON f.norm_artist = a.norm_name"
+        " WHERE a.norm_name <> ''"
+        "   AND (f.fetched_at IS NULL OR f.fetched_at < :cutoff)"
+        " ORDER BY a.name",
+        {"cutoff": now - TAG_REFRESH_SECONDS},
+    ).fetchall()
+    if not stale:
+        return 0
+
+    if limit > 0 and len(stale) > limit:
+        logger.info("tagging %d of %d artists this pass; the rest follow next", limit, len(stale))
+        stale = stale[:limit]
+    else:
+        logger.info("fetching tags for %d artists", len(stale))
+
+    resolved = 0
+    unknown = 0
+    for artist in stale:
+        try:
+            pairs = fetch_tags(client, artist["name"], artist["mbid"])
+        except LastfmError as error:
+            # Something went wrong rather than the artist not existing. Left
+            # unrecorded on purpose, so the next pass tries again.
+            logger.warning("no tags for %s: %s", artist["name"], error)
+            continue
+        store.replace_tags(artist["norm_name"], pairs, now)
+        resolved += 1
+        if not pairs:
+            unknown += 1
+
+    logger.info("tagged %d artists (%d with nothing to say about them)", resolved, unknown)
+    return resolved
+
+
+def build_vibe_playlists(store, vibes, mirror_root, library_root, limit, now):
+    """Write one playlist per mood. Returns how many tracks were listed."""
+    if not store.scalar("SELECT COUNT(*) FROM artist_tag"):
+        return 0, []
+
+    directory = Path(mirror_root) / PLAYLIST_DIRECTORY
+    week = now // ROTATION_PERIOD_SECONDS
+    total = 0
+    produced = []
+
+    for vibe in vibes:
+        tags = [str(tag).strip().casefold() for tag in vibe["tags"]]
+        placeholders = ",".join("?" * len(tags))
+        parameters = [*tags, vibe.get("min_score", VIBE_MIN_SCORE)]
+
+        # An exclusion drops the artist outright rather than docking their
+        # score. It is the only way to write "the eighties, but not the stadium
+        # rock": those artists carry `80s` as heavily as the synth acts do, so
+        # no amount of weighting separates them -- but they also carry `hard
+        # rock`, and the synth acts do not.
+        excluded = [str(tag).strip().casefold() for tag in vibe.get("exclude", [])]
+        exclude_clause = ""
+        if excluded:
+            exclude_clause = (
+                " AND NOT EXISTS (SELECT 1 FROM artist_tag x"
+                "                  WHERE x.norm_artist = a.norm_name"
+                f"                   AND x.tag IN ({','.join('?' * len(excluded))}))"
+            )
+            parameters += excluded
+
+        years = vibe.get("years")
+        year_clause = ""
+        if years:
+            year_clause = (
+                " AND CAST(substr(al.release_date, 1, 4) AS INTEGER) BETWEEN ? AND ?"
+            )
+            parameters += [int(years[0]), int(years[1])]
+        parameters += [week, limit]
+
+        sql = f"""
+            WITH vibe AS (
+                SELECT norm_artist, SUM(weight) AS score
+                  FROM artist_tag
+                 WHERE tag IN ({placeholders})
+                 GROUP BY norm_artist
+                HAVING SUM(weight) >= ?
+            )
+            SELECT t.id AS id, a.name AS artist, t.title AS title,
+                   t.duration AS duration, t.path AS path
+              FROM lidarr_track t
+              JOIN lidarr_artist a ON a.id = t.artist_id
+              JOIN vibe v ON v.norm_artist = a.norm_name
+              LEFT JOIN lidarr_album al ON al.id = t.album_id
+             WHERE {PLAYABLE}{exclude_clause}{year_clause}
+             ORDER BY ((t.id * {SHUFFLE_MULTIPLIER}) + ?) % {SHUFFLE_MODULUS}
+             LIMIT ?
+        """
+
+        entries = []
+        for row in store.connection.execute(sql, parameters):
+            mirror = mirror_path_for(row["path"], library_root, mirror_root)
+            if mirror is None or not mirror.is_file():
+                continue
+            entries.append({**dict(row), "mirror": mirror})
+
+        write_playlist(
+            directory / f"{vibe['name']}{PLAYLIST_SUFFIX}", entries, Path(mirror_root)
+        )
+        produced.append(f"{vibe['name']}{PLAYLIST_SUFFIX}")
+        total += len(entries)
+        logger.info("playlist %-20s %4d tracks -- by tag", vibe["name"], len(entries))
+
+    return total, produced
+
+
 def library_root_of(store):
     """Return the directory Lidarr's artist folders sit under.
 
@@ -1208,7 +1599,40 @@ def mirror_path_for(source, library_root, mirror_root):
     return (Path(mirror_root) / relative).with_suffix(MIRROR_SUFFIX)
 
 
-def write_playlist(path, entries):
+def set_ownership(path, uid, gid):
+    """Give a path an owner and group. Returns whether anything changed."""
+    try:
+        current = path.stat()
+        if (current.st_uid, current.st_gid) == (uid, gid):
+            return False
+        os.chown(path, uid, gid)
+    except OSError:
+        # Not permitted unless running as root, which is the case where the
+        # ownership is already whatever the caller runs as.
+        return False
+    return True
+
+
+def match_ownership(path, reference):
+    """Give a path the owner and group of the tree it is joining.
+
+    The image runs as root by default, so that a bind mount of any ownership
+    stays writable. The cost is that everything it writes comes out root-owned,
+    and a root-owned playlist inside a mirror owned by the apps account is
+    unreadable to the thing that serves it -- the group bit does not help when
+    the group is root.
+
+    Copying the mirror's own ownership avoids having to be told what it should
+    be, and is a no-op when the two already agree.
+    """
+    try:
+        wanted = reference.stat()
+    except OSError:
+        return False
+    return set_ownership(path, wanted.st_uid, wanted.st_gid)
+
+
+def write_playlist(path, entries, reference=None):
     """Write one extended M3U, atomically.
 
     Paths are relative to the playlist file, so the same playlist works from the
@@ -1220,8 +1644,12 @@ def write_playlist(path, entries):
         lines.append(f"#EXTINF:{seconds},{entry['artist']} - {entry['title']}")
         lines.append(os.path.relpath(entry["mirror"], path.parent))
 
+    fresh = not path.parent.exists()
     path.parent.mkdir(parents=True, exist_ok=True)
-    handle, temporary = tempfile.mkstemp(dir=path.parent, suffix=".m3u.part")
+    if fresh and reference is not None:
+        match_ownership(path.parent, reference)
+
+    handle, temporary = tempfile.mkstemp(dir=path.parent, suffix=".m3u8.part")
     os.close(handle)
     temporary = Path(temporary)
     try:
@@ -1231,11 +1659,31 @@ def write_playlist(path, entries):
         mode = temporary.stat().st_mode
         if not mode & GROUP_READ:
             temporary.chmod(mode | GROUP_READ)
+        # Before the rename, so the playlist is never briefly visible owned by
+        # the wrong account.
+        if reference is not None:
+            match_ownership(temporary, reference)
         os.replace(temporary, path)
     finally:
         temporary.unlink(missing_ok=True)
 
 
+def prune_playlists(directory, produced, store):
+    """Delete playlists this build no longer produces.
+
+    Driven by a record of what was written last time rather than by "every M3U
+    that is not one of ours", so a playlist put there by hand is left alone. A
+    renamed mood otherwise leaves its old file on the device for ever.
+    """
+    previous = set(json.loads(store.get_state("playlist_files") or "[]"))
+    for name in sorted(previous - set(produced)):
+        stale = directory / name
+        if stale.is_file():
+            logger.info("removing playlist %s, no longer produced", name)
+            stale.unlink(missing_ok=True)
+    store.set_state("playlist_files", json.dumps(sorted(produced)))
+
+
 def build_playlists(store, mirror_root, library_root, limit, now):
     """Write every playlist into the mirror. Returns how many tracks were listed.
 
@@ -1253,6 +1701,7 @@ def build_playlists(store, mirror_root, library_root, limit, now):
 
     total = 0
     missing = 0
+    produced = []
     for name, description, sql in PLAYLISTS:
         entries = []
         for row in store.connection.execute(sql, parameters):
@@ -1261,7 +1710,8 @@ def build_playlists(store, mirror_root, library_root, limit, now):
                 missing += 1
                 continue
             entries.append({**dict(row), "mirror": mirror})
-        write_playlist(directory / f"{name}.m3u", entries)
+        write_playlist(directory / f"{name}{PLAYLIST_SUFFIX}", entries, Path(mirror_root))
+        produced.append(f"{name}{PLAYLIST_SUFFIX}")
         total += len(entries)
         logger.info("playlist %-20s %4d tracks -- %s", name, len(entries), description)
 
@@ -1273,7 +1723,7 @@ def build_playlists(store, mirror_root, library_root, limit, now):
             " place, since the paths are then being mapped to nothing.",
             missing,
         )
-    return total
+    return total, produced
 
 
 def coverage_report(store):
@@ -1447,7 +1897,156 @@ def coverage_report(store):
         logger.info("  unmatched %2d: %-60s %d plays", position, label[:60], row["plays"])
 
 
-def report(store, now):
+# An album has to have sat unplayed for at least this long before it counts as
+# cold. Measured from the newest file in it, not from the release date: what
+# matters is how long it has been available to play, not how old the record is.
+COLD_AFTER_DAYS = 365
+
+
+def human_bytes(count):
+    """Return a size that can be read at a glance."""
+    size = float(count or 0)
+    for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
+        if size < 1024 or unit == "TiB":
+            return f"{size:.1f} {unit}"
+        size /= 1024
+
+
+def cull_is_safe(store):
+    """Return why a cull must not run, or None if it may.
+
+    Every one of these makes played music look unplayed, which is the single
+    failure that costs a library. They are checked rather than trusted because
+    the report they gate is the one that ends in deletion.
+    """
+    if store.get_state("backfill_complete") != "yes":
+        return "the scrobble history is still being backfilled"
+    if int(store.get_state("index_skipped") or 0):
+        return "some artists could not be indexed, so their tracks are missing"
+    if int(store.get_state("index_albums_skipped") or 0):
+        return "some artists have no albums indexed"
+    if not store.scalar("SELECT COUNT(*) FROM lidarr_track"):
+        return "the library has not been indexed"
+    if not store.scalar("SELECT COUNT(*) FROM scrobble"):
+        return "there is no listening history to judge against"
+    return None
+
+
+COLD_ALBUMS = """
+WITH played AS (
+    SELECT DISTINCT sk.track_id AS track_id
+      FROM scrobble_key sk
+     WHERE sk.track_id IS NOT NULL
+),
+album AS (
+    SELECT t.album_id                                  AS album_id,
+           COUNT(*)                                    AS tracks,
+           SUM(COALESCE(t.size, 0))                    AS bytes,
+           MAX(COALESCE(t.added, 0))                   AS newest,
+           SUM(CASE WHEN p.track_id IS NULL THEN 0 ELSE 1 END) AS plays
+      FROM lidarr_track t
+      LEFT JOIN played p ON p.track_id = t.id
+     WHERE t.has_file = 1
+     GROUP BY t.album_id
+)
+SELECT al.id       AS album_id,
+       al.title    AS title,
+       ar.id       AS artist_id,
+       ar.name     AS artist,
+       album.tracks AS tracks,
+       album.bytes  AS bytes,
+       album.newest AS newest
+  FROM album
+  JOIN lidarr_album al ON al.id = album.album_id
+  JOIN lidarr_artist ar ON ar.id = al.artist_id
+ WHERE album.plays = 0
+   AND album.newest > 0
+   AND album.newest < :cutoff
+"""
+
+
+def cold_albums(store, cutoff):
+    """Return every album with files, none of them ever played, old enough to judge."""
+    return store.connection.execute(COLD_ALBUMS, {"cutoff": cutoff}).fetchall()
+
+
+def cold_report(store, now, cold_after_days=COLD_AFTER_DAYS):
+    """Report what has never been played. Writes nothing, anywhere.
+
+    Album-level rather than track-level: a record with two played tracks is a
+    record that gets played, and picking the other ten off it leaves gaps rather
+    than reclaiming anything worth having.
+    """
+    refusal = cull_is_safe(store)
+    if refusal is not None:
+        logger.warning("no cold report: %s", refusal)
+        return []
+
+    cutoff = now - cold_after_days * 86400
+    rows = cold_albums(store, cutoff)
+    if not rows:
+        logger.info("--- cold albums --- none: everything with a file has been played")
+        return []
+
+    total_bytes = sum(row["bytes"] or 0 for row in rows)
+    total_tracks = sum(row["tracks"] or 0 for row in rows)
+    albums_with_files = store.scalar(
+        "SELECT COUNT(DISTINCT album_id) FROM lidarr_track WHERE has_file = 1"
+    )
+
+    logger.info("--- cold albums (never played, files older than %d days) ---", cold_after_days)
+    logger.info(
+        "%d of %d albums, %d tracks, %s",
+        len(rows),
+        albums_with_files,
+        total_tracks,
+        human_bytes(total_bytes),
+    )
+
+    # An artist every one of whose albums is cold is a different proposition
+    # from one cold record by someone otherwise played, and Lidarr can only tag
+    # at artist level anyway.
+    cold_by_artist = {}
+    for row in rows:
+        held = cold_by_artist.setdefault(row["artist_id"], {"name": row["artist"], "albums": 0,
+                                                            "bytes": 0})
+        held["albums"] += 1
+        held["bytes"] += row["bytes"] or 0
+
+    owned = dict(
+        store.connection.execute(
+            "SELECT al.artist_id, COUNT(DISTINCT al.id) FROM lidarr_album al"
+            "  JOIN lidarr_track t ON t.album_id = al.id AND t.has_file = 1"
+            " GROUP BY al.artist_id"
+        ).fetchall()
+    )
+    entirely = [
+        held for artist_id, held in cold_by_artist.items()
+        if held["albums"] == owned.get(artist_id)
+    ]
+    logger.info(
+        "%d artists are cold in their entirety (%s), %d have some cold records",
+        len(entirely),
+        human_bytes(sum(held["bytes"] for held in entirely)),
+        len(cold_by_artist) - len(entirely),
+    )
+
+    for position, held in enumerate(
+        sorted(cold_by_artist.values(), key=lambda held: -held["bytes"])[:20], start=1
+    ):
+        logger.info(
+            "  cold %2d: %-40s %2d albums, %s",
+            position,
+            held["name"][:40],
+            held["albums"],
+            human_bytes(held["bytes"]),
+        )
+
+    logger.info("nothing was changed in Lidarr: this report does not write")
+    return rows
+
+
+def report(store, now, cold_after=COLD_AFTER_DAYS):
     """Log what the store holds.
 
     The MBID coverage line is the one to watch: scrobbles carrying a MusicBrainz
@@ -1481,6 +2080,7 @@ def report(store, now):
         logger.info("  top artist %2d: %-40s %d", position, row["artist"][:40], row["plays"])
 
     coverage_report(store)
+    cold_report(store, now, cold_after)
 
     recent = store.connection.execute(
         "SELECT artist, track, COUNT(*) AS plays FROM scrobble WHERE uts >= ?"
@@ -1494,7 +2094,7 @@ def report(store, now):
 
 def run_once(
     client, store, user, now, backfill_limit, lidarr=None, mirror=None,
-    library_root=None, playlist_limit=100,
+    library_root=None, playlist_limit=100, vibes=DEFAULT_VIBES, tag_limit=0,
 ):
     """Run a single pass. Returns the number of scrobbles added."""
     started = time.monotonic()
@@ -1516,7 +2116,14 @@ def run_once(
                     " written; set --library-root"
                 )
             else:
-                build_playlists(store, mirror, root, playlist_limit, now)
+                _, from_history = build_playlists(store, mirror, root, playlist_limit, now)
+                sync_tags(client, store, now, limit=tag_limit)
+                _, from_tags = build_vibe_playlists(
+                    store, vibes, mirror, root, playlist_limit, now
+                )
+                prune_playlists(
+                    Path(mirror) / PLAYLIST_DIRECTORY, from_history + from_tags, store
+                )
 
     logger.info(
         "pass complete in %.1fs: %d scrobbles added, %d loved",
@@ -1608,6 +2215,25 @@ def build_parser():
         default=int(os.getenv("MUSIC_CURATOR_PLAYLIST_LIMIT", "100")),
         help="most tracks to put in any one playlist (env MUSIC_CURATOR_PLAYLIST_LIMIT)",
     )
+    parser.add_argument(
+        "--vibes",
+        default=os.getenv("MUSIC_CURATOR_VIBES"),
+        help="JSON file of mood definitions, replacing the built-in set"
+        " (env MUSIC_CURATOR_VIBES)",
+    )
+    parser.add_argument(
+        "--tag-limit",
+        type=int,
+        default=int(os.getenv("MUSIC_CURATOR_TAG_LIMIT", "0")),
+        help="cap artist tag lookups per pass; 0 for no cap (env MUSIC_CURATOR_TAG_LIMIT)",
+    )
+    parser.add_argument(
+        "--cold-after",
+        type=int,
+        default=int(os.getenv("MUSIC_CURATOR_COLD_AFTER", str(COLD_AFTER_DAYS))),
+        help="days a file must sit unplayed before its album counts as cold"
+        " (env MUSIC_CURATOR_COLD_AFTER)",
+    )
     parser.add_argument(
         "--skip-index",
         action="store_true",
@@ -1632,6 +2258,7 @@ def main(argv=None, clock=time.time):
 
     try:
         interval = parse_interval(args.interval) if args.interval else None
+        vibes = load_vibes(args.vibes)
     except ValueError as error:
         logger.error("%s", error)
         return 2
@@ -1645,7 +2272,7 @@ def main(argv=None, clock=time.time):
     store = Store(database)
 
     if args.report_only:
-        report(store, int(clock()))
+        report(store, int(clock()), args.cold_after)
         store.close()
         lock.close()
         return 0
@@ -1682,12 +2309,14 @@ def main(argv=None, clock=time.time):
                     args.mirror,
                     args.library_root,
                     args.playlist_limit,
+                    vibes,
+                    args.tag_limit,
                 )
             except (LastfmError, LidarrError) as error:
                 logger.error("%s", error)
                 if interval is None:
                     return 1
-            report(store, now)
+            report(store, now, args.cold_after)
 
             if interval is None or stopping:
                 return 0

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "music-curator"
-version = "0.4.0"
+version = "0.7.0"
 description = "Ingest a Last.fm listening history and curate a music library from it"
 readme = "README.md"
 requires-python = ">=3.11"

tests/conftest.py

@@ -56,7 +56,9 @@ class FakeLastfm:
     is prepended to the first page with no `date`.
     """
 
-    def __init__(self, tracks=(), loved=(), nowplaying=None, outcomes=()):
+    def __init__(self, tracks=(), loved=(), nowplaying=None, outcomes=(), tags=None):
+        # Keyed by mbid or by artist name, whichever the caller asked with.
+        self.tags = tags or {}
         self.tracks = sorted(tracks, key=lambda track: int(track["date"]["uts"]), reverse=True)
         self.loved = list(loved)
         self.nowplaying = nowplaying
@@ -82,6 +84,15 @@ class FakeLastfm:
             return json.dumps(self._recent(query))
         if method == "user.getlovedtracks":
             return json.dumps(self._loved(query))
+        if method == "artist.gettoptags":
+            key = query.get("mbid") or query.get("artist", "")
+            if key not in self.tags:
+                # What the real service says for a key it cannot resolve, which
+                # for mbids is a great many artists whose pages plainly exist.
+                return json.dumps(
+                    {"error": 6, "message": "The artist you supplied could not be found"}
+                )
+            return json.dumps({"toptags": {"tag": self.tags[key]}})
         raise AssertionError(f"unexpected method {method}")
 
     def _recent(self, query):

tests/test_music_curator.py

@@ -1,6 +1,8 @@
 import json
+import os
 import stat
 import urllib.error
+from datetime import datetime, timezone
 from pathlib import Path
 
 import pytest
@@ -873,23 +875,23 @@ def test_playlists_are_written_into_the_mirror(tmp_path):
 
     music_curator.build_playlists(store, mirror, str(source), 100, NOW)
 
-    written = sorted(p.name for p in (mirror / "_playlists").glob("*.m3u"))
+    written = sorted(p.name for p in (mirror / "_playlists").glob("*.m3u8"))
     assert written == [
-        "all-time.m3u",
+        "all-time.m3u8",
-        "deep-cuts.m3u",
+        "deep-cuts.m3u8",
-        "heavy-rotation.m3u",
+        "heavy-rotation.m3u8",
-        "neglected.m3u",
+        "neglected.m3u8",
-        "unheard-favourites.m3u",
+        "unheard-favourites.m3u8",
-        "unheard.m3u",
+        "unheard.m3u8",
     ]
 
-    played = (mirror / "_playlists" / "all-time.m3u").read_text().splitlines()
+    played = (mirror / "_playlists" / "all-time.m3u8").read_text().splitlines()
     assert played[0] == "#EXTM3U"
     assert played[1].startswith("#EXTINF:")
     assert "Played Band - Hit" in played[1]
     # Relative to the playlist file, so the same file works from any mount.
     assert played[2] == "../Played Band/Known/Hit.mp3"
-    assert (mirror / "_playlists" / "all-time.m3u").parent.joinpath(played[2]).resolve().is_file()
+    assert (mirror / "_playlists" / "all-time.m3u8").parent.joinpath(played[2]).resolve().is_file()
 
 
 def test_an_unplayed_track_lands_in_the_unheard_playlist(tmp_path):
@@ -903,7 +905,7 @@ def test_an_unplayed_track_lands_in_the_unheard_playlist(tmp_path):
 
     music_curator.build_playlists(store, mirror, str(source), 100, NOW)
 
-    unheard = (mirror / "_playlists" / "unheard.m3u").read_text()
+    unheard = (mirror / "_playlists" / "unheard.m3u8").read_text()
     assert "Never Heard" in unheard
     assert "Album Track" in unheard
     # The one thing that was played must not be in it.
@@ -923,10 +925,11 @@ def test_a_track_with_no_mirror_file_is_left_out(tmp_path):
     )
     music_curator.match_library(store)
 
-    total = music_curator.build_playlists(store, mirror, str(source), 100, NOW)
+    total, produced = music_curator.build_playlists(store, mirror, str(source), 100, NOW)
 
     assert total == 0
-    assert (mirror / "_playlists" / "all-time.m3u").read_text() == "#EXTM3U\n"
+    assert produced, "the playlists are still written, they are simply empty"
+    assert (mirror / "_playlists" / "all-time.m3u8").read_text() == "#EXTM3U\n"
 
 
 def test_the_library_root_is_derived_from_the_artist_folders(tmp_path):
@@ -949,7 +952,7 @@ def test_the_playlist_limit_is_honoured(tmp_path):
 
     music_curator.build_playlists(store, mirror, str(source), 1, NOW)
 
-    unheard = (mirror / "_playlists" / "unheard.m3u").read_text().splitlines()
+    unheard = (mirror / "_playlists" / "unheard.m3u8").read_text().splitlines()
     assert len([line for line in unheard if line.startswith("#EXTINF")]) == 1
 
 
@@ -965,7 +968,7 @@ def test_the_rotation_moves_weekly_not_every_pass(tmp_path):
 
     def unheard_at(when):
         music_curator.build_playlists(store, mirror, str(source), 100, when)
-        return (mirror / "_playlists" / "unheard.m3u").read_text()
+        return (mirror / "_playlists" / "unheard.m3u8").read_text()
 
     same_week = unheard_at(NOW), unheard_at(NOW + 3600)
     assert same_week[0] == same_week[1]
@@ -981,5 +984,523 @@ def test_playlists_are_group_readable(tmp_path):
 
     music_curator.build_playlists(store, mirror, str(source), 100, NOW)
 
-    for playlist in (mirror / "_playlists").glob("*.m3u"):
+    for playlist in (mirror / "_playlists").glob("*.m3u8"):
         assert playlist.stat().st_mode & stat.S_IRGRP, playlist
+
+
+def test_tag_weights_fall_back_to_rank_when_no_count_is_sent():
+    """The documented sample carries only a name and a URL; a live response also
+    carries a count. Neither may be relied on alone."""
+    with_count = music_curator.parse_tags(
+        {"toptags": {"tag": [{"name": "Screamo", "count": 100}, {"name": "emo", "count": 40}]}}
+    )
+    assert with_count == [("screamo", 100), ("emo", 40)]
+
+    without = music_curator.parse_tags(
+        {"toptags": {"tag": [{"name": "screamo"}, {"name": "emo"}]}}
+    )
+    assert [tag for tag, _ in without] == ["screamo", "emo"]
+    assert without[0][1] > without[1][1]
+
+
+def test_a_lone_tag_is_not_a_list():
+    assert music_curator.parse_tags({"toptags": {"tag": {"name": "dnb", "count": 90}}}) == [
+        ("dnb", 90)
+    ]
+
+
+def test_an_artist_lastfm_cannot_answer_for_is_not_asked_again(tmp_path):
+    """Recording the fetch even when it returns nothing is what stops a pass
+    spending a request per unknown artist, forever."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(tags={})
+
+    assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 2
+    before = len(lastfm.calls)
+
+    assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 0
+    assert len(lastfm.calls) == before
+
+
+def test_tags_are_refetched_once_they_go_stale(tmp_path):
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(tags={})
+    music_curator.sync_tags(client_for(lastfm), store, NOW)
+
+    later = NOW + music_curator.TAG_REFRESH_SECONDS + 1
+    assert music_curator.sync_tags(client_for(lastfm), store, later) == 2
+
+
+def tagged_store(tmp_path, tags):
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    music_curator.sync_tags(client_for(FakeLastfm(tags=tags)), store, NOW)
+    return store, source, mirror
+
+
+def test_a_vibe_selects_by_tag(tmp_path):
+    store, source, mirror = tagged_store(
+        tmp_path,
+        {
+            "Played Band": [{"name": "screamo", "count": 100}],
+            "Silent Band": [{"name": "classic rock", "count": 100}],
+        },
+    )
+    vibes = [{"name": "screamo", "tags": ["screamo"]}]
+
+    music_curator.build_vibe_playlists(store, vibes, mirror, str(source), 100, NOW)
+
+    written = (mirror / "_playlists" / "screamo.m3u8").read_text()
+    assert "Played Band" in written
+    assert "Silent Band" not in written
+
+
+def test_a_weakly_tagged_artist_is_below_the_threshold(tmp_path):
+    """A single low-weight tag is not a genre, it is somebody's stray opinion."""
+    store, source, mirror = tagged_store(
+        tmp_path, {"Played Band": [{"name": "screamo", "count": 3}]}
+    )
+    vibes = [{"name": "screamo", "tags": ["screamo"]}]
+
+    music_curator.build_vibe_playlists(store, vibes, mirror, str(source), 100, NOW)
+
+    assert (mirror / "_playlists" / "screamo.m3u8").read_text() == "#EXTM3U\n"
+
+
+def test_a_vibe_can_be_restricted_by_release_year(tmp_path):
+    """What separates eighties synth records from everything else a synthpop
+    tag drags in."""
+    store, source, mirror = tagged_store(
+        tmp_path, {"Played Band": [{"name": "synthpop", "count": 100}]}
+    )
+    inside = [{"name": "eighties", "tags": ["synthpop"], "years": [1975, 1992]}]
+    outside = [{"name": "nineties", "tags": ["synthpop"], "years": [1993, 1999]}]
+
+    # FakeLidarr dates every album 2019, so neither window should catch it.
+    music_curator.build_vibe_playlists(store, inside, mirror, str(source), 100, NOW)
+    music_curator.build_vibe_playlists(store, outside, mirror, str(source), 100, NOW)
+    assert (mirror / "_playlists" / "eighties.m3u8").read_text() == "#EXTM3U\n"
+    assert (mirror / "_playlists" / "nineties.m3u8").read_text() == "#EXTM3U\n"
+
+    modern = [{"name": "modern", "tags": ["synthpop"], "years": [2000, 2030]}]
+    music_curator.build_vibe_playlists(store, modern, mirror, str(source), 100, NOW)
+    assert "Played Band" in (mirror / "_playlists" / "modern.m3u8").read_text()
+
+
+def test_the_built_in_vibes_are_all_usable_filenames():
+    for vibe in music_curator.DEFAULT_VIBES:
+        assert music_curator.SAFE_VIBE_NAME.fullmatch(vibe["name"]), vibe["name"]
+        assert vibe["tags"]
+
+
+def test_a_vibes_file_replaces_the_built_in_set(tmp_path):
+    path = tmp_path / "vibes.json"
+    path.write_text(json.dumps([{"name": "mine", "tags": ["shoegaze"]}]))
+
+    assert music_curator.load_vibes(str(path)) == ({"name": "mine", "tags": ["shoegaze"]},)
+    assert music_curator.load_vibes(None) is music_curator.DEFAULT_VIBES
+
+
+@pytest.mark.parametrize(
+    "content",
+    [
+        '{"not": "a list"}',
+        '[{"tags": ["x"]}]',
+        '[{"name": "../escape", "tags": ["x"]}]',
+        '[{"name": "ok"}]',
+        "not json at all",
+    ],
+)
+def test_a_bad_vibes_file_is_refused_up_front(tmp_path, content):
+    """A bad name would otherwise surface as a file written somewhere
+    unintended, which is a poor way to learn about a typo."""
+    path = tmp_path / "vibes.json"
+    path.write_text(content)
+
+    with pytest.raises(ValueError):
+        music_curator.load_vibes(str(path))
+
+
+def test_tags_are_looked_up_by_name_not_by_mbid(tmp_path):
+    """Last.fm's mbid index is stale: it cannot find Devo or Escape the Fate by
+    one, though their pages plainly exist. Its name index can."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(tags={"Played Band": [{"name": "screamo", "count": 90}]})
+
+    music_curator.sync_tags(client_for(lastfm), store, NOW)
+
+    asked = [call for call in lastfm.calls if call.get("method") == "artist.getTopTags"]
+    # The name is always tried first; the mbid only appears as a fallback for
+    # the artist that the name could not resolve.
+    assert "artist" in asked[0]
+    assert [call for call in asked if call.get("artist") == "Played Band"]
+    assert not [call for call in asked if call.get("mbid") == "artist-mbid-1"]
+    assert store.scalar("SELECT COUNT(*) FROM artist_tag WHERE tag = 'screamo'") == 1
+
+
+def test_the_mbid_is_tried_when_the_name_is_not_found(tmp_path):
+    """Kept only for a name Lidarr spells differently to Last.fm."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(tags={"artist-mbid-1": [{"name": "dnb", "count": 80}]})
+
+    music_curator.sync_tags(client_for(lastfm), store, NOW)
+
+    assert store.scalar("SELECT COUNT(*) FROM artist_tag WHERE tag = 'dnb'") == 1
+    asked = [call for call in lastfm.calls if call.get("method") == "artist.getTopTags"]
+    assert any("mbid" in call for call in asked)
+
+
+def test_an_artist_neither_key_resolves_is_recorded_and_not_retried(tmp_path):
+    """Otherwise every pass spends a request on it again, for ever."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(tags={})
+
+    assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 2
+    spent = len(lastfm.calls)
+
+    assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 0
+    assert len(lastfm.calls) == spent
+
+
+def test_a_real_failure_is_not_recorded_so_the_next_pass_retries(tmp_path):
+    """A rate limit is not the same as an artist not existing."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    lastfm = FakeLastfm(
+        tags={"Played Band": [], "Silent Band": []},
+        outcomes=[{"error": 10, "message": "Invalid API key"}],
+    )
+
+    music_curator.sync_tags(client_for(lastfm), store, NOW)
+
+    # One artist failed hard and must still be pending.
+    assert store.scalar("SELECT COUNT(*) FROM artist_tag_fetched") == 1
+
+
+# Tags that carry real weight in a real library, against tags that describe a
+# passport, span most of the collection, or are somebody's artist name.
+USELESS_TAGS = {
+    "american", "british", "australian", "canadian", "swedish", "dutch", "german",
+    "scottish", "english", "uk", "usa", "canada",
+    "rock", "electronic", "pop", "alternative", "metal ", "all", "heavy",
+    "female vocalists", "male vocalists", "female vocalist",
+    "my top songs", "cover", "covers", "not emo",
+    "green day", "paramore", "queen", "bon jovi", "shinedown", "aerosmith",
+    "journey", "fleetwood mac",
+}
+
+
+def test_no_mood_selects_on_a_useless_tag():
+    """Nationality is not a sound; `rock` and `electronic` span most of the
+    library; and Last.fm's top tag for an artist is often their own name."""
+    for vibe in music_curator.DEFAULT_VIBES:
+        overlap = {tag.casefold() for tag in vibe["tags"]} & USELESS_TAGS
+        assert not overlap, f"{vibe['name']} selects on {overlap}"
+
+
+def test_mood_names_are_unique():
+    names = [vibe["name"] for vibe in music_curator.DEFAULT_VIBES]
+    assert len(names) == len(set(names))
+
+
+ROCK_TAGS = {
+    "classic rock", "hard rock", "blues rock", "southern rock", "arena rock",
+    "glam rock", "hair metal", "glam metal", "heavy metal", "metal", "art rock",
+    "psychedelic rock", "progressive rock", "rock and roll", "rock n roll",
+}
+
+
+def test_a_decade_tag_in_a_non_rock_mood_must_exclude_the_rock():
+    """`80s` sits on Def Leppard and Bon Jovi as heavily as on Eurythmics, so a
+    mood that reaches for a decade without wanting rock has to say so. A mood
+    that does want it -- classic-rock reaching for 70s -- is exempt."""
+    for vibe in music_curator.DEFAULT_VIBES:
+        tags = {tag.casefold() for tag in vibe["tags"]}
+        decades = {"60s", "70s", "80s", "90s"} & tags
+        if not decades or tags & ROCK_TAGS:
+            continue
+        excluded = {tag.casefold() for tag in vibe.get("exclude", [])}
+        missing = {"hard rock", "hair metal"} - excluded
+        assert not missing, f"{vibe['name']} selects on {decades} without excluding {missing}"
+
+
+def test_the_eighties_mood_covers_the_canon():
+    """Depeche Mode, Duran Duran, Eurythmics and Frankie Goes to Hollywood --
+    checked against their live Last.fm tags. Three of the four carry "synth pop"
+    with a space; only one carries it without."""
+    synths = next(v for v in music_curator.DEFAULT_VIBES if v["name"] == "80s-synths")
+    tags = {t.casefold() for t in synths["tags"]}
+    for artist_tags in (
+        {"80s", "new wave", "pop", "female vocalists", "synth pop"},   # Eurythmics
+        {"80s", "new wave", "pop", "british", "dance"},                # Frankie
+        {"electronic", "synthpop", "new wave", "80s", "synth pop"},    # Depeche Mode
+        {"new wave", "80s", "pop", "synth pop", "rock"},               # Duran Duran
+    ):
+        assert tags & artist_tags, artist_tags
+    assert synths["years"] == [1975, 1992]
+
+
+def test_an_excluded_tag_drops_the_artist(tmp_path):
+    """Weighting cannot separate the eighties synth acts from the eighties
+    stadium rock, because the tag they would be weighted on is the one they
+    share."""
+    store, source, mirror = tagged_store(
+        tmp_path,
+        {
+            "Played Band": [{"name": "80s", "count": 100}, {"name": "hard rock", "count": 90}],
+            "Silent Band": [{"name": "80s", "count": 100}, {"name": "synth pop", "count": 90}],
+        },
+    )
+    vibes = [{"name": "eighties", "tags": ["80s", "synth pop"], "exclude": ["hard rock"]}]
+
+    music_curator.build_vibe_playlists(store, vibes, mirror, str(source), 100, NOW)
+
+    written = (mirror / "_playlists" / "eighties.m3u8").read_text()
+    assert "Silent Band" in written
+    assert "Played Band" not in written
+
+
+def test_a_vibe_cannot_both_select_and_exclude_a_tag(tmp_path):
+    path = tmp_path / "vibes.json"
+    path.write_text(json.dumps([{"name": "x", "tags": ["80s"], "exclude": ["80s"]}]))
+
+    with pytest.raises(ValueError, match="selects on and excludes"):
+        music_curator.load_vibes(str(path))
+
+
+def test_matching_ownership_is_a_no_op_when_it_already_agrees(tmp_path):
+    target = tmp_path / "file"
+    target.write_text("x")
+
+    assert music_curator.match_ownership(target, tmp_path) is False
+
+
+def test_ownership_failure_is_tolerated(tmp_path):
+    """Not permitted unless running as root -- which is exactly the case where
+    the ownership is already whatever the caller runs as."""
+    target = tmp_path / "file"
+    target.write_text("x")
+
+    # uid 0 from a non-root test process: refused, and must not raise.
+    assert music_curator.set_ownership(target, 0, 0) is False
+
+
+def test_a_playlist_is_chowned_to_match_the_mirror(tmp_path, monkeypatch):
+    """The image runs as root, so its output is root-owned, and a root-owned
+    playlist in an apps-owned mirror is unreadable to whatever serves it."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    music_curator.match_library(store)
+
+    attempted = []
+    real_stat = music_curator.Path.stat
+
+    def pretend_mirror_is_owned_by_568(self, *args, **kwargs):
+        info = real_stat(self, *args, **kwargs)
+        if self == mirror:
+            return os.stat_result(
+                (info.st_mode, info.st_ino, info.st_dev, info.st_nlink, 568, 568,
+                 info.st_size, int(info.st_atime), int(info.st_mtime), int(info.st_ctime))
+            )
+        return info
+
+    monkeypatch.setattr(music_curator.Path, "stat", pretend_mirror_is_owned_by_568)
+    monkeypatch.setattr(
+        music_curator.os, "chown", lambda p, u, g: attempted.append((str(p), u, g))
+    )
+
+    music_curator.build_playlists(store, mirror, str(source), 100, NOW)
+
+    assert attempted, "no ownership was applied"
+    assert all(tuple(owner) == (568, 568) for _, *owner in attempted)
+    # The temporary file, before the rename, never the finished playlist.
+    assert all(path.endswith(".part") or path.endswith("_playlists") for path, *_ in attempted)
+
+
+def test_a_renamed_mood_does_not_leave_its_old_playlist_behind(tmp_path):
+    """Otherwise the dead file stays on the device for ever."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    directory = mirror / "_playlists"
+    directory.mkdir(parents=True, exist_ok=True)
+    (directory / "high-energy-rock.m3u").write_text("#EXTM3U\n")
+    (directory / "screamo.m3u8").write_text("#EXTM3U\n")
+    store.set_state(
+        "playlist_files", json.dumps(["high-energy-rock.m3u", "screamo.m3u8"])
+    )
+
+    music_curator.prune_playlists(directory, ["screamo.m3u8"], store)
+
+    assert not (directory / "high-energy-rock.m3u").exists()
+    assert (directory / "screamo.m3u8").exists()
+
+
+def test_pruning_leaves_a_playlist_it_never_wrote(tmp_path):
+    """A hand-made playlist in that directory is not ours to delete."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    directory = mirror / "_playlists"
+    directory.mkdir(parents=True, exist_ok=True)
+    (directory / "lyras-own-mix.m3u").write_text("#EXTM3U\n")
+    store.set_state("playlist_files", json.dumps(["screamo.m3u8"]))
+
+    music_curator.prune_playlists(directory, [], store)
+
+    assert (directory / "lyras-own-mix.m3u").exists()
+
+
+def cold_store(tmp_path, played=(), added="2020-05-01T12:00:00Z"):
+    """A library where nothing is played unless named, indexed and matched."""
+    library = [
+        {
+            "name": "Played Band",
+            "albums": [{"title": "Known", "tracks": [{"title": "Hit", "added": added}]}],
+        },
+        {
+            "name": "Silent Band",
+            "albums": [
+                {"title": "Unknown", "tracks": [{"title": "Never Heard", "added": added}]}
+            ],
+        },
+    ]
+    store = store_at(tmp_path)
+    ingest(store, FakeLastfm([scrobble_of(a, t) for a, t in played]))
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=FakeLidarr(library)), store
+    )
+    music_curator.match_library(store)
+    return store
+
+
+def test_an_album_with_no_plays_is_cold(tmp_path):
+    store = cold_store(tmp_path, played=[("Played Band", "Hit")])
+
+    cold = music_curator.cold_report(store, NOW)
+
+    assert [row["artist"] for row in cold] == ["Silent Band"]
+
+
+def test_an_album_with_any_play_is_not_cold(tmp_path):
+    """A record with one played track is a record that gets played; picking the
+    rest off it leaves gaps rather than reclaiming anything."""
+    store = cold_store(tmp_path, played=[("Played Band", "Hit")])
+
+    cold = music_curator.cold_report(store, NOW)
+
+    assert "Played Band" not in [row["artist"] for row in cold]
+
+
+def test_a_recent_arrival_is_too_young_to_judge(tmp_path):
+    """It has not had a chance to be played yet."""
+    recent = datetime.fromtimestamp(NOW - 30 * 86400, tz=timezone.utc).isoformat()
+    store = cold_store(tmp_path, played=[("Played Band", "Hit")], added=recent)
+
+    assert music_curator.cold_report(store, NOW) == []
+
+
+def test_the_cull_refuses_to_run_on_an_incomplete_index(tmp_path):
+    """A missing artist makes their played music look unplayed, which is
+    precisely how this would delete something you like."""
+    store = cold_store(tmp_path, played=[("Played Band", "Hit")])
+    store.set_state("index_skipped", "1")
+
+    assert music_curator.cull_is_safe(store) is not None
+    assert music_curator.cold_report(store, NOW) == []
+
+
+def test_the_cull_refuses_while_the_backfill_is_unfinished(tmp_path):
+    store = cold_store(tmp_path, played=[("Played Band", "Hit")])
+    store.set_state("backfill_complete", "no")
+
+    assert music_curator.cold_report(store, NOW) == []
+
+
+def test_the_cold_report_writes_nothing_to_lidarr(tmp_path):
+    """Stage four is read-only until the list it produces has been looked at."""
+    store = cold_store(tmp_path, played=[("Played Band", "Hit")])
+    before = store.scalar("SELECT COUNT(*) FROM lidarr_album WHERE monitored = 1")
+
+    music_curator.cold_report(store, NOW)
+
+    assert store.scalar("SELECT COUNT(*) FROM lidarr_album WHERE monitored = 1") == before
+
+
+def test_human_bytes_reads_at_a_glance():
+    assert music_curator.human_bytes(0) == "0.0 B"
+    assert music_curator.human_bytes(1536) == "1.5 KiB"
+    assert music_curator.human_bytes(3 * 1024**3) == "3.0 GiB"
+
+
+def test_playlists_are_written_as_m3u8(tmp_path):
+    """Rockbox's is_m3u8_name() treats every extension as UTF-8 except an
+    explicit ".m3u", which it decodes through the configured codepage instead.
+    A library with accented names needs the other extension."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    music_curator.index_library(
+        music_curator.Lidarr("http://lidarr", "key", transport=api), store
+    )
+    music_curator.match_library(store)
+
+    music_curator.build_playlists(store, mirror, str(source), 100, NOW)
+
+    written = sorted(p.suffix for p in (mirror / "_playlists").iterdir())
+    assert set(written) == {".m3u8"}
+
+
+def test_playlists_are_written_as_utf8(tmp_path):
+    playlist = tmp_path / "_playlists" / "accents.m3u8"
+    music_curator.write_playlist(
+        playlist,
+        [{"artist": "Mötley Crüe", "title": "Kickstart My Heart",
+          "duration": 283000, "mirror": tmp_path / "x.mp3"}],
+    )
+
+    # Decodes as UTF-8, and carries no BOM: Rockbox does not need one at this
+    # extension, and a BOM confuses players that do not expect it.
+    raw = playlist.read_bytes()
+    assert not raw.startswith(b"\xef\xbb\xbf")
+    assert "Mötley Crüe" in raw.decode("utf-8")
+
+
+def test_the_old_m3u_playlists_are_pruned_after_the_rename(tmp_path):
+    """Without this the seventeen dead .m3u files stay on the device for ever."""
+    api, source, mirror = playlist_library(tmp_path)
+    store = store_at(tmp_path)
+    directory = mirror / "_playlists"
+    directory.mkdir(parents=True, exist_ok=True)
+    (directory / "screamo.m3u").write_text("#EXTM3U\n")
+    store.set_state("playlist_files", json.dumps(["screamo.m3u"]))
+
+    music_curator.prune_playlists(directory, ["screamo.m3u8"], store)
+
+    assert not (directory / "screamo.m3u").exists()