diff --git a/README.md b/README.md index 852ccb9..b0b4fb4 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,10 @@ which keeps an MP3 copy of a lossless library for an iPod. This one answers the question that mirror cannot: which of it is worth carrying, and which of it has not been played in years. -**This is stage two.** It ingests the scrobble history, indexes the library from -Lidarr, and matches one to the other. There are no playlists yet, and it writes -nothing back — every Lidarr call is a `GET`. See "Where this is going" below. +**This is stage three.** It ingests the scrobble history, indexes the library +from Lidarr, matches one to the other, and writes playlists into the mirror. +Nothing is written back to Lidarr — every call there is a `GET`. See "Where this +is going" below. ## What it does today @@ -18,6 +19,7 @@ nothing back — every Lidarr call is a `GET`. See "Where this is going" below. - Indexes every artist, album and track Lidarr knows about, with file paths and the date each file landed. - Ties the two together and reports how well it managed. +- Writes M3U playlists into the mirror, from the listening history. ## Matching @@ -126,6 +128,50 @@ and a shared title is a weak proxy for a shared song: Only the first is worth chasing. Counting all three as matcher failures overstates the problem and would over-block the cull. +## Playlists + +Written into `/_playlists/` as extended M3U, rebuilt every pass. Six +rules, capped at `--playlist-limit` tracks each: + +| Playlist | Rule | +| -------------------- | --------------------------------------------------------- | +| `heavy-rotation` | Most played over the last twelve months | +| `all-time` | Most played ever | +| `neglected` | Played heavily once, silent for twelve months | +| `deep-cuts` | Never played, from albums whose other tracks you play constantly | +| `unheard-favourites` | Never played, by the artists you play most | +| `unheard` | Never played, anywhere in the library | + +Ninety days was the obvious window for "recent" and is the wrong one: on a real +history it holds a few hundred plays spread thinly across a twenty-thousand +track rotation, so nothing ranks meaningfully. Twelve months does. + +The two `unheard` playlists rotate **weekly**, not per pass. A pass runs every +few hours, and a playlist that reorders itself each time is one that has to be +re-imported each time — the Music app imports a snapshot of a file, it does not +track it. + +### Paths + +Lidarr knows where the lossless source is; the playlists have to point at the +MP3s music-mirror made from it. The mapping strips a library root from Lidarr's +track paths and re-roots them under the mirror, with the suffix changed. + +`--library-root` is derived from the common parent of the indexed artist folders +when unset, so it agrees with Lidarr by construction rather than by being kept +in step by hand. Override it if that guess is wrong. + +Entries are written **relative to the playlist file**, so one playlist works +from the NAS, from a Mac over SMB, and from Linux, without rewriting. + +A track is only listed once its mirror file has been confirmed to exist. Lidarr +holding the FLAC says nothing about whether the MP3 has been encoded yet. If a +large number are missing, the run says so — that is what a wrong `--library-root` +or `--mirror` looks like, since the paths then map to nothing at all. + +`_playlists/` survives music-mirror's prune: it only deletes `*.mp3`, and its +empty-directory sweep skips a directory holding M3Us. + ## How the ingest works Two halves, both taking their bounds from the database rather than from a saved @@ -180,6 +226,9 @@ music-curator --report-only # report on the store, fetch nothing | `--backfill-limit` | `MUSIC_CURATOR_BACKFILL_LIMIT` | `0` | Cap backfill requests per pass; 0 for no cap | | `--lidarr-url` | `MUSIC_CURATOR_LIDARR_URL` | unset | Lidarr base URL, e.g. `http://lidarr:8686` | | `--lidarr-api-key` | `MUSIC_CURATOR_LIDARR_API_KEY` | unset | Lidarr API key | +| `--mirror` | `MUSIC_CURATOR_MIRROR` | unset | Root of the MP3 mirror; playlists go here | +| `--library-root` | `MUSIC_CURATOR_LIBRARY_ROOT` | derived | Prefix to strip from Lidarr's paths | +| `--playlist-limit` | `MUSIC_CURATOR_PLAYLIST_LIMIT` | `100` | Most tracks in any one playlist | | `--skip-index` | — | off | Match against the index already held | | `--report-only` | — | off | Report without fetching | @@ -228,7 +277,8 @@ nix shell nixpkgs#python3Packages.pytest -c pytest | ------------------------------------------------ | ------------ | | Last.fm ingest and store | done | | Lidarr index and the scrobble-to-track matcher | done | -| M3U playlists written into the mirror | next | +| M3U playlists from the listening history | done | +| Genre and mood playlists from Last.fm tags | next | | Cold-music report, unmonitoring what is not played | last | The cull will unmonitor cold albums in Lidarr and tag their artists. It will diff --git a/compose.yaml b/compose.yaml index c0168d7..7405862 100644 --- a/compose.yaml +++ b/compose.yaml @@ -30,5 +30,11 @@ services: # Cap the backfill at this many requests per pass. Unlimited by default, # which finishes a long history in one go. # MUSIC_CURATOR_BACKFILL_LIMIT: "0" + # The MP3 mirror music-mirror maintains. Playlists are written into + # _playlists/ inside it; leave unset to skip them. + MUSIC_CURATOR_MIRROR: /mirror + # Most tracks in any one playlist. + # MUSIC_CURATOR_PLAYLIST_LIMIT: "100" volumes: - /mnt/tank/apps/music-curator:/data + - /mnt/tank/media/music-mp3:/mirror diff --git a/music_curator.py b/music_curator.py index 55546f4..9d4d930 100644 --- a/music_curator.py +++ b/music_curator.py @@ -26,6 +26,7 @@ import re import signal import sqlite3 import sys +import tempfile import time import unicodedata import urllib.error @@ -62,6 +63,16 @@ REQUEST_DELAY_SECONDS = 0.25 BACKOFF_SECONDS = 2.0 BACKOFF_CEILING_SECONDS = 60.0 +# What music-mirror names its output, and where playlists are written inside the +# mirror. music-mirror's prune only deletes `*.mp3` and only removes directories +# it finds empty, so a directory of M3Us survives it untouched. +MIRROR_SUFFIX = ".mp3" +PLAYLIST_DIRECTORY = "_playlists" + +# Group-readable, for the same reason music-mirror sets it: the mirror is read +# back by whatever serves it, and a playlist nobody can read is not a playlist. +GROUP_READ = 0o040 + SCHEMA_VERSION = "2" # Versions this build upgrades in place. Everything added since version 1 is a @@ -1046,6 +1057,225 @@ def format_time(uts): return datetime.fromtimestamp(uts, tz=timezone.utc).strftime("%Y-%m-%d %H:%M") +# Playlists written into the mirror. Each is a rule over the listening history +# and the library index; none of them look at the audio. +# +# The rotating ones are shuffled by week rather than by pass. A pass runs every +# few hours, and a playlist that reorders itself every time is one that has to +# be re-imported every time -- the Music app does not track a file, it imports a +# snapshot of one. +ROTATION_PERIOD_SECONDS = 7 * 86400 +SHUFFLE_MULTIPLIER = 2654435761 +SHUFFLE_MODULUS = 104729 + +# Common table expressions the rules share. `played` is every library track that +# has ever been matched to a scrobble, with its count and the last time it was +# heard; `recent` is the same restricted to a window. +PLAYED_CTE = """ +WITH played AS ( + SELECT sk.track_id AS track_id, COUNT(*) AS plays, MAX(s.uts) AS last_uts + FROM scrobble s + JOIN scrobble_key sk ON sk.artist = s.artist AND sk.track = s.track + WHERE sk.track_id IS NOT NULL + GROUP BY sk.track_id +) +""" +RECENT_CTE = """ +WITH recent AS ( + SELECT sk.track_id AS track_id, COUNT(*) AS plays + FROM scrobble s + JOIN scrobble_key sk ON sk.artist = s.artist AND sk.track = s.track + WHERE sk.track_id IS NOT NULL AND s.uts >= :year_ago + GROUP BY sk.track_id +) +""" +SELECT_TRACK = """ +SELECT t.id AS id, a.name AS artist, t.title AS title, + t.duration AS duration, t.path AS path + FROM lidarr_track t + JOIN lidarr_artist a ON a.id = t.artist_id +""" +PLAYABLE = " t.has_file = 1 AND t.path IS NOT NULL AND t.path <> '' " +SHUFFLE = f" ((t.id * {SHUFFLE_MULTIPLIER}) + :week) % {SHUFFLE_MODULUS} " + +PLAYLISTS = ( + ( + "heavy-rotation", + "most played over the last twelve months", + RECENT_CTE + SELECT_TRACK + f""" + JOIN recent r ON r.track_id = t.id + WHERE {PLAYABLE} + ORDER BY r.plays DESC, a.name, t.title + LIMIT :limit + """, + ), + ( + "all-time", + "most played ever", + PLAYED_CTE + SELECT_TRACK + f""" + JOIN played p ON p.track_id = t.id + WHERE {PLAYABLE} + ORDER BY p.plays DESC, a.name, t.title + LIMIT :limit + """, + ), + ( + "neglected", + "played heavily once, silent for twelve months", + PLAYED_CTE + SELECT_TRACK + f""" + JOIN played p ON p.track_id = t.id + WHERE {PLAYABLE} AND p.last_uts < :year_ago + ORDER BY p.plays DESC, a.name, t.title + LIMIT :limit + """, + ), + ( + "deep-cuts", + "never played, from albums whose other tracks you play constantly", + PLAYED_CTE + """, + album_plays AS ( + SELECT t.album_id AS album_id, SUM(p.plays) AS plays + FROM played p JOIN lidarr_track t ON t.id = p.track_id + GROUP BY t.album_id + ) + """ + SELECT_TRACK + f""" + JOIN album_plays ap ON ap.album_id = t.album_id + LEFT JOIN played p ON p.track_id = t.id + WHERE {PLAYABLE} AND p.track_id IS NULL + ORDER BY ap.plays DESC, t.id + LIMIT :limit + """, + ), + ( + "unheard-favourites", + "never played, by the artists you play most; rotates weekly", + PLAYED_CTE + """, + artist_plays AS ( + SELECT t.artist_id AS artist_id, SUM(p.plays) AS plays + FROM played p JOIN lidarr_track t ON t.id = p.track_id + GROUP BY t.artist_id + ) + """ + SELECT_TRACK + f""" + LEFT JOIN played p ON p.track_id = t.id + WHERE {PLAYABLE} AND p.track_id IS NULL + AND t.artist_id IN (SELECT artist_id FROM artist_plays + ORDER BY plays DESC LIMIT 50) + ORDER BY {SHUFFLE} + LIMIT :limit + """, + ), + ( + "unheard", + "never played, anywhere in the library; rotates weekly", + PLAYED_CTE + SELECT_TRACK + f""" + LEFT JOIN played p ON p.track_id = t.id + WHERE {PLAYABLE} AND p.track_id IS NULL + ORDER BY {SHUFFLE} + LIMIT :limit + """, + ), +) + + +def library_root_of(store): + """Return the directory Lidarr's artist folders sit under. + + Derived rather than configured, because it has to agree with what Lidarr + reports and no one wants to keep a second copy of that in step by hand. + """ + paths = [ + row["path"] + for row in store.connection.execute( + "SELECT path FROM lidarr_artist WHERE path IS NOT NULL AND path <> ''" + ) + ] + if not paths: + return None + if len(paths) == 1: + return str(Path(paths[0]).parent) + try: + return os.path.commonpath(paths) + except ValueError: + return None + + +def mirror_path_for(source, library_root, mirror_root): + """Return where music-mirror would have put the MP3 for a source file.""" + try: + relative = Path(source).relative_to(library_root) + except ValueError: + return None + return (Path(mirror_root) / relative).with_suffix(MIRROR_SUFFIX) + + +def write_playlist(path, entries): + """Write one extended M3U, atomically. + + Paths are relative to the playlist file, so the same playlist works from the + NAS, from a Mac over SMB and from Linux without rewriting. + """ + lines = ["#EXTM3U"] + for entry in entries: + seconds = round((entry["duration"] or 0) / 1000) + lines.append(f"#EXTINF:{seconds},{entry['artist']} - {entry['title']}") + lines.append(os.path.relpath(entry["mirror"], path.parent)) + + path.parent.mkdir(parents=True, exist_ok=True) + handle, temporary = tempfile.mkstemp(dir=path.parent, suffix=".m3u.part") + os.close(handle) + temporary = Path(temporary) + try: + temporary.write_text("\n".join(lines) + "\n", encoding="utf-8") + # The mirror is read back by something else; see music-mirror, which had + # to learn this the hard way. + mode = temporary.stat().st_mode + if not mode & GROUP_READ: + temporary.chmod(mode | GROUP_READ) + os.replace(temporary, path) + finally: + temporary.unlink(missing_ok=True) + + +def build_playlists(store, mirror_root, library_root, limit, now): + """Write every playlist into the mirror. Returns how many tracks were listed. + + A track is only listed once its mirror file has been confirmed to exist. The + index knows what Lidarr holds, which is the lossless source; whether the MP3 + beside it has been encoded yet is music-mirror's business and is checked + rather than assumed. + """ + directory = Path(mirror_root) / PLAYLIST_DIRECTORY + parameters = { + "limit": limit, + "year_ago": now - 365 * 86400, + "week": now // ROTATION_PERIOD_SECONDS, + } + + total = 0 + missing = 0 + for name, description, sql in PLAYLISTS: + entries = [] + for row in store.connection.execute(sql, parameters): + mirror = mirror_path_for(row["path"], library_root, mirror_root) + if mirror is None or not mirror.is_file(): + missing += 1 + continue + entries.append({**dict(row), "mirror": mirror}) + write_playlist(directory / f"{name}.m3u", entries) + total += len(entries) + logger.info("playlist %-20s %4d tracks -- %s", name, len(entries), description) + + if missing: + logger.warning( + "%d selected tracks had no file in the mirror and were left out." + " A handful means music-mirror has not encoded them yet; a large" + " number means --library-root or --mirror is pointing at the wrong" + " place, since the paths are then being mapped to nothing.", + missing, + ) + return total + + def coverage_report(store): """Log how much of the listening history could be tied to the library. @@ -1262,7 +1492,10 @@ def report(store, now): logger.info(" top track 90d %2d: %-50s %d", position, label[:50], row["plays"]) -def run_once(client, store, user, now, backfill_limit, lidarr=None): +def run_once( + client, store, user, now, backfill_limit, lidarr=None, mirror=None, + library_root=None, playlist_limit=100, +): """Run a single pass. Returns the number of scrobbles added.""" started = time.monotonic() added = catch_up(client, store, user, now) @@ -1275,6 +1508,15 @@ def run_once(client, store, user, now, backfill_limit, lidarr=None): # arrived, and they need a verdict too. if store.scalar("SELECT COUNT(*) FROM lidarr_track"): match_library(store) + if mirror is not None: + root = library_root or library_root_of(store) + if root is None: + logger.warning( + "cannot work out where Lidarr's music lives, so no playlists were" + " written; set --library-root" + ) + else: + build_playlists(store, mirror, root, playlist_limit, now) logger.info( "pass complete in %.1fs: %d scrobbles added, %d loved", @@ -1347,6 +1589,25 @@ def build_parser(): default=os.getenv("MUSIC_CURATOR_LIDARR_API_KEY"), help="Lidarr API key (env MUSIC_CURATOR_LIDARR_API_KEY)", ) + parser.add_argument( + "--mirror", + default=os.getenv("MUSIC_CURATOR_MIRROR"), + help="root of the MP3 mirror; playlists are written into it" + " (env MUSIC_CURATOR_MIRROR)", + ) + parser.add_argument( + "--library-root", + default=os.getenv("MUSIC_CURATOR_LIBRARY_ROOT"), + help="prefix to strip from Lidarr's track paths when mapping them into the" + " mirror; derived from the indexed artist folders when unset" + " (env MUSIC_CURATOR_LIBRARY_ROOT)", + ) + parser.add_argument( + "--playlist-limit", + type=int, + default=int(os.getenv("MUSIC_CURATOR_PLAYLIST_LIMIT", "100")), + help="most tracks to put in any one playlist (env MUSIC_CURATOR_PLAYLIST_LIMIT)", + ) parser.add_argument( "--skip-index", action="store_true", @@ -1411,7 +1672,17 @@ def main(argv=None, clock=time.time): while True: now = int(clock()) try: - run_once(client, store, args.user, now, args.backfill_limit, lidarr) + run_once( + client, + store, + args.user, + now, + args.backfill_limit, + lidarr, + args.mirror, + args.library_root, + args.playlist_limit, + ) except (LastfmError, LidarrError) as error: logger.error("%s", error) if interval is None: diff --git a/tests/test_music_curator.py b/tests/test_music_curator.py index 4ecfa85..f31e8a2 100644 --- a/tests/test_music_curator.py +++ b/tests/test_music_curator.py @@ -1,5 +1,7 @@ import json +import stat import urllib.error +from pathlib import Path import pytest from conftest import FakeLastfm, FakeLidarr, make_loved, make_tracks @@ -825,3 +827,159 @@ def test_the_title_index_is_used_for_the_report_lookup(tmp_path): ) ) assert "lidarr_track_title" in plan, plan + + +def playlist_library(tmp_path): + """A library on disk, with mirror MP3s beside the Lidarr source paths.""" + source = tmp_path / "music" + mirror = tmp_path / "mirror" + library = [ + { + "name": "Played Band", + "albums": [ + { + "title": "Known", + "tracks": [{"title": "Hit"}, {"title": "Album Track"}], + } + ], + }, + { + "name": "Silent Band", + "albums": [{"title": "Unknown", "tracks": [{"title": "Never Heard"}]}], + }, + ] + api = FakeLidarr(library) + # FakeLidarr invents /music///.flac; put the mirror + # MP3s at the paths music-mirror would have produced from those. + for handle in api.files: + relative = Path(handle["path"]).relative_to("/music") + handle["path"] = str(source / relative) + mp3 = (mirror / relative).with_suffix(".mp3") + mp3.parent.mkdir(parents=True, exist_ok=True) + mp3.write_bytes(b"not really an mp3") + for artist in api.artists: + artist["path"] = str(source / Path(artist["path"]).name) + return api, source, mirror + + +def test_playlists_are_written_into_the_mirror(tmp_path): + api, source, mirror = playlist_library(tmp_path) + store = store_at(tmp_path) + ingest(store, FakeLastfm([scrobble_of("Played Band", "Hit")] * 1)) + music_curator.index_library( + music_curator.Lidarr("http://lidarr", "key", transport=api), store + ) + music_curator.match_library(store) + + music_curator.build_playlists(store, mirror, str(source), 100, NOW) + + written = sorted(p.name for p in (mirror / "_playlists").glob("*.m3u")) + assert written == [ + "all-time.m3u", + "deep-cuts.m3u", + "heavy-rotation.m3u", + "neglected.m3u", + "unheard-favourites.m3u", + "unheard.m3u", + ] + + played = (mirror / "_playlists" / "all-time.m3u").read_text().splitlines() + assert played[0] == "#EXTM3U" + assert played[1].startswith("#EXTINF:") + assert "Played Band - Hit" in played[1] + # Relative to the playlist file, so the same file works from any mount. + assert played[2] == "../Played Band/Known/Hit.mp3" + assert (mirror / "_playlists" / "all-time.m3u").parent.joinpath(played[2]).resolve().is_file() + + +def test_an_unplayed_track_lands_in_the_unheard_playlist(tmp_path): + api, source, mirror = playlist_library(tmp_path) + store = store_at(tmp_path) + ingest(store, FakeLastfm([scrobble_of("Played Band", "Hit")])) + music_curator.index_library( + music_curator.Lidarr("http://lidarr", "key", transport=api), store + ) + music_curator.match_library(store) + + music_curator.build_playlists(store, mirror, str(source), 100, NOW) + + unheard = (mirror / "_playlists" / "unheard.m3u").read_text() + assert "Never Heard" in unheard + assert "Album Track" in unheard + # The one thing that was played must not be in it. + assert "Played Band - Hit\n" not in unheard + + +def test_a_track_with_no_mirror_file_is_left_out(tmp_path): + """The index knows what Lidarr holds; whether music-mirror has encoded the + MP3 yet is a different question, and is checked rather than assumed.""" + api, source, mirror = playlist_library(tmp_path) + for mp3 in mirror.rglob("*.mp3"): + mp3.unlink() + store = store_at(tmp_path) + ingest(store, FakeLastfm([scrobble_of("Played Band", "Hit")])) + music_curator.index_library( + music_curator.Lidarr("http://lidarr", "key", transport=api), store + ) + music_curator.match_library(store) + + total = music_curator.build_playlists(store, mirror, str(source), 100, NOW) + + assert total == 0 + assert (mirror / "_playlists" / "all-time.m3u").read_text() == "#EXTM3U\n" + + +def test_the_library_root_is_derived_from_the_artist_folders(tmp_path): + api, source, mirror = playlist_library(tmp_path) + store = store_at(tmp_path) + music_curator.index_library( + music_curator.Lidarr("http://lidarr", "key", transport=api), store + ) + + assert music_curator.library_root_of(store) == str(source) + + +def test_the_playlist_limit_is_honoured(tmp_path): + api, source, mirror = playlist_library(tmp_path) + store = store_at(tmp_path) + music_curator.index_library( + music_curator.Lidarr("http://lidarr", "key", transport=api), store + ) + music_curator.match_library(store) + + music_curator.build_playlists(store, mirror, str(source), 1, NOW) + + unheard = (mirror / "_playlists" / "unheard.m3u").read_text().splitlines() + assert len([line for line in unheard if line.startswith("#EXTINF")]) == 1 + + +def test_the_rotation_moves_weekly_not_every_pass(tmp_path): + """A playlist that reorders on every pass is one that has to be re-imported + on every pass; the Music app imports a snapshot, it does not track a file.""" + api, source, mirror = playlist_library(tmp_path) + store = store_at(tmp_path) + music_curator.index_library( + music_curator.Lidarr("http://lidarr", "key", transport=api), store + ) + music_curator.match_library(store) + + def unheard_at(when): + music_curator.build_playlists(store, mirror, str(source), 100, when) + return (mirror / "_playlists" / "unheard.m3u").read_text() + + same_week = unheard_at(NOW), unheard_at(NOW + 3600) + assert same_week[0] == same_week[1] + + +def test_playlists_are_group_readable(tmp_path): + api, source, mirror = playlist_library(tmp_path) + store = store_at(tmp_path) + music_curator.index_library( + music_curator.Lidarr("http://lidarr", "key", transport=api), store + ) + music_curator.match_library(store) + + music_curator.build_playlists(store, mirror, str(source), 100, NOW) + + for playlist in (mirror / "_playlists").glob("*.m3u"): + assert playlist.stat().st_mode & stat.S_IRGRP, playlist