lyrathorpe/music-curator
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat: index the library from Lidarr and match it against the scrobbles
Build and publish container / build (pull_request) Successful in 10m2s
Details

Browse Source 
Stage two. The scrobble history says what was played by name; Lidarr says what
is owned, and where the files are. Neither is useful for curation until the two
are tied together, and the quality of that join is what decides whether the
later cull can be trusted at all.

The index is a wholesale rebuild of every artist, album and track Lidarr holds,
including file paths and the date each file landed -- the latter for the age
floor a cull will need. It is rebuilt rather than reconciled because Lidarr is
the authority and a deletion there has to disappear here, not linger as a
library entry with no file behind it. Every call is a GET; nothing is written
back.

Matching runs at the level of the distinct artist/track pair rather than the
individual play, because a verdict is a property of the name pair and there are
three plays for every one of them. Two tiers: a MusicBrainz recording id, which
Last.fm supplies per scrobble and Lidarr exposes as ForeignRecordingId, gives an
exact join; everything else falls to a normalised name comparison. There is
deliberately no third tier. A near-miss guess is worse than an admitted one,
since the entire purpose of the resulting number is to state how far the
matching can be relied on.

Normalisation folds the ways the two sides habitually disagree: case, accents,
guest credits that Last.fm puts in the artist field, trailing version suffixes,
ampersands, and a leading article. Punctuation needs two opposing rules and both
are load-bearing -- apostrophes are deleted so "Don't" meets "Dont", while every
other mark becomes a space so "AC/DC", "AC-DC" and "AC DC" meet as well. It errs
towards collapsing too much: a false match makes a track look played, a missed
match makes it look abandoned, and only the second one loses music.

The coverage report deliberately does not lead with matched versus unmatched.
Most unmatched listening is music that was never in the library and says nothing
about the matcher. The figure that matters is unmatched listening by an artist
the library does hold: a track that was played, sitting next to a file it should
have matched. The worst fifteen are listed by play count.

The schema gains its tables additively and migrates a version 1 store in place,
because rebuilding a nine-year history costs several thousand API requests.
This commit is contained in:
Emma Thorpe
2026-08-24 13:31:08 +01:00
parent 18f05d3d55
commit 5c4797ef38
5 changed files with 847 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
music_curator.py
+453 -8
View File
@@ -25,6 +25,7 @@ import signal
import sqlite3
import sys
import time
import unicodedata
import urllib.error
import urllib.parse
import urllib.request
@@ -54,7 +55,12 @@ REQUEST_DELAY_SECONDS = 0.25
BACKOFF_SECONDS = 2.0
BACKOFF_CEILING_SECONDS = 60.0
SCHEMA_VERSION = "1"
SCHEMA_VERSION = "2"
# Versions this build upgrades in place. Everything added since version 1 is a
# new table, and the schema script only ever creates what is missing, so running
# it is the whole migration -- an existing history is not re-downloaded.
MIGRATABLE_FROM = {"1"}
SCHEMA = """
-- One row per scrobble. The primary key collapses two plays of the same track
@@ -89,13 +95,149 @@ CREATE TABLE IF NOT EXISTS state (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
-- The library as Lidarr sees it. Rebuilt from the API rather than by walking
-- the files, because Lidarr is the only thing holding the MusicBrainz ids that
-- make an exact join possible.
CREATE TABLE IF NOT EXISTS lidarr_artist (
id INTEGER PRIMARY KEY,
mbid TEXT,
name TEXT NOT NULL,
norm_name TEXT NOT NULL,
path TEXT,
monitored INTEGER NOT NULL DEFAULT 1
);
CREATE INDEX IF NOT EXISTS lidarr_artist_norm ON lidarr_artist (norm_name);
CREATE TABLE IF NOT EXISTS lidarr_album (
id INTEGER PRIMARY KEY,
artist_id INTEGER NOT NULL,
mbid TEXT,
title TEXT NOT NULL,
monitored INTEGER NOT NULL DEFAULT 1,
release_date TEXT
);
CREATE INDEX IF NOT EXISTS lidarr_album_artist ON lidarr_album (artist_id);
CREATE TABLE IF NOT EXISTS lidarr_track (
id INTEGER PRIMARY KEY,
artist_id INTEGER NOT NULL,
album_id INTEGER NOT NULL,
recording_mbid TEXT,
title TEXT NOT NULL,
norm_artist TEXT NOT NULL,
norm_title TEXT NOT NULL,
has_file INTEGER NOT NULL DEFAULT 0,
path TEXT,
added INTEGER,
duration INTEGER
);
CREATE INDEX IF NOT EXISTS lidarr_track_recording ON lidarr_track (recording_mbid);
CREATE INDEX IF NOT EXISTS lidarr_track_norm ON lidarr_track (norm_artist, norm_title);
CREATE INDEX IF NOT EXISTS lidarr_track_album ON lidarr_track (album_id);
-- One row per distinct thing listened to, with the verdict on whether it could
-- be tied to the library. Matching belongs at this granularity rather than per
-- play: it is a property of the name pair, and there are three plays for every
-- pair.
CREATE TABLE IF NOT EXISTS scrobble_key (
artist TEXT NOT NULL,
track TEXT NOT NULL,
norm_artist TEXT NOT NULL,
norm_track TEXT NOT NULL,
track_mbid TEXT,
plays INTEGER NOT NULL,
first_uts INTEGER NOT NULL,
last_uts INTEGER NOT NULL,
track_id INTEGER,
method TEXT NOT NULL DEFAULT 'none',
PRIMARY KEY (artist, track)
);
CREATE INDEX IF NOT EXISTS scrobble_key_track ON scrobble_key (track_id);
"""
# Suffixes that a file tag carries and a scrobble does not, or the reverse.
# Stripped only from a trailing bracket or after a trailing dash, so a title
# like "(Don't Fear) The Reaper" keeps its opening parenthetical.
VERSION_WORDS = (
"remaster",
"remastered",
"live",
"mono",
"stereo",
"version",
"edit",
"mix",
"remix",
"deluxe",
"bonus",
"explicit",
"acoustic",
"demo",
"radio",
"single",
"anniversary",
"reissue",
"instrumental",
)
_VERSIONS = "|".join(VERSION_WORDS)
BRACKETED_VERSION = re.compile(
rf"\s*[\(\[][^\)\]]*\b(?:{_VERSIONS})\b[^\)\]]*[\)\]]\s*$", re.IGNORECASE
)
TRAILING_VERSION = re.compile(rf"\s+-\s+[^-]*\b(?:{_VERSIONS})\b.*$", re.IGNORECASE)
# Last.fm routinely carries the guest credit in the artist field where the file
# tag holds only the primary artist -- "Yellowcard feat. Tay Jardine" against a
# tag of "Yellowcard". `with` is deliberately absent: it appears in far too many
# real titles to cut on sight.
GUEST_CREDIT = re.compile(r"\s+(?:feat|ft|featuring)\b.*$", re.IGNORECASE)
LEADING_ARTICLE = re.compile(r"^the\s+")
# Deleted rather than spaced, so "Don't" and "Dont" agree. Every other mark
# becomes a space instead, so "AC/DC", "AC-DC" and "AC DC" all agree too; the
# two rules pull opposite ways and both are needed.
APOSTROPHES = re.compile(r"['‘’ʼ`´]")
NOT_ALPHANUMERIC = re.compile(r"[^0-9a-z ]+")
WHITESPACE = re.compile(r"\s+")
class LastfmError(Exception):
"""A Last.fm request that failed in a way retrying will not fix."""
class LidarrError(Exception):
"""A Lidarr request that failed."""
def normalise(text):
"""Return a comparison key for an artist or a track title.
Scrobbles and file tags disagree in predictable ways, and every one of them
has to be flattened or a track that was played reads as one that was not.
That is the direction that costs music, so this leans towards collapsing too
much rather than too little: a false match makes something look played, a
missed match makes something look abandoned.
"""
if not text:
return ""
text = unicodedata.normalize("NFKD", str(text)).casefold()
text = "".join(character for character in text if not unicodedata.combining(character))
text = GUEST_CREDIT.sub("", text)
# A title can carry more than one, e.g. "Song (Live) (Remastered)".
while True:
stripped = BRACKETED_VERSION.sub("", text)
if stripped == text:
break
text = stripped
text = TRAILING_VERSION.sub("", text)
text = text.replace("&", " and ")
text = APOSTROPHES.sub("", text)
text = NOT_ALPHANUMERIC.sub(" ", text)
text = WHITESPACE.sub(" ", text).strip()
return LEADING_ARTICLE.sub("", text)
@dataclass(frozen=True)
class Scrobble:
"""One play, as Last.fm recorded it."""
@@ -176,9 +318,10 @@ def parse_scrobble(entry):
)
def http_get(url, timeout=30):
def http_get(url, timeout=30, headers=None):
"""Fetch a URL and return its body. Replaced in tests."""
with urllib.request.urlopen(url, timeout=timeout) as response: # noqa: S310 - fixed https root
request = urllib.request.Request(url, headers=headers or {})
with urllib.request.urlopen(request, timeout=timeout) as response: # noqa: S310
return response.read().decode("utf-8")
@@ -267,12 +410,17 @@ class Store:
# the tail of it.
self.connection.execute("PRAGMA journal_mode = WAL")
self.connection.execute("PRAGMA synchronous = NORMAL")
# Registered so matching runs in SQL, rather than pulling twenty
# thousand name pairs into Python to compare them one at a time.
self.connection.create_function("normalise", 1, normalise, deterministic=True)
self.connection.executescript(SCHEMA)
self._check_version()
def _check_version(self):
held = self.get_state("schema_version")
if held is None:
if held is None or held in MIGRATABLE_FROM:
if held is not None:
logger.info("store migrated from schema version %s to %s", held, SCHEMA_VERSION)
self.set_state("schema_version", SCHEMA_VERSION)
elif held != SCHEMA_VERSION:
raise RuntimeError(
@@ -330,6 +478,58 @@ class Store:
rows,
)
def replace_library(self, artists, albums, tracks):
"""Swap in a freshly indexed library, in one transaction."""
with self.connection:
for table in ("lidarr_track", "lidarr_album", "lidarr_artist"):
self.connection.execute(f"DELETE FROM {table}")
self.connection.executemany(
"INSERT INTO lidarr_artist (id, mbid, name, norm_name, path, monitored)"
" VALUES (?, ?, ?, ?, ?, ?)",
artists,
)
self.connection.executemany(
"INSERT INTO lidarr_album"
" (id, artist_id, mbid, title, monitored, release_date)"
" VALUES (?, ?, ?, ?, ?, ?)",
albums,
)
self.connection.executemany(
"INSERT INTO lidarr_track"
" (id, artist_id, album_id, recording_mbid, title, norm_artist, norm_title,"
" has_file, path, added, duration)"
" VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
tracks,
)
def rebuild_keys(self):
"""Collapse the scrobble history into one row per distinct track.
Rows whose plays have not changed keep their existing verdict until the
matcher overwrites it; rows for tracks no longer in the history are
dropped, so an edited history does not leave orphans behind.
"""
with self.connection:
self.connection.execute(
"DELETE FROM scrobble_key WHERE (artist, track) NOT IN"
" (SELECT artist, track FROM scrobble)"
)
self.connection.execute(
"INSERT INTO scrobble_key"
" (artist, track, norm_artist, norm_track, track_mbid, plays,"
" first_uts, last_uts)"
" SELECT artist, track, normalise(artist), normalise(track),"
" MAX(track_mbid), COUNT(*), MIN(uts), MAX(uts)"
" FROM scrobble GROUP BY artist, track"
" ON CONFLICT (artist, track) DO UPDATE SET"
" norm_artist = excluded.norm_artist,"
" norm_track = excluded.norm_track,"
" track_mbid = excluded.track_mbid,"
" plays = excluded.plays,"
" first_uts = excluded.first_uts,"
" last_uts = excluded.last_uts"
)
def scalar(self, sql):
return self.connection.execute(sql).fetchone()[0]
@@ -445,6 +645,152 @@ def sync_loved(client, store, user):
return len(rows)
class Lidarr:
"""Minimal read-only Lidarr client.
No retries: Lidarr is on the same LAN as this, and a failure there means it
is down or the key is wrong, neither of which improves on a second attempt.
"""
def __init__(self, url, api_key, timeout=60, transport=None):
self.root = url.rstrip("/")
self.api_key = api_key
self.timeout = timeout
self.transport = transport or http_get
def get(self, path, params=None):
"""Return the decoded response for one API path."""
query = urllib.parse.urlencode(params or {})
url = f"{self.root}/api/v1/{path}" + (f"?{query}" if query else "")
try:
body = self.transport(url, timeout=self.timeout, headers={"X-Api-Key": self.api_key})
except urllib.error.HTTPError as error:
raise LidarrError(f"{path}: HTTP {error.code}") from error
except (urllib.error.URLError, TimeoutError) as error:
raise LidarrError(f"{path}: {error}") from error
try:
return json.loads(body)
except json.JSONDecodeError as error:
raise LidarrError(f"{path}: malformed response") from error
def parse_added(value):
"""Return a unix timestamp for a Lidarr ISO-8601 date, or None."""
if not value:
return None
try:
return int(datetime.fromisoformat(str(value).replace("Z", "+00:00")).timestamp())
except ValueError:
return None
def index_library(client, store):
"""Rebuild the library index from Lidarr. Returns (artists, tracks).
Rebuilt wholesale rather than reconciled: Lidarr is the authority, the index
is small, and a delete in Lidarr has to disappear here or it lingers as a
library entry with no file behind it.
"""
artists = client.get("artist")
artist_rows, album_rows, track_rows = [], [], []
for artist in artists:
artist_id = artist["id"]
name = artist.get("artistName") or ""
artist_rows.append(
(
artist_id,
mbid_of(artist.get("foreignArtistId")),
name,
normalise(name),
artist.get("path"),
1 if artist.get("monitored") else 0,
)
)
for album in client.get("album", {"artistId": artist_id}):
album_rows.append(
(
album["id"],
artist_id,
mbid_of(album.get("foreignAlbumId")),
album.get("title") or "",
1 if album.get("monitored") else 0,
(album.get("releaseDate") or "")[:10] or None,
)
)
# Paths and the date a track landed live on the file, not the track.
files = {
handle["id"]: handle for handle in client.get("trackfile", {"artistId": artist_id})
}
for track in client.get("track", {"artistId": artist_id}):
handle = files.get(track.get("trackFileId") or 0) or {}
title = track.get("title") or ""
track_rows.append(
(
track["id"],
artist_id,
track.get("albumId") or 0,
mbid_of(track.get("foreignRecordingId")),
title,
normalise(name),
normalise(title),
1 if track.get("hasFile") else 0,
handle.get("path"),
parse_added(handle.get("dateAdded")),
track.get("duration"),
)
)
store.replace_library(artist_rows, album_rows, track_rows)
logger.info(
"library indexed: %d artists, %d albums, %d tracks (%d with files)",
len(artist_rows),
len(album_rows),
len(track_rows),
sum(row[7] for row in track_rows),
)
return len(artist_rows), len(track_rows)
def match_library(store):
"""Tie every distinct thing listened to back to a library track.
Two tiers. A MusicBrainz recording id is an exact join and is tried first;
everything else falls to the normalised name pair. There is no third tier on
purpose -- a guess that is nearly right is worse here than an admitted miss,
because the whole point of the number this produces is to say how far the
matching can be trusted.
"""
store.rebuild_keys()
with store.connection:
store.connection.execute("UPDATE scrobble_key SET track_id = NULL, method = 'none'")
store.connection.execute(
"UPDATE scrobble_key SET"
" track_id = (SELECT t.id FROM lidarr_track t"
" WHERE t.recording_mbid = scrobble_key.track_mbid"
" ORDER BY t.has_file DESC, t.id LIMIT 1),"
" method = 'mbid'"
" WHERE track_mbid IS NOT NULL"
" AND EXISTS (SELECT 1 FROM lidarr_track t"
" WHERE t.recording_mbid = scrobble_key.track_mbid)"
)
store.connection.execute(
"UPDATE scrobble_key SET"
" track_id = (SELECT t.id FROM lidarr_track t"
" WHERE t.norm_artist = scrobble_key.norm_artist"
" AND t.norm_title = scrobble_key.norm_track"
" ORDER BY t.has_file DESC, t.id LIMIT 1),"
" method = 'name'"
" WHERE track_id IS NULL"
" AND norm_artist <> '' AND norm_track <> ''"
" AND EXISTS (SELECT 1 FROM lidarr_track t"
" WHERE t.norm_artist = scrobble_key.norm_artist"
" AND t.norm_title = scrobble_key.norm_track)"
)
def format_time(uts):
"""Return a UTC timestamp as a readable date."""
if uts is None:
@@ -452,6 +798,74 @@ def format_time(uts):
return datetime.fromtimestamp(uts, tz=timezone.utc).strftime("%Y-%m-%d %H:%M")
def coverage_report(store):
"""Log how much of the listening history could be tied to the library.
The split that matters is not matched against unmatched. Most unmatched
listening is music that was never in the library at all, which says nothing
about the matcher. The number to watch is unmatched listening by an artist
the library *does* hold: that is a track that was played, sitting next to a
file it should have matched, and every one of those is a candidate for being
wrongly called cold later on.
"""
tracks = store.scalar("SELECT COUNT(*) FROM lidarr_track")
if not tracks:
return
pairs = store.scalar("SELECT COUNT(*) FROM scrobble_key")
if not pairs:
return
logger.info("--- match coverage ---")
for method in ("mbid", "name", "none"):
row = store.connection.execute(
"SELECT COUNT(*) AS pairs, COALESCE(SUM(plays), 0) AS plays"
" FROM scrobble_key WHERE method = ?",
(method,),
).fetchone()
logger.info(
"%-5s: %6d pairs (%4.1f%%), %7d plays",
method,
row["pairs"],
100 * row["pairs"] / pairs,
row["plays"],
)
suspect = store.connection.execute(
"SELECT COUNT(*) AS pairs, COALESCE(SUM(plays), 0) AS plays FROM scrobble_key k"
" WHERE k.track_id IS NULL"
" AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)"
).fetchone()
logger.info(
"unmatched by an artist the library holds: %d pairs, %d plays -- these are the"
" matcher's misses, not music you do not own",
suspect["pairs"],
suspect["plays"],
)
with_files = store.scalar("SELECT COUNT(*) FROM lidarr_track WHERE has_file = 1")
played = store.scalar(
"SELECT COUNT(DISTINCT k.track_id) FROM scrobble_key k"
" JOIN lidarr_track t ON t.id = k.track_id WHERE t.has_file = 1"
)
logger.info(
"library tracks with a file: %d, of which played at least once: %d (%.1f%%)",
with_files,
played,
100 * played / with_files if with_files else 0,
)
worst = store.connection.execute(
"SELECT k.artist, k.track, k.plays FROM scrobble_key k"
" WHERE k.track_id IS NULL"
" AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)"
" ORDER BY k.plays DESC, k.artist LIMIT 15"
).fetchall()
for position, row in enumerate(worst, start=1):
label = f"{row['artist']} - {row['track']}"
logger.info(" unmatched %2d: %-60s %d plays", position, label[:60], row["plays"])
def report(store, now):
"""Log what the store holds.
@@ -485,6 +899,8 @@ def report(store, now):
for position, row in enumerate(top, start=1):
logger.info(" top artist %2d: %-40s %d", position, row["artist"][:40], row["plays"])
coverage_report(store)
recent = store.connection.execute(
"SELECT artist, track, COUNT(*) AS plays FROM scrobble WHERE uts >= ?"
" GROUP BY artist, track ORDER BY plays DESC, artist LIMIT 10",
@@ -495,12 +911,20 @@ def report(store, now):
logger.info(" top track 90d %2d: %-50s %d", position, label[:50], row["plays"])
def run_once(client, store, user, now, backfill_limit):
"""Run a single ingest pass. Returns the number of scrobbles added."""
def run_once(client, store, user, now, backfill_limit, lidarr=None):
"""Run a single pass. Returns the number of scrobbles added."""
started = time.monotonic()
added = catch_up(client, store, user, now)
added += backfill(client, store, user, now, limit=backfill_limit)
loved = sync_loved(client, store, user)
if lidarr is not None:
index_library(lidarr, store)
# Matching is worth redoing even without a fresh index: new scrobbles have
# arrived, and they need a verdict too.
if store.scalar("SELECT COUNT(*) FROM lidarr_track"):
match_library(store)
logger.info(
"pass complete in %.1fs: %d scrobbles added, %d loved",
time.monotonic() - started,
@@ -562,6 +986,21 @@ def build_parser():
help="cap the backfill at this many requests per pass; 0 for no cap"
" (env MUSIC_CURATOR_BACKFILL_LIMIT)",
)
parser.add_argument(
"--lidarr-url",
default=os.getenv("MUSIC_CURATOR_LIDARR_URL"),
help="base URL of Lidarr, e.g. http://lidarr:8686 (env MUSIC_CURATOR_LIDARR_URL)",
)
parser.add_argument(
"--lidarr-api-key",
default=os.getenv("MUSIC_CURATOR_LIDARR_API_KEY"),
help="Lidarr API key (env MUSIC_CURATOR_LIDARR_API_KEY)",
)
parser.add_argument(
"--skip-index",
action="store_true",
help="do not re-index the library this pass; match against the index already held",
)
parser.add_argument(
"--report-only",
action="store_true",
@@ -601,6 +1040,12 @@ def main(argv=None, clock=time.time):
client = Lastfm(args.api_key, delay=args.request_delay)
lidarr = None
if args.lidarr_url and args.lidarr_api_key and not args.skip_index:
lidarr = Lidarr(args.lidarr_url, args.lidarr_api_key)
elif not args.lidarr_url:
logger.info("no Lidarr URL configured; ingesting scrobbles only")
stopping = False
def stop(signum, _frame):
@@ -615,8 +1060,8 @@ def main(argv=None, clock=time.time):
while True:
now = int(clock())
try:
run_once(client, store, args.user, now, args.backfill_limit)
except LastfmError as error:
run_once(client, store, args.user, now, args.backfill_limit, lidarr)
except (LastfmError, LidarrError) as error:
logger.error("%s", error)
if interval is None:
return 1