lyrathorpe/music-curator
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat: ingest a Last.fm scrobble history into a local store
Build and publish container / build (push) Failing after 2m16s
Details

Browse Source 
First stage of a curation tool for the music library that music-mirror
mirrors. Before anything can build playlists or decide what has gone cold,
there has to be a local, queryable record of what is actually played; an API
call per question does not scale to a library-sized analysis.

Ingest is in two halves. A catch-up fetches everything scrobbled since the
newest scrobble held, and a backfill walks the history backwards until it runs
out. Both take their bounds from the database rather than from a saved cursor,
so an interrupted run resumes from what it actually has, and both windows are
bounded at each end so paging cannot shift under the fetch while new scrobbles
arrive mid-run.

Scrobbles carry no identifier, so the primary key is timestamp, artist and
track. Two plays of one track in the same second collapse into a single row:
they are indistinguishable in the data, and a surrogate key would make
re-ingest non-idempotent, which is the worse trade.

Three API behaviours are handled explicitly because each fails silently:
the currently-playing track arrives with no timestamp and would be re-ingested
on every pass; a lone result is returned as a bare object rather than a
one-item list; and MBIDs are empty strings rather than absent when unknown,
which would later look like a usable join key.

Retries cover the rate limit and the transient backend errors with an
exponential backoff. An invalid or suspended key fails immediately.

The report exists to surface one number before the next stage is built: the
share of scrobbles carrying a MusicBrainz recording id. Lidarr exposes the same
identifier per track, so those can be joined exactly and the rest must go
through name matching. That percentage bounds how far the matcher can be
trusted.

No runtime dependencies, and the tests run against a fake transport that
reproduces the service's paging and response shapes, so they need neither
network nor credentials.
This commit is contained in:
Emma Thorpe
2026-08-24 12:03:08 +01:00
commit 18f05d3d55
11 changed files with 1458 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dockerignore
+11
View File
@@ -0,0 +1,11 @@
.git
.gitignore
result
result-*
__pycache__
*.pyc
.pytest_cache
compose.yaml
*.db
*.db-wal
*.db-shm
.gitea/workflows/build-and-publish.yaml
+199
View File
@@ -0,0 +1,199 @@
name: Build and publish container
on:
# On merge to main, only build/release when image-affecting files change;
# CI-config and docs changes do not produce a new image. pyproject.toml is
# deliberately absent: the release step below commits to it, and that commit
# must not start another run.
push:
branches: [main]
paths:
- "Dockerfile"
- ".dockerignore"
- "music_curator.py"
# Pull requests always run (tests and the image build are the checks); no
# path filter.
pull_request:
branches: [main]
workflow_dispatch:
# A newer run cancels an older in-flight run in the same group (keyed by ref),
# so a fresh merge to main supersedes the previous build and only the latest
# release is produced. Each pull request likewise supersedes only its own
# earlier runs.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
defaults:
run:
shell: bash
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: write
packages: write
steps:
- name: Checkout
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
with:
# Full history and tags are required to derive the next version
# from the conventional-commit messages since the last release.
fetch-depth: 0
# The suite runs inside the image, against the interpreter that ships,
# rather than against whatever the runner happens to provide. A failing
# test fails the build. Layers are shared with the push build below.
- name: Run the test suite inside the image
run: docker build --target test -t music-curator:test .
- name: Determine registry host
run: echo "REGISTRY=${GITHUB_SERVER_URL#*://}" >> "$GITHUB_ENV"
# Derive the release version from conventional commits since the last
# v* tag: feat -> minor, fix/perf -> patch, ! or BREAKING CHANGE -> major.
# Anything else (chore, ci, docs, build) produces no release; those builds
# are published under a sha-<short> tag only.
- name: Compute version and image tags
id: version
run: |
set -euo pipefail
image="${REGISTRY}/${GITHUB_REPOSITORY,,}"
last_tag="$(git tag --list 'v*' --sort=-v:refname | head -n1 || true)"
if [ -n "$last_tag" ]; then
range="${last_tag}..HEAD"
base="${last_tag#v}"
else
range=""
base="0.0.0"
fi
subjects="$(git log ${range} --format='%s')"
bodies="$(git log ${range} --format='%B')"
bump="none"
if printf '%s\n' "$bodies" | grep -qiE 'BREAKING[ -]CHANGE' \
|| printf '%s\n' "$subjects" | grep -qE '^[a-z]+([(][^)]*[)])?!:'; then
bump="major"
elif printf '%s\n' "$subjects" | grep -qE '^feat([(][^)]*[)])?:'; then
bump="minor"
elif printf '%s\n' "$subjects" | grep -qE '^(fix|perf)([(][^)]*[)])?:'; then
bump="patch"
fi
major="${base%%.*}"
rest="${base#*.}"
minor="${rest%%.*}"
patch="${rest##*.}"
# workflow_dispatch releases too, so a release can be cut without a
# code change -- the push filter above ignores workflow and doc edits.
# Restricted to main: a manual run elsewhere must not tag a commit
# that is not on the default branch.
release="false"
if [ "${GITHUB_EVENT_NAME}" != "pull_request" ] \
&& [ "${GITHUB_REF_NAME}" = "main" ] \
&& [ "$bump" != "none" ]; then
release="true"
case "$bump" in
major) major=$((major + 1)); minor=0; patch=0 ;;
minor) minor=$((minor + 1)); patch=0 ;;
patch) patch=$((patch + 1)) ;;
esac
version="${major}.${minor}.${patch}"
{
echo "tags<<__EOT__"
echo "${image}:${version}"
echo "${image}:${major}.${minor}"
echo "${image}:${major}"
echo "${image}:latest"
echo "__EOT__"
} >> "$GITHUB_OUTPUT"
echo "version=${version}" >> "$GITHUB_OUTPUT"
else
short="$(git rev-parse --short HEAD)"
{
echo "tags<<__EOT__"
echo "${image}:sha-${short}"
echo "__EOT__"
} >> "$GITHUB_OUTPUT"
fi
echo "release=${release}" >> "$GITHUB_OUTPUT"
echo "Computed bump=${bump}, release=${release}, base=${base}"
- name: Set up Buildx
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4
- name: Log in to the Gitea container registry
if: github.event_name != 'pull_request'
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.repository_owner }}
password: ${{ secrets.PACKAGES_TOKEN }}
- name: Build and push
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7
with:
context: .
# Without this the last stage in the Dockerfile -- the test stage --
# would be what gets published.
target: runtime
# The NAS is the only host this runs on. Building arm64 as well would
# mean emulating it under QEMU for no consumer.
platforms: linux/amd64
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.version.outputs.tags }}
labels: |
org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}
org.opencontainers.image.revision=${{ github.sha }}
# Record the release: write the computed version into pyproject.toml, then
# commit and tag it, so the packaging metadata always matches the release
# instead of drifting behind it. The version is derived from commit
# messages and only known here, after the build, so it cannot be set by
# hand in the pull request that causes the release.
- name: Record and tag the release
if: steps.version.outputs.release == 'true'
env:
VERSION: ${{ steps.version.outputs.version }}
run: |
set -euo pipefail
python3 - "$VERSION" <<'PY'
import pathlib
import re
import sys
version = sys.argv[1]
path = pathlib.Path("pyproject.toml")
text = path.read_text()
text, count = re.subn(
r'(?m)^version = ".*"$', f'version = "{version}"', text, count=1
)
if count != 1:
raise SystemExit("no version line found in pyproject.toml")
path.write_text(text)
PY
git config user.name "${{ github.actor }}"
git config user.email "${{ github.actor }}@users.noreply.${REGISTRY}"
git add pyproject.toml
# The file may already carry this version, in which case there is
# nothing to commit and `git commit` would fail the job.
if git diff --cached --quiet; then
echo "pyproject.toml is already at ${VERSION}"
else
git commit -m "chore(release): v${VERSION}"
# Push the branch before the tag. If main has moved on and this push
# is rejected, the job fails without having left a tag pointing at a
# commit that is not on main.
git push origin "HEAD:${GITHUB_REF_NAME}"
fi
git tag -a "v${VERSION}" -m "v${VERSION}"
git push origin "v${VERSION}"
.gitignore
+11
View File
@@ -0,0 +1,11 @@
result
result-*
__pycache__/
*.pyc
.venv/
.pytest_cache/
*.egg-info/
*.db
*.db-wal
*.db-shm
*.lock
Dockerfile
+32
View File
@@ -0,0 +1,32 @@
# Alpine because there is nothing to compile and nothing to link against: the
# application is one Python module with no dependencies outside the standard
# library, so the base image is almost the whole image.
FROM python:3.13-alpine AS runtime
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY pyproject.toml README.md ./
COPY music_curator.py ./
RUN pip install --no-cache-dir . \
&& rm -rf build music_curator.egg-info
# The store lives here. Mount a volume over it, or the history is re-downloaded
# from Last.fm every time the container is recreated.
ENV MUSIC_CURATOR_DB=/data/curator.db
VOLUME ["/data"]
# Runs as root by default so a bind-mounted dataset of any ownership is
# writable. Override with `user:` in compose to run as the dataset's owner.
ENTRYPOINT ["music-curator"]
# Test stage: build it with `--target test`; a failing test fails the build.
# The suite talks to a fake transport, never to Last.fm, so it needs no
# credentials and no network. The published image is the `runtime` stage above
# and carries none of this.
FROM runtime AS test
RUN pip install --no-cache-dir pytest
COPY pytest.ini ./
COPY tests ./tests
RUN python -m pytest
README.md
+131
View File
@@ -0,0 +1,131 @@
# music-curator
Build a local record of what actually gets listened to, from Last.fm.
Companion to [music-mirror](https://code.emmathe.dev/lyrathorpe/music-mirror),
which keeps an MP3 copy of a lossless library for an iPod. This one answers the
question that mirror cannot: which of it is worth carrying, and which of it has
not been played in years.
**This is stage one.** It ingests the scrobble history and nothing else. There
are no playlists yet, and nothing touches Lidarr or the music library. See
"Where this is going" below.
## What it does today
- Pulls the full Last.fm scrobble history into SQLite, then keeps it current.
- Tracks loved tracks separately, as the protected set for later stages.
- Reports what it holds, including the number that matters most: how many
scrobbles carry a MusicBrainz recording id.
That last figure decides the next stage. Lidarr exposes a `ForeignRecordingId`
on every track, which is the same identifier, so scrobbles carrying one can be
joined to the library exactly. The rest have to go through name matching, which
is where a curation tool goes wrong and starts recommending the deletion of
music you love. Measure the join rate before trusting the verdict.
## How the ingest works
Two halves, both taking their bounds from the database rather than from a saved
cursor, so an interrupted run resumes from what it actually has.
| Half | Window | Purpose |
| --------- | ---------------------- | ---------------------------------------- |
| Catch-up | `newest held` → `now` | New scrobbles since the last pass |
| Backfill | start → `oldest held` | Walks towards the beginning of the history |
The backfill asks repeatedly for the newest page of everything at or before a
cursor, and moves the cursor to the oldest scrobble that came back. When a whole
page shares a single second the cursor cannot move without stepping over the
rest of that second, so it takes the next page of the same window instead.
Both windows are bounded at both ends, so paging cannot shift under the fetch
while new scrobbles arrive mid-run.
Three details of the API that are easy to get wrong, all handled:
- The **currently-playing** track is prepended to the first page with no
timestamp at all. Stored once, it would come back on every pass forever.
- A **lone result** is returned as a bare object, not a one-item list.
- **MBIDs are empty strings** rather than absent when unknown, and an empty
string looks like a usable join key right up until it silently matches
everything.
Scrobbles have no identifier, so the primary key is timestamp plus artist plus
track. Two plays of the same track in the same second collapse into one; they
are genuinely indistinguishable, and a surrogate key would make re-ingest
non-idempotent, which is a far worse trade.
Retries cover error 29 (rate limit) and the backend failures, 8, 11 and 16,
with an exponential backoff. An invalid or suspended key fails immediately
rather than retrying four more times to reach the same conclusion.
## Usage
```sh
music-curator # one pass, then report
music-curator --interval 6h # keep running
music-curator --report-only # report on the store, fetch nothing
```
| Option | Environment variable | Default | Meaning |
| ------------------ | ----------------------------- | ------------------ | ------------------------------------------- |
| `--user` | `MUSIC_CURATOR_LASTFM_USER` | — | Last.fm username to read |
| `--api-key` | `MUSIC_CURATOR_LASTFM_API_KEY` | — | Last.fm API key |
| `--db` | `MUSIC_CURATOR_DB` | `/data/curator.db` | Path to the SQLite store |
| `--interval` | `MUSIC_CURATOR_INTERVAL` | unset | Repeat forever, e.g. `45m`, `6h`, `1d` |
| `--request-delay` | `MUSIC_CURATOR_REQUEST_DELAY` | `0.25` | Seconds between API requests |
| `--backfill-limit` | `MUSIC_CURATOR_BACKFILL_LIMIT` | `0` | Cap backfill requests per pass; 0 for no cap |
| `--report-only` | — | off | Report without fetching |
A [Last.fm API key](https://www.last.fm/api/account/create) is all that is
needed. None of the endpoints used here authenticate a user, so there is no
shared secret, no session key and no signing.
The first pass over a long history is thousands of requests at 200 scrobbles
each. `--backfill-limit` spreads that over several passes if you would rather
not do it in one.
## Running it on TrueNAS Scale
`compose.yaml` is a Custom App definition. Adjust the host path and the `user:`
to match your pool, set the API key through the TrueNAS UI rather than in the
file, then add it as a custom app. The image is published to this Gitea's
registry on every release:
```
code.emmathe.dev/lyrathorpe/music-curator:latest
```
Give the store its own dataset. It is derived data and can be rebuilt from
Last.fm, but rebuilding means downloading the whole history again.
## Tests
```sh
docker build --target test . # what CI runs
pytest # needs pytest on PATH
```
The suite runs against a fake transport that reproduces the real service's
paging, its `from`/`to` semantics and its awkward response shapes. No network,
no credentials, no rate limit. On a Nix machine:
```sh
nix shell nixpkgs#python3Packages.pytest -c pytest
```
## Where this is going
| Stage | Status |
| ------------------------------------------------ | ------------ |
| Last.fm ingest and store | done |
| Lidarr index and the scrobble-to-track matcher | next |
| M3U playlists written into the mirror | after that |
| Cold-music report, unmonitoring what is not played | last |
The cull will unmonitor cold albums in Lidarr and tag their artists. It will
never delete files: Lidarr's `AlbumResource` has no tags at all, so tagging
only works per artist, and an artist is only tagged when every one of their
albums qualifies. It will also refuse to run at all if the matcher's coverage
is poor, because an unmatched track is not an unplayed track.
compose.yaml
+30
View File
@@ -0,0 +1,30 @@
# TrueNAS Scale "Custom App" definition.
#
# Adjust the host path and the user to match your pool. The Last.fm API key is
# a credential: set it through the app's environment in the TrueNAS UI rather
# than committing it here.
services:
music-curator:
image: code.emmathe.dev/lyrathorpe/music-curator:latest
container_name: music-curator
restart: unless-stopped
# The dataset owner, so the store is not written as root. `id apps` or the
# ownership of the dataset will tell you the right numbers.
user: "568:568"
environment:
MUSIC_CURATOR_LASTFM_USER: your-lastfm-username
# Get one at https://www.last.fm/api/account/create -- a key alone is
# enough, none of the endpoints used here authenticate a user.
MUSIC_CURATOR_LASTFM_API_KEY: set-me-in-the-truenas-ui
MUSIC_CURATOR_DB: /data/curator.db
# How long to wait between passes. Each one catches up on new scrobbles
# and continues the backfill if it has not finished.
MUSIC_CURATOR_INTERVAL: 6h
# Seconds between API requests. Last.fm asks for no more than five a
# second; the default leaves plenty of room.
# MUSIC_CURATOR_REQUEST_DELAY: "0.25"
# Cap the backfill at this many requests per pass. Unlimited by default,
# which finishes a long history in one go.
# MUSIC_CURATOR_BACKFILL_LIMIT: "0"
volumes:
- /mnt/tank/apps/music-curator:/data
music_curator.py
+643
View File
@@ -0,0 +1,643 @@
"""Build a local record of what actually gets listened to.
Ingests a Last.fm scrobble history into SQLite and keeps it current, so that
the later stages -- building playlists, and deciding what in the library has
gone cold -- can query listening habits locally instead of spending an API call
per question.
The store is derived state. It can be deleted and rebuilt from Last.fm at any
time. Nothing here writes to Last.fm, and nothing here touches the music
library or Lidarr.
Ingest is in two halves. A backfill walks the history backwards until it runs
out, and a catch-up fetches everything scrobbled since the newest scrobble
already held. Both take their bounds from the database rather than from a saved
cursor, so an interrupted run resumes from what it actually has.
"""
import argparse
import fcntl
import json
import logging
import os
import re
import signal
import sqlite3
import sys
import time
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
logger = logging.getLogger("music-curator")
API_ROOT = "https://ws.audioscrobbler.com/2.0/"
# The documented ceiling for user.getRecentTracks. getLovedTracks does not state
# one; asking for more than it allows is harmless, because paging is driven by
# the totalPages the response reports rather than by arithmetic on this number.
PAGE_SIZE = 200
# Error codes worth another attempt: backend failure, service offline, service
# temporarily unavailable, rate limit. Everything else is a fault in the request
# or the key and will fail again identically.
RETRYABLE_ERRORS = {8, 11, 16, 29}
RETRYABLE_STATUS = {429, 500, 502, 503, 504}
# Last.fm asks for no more than five requests a second averaged over five
# minutes. A full backfill is thousands of requests, so it is worth staying
# well inside that rather than discovering error 29 halfway through.
REQUEST_DELAY_SECONDS = 0.25
BACKOFF_SECONDS = 2.0
BACKOFF_CEILING_SECONDS = 60.0
SCHEMA_VERSION = "1"
SCHEMA = """
-- One row per scrobble. The primary key collapses two plays of the same track
-- in the same second into one: Last.fm gives scrobbles no identifier, so they
-- are genuinely indistinguishable, and one lost play a decade is not worth a
-- surrogate key that would make re-ingest non-idempotent.
CREATE TABLE IF NOT EXISTS scrobble (
uts INTEGER NOT NULL,
artist TEXT NOT NULL,
track TEXT NOT NULL,
album TEXT NOT NULL DEFAULT '',
artist_mbid TEXT,
album_mbid TEXT,
track_mbid TEXT,
PRIMARY KEY (uts, artist, track)
);
CREATE INDEX IF NOT EXISTS scrobble_uts ON scrobble (uts);
CREATE INDEX IF NOT EXISTS scrobble_artist_track ON scrobble (artist, track);
CREATE INDEX IF NOT EXISTS scrobble_track_mbid ON scrobble (track_mbid);
-- Loved tracks are current state, not history: a track can be unloved again.
-- The table is replaced on every pass rather than accumulated.
CREATE TABLE IF NOT EXISTS loved (
artist TEXT NOT NULL,
track TEXT NOT NULL,
track_mbid TEXT,
loved_at INTEGER,
PRIMARY KEY (artist, track)
);
CREATE TABLE IF NOT EXISTS state (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
"""
class LastfmError(Exception):
"""A Last.fm request that failed in a way retrying will not fix."""
@dataclass(frozen=True)
class Scrobble:
"""One play, as Last.fm recorded it."""
uts: int
artist: str
track: str
album: str = ""
artist_mbid: str | None = None
album_mbid: str | None = None
track_mbid: str | None = None
def parse_interval(interval):
"""Return seconds for an interval such as ``30m``, ``6h`` or ``90``."""
text = str(interval).strip().lower()
match = re.fullmatch(r"([0-9]+)([smhd]?)", text)
if not match:
raise ValueError(f"unrecognised interval {interval!r}: expected e.g. 45m, 6h, 1d")
value = int(match.group(1))
return value * {"": 1, "s": 1, "m": 60, "h": 3600, "d": 86400}[match.group(2)]
def as_list(value):
"""Return a list for a field Last.fm renders as a bare object when singular."""
if value is None:
return []
if isinstance(value, list):
return value
return [value]
def name_of(value):
"""Return the name out of a Last.fm sub-object.
The same conceptual field is ``#text`` in some responses (recent tracks) and
``name`` in others (loved tracks, and anything fetched with extended=1).
"""
if isinstance(value, dict):
return (value.get("name") or value.get("#text") or "").strip()
return (value or "").strip()
def mbid_of(value):
"""Return an MBID, or None where the field is present but empty.
Last.fm sends an empty string rather than omitting the key, and an empty
string would otherwise look like a usable join key further down the line.
"""
if isinstance(value, dict):
value = value.get("mbid")
value = (value or "").strip()
return value or None
def parse_scrobble(entry):
"""Return a Scrobble for one recent-tracks entry, or None to skip it.
The currently-playing track comes back with no ``date`` at all. Storing it
would mean a scrobble with no timestamp, and it would arrive again on every
pass until the song ended, so it is dropped until it has a real one.
"""
date = entry.get("date") or {}
if "uts" not in date:
return None
artist = name_of(entry.get("artist"))
track = (entry.get("name") or "").strip()
if not artist or not track:
return None
return Scrobble(
uts=int(date["uts"]),
artist=artist,
track=track,
album=name_of(entry.get("album")),
artist_mbid=mbid_of(entry.get("artist")),
album_mbid=mbid_of(entry.get("album")),
track_mbid=mbid_of(entry.get("mbid")),
)
def http_get(url, timeout=30):
"""Fetch a URL and return its body. Replaced in tests."""
with urllib.request.urlopen(url, timeout=timeout) as response: # noqa: S310 - fixed https root
return response.read().decode("utf-8")
class Lastfm:
"""Minimal read-only Last.fm client.
Only an API key is needed: none of the methods used here authenticate a
user, so there is no session key, no signing and no secret to hold.
"""
def __init__(
self,
api_key,
delay=REQUEST_DELAY_SECONDS,
attempts=5,
backoff=BACKOFF_SECONDS,
transport=None,
):
self.api_key = api_key
self.delay = delay
self.attempts = attempts
self.backoff = backoff
# Resolved here rather than as a default argument so the tests can
# replace the module-level fetcher.
self.transport = transport or http_get
self._previous_request = None
def call(self, method, params):
"""Return the decoded response for one API method."""
query = {key: value for key, value in params.items() if value is not None}
query.update(method=method, api_key=self.api_key, format="json")
url = f"{API_ROOT}?{urllib.parse.urlencode(query)}"
for attempt in range(1, self.attempts + 1):
self._throttle()
try:
payload = json.loads(self.transport(url))
except urllib.error.HTTPError as error:
if error.code not in RETRYABLE_STATUS:
raise LastfmError(f"{method}: HTTP {error.code}") from error
self._retry_or_raise(method, attempt, f"HTTP {error.code}", error)
continue
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError) as error:
self._retry_or_raise(method, attempt, str(error), error)
continue
code = payload.get("error")
if code is None:
return payload
detail = f"error {code}: {payload.get('message', '')}".strip()
if code not in RETRYABLE_ERRORS:
raise LastfmError(f"{method}: {detail}")
self._retry_or_raise(method, attempt, detail, None)
raise LastfmError(f"{method}: gave up after {self.attempts} attempts")
def _retry_or_raise(self, method, attempt, detail, cause):
"""Sleep before the next attempt, or give up if this was the last one."""
if attempt >= self.attempts:
raise LastfmError(f"{method}: {detail}") from cause
pause = min(self.backoff * 2 ** (attempt - 1), BACKOFF_CEILING_SECONDS)
logger.warning("%s: %s; retrying in %.0fs", method, detail, pause)
time.sleep(pause)
def _throttle(self):
"""Keep consecutive requests at least `delay` apart."""
if self.delay <= 0:
return
if self._previous_request is not None:
waited = time.monotonic() - self._previous_request
if waited < self.delay:
time.sleep(self.delay - waited)
self._previous_request = time.monotonic()
class Store:
"""The local scrobble database."""
def __init__(self, path):
self.path = Path(path)
self.path.parent.mkdir(parents=True, exist_ok=True)
self.connection = sqlite3.connect(self.path)
self.connection.row_factory = sqlite3.Row
# WAL so a long backfill does not lock a reader out, and a relaxed sync
# because the whole store can be rebuilt from Last.fm if a crash eats
# the tail of it.
self.connection.execute("PRAGMA journal_mode = WAL")
self.connection.execute("PRAGMA synchronous = NORMAL")
self.connection.executescript(SCHEMA)
self._check_version()
def _check_version(self):
held = self.get_state("schema_version")
if held is None:
self.set_state("schema_version", SCHEMA_VERSION)
elif held != SCHEMA_VERSION:
raise RuntimeError(
f"{self.path} is schema version {held}, this build expects {SCHEMA_VERSION}; "
"delete it and let it rebuild"
)
def close(self):
self.connection.close()
def get_state(self, key):
row = self.connection.execute("SELECT value FROM state WHERE key = ?", (key,)).fetchone()
return row["value"] if row else None
def set_state(self, key, value):
with self.connection:
self.connection.execute(
"INSERT INTO state (key, value) VALUES (?, ?)"
" ON CONFLICT (key) DO UPDATE SET value = excluded.value",
(key, value),
)
def add_scrobbles(self, scrobbles):
"""Insert scrobbles, ignoring any already held. Returns how many were new."""
if not scrobbles:
return 0
with self.connection:
before = self.connection.total_changes
self.connection.executemany(
"INSERT OR IGNORE INTO scrobble"
" (uts, artist, track, album, artist_mbid, album_mbid, track_mbid)"
" VALUES (?, ?, ?, ?, ?, ?, ?)",
[
(
item.uts,
item.artist,
item.track,
item.album,
item.artist_mbid,
item.album_mbid,
item.track_mbid,
)
for item in scrobbles
],
)
return self.connection.total_changes - before
def replace_loved(self, rows):
"""Replace the loved-track list wholesale, in one transaction."""
with self.connection:
self.connection.execute("DELETE FROM loved")
self.connection.executemany(
"INSERT OR IGNORE INTO loved (artist, track, track_mbid, loved_at)"
" VALUES (?, ?, ?, ?)",
rows,
)
def scalar(self, sql):
return self.connection.execute(sql).fetchone()[0]
def newest_uts(self):
return self.scalar("SELECT MAX(uts) FROM scrobble")
def oldest_uts(self):
return self.scalar("SELECT MIN(uts) FROM scrobble")
def count(self):
return self.scalar("SELECT COUNT(*) FROM scrobble")
def fetch_page(client, user, to=None, since=None, page=1):
"""Return the scrobbles on one page of the recent-tracks history."""
payload = client.call(
"user.getRecentTracks",
{"user": user, "limit": PAGE_SIZE, "page": page, "from": since, "to": to},
)
block = payload.get("recenttracks") or {}
entries = as_list(block.get("track"))
parsed = [scrobble for scrobble in map(parse_scrobble, entries) if scrobble is not None]
attributes = block.get("@attr") or {}
return parsed, int(attributes.get("totalPages") or 1)
def backfill(client, store, user, now, limit=0):
"""Walk the history backwards until it runs out. Returns scrobbles added.
Each request asks for the newest page of everything at or before a cursor,
and the cursor is then moved to the oldest scrobble that came back. There is
no page number to keep across runs, so an interrupted backfill resumes from
whatever the database holds.
"""
if store.get_state("backfill_complete") == "yes":
return 0
cursor = store.oldest_uts()
if cursor is None:
cursor = now
page = 1
added = 0
requests = 0
while limit <= 0 or requests < limit:
scrobbles, _ = fetch_page(client, user, to=cursor, page=page)
requests += 1
if not scrobbles:
store.set_state("backfill_complete", "yes")
logger.info("backfill complete: %d scrobbles held", store.count())
return added
added += store.add_scrobbles(scrobbles)
page_oldest = min(scrobble.uts for scrobble in scrobbles)
if page_oldest < cursor:
cursor, page = page_oldest, 1
else:
# Every scrobble on this page shares the cursor's second. Moving the
# window would step over the rest of them, so take the next page of
# the same window instead.
page += 1
logger.info("backfill: %d added, reached %s", added, format_time(page_oldest))
logger.info("backfill paused after %d requests; resumes next pass", requests)
return added
def catch_up(client, store, user, now):
"""Fetch everything scrobbled since the newest scrobble held.
Bounded at both ends, so the window cannot shift under the paging while new
scrobbles arrive mid-run; anything that lands during the pass is picked up
by the next one.
"""
newest = store.newest_uts()
if newest is None:
return 0
added = 0
page = 1
while True:
scrobbles, total_pages = fetch_page(client, user, to=now, since=newest, page=page)
added += store.add_scrobbles(scrobbles)
if page >= total_pages:
return added
page += 1
def sync_loved(client, store, user):
"""Rebuild the loved-track table. Returns how many are loved."""
rows = []
page = 1
while True:
payload = client.call(
"user.getLovedTracks", {"user": user, "limit": PAGE_SIZE, "page": page}
)
block = payload.get("lovedtracks") or {}
for entry in as_list(block.get("track")):
artist = name_of(entry.get("artist"))
track = (entry.get("name") or "").strip()
if not artist or not track:
continue
loved_at = (entry.get("date") or {}).get("uts")
rows.append((artist, track, mbid_of(entry.get("mbid")), int(loved_at or 0) or None))
total_pages = int((block.get("@attr") or {}).get("totalPages") or 1)
if page >= total_pages:
break
page += 1
# Written only once the whole list has been fetched: a failure halfway
# through must not leave a truncated set of protected tracks behind.
store.replace_loved(rows)
return len(rows)
def format_time(uts):
"""Return a UTC timestamp as a readable date."""
if uts is None:
return "never"
return datetime.fromtimestamp(uts, tz=timezone.utc).strftime("%Y-%m-%d %H:%M")
def report(store, now):
"""Log what the store holds.
The MBID coverage line is the one to watch: scrobbles carrying a MusicBrainz
recording id can be joined to the library exactly, and everything else has
to go through name matching. That percentage predicts how well the next
stage will work, and whether it can be trusted to decide what is unplayed.
"""
total = store.count()
logger.info("scrobbles: %d", total)
if not total:
return
with_mbid = store.scalar("SELECT COUNT(*) FROM scrobble WHERE track_mbid IS NOT NULL")
logger.info(
"range: %s to %s", format_time(store.oldest_uts()), format_time(store.newest_uts())
)
logger.info("distinct artists: %d", store.scalar("SELECT COUNT(DISTINCT artist) FROM scrobble"))
logger.info(
"distinct tracks: %d",
store.scalar("SELECT COUNT(DISTINCT artist || ' - ' || track) FROM scrobble"),
)
logger.info("recording MBID present: %d (%.1f%%)", with_mbid, 100 * with_mbid / total)
logger.info("loved tracks: %d", store.scalar("SELECT COUNT(*) FROM loved"))
logger.info("backfill complete: %s", store.get_state("backfill_complete") == "yes")
top = store.connection.execute(
"SELECT artist, COUNT(*) AS plays FROM scrobble"
" GROUP BY artist ORDER BY plays DESC, artist LIMIT 10"
).fetchall()
for position, row in enumerate(top, start=1):
logger.info(" top artist %2d: %-40s %d", position, row["artist"][:40], row["plays"])
recent = store.connection.execute(
"SELECT artist, track, COUNT(*) AS plays FROM scrobble WHERE uts >= ?"
" GROUP BY artist, track ORDER BY plays DESC, artist LIMIT 10",
(now - 90 * 86400,),
).fetchall()
for position, row in enumerate(recent, start=1):
label = f"{row['artist']} - {row['track']}"
logger.info(" top track 90d %2d: %-50s %d", position, label[:50], row["plays"])
def run_once(client, store, user, now, backfill_limit):
"""Run a single ingest pass. Returns the number of scrobbles added."""
started = time.monotonic()
added = catch_up(client, store, user, now)
added += backfill(client, store, user, now, limit=backfill_limit)
loved = sync_loved(client, store, user)
logger.info(
"pass complete in %.1fs: %d scrobbles added, %d loved",
time.monotonic() - started,
added,
loved,
)
return added
def acquire_lock(database):
"""Take an exclusive lock so two passes cannot ingest into one store."""
database.parent.mkdir(parents=True, exist_ok=True)
handle = open(database.with_suffix(".lock"), "w") # noqa: SIM115 - held for the process
try:
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
handle.close()
return None
return handle
def build_parser():
"""Return the argument parser. Every option also reads an environment
variable, so the container can be configured without a command line."""
parser = argparse.ArgumentParser(
prog="music-curator",
description="Ingest a Last.fm scrobble history into a local store.",
)
parser.add_argument(
"--user",
default=os.getenv("MUSIC_CURATOR_LASTFM_USER"),
help="Last.fm username to read (env MUSIC_CURATOR_LASTFM_USER)",
)
parser.add_argument(
"--api-key",
default=os.getenv("MUSIC_CURATOR_LASTFM_API_KEY"),
help="Last.fm API key (env MUSIC_CURATOR_LASTFM_API_KEY)",
)
parser.add_argument(
"--db",
default=os.getenv("MUSIC_CURATOR_DB", "/data/curator.db"),
help="path to the SQLite store (env MUSIC_CURATOR_DB)",
)
parser.add_argument(
"--interval",
default=os.getenv("MUSIC_CURATOR_INTERVAL"),
help="repeat forever, waiting this long between passes, e.g. 6h (env MUSIC_CURATOR_INTERVAL)",
)
parser.add_argument(
"--request-delay",
type=float,
default=float(os.getenv("MUSIC_CURATOR_REQUEST_DELAY", REQUEST_DELAY_SECONDS)),
help="seconds between API requests (env MUSIC_CURATOR_REQUEST_DELAY)",
)
parser.add_argument(
"--backfill-limit",
type=int,
default=int(os.getenv("MUSIC_CURATOR_BACKFILL_LIMIT", "0")),
help="cap the backfill at this many requests per pass; 0 for no cap"
" (env MUSIC_CURATOR_BACKFILL_LIMIT)",
)
parser.add_argument(
"--report-only",
action="store_true",
help="report on the existing store without fetching anything",
)
return parser
def main(argv=None, clock=time.time):
"""Entry point. Returns a process exit code."""
logging.basicConfig(format="%(asctime)s %(levelname)s %(message)s", level=logging.INFO)
args = build_parser().parse_args(argv)
if not args.report_only and (not args.user or not args.api_key):
logger.error("both a Last.fm user and an API key are required")
return 2
try:
interval = parse_interval(args.interval) if args.interval else None
except ValueError as error:
logger.error("%s", error)
return 2
database = Path(args.db).resolve()
lock = acquire_lock(database)
if lock is None:
logger.error("another pass is already using %s", database)
return 3
store = Store(database)
if args.report_only:
report(store, int(clock()))
store.close()
lock.close()
return 0
client = Lastfm(args.api_key, delay=args.request_delay)
stopping = False
def stop(signum, _frame):
nonlocal stopping
stopping = True
logger.info("signal %d received; finishing the current pass", signum)
signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)
try:
while True:
now = int(clock())
try:
run_once(client, store, args.user, now, args.backfill_limit)
except LastfmError as error:
logger.error("%s", error)
if interval is None:
return 1
report(store, now)
if interval is None or stopping:
return 0
logger.info("sleeping %ds", interval)
for _ in range(interval):
if stopping:
return 0
time.sleep(1)
finally:
store.close()
lock.close()
def run():
"""Console-script entry point."""
sys.exit(main())
if __name__ == "__main__":
run()
pyproject.toml
+21
View File
@@ -0,0 +1,21 @@
[build-system]
requires = ["setuptools>=77"]
build-backend = "setuptools.build_meta"
[project]
name = "music-curator"
version = "0.1.0"
description = "Ingest a Last.fm listening history and curate a music library from it"
readme = "README.md"
requires-python = ">=3.11"
# No runtime Python dependencies: urllib, sqlite3 and json cover all of it.
dependencies = []
[project.scripts]
music-curator = "music_curator:run"
[project.urls]
Homepage = "https://code.emmathe.dev/lyrathorpe/music-curator"
[tool.setuptools]
py-modules = ["music_curator"]
pytest.ini
+5
View File
@@ -0,0 +1,5 @@
[pytest]
minversion = 7.0
testpaths = tests
python_files = test_*.py
addopts = -q
tests/conftest.py
+132
View File
@@ -0,0 +1,132 @@
import json
import os
import sys
import urllib.parse
import pytest
# Ensure the project root is on sys.path when running tests.
ROOT = os.path.dirname(os.path.dirname(__file__))
if ROOT not in sys.path:
sys.path.insert(0, ROOT)
def make_tracks(count, start=1_600_000_000, step=300, artists=5):
"""Return a synthetic recent-tracks history, oldest first.
Mirrors the real payload's quirks: artist and album are sub-objects keyed
`#text`, MBIDs are present-but-empty rather than absent when unknown, and
only some entries carry one.
"""
return [
{
"artist": {"#text": f"Artist {index % artists}", "mbid": f"artist-{index % artists}"},
"album": {"#text": f"Album {index % 7}", "mbid": ""},
"name": f"Track {index}",
"mbid": f"recording-{index}" if index % 3 == 0 else "",
"date": {"uts": str(start + index * step), "#text": "whenever"},
}
for index in range(count)
]
def make_loved(names):
"""Return loved-track entries, which key the artist as `name` not `#text`."""
return [
{
"name": name,
"mbid": "",
"artist": {"name": "Artist 0", "mbid": "artist-0"},
"date": {"uts": "1600000000"},
}
for name in names
]
class FakeLastfm:
"""A transport serving a canned history, so the tests need no network.
Honours `from`, `to`, `limit` and `page` the way the real service does, and
reproduces the two shapes that catch clients out: a lone result comes back
as a bare object rather than a one-item list, and a currently-playing track
is prepended to the first page with no `date`.
"""
def __init__(self, tracks=(), loved=(), nowplaying=None, outcomes=()):
self.tracks = sorted(tracks, key=lambda track: int(track["date"]["uts"]), reverse=True)
self.loved = list(loved)
self.nowplaying = nowplaying
# Served in order before any real response; an Exception is raised.
self.outcomes = list(outcomes)
self.calls = []
def __call__(self, url, timeout=None):
query = {
key: value[0]
for key, value in urllib.parse.parse_qs(urllib.parse.urlparse(url).query).items()
}
self.calls.append(query)
if self.outcomes:
outcome = self.outcomes.pop(0)
if isinstance(outcome, Exception):
raise outcome
return json.dumps(outcome)
method = query["method"].lower()
if method == "user.getrecenttracks":
return json.dumps(self._recent(query))
if method == "user.getlovedtracks":
return json.dumps(self._loved(query))
raise AssertionError(f"unexpected method {method}")
def _recent(self, query):
selected = self.tracks
if "to" in query:
selected = [t for t in selected if int(t["date"]["uts"]) <= int(query["to"])]
if "from" in query:
selected = [t for t in selected if int(t["date"]["uts"]) >= int(query["from"])]
window = self._page(selected, query)
if int(query.get("page", 1)) == 1 and self.nowplaying is not None:
window = [self.nowplaying, *window]
return {"recenttracks": self._wrap(window, selected, query)}
def _loved(self, query):
window = self._page(self.loved, query)
return {"lovedtracks": self._wrap(window, self.loved, query)}
@staticmethod
def _page(items, query):
limit = int(query.get("limit", 50))
page = int(query.get("page", 1))
start = (page - 1) * limit
return items[start : start + limit]
@staticmethod
def _wrap(window, selected, query):
limit = int(query.get("limit", 50))
total = len(selected)
return {
# A single result is not wrapped in a list by the real service.
"track": window[0] if len(window) == 1 else window,
"@attr": {
"user": query.get("user", ""),
"page": query.get("page", "1"),
"perPage": str(limit),
"totalPages": str(max(1, -(-total // limit))),
"total": str(total),
},
}
@pytest.fixture
def now_playing():
"""The entry Last.fm prepends for a track in progress: no `date` at all."""
return {
"artist": {"#text": "Artist 0", "mbid": "artist-0"},
"album": {"#text": "Album 0", "mbid": ""},
"name": "Currently Playing",
"mbid": "",
"@attr": {"nowplaying": "true"},
}
tests/test_music_curator.py
+243
View File
@@ -0,0 +1,243 @@
import urllib.error
import pytest
from conftest import FakeLastfm, make_loved, make_tracks
import music_curator
NOW = 1_700_000_000
def store_at(tmp_path):
return music_curator.Store(tmp_path / "curator.db")
def client_for(api, **kwargs):
kwargs.setdefault("delay", 0)
kwargs.setdefault("backoff", 0)
return music_curator.Lastfm("key", transport=api, **kwargs)
def ingest(store, api, backfill_limit=0):
client = client_for(api)
return music_curator.run_once(client, store, "lyra", NOW, backfill_limit)
def test_parse_interval_units():
assert music_curator.parse_interval("90") == 90
assert music_curator.parse_interval("30m") == 1800
assert music_curator.parse_interval("6h") == 21600
with pytest.raises(ValueError):
music_curator.parse_interval("soon")
def test_artist_name_is_read_from_either_shape():
"""Recent tracks key it `#text`; loved tracks and extended=1 key it `name`."""
assert music_curator.name_of({"#text": "Autechre"}) == "Autechre"
assert music_curator.name_of({"name": "Autechre"}) == "Autechre"
assert music_curator.name_of(None) == ""
def test_empty_mbid_becomes_none():
"""An empty string would otherwise look like a usable join key later on."""
assert music_curator.mbid_of({"mbid": ""}) is None
assert music_curator.mbid_of("") is None
assert music_curator.mbid_of({"mbid": "abc"}) == "abc"
def test_backfill_ingests_the_whole_history(tmp_path):
api = FakeLastfm(make_tracks(450))
store = store_at(tmp_path)
ingest(store, api)
assert store.count() == 450
assert store.get_state("backfill_complete") == "yes"
def test_backfill_resumes_after_a_capped_pass(tmp_path):
"""An interrupted backfill picks up from what the database holds."""
api = FakeLastfm(make_tracks(450))
store = store_at(tmp_path)
ingest(store, api, backfill_limit=1)
partial = store.count()
assert 0 < partial < 450
assert store.get_state("backfill_complete") != "yes"
ingest(store, api)
assert store.count() == 450
assert store.get_state("backfill_complete") == "yes"
def test_now_playing_is_not_stored(tmp_path, now_playing):
"""It has no timestamp, and it would arrive again on every single pass."""
api = FakeLastfm(make_tracks(10), nowplaying=now_playing)
store = store_at(tmp_path)
ingest(store, api)
assert store.count() == 10
titles = {row["track"] for row in store.connection.execute("SELECT track FROM scrobble")}
assert "Currently Playing" not in titles
def test_a_pass_over_unchanged_history_adds_nothing(tmp_path):
api = FakeLastfm(make_tracks(300))
store = store_at(tmp_path)
ingest(store, api)
before = store.count()
assert ingest(store, api) == 0
assert store.count() == before
def test_catch_up_adds_only_what_is_new(tmp_path):
history = make_tracks(300)
api = FakeLastfm(history)
store = store_at(tmp_path)
ingest(store, api)
newest = int(history[-1]["date"]["uts"])
api.tracks = sorted(
[*history, *make_tracks(5, start=newest + 3600)],
key=lambda track: int(track["date"]["uts"]),
reverse=True,
)
assert ingest(store, api) == 5
assert store.count() == 305
def test_a_lone_result_is_not_a_list(tmp_path):
"""Last.fm returns a bare object rather than a one-item list."""
api = FakeLastfm(make_tracks(1))
store = store_at(tmp_path)
ingest(store, api)
assert store.count() == 1
def test_a_full_page_sharing_one_second_is_paged_not_skipped(tmp_path):
"""Moving the window past a same-second cluster would lose the rest of it."""
api = FakeLastfm(make_tracks(250, step=0))
store = store_at(tmp_path)
ingest(store, api)
# 250 plays at one timestamp collapse to one row per distinct track.
assert store.count() == 250
assert store.oldest_uts() == store.newest_uts()
def test_loved_tracks_are_replaced_not_accumulated(tmp_path):
api = FakeLastfm(make_tracks(5), loved=make_loved(["One", "Two", "Three"]))
store = store_at(tmp_path)
ingest(store, api)
assert store.scalar("SELECT COUNT(*) FROM loved") == 3
api.loved = make_loved(["One"])
ingest(store, api)
assert store.scalar("SELECT COUNT(*) FROM loved") == 1
def test_rate_limiting_is_retried(tmp_path):
"""Error 29 is the service asking for patience, not a broken request."""
api = FakeLastfm(make_tracks(5), outcomes=[{"error": 29, "message": "Rate Limit Exceded"}])
store = store_at(tmp_path)
ingest(store, api)
assert store.count() == 5
def test_server_errors_are_retried(tmp_path):
api = FakeLastfm(
make_tracks(5),
outcomes=[urllib.error.HTTPError("url", 503, "unavailable", {}, None)],
)
store = store_at(tmp_path)
ingest(store, api)
assert store.count() == 5
def test_an_invalid_api_key_is_not_retried():
"""Error 10 will fail identically forever; failing fast says why."""
api = FakeLastfm(outcomes=[{"error": 10, "message": "Invalid API key"}])
client = client_for(api)
with pytest.raises(music_curator.LastfmError, match="error 10"):
client.call("user.getRecentTracks", {"user": "lyra"})
assert len(api.calls) == 1
def test_giving_up_reports_the_last_failure():
api = FakeLastfm(outcomes=[{"error": 29, "message": "slow down"}] * 3)
client = client_for(api, attempts=3)
with pytest.raises(music_curator.LastfmError, match="error 29"):
client.call("user.getRecentTracks", {"user": "lyra"})
assert len(api.calls) == 3
def test_the_store_rejects_a_schema_it_does_not_understand(tmp_path):
store = store_at(tmp_path)
store.set_state("schema_version", "999")
store.close()
with pytest.raises(RuntimeError, match="schema version"):
store_at(tmp_path)
def test_missing_credentials_are_refused(tmp_path, monkeypatch):
monkeypatch.delenv("MUSIC_CURATOR_LASTFM_USER", raising=False)
monkeypatch.delenv("MUSIC_CURATOR_LASTFM_API_KEY", raising=False)
assert music_curator.main(["--db", str(tmp_path / "curator.db")]) == 2
def test_report_only_needs_no_credentials(tmp_path):
assert music_curator.main(["--db", str(tmp_path / "curator.db"), "--report-only"]) == 0
def test_a_second_pass_will_not_start_while_one_is_running(tmp_path, monkeypatch):
database = tmp_path / "curator.db"
held = music_curator.acquire_lock(database)
assert held is not None
try:
assert music_curator.main(["--db", str(database), "--report-only"]) == 3
finally:
held.close()
def test_main_ingests_through_the_module_level_fetcher(tmp_path, monkeypatch):
api = FakeLastfm(make_tracks(20))
monkeypatch.setattr(music_curator, "http_get", api)
code = music_curator.main(
[
"--db",
str(tmp_path / "curator.db"),
"--user",
"lyra",
"--api-key",
"key",
"--request-delay",
"0",
],
clock=lambda: NOW,
)
assert code == 0
assert store_at(tmp_path).count() == 20