Compare commits
6
Commits
88b96b8159
...
v0.5.1
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fbce764dc5 | ||
|
|
71c7115507 | ||
|
|
40dfce8a4c | ||
|
|
ddd126cc0d | ||
|
|
97b27f8daa | ||
|
|
15e5ee5aea |
@@ -8,7 +8,8 @@ question that mirror cannot: which of it is worth carrying, and which of it has
|
|||||||
not been played in years.
|
not been played in years.
|
||||||
|
|
||||||
**This is stage three.** It ingests the scrobble history, indexes the library
|
**This is stage three.** It ingests the scrobble history, indexes the library
|
||||||
from Lidarr, matches one to the other, and writes playlists into the mirror.
|
from Lidarr, matches one to the other, and writes playlists into the mirror —
|
||||||
|
by listening history and by mood.
|
||||||
Nothing is written back to Lidarr — every call there is a `GET`. See "Where this
|
Nothing is written back to Lidarr — every call there is a `GET`. See "Where this
|
||||||
is going" below.
|
is going" below.
|
||||||
|
|
||||||
@@ -151,6 +152,57 @@ few hours, and a playlist that reorders itself each time is one that has to be
|
|||||||
re-imported each time — the Music app imports a snapshot of a file, it does not
|
re-imported each time — the Music app imports a snapshot of a file, it does not
|
||||||
track it.
|
track it.
|
||||||
|
|
||||||
|
### Moods
|
||||||
|
|
||||||
|
A second set of playlists selects by **Last.fm's crowd tags** rather than by
|
||||||
|
listening history. MusicBrainz genres arrive free with the Lidarr index and are
|
||||||
|
no use for this: they are sparse and formal, and will not tell you a record is
|
||||||
|
screamo or synthwave. People typing tags will.
|
||||||
|
|
||||||
|
Tags are fetched once per artist — one `artist.getTopTags` call each — and
|
||||||
|
refreshed every ninety days. `--tag-limit` spreads the first sweep over several
|
||||||
|
passes.
|
||||||
|
|
||||||
|
Artists are looked up **by name**, not by MusicBrainz id, despite Lidarr having
|
||||||
|
an id for every one of them. Last.fm's mbid index is stale and partial: it
|
||||||
|
answers "the artist you supplied could not be found" for Devo, Escape the Fate,
|
||||||
|
Blasterjaxx and a few hundred others whose pages plainly exist and carry exactly
|
||||||
|
the tags wanted. Its name index is the one its own site runs on. The id is kept
|
||||||
|
only as a fallback, for a name Lidarr spells differently.
|
||||||
|
|
||||||
|
An artist neither key resolves is recorded as fetched with no tags, so the next
|
||||||
|
pass does not spend a request on it again. A genuine failure — a rate limit, a
|
||||||
|
bad key — is *not* recorded, so that one is retried.
|
||||||
|
|
||||||
|
The built-in moods are chosen for this library rather than as a general
|
||||||
|
taxonomy:
|
||||||
|
|
||||||
|
| Mood | Selected on |
|
||||||
|
| ------------------ | ------------------------------------------------- |
|
||||||
|
| `80s-synths` | synthpop, new wave, synthwave — released 1975-1992 |
|
||||||
|
| `high-energy-rock` | hard rock, punk, pop punk, alternative |
|
||||||
|
| `screamo` | screamo, post-hardcore, metalcore, emo |
|
||||||
|
| `drum-and-bass` | drum and bass, liquid funk, neurofunk, jungle |
|
||||||
|
| `dance` | house, big room, hardstyle, trance, dubstep |
|
||||||
|
| `classic-rock` | classic rock, prog, psychedelic, blues rock |
|
||||||
|
|
||||||
|
An artist qualifies when their tag weights inside a mood sum to at least 30 out
|
||||||
|
of Last.fm's 0-100 scale. One low-weight tag is not a genre, it is somebody's
|
||||||
|
stray opinion.
|
||||||
|
|
||||||
|
`years` filters on the album's release date, which is what separates eighties
|
||||||
|
synth records from everything else a synthpop tag drags in.
|
||||||
|
|
||||||
|
`--vibes` replaces the whole set with a JSON file of the same shape, so a new
|
||||||
|
mood does not need a new release:
|
||||||
|
|
||||||
|
```json
|
||||||
|
[{ "name": "shoegaze", "tags": ["shoegaze", "dream pop"], "min_score": 40 }]
|
||||||
|
```
|
||||||
|
|
||||||
|
Names are validated when the file is read, not when the file is written. A bad
|
||||||
|
one would otherwise surface as a playlist created somewhere unintended.
|
||||||
|
|
||||||
### Paths
|
### Paths
|
||||||
|
|
||||||
Lidarr knows where the lossless source is; the playlists have to point at the
|
Lidarr knows where the lossless source is; the playlists have to point at the
|
||||||
@@ -229,6 +281,8 @@ music-curator --report-only # report on the store, fetch nothing
|
|||||||
| `--mirror` | `MUSIC_CURATOR_MIRROR` | unset | Root of the MP3 mirror; playlists go here |
|
| `--mirror` | `MUSIC_CURATOR_MIRROR` | unset | Root of the MP3 mirror; playlists go here |
|
||||||
| `--library-root` | `MUSIC_CURATOR_LIBRARY_ROOT` | derived | Prefix to strip from Lidarr's paths |
|
| `--library-root` | `MUSIC_CURATOR_LIBRARY_ROOT` | derived | Prefix to strip from Lidarr's paths |
|
||||||
| `--playlist-limit` | `MUSIC_CURATOR_PLAYLIST_LIMIT` | `100` | Most tracks in any one playlist |
|
| `--playlist-limit` | `MUSIC_CURATOR_PLAYLIST_LIMIT` | `100` | Most tracks in any one playlist |
|
||||||
|
| `--vibes` | `MUSIC_CURATOR_VIBES` | built-in | JSON file of mood definitions |
|
||||||
|
| `--tag-limit` | `MUSIC_CURATOR_TAG_LIMIT` | `0` | Cap artist tag lookups per pass |
|
||||||
| `--skip-index` | — | off | Match against the index already held |
|
| `--skip-index` | — | off | Match against the index already held |
|
||||||
| `--report-only` | — | off | Report without fetching |
|
| `--report-only` | — | off | Report without fetching |
|
||||||
|
|
||||||
@@ -278,7 +332,7 @@ nix shell nixpkgs#python3Packages.pytest -c pytest
|
|||||||
| Last.fm ingest and store | done |
|
| Last.fm ingest and store | done |
|
||||||
| Lidarr index and the scrobble-to-track matcher | done |
|
| Lidarr index and the scrobble-to-track matcher | done |
|
||||||
| M3U playlists from the listening history | done |
|
| M3U playlists from the listening history | done |
|
||||||
| Genre and mood playlists from Last.fm tags | next |
|
| Genre and mood playlists from Last.fm tags | done |
|
||||||
| Cold-music report, unmonitoring what is not played | last |
|
| Cold-music report, unmonitoring what is not played | last |
|
||||||
|
|
||||||
The cull will unmonitor cold albums in Lidarr and tag their artists. It will
|
The cull will unmonitor cold albums in Lidarr and tag their artists. It will
|
||||||
|
|||||||
+302
-3
@@ -69,6 +69,18 @@ BACKOFF_CEILING_SECONDS = 60.0
|
|||||||
MIRROR_SUFFIX = ".mp3"
|
MIRROR_SUFFIX = ".mp3"
|
||||||
PLAYLIST_DIRECTORY = "_playlists"
|
PLAYLIST_DIRECTORY = "_playlists"
|
||||||
|
|
||||||
|
# Tags are re-fetched this often. They move slowly, and the first pass over a
|
||||||
|
# library already costs one request per artist.
|
||||||
|
TAG_REFRESH_SECONDS = 90 * 86400
|
||||||
|
|
||||||
|
# How much of an artist's tag weight has to fall inside a vibe before their
|
||||||
|
# tracks are eligible for it. Weights are Last.fm's own 0-100 popularity, summed
|
||||||
|
# across whichever of the vibe's tags the artist carries.
|
||||||
|
VIBE_MIN_SCORE = 30
|
||||||
|
|
||||||
|
# A vibe writes a file named after itself, so the name has to be a filename.
|
||||||
|
SAFE_VIBE_NAME = re.compile(r"^[a-z0-9][a-z0-9-]*$")
|
||||||
|
|
||||||
# Group-readable, for the same reason music-mirror sets it: the mirror is read
|
# Group-readable, for the same reason music-mirror sets it: the mirror is read
|
||||||
# back by whatever serves it, and a playlist nobody can read is not a playlist.
|
# back by whatever serves it, and a playlist nobody can read is not a playlist.
|
||||||
GROUP_READ = 0o040
|
GROUP_READ = 0o040
|
||||||
@@ -177,6 +189,24 @@ CREATE TABLE IF NOT EXISTS scrobble_key (
|
|||||||
PRIMARY KEY (artist, track)
|
PRIMARY KEY (artist, track)
|
||||||
);
|
);
|
||||||
CREATE INDEX IF NOT EXISTS scrobble_key_track ON scrobble_key (track_id);
|
CREATE INDEX IF NOT EXISTS scrobble_key_track ON scrobble_key (track_id);
|
||||||
|
|
||||||
|
-- Last.fm's crowd tags for each library artist. MusicBrainz genres come free
|
||||||
|
-- with the Lidarr index and are useless for this: they are sparse and formal,
|
||||||
|
-- and will not tell you a record is screamo or synthwave. People typing tags
|
||||||
|
-- will. Keyed on the normalised name rather than the Lidarr id so it survives
|
||||||
|
-- an artist being removed and re-added there.
|
||||||
|
CREATE TABLE IF NOT EXISTS artist_tag (
|
||||||
|
norm_artist TEXT NOT NULL,
|
||||||
|
tag TEXT NOT NULL,
|
||||||
|
weight INTEGER NOT NULL,
|
||||||
|
PRIMARY KEY (norm_artist, tag)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS artist_tag_tag ON artist_tag (tag);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS artist_tag_fetched (
|
||||||
|
norm_artist TEXT PRIMARY KEY,
|
||||||
|
fetched_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
@@ -236,7 +266,16 @@ WHITESPACE = re.compile(r"\s+")
|
|||||||
|
|
||||||
|
|
||||||
class LastfmError(Exception):
|
class LastfmError(Exception):
|
||||||
"""A Last.fm request that failed in a way retrying will not fix."""
|
"""A Last.fm request that failed in a way retrying will not fix.
|
||||||
|
|
||||||
|
`code` is the service's own error number where the failure came from the
|
||||||
|
API rather than the transport. Callers need it to tell "this thing does not
|
||||||
|
exist", which is final, from "something went wrong", which is not.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, message, code=None):
|
||||||
|
super().__init__(message)
|
||||||
|
self.code = code
|
||||||
|
|
||||||
|
|
||||||
class LidarrError(Exception):
|
class LidarrError(Exception):
|
||||||
@@ -417,7 +456,7 @@ class Lastfm:
|
|||||||
return payload
|
return payload
|
||||||
detail = f"error {code}: {payload.get('message', '')}".strip()
|
detail = f"error {code}: {payload.get('message', '')}".strip()
|
||||||
if code not in RETRYABLE_ERRORS:
|
if code not in RETRYABLE_ERRORS:
|
||||||
raise LastfmError(f"{method}: {detail}")
|
raise LastfmError(f"{method}: {detail}", code=code)
|
||||||
self._retry_or_raise(method, attempt, detail, None)
|
self._retry_or_raise(method, attempt, detail, None)
|
||||||
|
|
||||||
raise LastfmError(f"{method}: gave up after {self.attempts} attempts")
|
raise LastfmError(f"{method}: gave up after {self.attempts} attempts")
|
||||||
@@ -546,6 +585,24 @@ class Store:
|
|||||||
tracks,
|
tracks,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def replace_tags(self, norm_artist, pairs, now):
|
||||||
|
"""Replace one artist's tags, and record that they were fetched.
|
||||||
|
|
||||||
|
The fetch is recorded even when nothing came back, so an artist Last.fm
|
||||||
|
has never heard of is not asked about again on every pass.
|
||||||
|
"""
|
||||||
|
with self.connection:
|
||||||
|
self.connection.execute("DELETE FROM artist_tag WHERE norm_artist = ?", (norm_artist,))
|
||||||
|
self.connection.executemany(
|
||||||
|
"INSERT OR IGNORE INTO artist_tag (norm_artist, tag, weight) VALUES (?, ?, ?)",
|
||||||
|
[(norm_artist, tag, weight) for tag, weight in pairs],
|
||||||
|
)
|
||||||
|
self.connection.execute(
|
||||||
|
"INSERT INTO artist_tag_fetched (norm_artist, fetched_at) VALUES (?, ?)"
|
||||||
|
" ON CONFLICT (norm_artist) DO UPDATE SET fetched_at = excluded.fetched_at",
|
||||||
|
(norm_artist, now),
|
||||||
|
)
|
||||||
|
|
||||||
def rebuild_keys(self):
|
def rebuild_keys(self):
|
||||||
"""Collapse the scrobble history into one row per distinct track.
|
"""Collapse the scrobble history into one row per distinct track.
|
||||||
|
|
||||||
@@ -1177,6 +1234,231 @@ PLAYLISTS = (
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Moods, as sets of Last.fm tags. Chosen for this library -- drum and bass,
|
||||||
|
# punk and its descendants, big-room dance, classic rock -- rather than as a
|
||||||
|
# general taxonomy. `--vibes` replaces the lot with a JSON file of the same
|
||||||
|
# shape, so a new one does not need a new release.
|
||||||
|
#
|
||||||
|
# `years` filters on the album's release date, which is what separates eighties
|
||||||
|
# synth records from everything a synthpop tag would otherwise drag in.
|
||||||
|
DEFAULT_VIBES = (
|
||||||
|
{
|
||||||
|
"name": "80s-synths",
|
||||||
|
"tags": [
|
||||||
|
"synthpop", "synth pop", "synth-pop", "new wave", "synthwave",
|
||||||
|
"new romantic", "electropop", "80s", "1980s",
|
||||||
|
],
|
||||||
|
"years": [1975, 1992],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "high-energy-rock",
|
||||||
|
"tags": [
|
||||||
|
"hard rock", "punk rock", "pop punk", "punk", "alternative rock",
|
||||||
|
"rock", "garage rock", "skate punk",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "screamo",
|
||||||
|
"tags": [
|
||||||
|
"screamo", "post-hardcore", "metalcore", "emo", "hardcore",
|
||||||
|
"melodic hardcore", "emocore",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "drum-and-bass",
|
||||||
|
"tags": [
|
||||||
|
"drum and bass", "drum n bass", "dnb", "liquid funk", "neurofunk",
|
||||||
|
"jungle", "breakbeat",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "dance",
|
||||||
|
"tags": [
|
||||||
|
"electro house", "house", "big room", "electronic dance music",
|
||||||
|
"edm", "hardstyle", "trance", "dubstep", "electro",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "classic-rock",
|
||||||
|
"tags": [
|
||||||
|
"classic rock", "progressive rock", "psychedelic rock",
|
||||||
|
"blues rock", "70s", "60s",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def load_vibes(path):
|
||||||
|
"""Return the vibe definitions, from a file when one is given.
|
||||||
|
|
||||||
|
Validated up front rather than at write time: a bad name would otherwise
|
||||||
|
surface as a file created somewhere unintended, which is a poor way to find
|
||||||
|
out about a typo.
|
||||||
|
"""
|
||||||
|
if not path:
|
||||||
|
return DEFAULT_VIBES
|
||||||
|
try:
|
||||||
|
loaded = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||||
|
except (OSError, json.JSONDecodeError) as error:
|
||||||
|
raise ValueError(f"cannot read vibes from {path}: {error}") from error
|
||||||
|
|
||||||
|
if not isinstance(loaded, list):
|
||||||
|
raise ValueError(f"{path}: expected a list of vibes")
|
||||||
|
for vibe in loaded:
|
||||||
|
name = vibe.get("name") if isinstance(vibe, dict) else None
|
||||||
|
if not name or not SAFE_VIBE_NAME.fullmatch(str(name)):
|
||||||
|
raise ValueError(f"{path}: {name!r} is not a usable vibe name (a-z, 0-9, -)")
|
||||||
|
if not vibe.get("tags"):
|
||||||
|
raise ValueError(f"{path}: vibe {name!r} lists no tags")
|
||||||
|
return tuple(loaded)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_tags(payload):
|
||||||
|
"""Return (tag, weight) pairs from an artist.getTopTags response.
|
||||||
|
|
||||||
|
The documented sample carries only a name and a URL per tag; a live response
|
||||||
|
also carries a 0-100 `count`. Rather than depend on which, the count is used
|
||||||
|
when present and the documented ordering -- by popularity -- stands in for it
|
||||||
|
when it is not.
|
||||||
|
"""
|
||||||
|
block = payload.get("toptags") or {}
|
||||||
|
pairs = []
|
||||||
|
for position, entry in enumerate(as_list(block.get("tag"))):
|
||||||
|
tag = (entry.get("name") or "").strip().casefold()
|
||||||
|
if not tag:
|
||||||
|
continue
|
||||||
|
weight = int(entry.get("count") or 0) or max(1, 100 - position * 5)
|
||||||
|
pairs.append((tag, weight))
|
||||||
|
return pairs
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_tags(client, name, mbid):
|
||||||
|
"""Return an artist's tags, asking by name first.
|
||||||
|
|
||||||
|
By name, not by MusicBrainz id, despite Lidarr having an id for everything.
|
||||||
|
Last.fm's mbid index is stale and partial -- it answers "the artist you
|
||||||
|
supplied could not be found" for Devo, Escape the Fate and a few hundred
|
||||||
|
others whose pages plainly exist -- while the name index is the one its own
|
||||||
|
site runs on. The id is kept only as a fallback for a name Lidarr spells
|
||||||
|
differently.
|
||||||
|
|
||||||
|
Returns an empty list when the artist is genuinely unknown, which the caller
|
||||||
|
records so it is not asked again.
|
||||||
|
"""
|
||||||
|
attempts = [{"artist": name}] if name else []
|
||||||
|
if mbid:
|
||||||
|
attempts.append({"mbid": mbid})
|
||||||
|
|
||||||
|
for query in attempts:
|
||||||
|
try:
|
||||||
|
return parse_tags(client.call("artist.getTopTags", {**query, "autocorrect": 1}))
|
||||||
|
except LastfmError as error:
|
||||||
|
# Error 6 here means "no such artist", which the next key may still
|
||||||
|
# answer. Anything else is a real failure and belongs to the caller.
|
||||||
|
if error.code != 6:
|
||||||
|
raise
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def sync_tags(client, store, now, limit=0):
|
||||||
|
"""Fetch crowd tags for library artists that have none, or stale ones.
|
||||||
|
|
||||||
|
One request per artist, once, and then only for artists newly added. An
|
||||||
|
artist Last.fm has never heard of is recorded as fetched with no tags, so it
|
||||||
|
is not asked about again every pass.
|
||||||
|
"""
|
||||||
|
stale = store.connection.execute(
|
||||||
|
"SELECT a.norm_name AS norm_name, a.name AS name, a.mbid AS mbid"
|
||||||
|
" FROM lidarr_artist a"
|
||||||
|
" LEFT JOIN artist_tag_fetched f ON f.norm_artist = a.norm_name"
|
||||||
|
" WHERE a.norm_name <> ''"
|
||||||
|
" AND (f.fetched_at IS NULL OR f.fetched_at < :cutoff)"
|
||||||
|
" ORDER BY a.name",
|
||||||
|
{"cutoff": now - TAG_REFRESH_SECONDS},
|
||||||
|
).fetchall()
|
||||||
|
if not stale:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
if limit > 0 and len(stale) > limit:
|
||||||
|
logger.info("tagging %d of %d artists this pass; the rest follow next", limit, len(stale))
|
||||||
|
stale = stale[:limit]
|
||||||
|
else:
|
||||||
|
logger.info("fetching tags for %d artists", len(stale))
|
||||||
|
|
||||||
|
resolved = 0
|
||||||
|
unknown = 0
|
||||||
|
for artist in stale:
|
||||||
|
try:
|
||||||
|
pairs = fetch_tags(client, artist["name"], artist["mbid"])
|
||||||
|
except LastfmError as error:
|
||||||
|
# Something went wrong rather than the artist not existing. Left
|
||||||
|
# unrecorded on purpose, so the next pass tries again.
|
||||||
|
logger.warning("no tags for %s: %s", artist["name"], error)
|
||||||
|
continue
|
||||||
|
store.replace_tags(artist["norm_name"], pairs, now)
|
||||||
|
resolved += 1
|
||||||
|
if not pairs:
|
||||||
|
unknown += 1
|
||||||
|
|
||||||
|
logger.info("tagged %d artists (%d with nothing to say about them)", resolved, unknown)
|
||||||
|
return resolved
|
||||||
|
|
||||||
|
|
||||||
|
def build_vibe_playlists(store, vibes, mirror_root, library_root, limit, now):
|
||||||
|
"""Write one playlist per mood. Returns how many tracks were listed."""
|
||||||
|
if not store.scalar("SELECT COUNT(*) FROM artist_tag"):
|
||||||
|
return 0
|
||||||
|
|
||||||
|
directory = Path(mirror_root) / PLAYLIST_DIRECTORY
|
||||||
|
week = now // ROTATION_PERIOD_SECONDS
|
||||||
|
total = 0
|
||||||
|
|
||||||
|
for vibe in vibes:
|
||||||
|
tags = [str(tag).strip().casefold() for tag in vibe["tags"]]
|
||||||
|
placeholders = ",".join("?" * len(tags))
|
||||||
|
years = vibe.get("years")
|
||||||
|
parameters = [*tags, vibe.get("min_score", VIBE_MIN_SCORE)]
|
||||||
|
year_clause = ""
|
||||||
|
if years:
|
||||||
|
year_clause = (
|
||||||
|
" AND CAST(substr(al.release_date, 1, 4) AS INTEGER) BETWEEN ? AND ?"
|
||||||
|
)
|
||||||
|
parameters += [int(years[0]), int(years[1])]
|
||||||
|
parameters += [week, limit]
|
||||||
|
|
||||||
|
sql = f"""
|
||||||
|
WITH vibe AS (
|
||||||
|
SELECT norm_artist, SUM(weight) AS score
|
||||||
|
FROM artist_tag
|
||||||
|
WHERE tag IN ({placeholders})
|
||||||
|
GROUP BY norm_artist
|
||||||
|
HAVING SUM(weight) >= ?
|
||||||
|
)
|
||||||
|
SELECT t.id AS id, a.name AS artist, t.title AS title,
|
||||||
|
t.duration AS duration, t.path AS path
|
||||||
|
FROM lidarr_track t
|
||||||
|
JOIN lidarr_artist a ON a.id = t.artist_id
|
||||||
|
JOIN vibe v ON v.norm_artist = a.norm_name
|
||||||
|
LEFT JOIN lidarr_album al ON al.id = t.album_id
|
||||||
|
WHERE {PLAYABLE}{year_clause}
|
||||||
|
ORDER BY ((t.id * {SHUFFLE_MULTIPLIER}) + ?) % {SHUFFLE_MODULUS}
|
||||||
|
LIMIT ?
|
||||||
|
"""
|
||||||
|
|
||||||
|
entries = []
|
||||||
|
for row in store.connection.execute(sql, parameters):
|
||||||
|
mirror = mirror_path_for(row["path"], library_root, mirror_root)
|
||||||
|
if mirror is None or not mirror.is_file():
|
||||||
|
continue
|
||||||
|
entries.append({**dict(row), "mirror": mirror})
|
||||||
|
|
||||||
|
write_playlist(directory / f"{vibe['name']}.m3u", entries)
|
||||||
|
total += len(entries)
|
||||||
|
logger.info("playlist %-20s %4d tracks -- by tag", vibe["name"], len(entries))
|
||||||
|
|
||||||
|
return total
|
||||||
|
|
||||||
|
|
||||||
def library_root_of(store):
|
def library_root_of(store):
|
||||||
"""Return the directory Lidarr's artist folders sit under.
|
"""Return the directory Lidarr's artist folders sit under.
|
||||||
|
|
||||||
@@ -1494,7 +1776,7 @@ def report(store, now):
|
|||||||
|
|
||||||
def run_once(
|
def run_once(
|
||||||
client, store, user, now, backfill_limit, lidarr=None, mirror=None,
|
client, store, user, now, backfill_limit, lidarr=None, mirror=None,
|
||||||
library_root=None, playlist_limit=100,
|
library_root=None, playlist_limit=100, vibes=DEFAULT_VIBES, tag_limit=0,
|
||||||
):
|
):
|
||||||
"""Run a single pass. Returns the number of scrobbles added."""
|
"""Run a single pass. Returns the number of scrobbles added."""
|
||||||
started = time.monotonic()
|
started = time.monotonic()
|
||||||
@@ -1517,6 +1799,8 @@ def run_once(
|
|||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
build_playlists(store, mirror, root, playlist_limit, now)
|
build_playlists(store, mirror, root, playlist_limit, now)
|
||||||
|
sync_tags(client, store, now, limit=tag_limit)
|
||||||
|
build_vibe_playlists(store, vibes, mirror, root, playlist_limit, now)
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"pass complete in %.1fs: %d scrobbles added, %d loved",
|
"pass complete in %.1fs: %d scrobbles added, %d loved",
|
||||||
@@ -1608,6 +1892,18 @@ def build_parser():
|
|||||||
default=int(os.getenv("MUSIC_CURATOR_PLAYLIST_LIMIT", "100")),
|
default=int(os.getenv("MUSIC_CURATOR_PLAYLIST_LIMIT", "100")),
|
||||||
help="most tracks to put in any one playlist (env MUSIC_CURATOR_PLAYLIST_LIMIT)",
|
help="most tracks to put in any one playlist (env MUSIC_CURATOR_PLAYLIST_LIMIT)",
|
||||||
)
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--vibes",
|
||||||
|
default=os.getenv("MUSIC_CURATOR_VIBES"),
|
||||||
|
help="JSON file of mood definitions, replacing the built-in set"
|
||||||
|
" (env MUSIC_CURATOR_VIBES)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--tag-limit",
|
||||||
|
type=int,
|
||||||
|
default=int(os.getenv("MUSIC_CURATOR_TAG_LIMIT", "0")),
|
||||||
|
help="cap artist tag lookups per pass; 0 for no cap (env MUSIC_CURATOR_TAG_LIMIT)",
|
||||||
|
)
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--skip-index",
|
"--skip-index",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
@@ -1632,6 +1928,7 @@ def main(argv=None, clock=time.time):
|
|||||||
|
|
||||||
try:
|
try:
|
||||||
interval = parse_interval(args.interval) if args.interval else None
|
interval = parse_interval(args.interval) if args.interval else None
|
||||||
|
vibes = load_vibes(args.vibes)
|
||||||
except ValueError as error:
|
except ValueError as error:
|
||||||
logger.error("%s", error)
|
logger.error("%s", error)
|
||||||
return 2
|
return 2
|
||||||
@@ -1682,6 +1979,8 @@ def main(argv=None, clock=time.time):
|
|||||||
args.mirror,
|
args.mirror,
|
||||||
args.library_root,
|
args.library_root,
|
||||||
args.playlist_limit,
|
args.playlist_limit,
|
||||||
|
vibes,
|
||||||
|
args.tag_limit,
|
||||||
)
|
)
|
||||||
except (LastfmError, LidarrError) as error:
|
except (LastfmError, LidarrError) as error:
|
||||||
logger.error("%s", error)
|
logger.error("%s", error)
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "music-curator"
|
name = "music-curator"
|
||||||
version = "0.4.0"
|
version = "0.5.1"
|
||||||
description = "Ingest a Last.fm listening history and curate a music library from it"
|
description = "Ingest a Last.fm listening history and curate a music library from it"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
|
|||||||
+12
-1
@@ -56,7 +56,9 @@ class FakeLastfm:
|
|||||||
is prepended to the first page with no `date`.
|
is prepended to the first page with no `date`.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, tracks=(), loved=(), nowplaying=None, outcomes=()):
|
def __init__(self, tracks=(), loved=(), nowplaying=None, outcomes=(), tags=None):
|
||||||
|
# Keyed by mbid or by artist name, whichever the caller asked with.
|
||||||
|
self.tags = tags or {}
|
||||||
self.tracks = sorted(tracks, key=lambda track: int(track["date"]["uts"]), reverse=True)
|
self.tracks = sorted(tracks, key=lambda track: int(track["date"]["uts"]), reverse=True)
|
||||||
self.loved = list(loved)
|
self.loved = list(loved)
|
||||||
self.nowplaying = nowplaying
|
self.nowplaying = nowplaying
|
||||||
@@ -82,6 +84,15 @@ class FakeLastfm:
|
|||||||
return json.dumps(self._recent(query))
|
return json.dumps(self._recent(query))
|
||||||
if method == "user.getlovedtracks":
|
if method == "user.getlovedtracks":
|
||||||
return json.dumps(self._loved(query))
|
return json.dumps(self._loved(query))
|
||||||
|
if method == "artist.gettoptags":
|
||||||
|
key = query.get("mbid") or query.get("artist", "")
|
||||||
|
if key not in self.tags:
|
||||||
|
# What the real service says for a key it cannot resolve, which
|
||||||
|
# for mbids is a great many artists whose pages plainly exist.
|
||||||
|
return json.dumps(
|
||||||
|
{"error": 6, "message": "The artist you supplied could not be found"}
|
||||||
|
)
|
||||||
|
return json.dumps({"toptags": {"tag": self.tags[key]}})
|
||||||
raise AssertionError(f"unexpected method {method}")
|
raise AssertionError(f"unexpected method {method}")
|
||||||
|
|
||||||
def _recent(self, query):
|
def _recent(self, query):
|
||||||
|
|||||||
@@ -983,3 +983,218 @@ def test_playlists_are_group_readable(tmp_path):
|
|||||||
|
|
||||||
for playlist in (mirror / "_playlists").glob("*.m3u"):
|
for playlist in (mirror / "_playlists").glob("*.m3u"):
|
||||||
assert playlist.stat().st_mode & stat.S_IRGRP, playlist
|
assert playlist.stat().st_mode & stat.S_IRGRP, playlist
|
||||||
|
|
||||||
|
|
||||||
|
def test_tag_weights_fall_back_to_rank_when_no_count_is_sent():
|
||||||
|
"""The documented sample carries only a name and a URL; a live response also
|
||||||
|
carries a count. Neither may be relied on alone."""
|
||||||
|
with_count = music_curator.parse_tags(
|
||||||
|
{"toptags": {"tag": [{"name": "Screamo", "count": 100}, {"name": "emo", "count": 40}]}}
|
||||||
|
)
|
||||||
|
assert with_count == [("screamo", 100), ("emo", 40)]
|
||||||
|
|
||||||
|
without = music_curator.parse_tags(
|
||||||
|
{"toptags": {"tag": [{"name": "screamo"}, {"name": "emo"}]}}
|
||||||
|
)
|
||||||
|
assert [tag for tag, _ in without] == ["screamo", "emo"]
|
||||||
|
assert without[0][1] > without[1][1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_lone_tag_is_not_a_list():
|
||||||
|
assert music_curator.parse_tags({"toptags": {"tag": {"name": "dnb", "count": 90}}}) == [
|
||||||
|
("dnb", 90)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_artist_lastfm_cannot_answer_for_is_not_asked_again(tmp_path):
|
||||||
|
"""Recording the fetch even when it returns nothing is what stops a pass
|
||||||
|
spending a request per unknown artist, forever."""
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
lastfm = FakeLastfm(tags={})
|
||||||
|
|
||||||
|
assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 2
|
||||||
|
before = len(lastfm.calls)
|
||||||
|
|
||||||
|
assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 0
|
||||||
|
assert len(lastfm.calls) == before
|
||||||
|
|
||||||
|
|
||||||
|
def test_tags_are_refetched_once_they_go_stale(tmp_path):
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
lastfm = FakeLastfm(tags={})
|
||||||
|
music_curator.sync_tags(client_for(lastfm), store, NOW)
|
||||||
|
|
||||||
|
later = NOW + music_curator.TAG_REFRESH_SECONDS + 1
|
||||||
|
assert music_curator.sync_tags(client_for(lastfm), store, later) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def tagged_store(tmp_path, tags):
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
music_curator.sync_tags(client_for(FakeLastfm(tags=tags)), store, NOW)
|
||||||
|
return store, source, mirror
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_vibe_selects_by_tag(tmp_path):
|
||||||
|
store, source, mirror = tagged_store(
|
||||||
|
tmp_path,
|
||||||
|
{
|
||||||
|
"Played Band": [{"name": "screamo", "count": 100}],
|
||||||
|
"Silent Band": [{"name": "classic rock", "count": 100}],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
vibes = [{"name": "screamo", "tags": ["screamo"]}]
|
||||||
|
|
||||||
|
music_curator.build_vibe_playlists(store, vibes, mirror, str(source), 100, NOW)
|
||||||
|
|
||||||
|
written = (mirror / "_playlists" / "screamo.m3u").read_text()
|
||||||
|
assert "Played Band" in written
|
||||||
|
assert "Silent Band" not in written
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_weakly_tagged_artist_is_below_the_threshold(tmp_path):
|
||||||
|
"""A single low-weight tag is not a genre, it is somebody's stray opinion."""
|
||||||
|
store, source, mirror = tagged_store(
|
||||||
|
tmp_path, {"Played Band": [{"name": "screamo", "count": 3}]}
|
||||||
|
)
|
||||||
|
vibes = [{"name": "screamo", "tags": ["screamo"]}]
|
||||||
|
|
||||||
|
music_curator.build_vibe_playlists(store, vibes, mirror, str(source), 100, NOW)
|
||||||
|
|
||||||
|
assert (mirror / "_playlists" / "screamo.m3u").read_text() == "#EXTM3U\n"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_vibe_can_be_restricted_by_release_year(tmp_path):
|
||||||
|
"""What separates eighties synth records from everything else a synthpop
|
||||||
|
tag drags in."""
|
||||||
|
store, source, mirror = tagged_store(
|
||||||
|
tmp_path, {"Played Band": [{"name": "synthpop", "count": 100}]}
|
||||||
|
)
|
||||||
|
inside = [{"name": "eighties", "tags": ["synthpop"], "years": [1975, 1992]}]
|
||||||
|
outside = [{"name": "nineties", "tags": ["synthpop"], "years": [1993, 1999]}]
|
||||||
|
|
||||||
|
# FakeLidarr dates every album 2019, so neither window should catch it.
|
||||||
|
music_curator.build_vibe_playlists(store, inside, mirror, str(source), 100, NOW)
|
||||||
|
music_curator.build_vibe_playlists(store, outside, mirror, str(source), 100, NOW)
|
||||||
|
assert (mirror / "_playlists" / "eighties.m3u").read_text() == "#EXTM3U\n"
|
||||||
|
assert (mirror / "_playlists" / "nineties.m3u").read_text() == "#EXTM3U\n"
|
||||||
|
|
||||||
|
modern = [{"name": "modern", "tags": ["synthpop"], "years": [2000, 2030]}]
|
||||||
|
music_curator.build_vibe_playlists(store, modern, mirror, str(source), 100, NOW)
|
||||||
|
assert "Played Band" in (mirror / "_playlists" / "modern.m3u").read_text()
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_built_in_vibes_are_all_usable_filenames():
|
||||||
|
for vibe in music_curator.DEFAULT_VIBES:
|
||||||
|
assert music_curator.SAFE_VIBE_NAME.fullmatch(vibe["name"]), vibe["name"]
|
||||||
|
assert vibe["tags"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_vibes_file_replaces_the_built_in_set(tmp_path):
|
||||||
|
path = tmp_path / "vibes.json"
|
||||||
|
path.write_text(json.dumps([{"name": "mine", "tags": ["shoegaze"]}]))
|
||||||
|
|
||||||
|
assert music_curator.load_vibes(str(path)) == ({"name": "mine", "tags": ["shoegaze"]},)
|
||||||
|
assert music_curator.load_vibes(None) is music_curator.DEFAULT_VIBES
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"content",
|
||||||
|
[
|
||||||
|
'{"not": "a list"}',
|
||||||
|
'[{"tags": ["x"]}]',
|
||||||
|
'[{"name": "../escape", "tags": ["x"]}]',
|
||||||
|
'[{"name": "ok"}]',
|
||||||
|
"not json at all",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_a_bad_vibes_file_is_refused_up_front(tmp_path, content):
|
||||||
|
"""A bad name would otherwise surface as a file written somewhere
|
||||||
|
unintended, which is a poor way to learn about a typo."""
|
||||||
|
path = tmp_path / "vibes.json"
|
||||||
|
path.write_text(content)
|
||||||
|
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
music_curator.load_vibes(str(path))
|
||||||
|
|
||||||
|
|
||||||
|
def test_tags_are_looked_up_by_name_not_by_mbid(tmp_path):
|
||||||
|
"""Last.fm's mbid index is stale: it cannot find Devo or Escape the Fate by
|
||||||
|
one, though their pages plainly exist. Its name index can."""
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
lastfm = FakeLastfm(tags={"Played Band": [{"name": "screamo", "count": 90}]})
|
||||||
|
|
||||||
|
music_curator.sync_tags(client_for(lastfm), store, NOW)
|
||||||
|
|
||||||
|
asked = [call for call in lastfm.calls if call.get("method") == "artist.getTopTags"]
|
||||||
|
# The name is always tried first; the mbid only appears as a fallback for
|
||||||
|
# the artist that the name could not resolve.
|
||||||
|
assert "artist" in asked[0]
|
||||||
|
assert [call for call in asked if call.get("artist") == "Played Band"]
|
||||||
|
assert not [call for call in asked if call.get("mbid") == "artist-mbid-1"]
|
||||||
|
assert store.scalar("SELECT COUNT(*) FROM artist_tag WHERE tag = 'screamo'") == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_mbid_is_tried_when_the_name_is_not_found(tmp_path):
|
||||||
|
"""Kept only for a name Lidarr spells differently to Last.fm."""
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
lastfm = FakeLastfm(tags={"artist-mbid-1": [{"name": "dnb", "count": 80}]})
|
||||||
|
|
||||||
|
music_curator.sync_tags(client_for(lastfm), store, NOW)
|
||||||
|
|
||||||
|
assert store.scalar("SELECT COUNT(*) FROM artist_tag WHERE tag = 'dnb'") == 1
|
||||||
|
asked = [call for call in lastfm.calls if call.get("method") == "artist.getTopTags"]
|
||||||
|
assert any("mbid" in call for call in asked)
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_artist_neither_key_resolves_is_recorded_and_not_retried(tmp_path):
|
||||||
|
"""Otherwise every pass spends a request on it again, for ever."""
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
lastfm = FakeLastfm(tags={})
|
||||||
|
|
||||||
|
assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 2
|
||||||
|
spent = len(lastfm.calls)
|
||||||
|
|
||||||
|
assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 0
|
||||||
|
assert len(lastfm.calls) == spent
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_real_failure_is_not_recorded_so_the_next_pass_retries(tmp_path):
|
||||||
|
"""A rate limit is not the same as an artist not existing."""
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
lastfm = FakeLastfm(
|
||||||
|
tags={"Played Band": [], "Silent Band": []},
|
||||||
|
outcomes=[{"error": 10, "message": "Invalid API key"}],
|
||||||
|
)
|
||||||
|
|
||||||
|
music_curator.sync_tags(client_for(lastfm), store, NOW)
|
||||||
|
|
||||||
|
# One artist failed hard and must still be pending.
|
||||||
|
assert store.scalar("SELECT COUNT(*) FROM artist_tag_fetched") == 1
|
||||||
|
|||||||
Reference in New Issue
Block a user