Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
88b96b8159 | ||
|
|
05cca3508c | ||
|
|
cda3d8463b |
@@ -7,9 +7,10 @@ which keeps an MP3 copy of a lossless library for an iPod. This one answers the
|
|||||||
question that mirror cannot: which of it is worth carrying, and which of it has
|
question that mirror cannot: which of it is worth carrying, and which of it has
|
||||||
not been played in years.
|
not been played in years.
|
||||||
|
|
||||||
**This is stage two.** It ingests the scrobble history, indexes the library from
|
**This is stage three.** It ingests the scrobble history, indexes the library
|
||||||
Lidarr, and matches one to the other. There are no playlists yet, and it writes
|
from Lidarr, matches one to the other, and writes playlists into the mirror.
|
||||||
nothing back — every Lidarr call is a `GET`. See "Where this is going" below.
|
Nothing is written back to Lidarr — every call there is a `GET`. See "Where this
|
||||||
|
is going" below.
|
||||||
|
|
||||||
## What it does today
|
## What it does today
|
||||||
|
|
||||||
@@ -18,6 +19,7 @@ nothing back — every Lidarr call is a `GET`. See "Where this is going" below.
|
|||||||
- Indexes every artist, album and track Lidarr knows about, with file paths and
|
- Indexes every artist, album and track Lidarr knows about, with file paths and
|
||||||
the date each file landed.
|
the date each file landed.
|
||||||
- Ties the two together and reports how well it managed.
|
- Ties the two together and reports how well it managed.
|
||||||
|
- Writes M3U playlists into the mirror, from the listening history.
|
||||||
|
|
||||||
## Matching
|
## Matching
|
||||||
|
|
||||||
@@ -126,6 +128,50 @@ and a shared title is a weak proxy for a shared song:
|
|||||||
Only the first is worth chasing. Counting all three as matcher failures
|
Only the first is worth chasing. Counting all three as matcher failures
|
||||||
overstates the problem and would over-block the cull.
|
overstates the problem and would over-block the cull.
|
||||||
|
|
||||||
|
## Playlists
|
||||||
|
|
||||||
|
Written into `<mirror>/_playlists/` as extended M3U, rebuilt every pass. Six
|
||||||
|
rules, capped at `--playlist-limit` tracks each:
|
||||||
|
|
||||||
|
| Playlist | Rule |
|
||||||
|
| -------------------- | --------------------------------------------------------- |
|
||||||
|
| `heavy-rotation` | Most played over the last twelve months |
|
||||||
|
| `all-time` | Most played ever |
|
||||||
|
| `neglected` | Played heavily once, silent for twelve months |
|
||||||
|
| `deep-cuts` | Never played, from albums whose other tracks you play constantly |
|
||||||
|
| `unheard-favourites` | Never played, by the artists you play most |
|
||||||
|
| `unheard` | Never played, anywhere in the library |
|
||||||
|
|
||||||
|
Ninety days was the obvious window for "recent" and is the wrong one: on a real
|
||||||
|
history it holds a few hundred plays spread thinly across a twenty-thousand
|
||||||
|
track rotation, so nothing ranks meaningfully. Twelve months does.
|
||||||
|
|
||||||
|
The two `unheard` playlists rotate **weekly**, not per pass. A pass runs every
|
||||||
|
few hours, and a playlist that reorders itself each time is one that has to be
|
||||||
|
re-imported each time — the Music app imports a snapshot of a file, it does not
|
||||||
|
track it.
|
||||||
|
|
||||||
|
### Paths
|
||||||
|
|
||||||
|
Lidarr knows where the lossless source is; the playlists have to point at the
|
||||||
|
MP3s music-mirror made from it. The mapping strips a library root from Lidarr's
|
||||||
|
track paths and re-roots them under the mirror, with the suffix changed.
|
||||||
|
|
||||||
|
`--library-root` is derived from the common parent of the indexed artist folders
|
||||||
|
when unset, so it agrees with Lidarr by construction rather than by being kept
|
||||||
|
in step by hand. Override it if that guess is wrong.
|
||||||
|
|
||||||
|
Entries are written **relative to the playlist file**, so one playlist works
|
||||||
|
from the NAS, from a Mac over SMB, and from Linux, without rewriting.
|
||||||
|
|
||||||
|
A track is only listed once its mirror file has been confirmed to exist. Lidarr
|
||||||
|
holding the FLAC says nothing about whether the MP3 has been encoded yet. If a
|
||||||
|
large number are missing, the run says so — that is what a wrong `--library-root`
|
||||||
|
or `--mirror` looks like, since the paths then map to nothing at all.
|
||||||
|
|
||||||
|
`_playlists/` survives music-mirror's prune: it only deletes `*.mp3`, and its
|
||||||
|
empty-directory sweep skips a directory holding M3Us.
|
||||||
|
|
||||||
## How the ingest works
|
## How the ingest works
|
||||||
|
|
||||||
Two halves, both taking their bounds from the database rather than from a saved
|
Two halves, both taking their bounds from the database rather than from a saved
|
||||||
@@ -180,6 +226,9 @@ music-curator --report-only # report on the store, fetch nothing
|
|||||||
| `--backfill-limit` | `MUSIC_CURATOR_BACKFILL_LIMIT` | `0` | Cap backfill requests per pass; 0 for no cap |
|
| `--backfill-limit` | `MUSIC_CURATOR_BACKFILL_LIMIT` | `0` | Cap backfill requests per pass; 0 for no cap |
|
||||||
| `--lidarr-url` | `MUSIC_CURATOR_LIDARR_URL` | unset | Lidarr base URL, e.g. `http://lidarr:8686` |
|
| `--lidarr-url` | `MUSIC_CURATOR_LIDARR_URL` | unset | Lidarr base URL, e.g. `http://lidarr:8686` |
|
||||||
| `--lidarr-api-key` | `MUSIC_CURATOR_LIDARR_API_KEY` | unset | Lidarr API key |
|
| `--lidarr-api-key` | `MUSIC_CURATOR_LIDARR_API_KEY` | unset | Lidarr API key |
|
||||||
|
| `--mirror` | `MUSIC_CURATOR_MIRROR` | unset | Root of the MP3 mirror; playlists go here |
|
||||||
|
| `--library-root` | `MUSIC_CURATOR_LIBRARY_ROOT` | derived | Prefix to strip from Lidarr's paths |
|
||||||
|
| `--playlist-limit` | `MUSIC_CURATOR_PLAYLIST_LIMIT` | `100` | Most tracks in any one playlist |
|
||||||
| `--skip-index` | — | off | Match against the index already held |
|
| `--skip-index` | — | off | Match against the index already held |
|
||||||
| `--report-only` | — | off | Report without fetching |
|
| `--report-only` | — | off | Report without fetching |
|
||||||
|
|
||||||
@@ -228,7 +277,8 @@ nix shell nixpkgs#python3Packages.pytest -c pytest
|
|||||||
| ------------------------------------------------ | ------------ |
|
| ------------------------------------------------ | ------------ |
|
||||||
| Last.fm ingest and store | done |
|
| Last.fm ingest and store | done |
|
||||||
| Lidarr index and the scrobble-to-track matcher | done |
|
| Lidarr index and the scrobble-to-track matcher | done |
|
||||||
| M3U playlists written into the mirror | next |
|
| M3U playlists from the listening history | done |
|
||||||
|
| Genre and mood playlists from Last.fm tags | next |
|
||||||
| Cold-music report, unmonitoring what is not played | last |
|
| Cold-music report, unmonitoring what is not played | last |
|
||||||
|
|
||||||
The cull will unmonitor cold albums in Lidarr and tag their artists. It will
|
The cull will unmonitor cold albums in Lidarr and tag their artists. It will
|
||||||
|
|||||||
@@ -30,5 +30,11 @@ services:
|
|||||||
# Cap the backfill at this many requests per pass. Unlimited by default,
|
# Cap the backfill at this many requests per pass. Unlimited by default,
|
||||||
# which finishes a long history in one go.
|
# which finishes a long history in one go.
|
||||||
# MUSIC_CURATOR_BACKFILL_LIMIT: "0"
|
# MUSIC_CURATOR_BACKFILL_LIMIT: "0"
|
||||||
|
# The MP3 mirror music-mirror maintains. Playlists are written into
|
||||||
|
# _playlists/ inside it; leave unset to skip them.
|
||||||
|
MUSIC_CURATOR_MIRROR: /mirror
|
||||||
|
# Most tracks in any one playlist.
|
||||||
|
# MUSIC_CURATOR_PLAYLIST_LIMIT: "100"
|
||||||
volumes:
|
volumes:
|
||||||
- /mnt/tank/apps/music-curator:/data
|
- /mnt/tank/apps/music-curator:/data
|
||||||
|
- /mnt/tank/media/music-mp3:/mirror
|
||||||
|
|||||||
+273
-2
@@ -26,6 +26,7 @@ import re
|
|||||||
import signal
|
import signal
|
||||||
import sqlite3
|
import sqlite3
|
||||||
import sys
|
import sys
|
||||||
|
import tempfile
|
||||||
import time
|
import time
|
||||||
import unicodedata
|
import unicodedata
|
||||||
import urllib.error
|
import urllib.error
|
||||||
@@ -62,6 +63,16 @@ REQUEST_DELAY_SECONDS = 0.25
|
|||||||
BACKOFF_SECONDS = 2.0
|
BACKOFF_SECONDS = 2.0
|
||||||
BACKOFF_CEILING_SECONDS = 60.0
|
BACKOFF_CEILING_SECONDS = 60.0
|
||||||
|
|
||||||
|
# What music-mirror names its output, and where playlists are written inside the
|
||||||
|
# mirror. music-mirror's prune only deletes `*.mp3` and only removes directories
|
||||||
|
# it finds empty, so a directory of M3Us survives it untouched.
|
||||||
|
MIRROR_SUFFIX = ".mp3"
|
||||||
|
PLAYLIST_DIRECTORY = "_playlists"
|
||||||
|
|
||||||
|
# Group-readable, for the same reason music-mirror sets it: the mirror is read
|
||||||
|
# back by whatever serves it, and a playlist nobody can read is not a playlist.
|
||||||
|
GROUP_READ = 0o040
|
||||||
|
|
||||||
SCHEMA_VERSION = "2"
|
SCHEMA_VERSION = "2"
|
||||||
|
|
||||||
# Versions this build upgrades in place. Everything added since version 1 is a
|
# Versions this build upgrades in place. Everything added since version 1 is a
|
||||||
@@ -1046,6 +1057,225 @@ def format_time(uts):
|
|||||||
return datetime.fromtimestamp(uts, tz=timezone.utc).strftime("%Y-%m-%d %H:%M")
|
return datetime.fromtimestamp(uts, tz=timezone.utc).strftime("%Y-%m-%d %H:%M")
|
||||||
|
|
||||||
|
|
||||||
|
# Playlists written into the mirror. Each is a rule over the listening history
|
||||||
|
# and the library index; none of them look at the audio.
|
||||||
|
#
|
||||||
|
# The rotating ones are shuffled by week rather than by pass. A pass runs every
|
||||||
|
# few hours, and a playlist that reorders itself every time is one that has to
|
||||||
|
# be re-imported every time -- the Music app does not track a file, it imports a
|
||||||
|
# snapshot of one.
|
||||||
|
ROTATION_PERIOD_SECONDS = 7 * 86400
|
||||||
|
SHUFFLE_MULTIPLIER = 2654435761
|
||||||
|
SHUFFLE_MODULUS = 104729
|
||||||
|
|
||||||
|
# Common table expressions the rules share. `played` is every library track that
|
||||||
|
# has ever been matched to a scrobble, with its count and the last time it was
|
||||||
|
# heard; `recent` is the same restricted to a window.
|
||||||
|
PLAYED_CTE = """
|
||||||
|
WITH played AS (
|
||||||
|
SELECT sk.track_id AS track_id, COUNT(*) AS plays, MAX(s.uts) AS last_uts
|
||||||
|
FROM scrobble s
|
||||||
|
JOIN scrobble_key sk ON sk.artist = s.artist AND sk.track = s.track
|
||||||
|
WHERE sk.track_id IS NOT NULL
|
||||||
|
GROUP BY sk.track_id
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
RECENT_CTE = """
|
||||||
|
WITH recent AS (
|
||||||
|
SELECT sk.track_id AS track_id, COUNT(*) AS plays
|
||||||
|
FROM scrobble s
|
||||||
|
JOIN scrobble_key sk ON sk.artist = s.artist AND sk.track = s.track
|
||||||
|
WHERE sk.track_id IS NOT NULL AND s.uts >= :year_ago
|
||||||
|
GROUP BY sk.track_id
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
SELECT_TRACK = """
|
||||||
|
SELECT t.id AS id, a.name AS artist, t.title AS title,
|
||||||
|
t.duration AS duration, t.path AS path
|
||||||
|
FROM lidarr_track t
|
||||||
|
JOIN lidarr_artist a ON a.id = t.artist_id
|
||||||
|
"""
|
||||||
|
PLAYABLE = " t.has_file = 1 AND t.path IS NOT NULL AND t.path <> '' "
|
||||||
|
SHUFFLE = f" ((t.id * {SHUFFLE_MULTIPLIER}) + :week) % {SHUFFLE_MODULUS} "
|
||||||
|
|
||||||
|
PLAYLISTS = (
|
||||||
|
(
|
||||||
|
"heavy-rotation",
|
||||||
|
"most played over the last twelve months",
|
||||||
|
RECENT_CTE + SELECT_TRACK + f"""
|
||||||
|
JOIN recent r ON r.track_id = t.id
|
||||||
|
WHERE {PLAYABLE}
|
||||||
|
ORDER BY r.plays DESC, a.name, t.title
|
||||||
|
LIMIT :limit
|
||||||
|
""",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"all-time",
|
||||||
|
"most played ever",
|
||||||
|
PLAYED_CTE + SELECT_TRACK + f"""
|
||||||
|
JOIN played p ON p.track_id = t.id
|
||||||
|
WHERE {PLAYABLE}
|
||||||
|
ORDER BY p.plays DESC, a.name, t.title
|
||||||
|
LIMIT :limit
|
||||||
|
""",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"neglected",
|
||||||
|
"played heavily once, silent for twelve months",
|
||||||
|
PLAYED_CTE + SELECT_TRACK + f"""
|
||||||
|
JOIN played p ON p.track_id = t.id
|
||||||
|
WHERE {PLAYABLE} AND p.last_uts < :year_ago
|
||||||
|
ORDER BY p.plays DESC, a.name, t.title
|
||||||
|
LIMIT :limit
|
||||||
|
""",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"deep-cuts",
|
||||||
|
"never played, from albums whose other tracks you play constantly",
|
||||||
|
PLAYED_CTE + """,
|
||||||
|
album_plays AS (
|
||||||
|
SELECT t.album_id AS album_id, SUM(p.plays) AS plays
|
||||||
|
FROM played p JOIN lidarr_track t ON t.id = p.track_id
|
||||||
|
GROUP BY t.album_id
|
||||||
|
)
|
||||||
|
""" + SELECT_TRACK + f"""
|
||||||
|
JOIN album_plays ap ON ap.album_id = t.album_id
|
||||||
|
LEFT JOIN played p ON p.track_id = t.id
|
||||||
|
WHERE {PLAYABLE} AND p.track_id IS NULL
|
||||||
|
ORDER BY ap.plays DESC, t.id
|
||||||
|
LIMIT :limit
|
||||||
|
""",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"unheard-favourites",
|
||||||
|
"never played, by the artists you play most; rotates weekly",
|
||||||
|
PLAYED_CTE + """,
|
||||||
|
artist_plays AS (
|
||||||
|
SELECT t.artist_id AS artist_id, SUM(p.plays) AS plays
|
||||||
|
FROM played p JOIN lidarr_track t ON t.id = p.track_id
|
||||||
|
GROUP BY t.artist_id
|
||||||
|
)
|
||||||
|
""" + SELECT_TRACK + f"""
|
||||||
|
LEFT JOIN played p ON p.track_id = t.id
|
||||||
|
WHERE {PLAYABLE} AND p.track_id IS NULL
|
||||||
|
AND t.artist_id IN (SELECT artist_id FROM artist_plays
|
||||||
|
ORDER BY plays DESC LIMIT 50)
|
||||||
|
ORDER BY {SHUFFLE}
|
||||||
|
LIMIT :limit
|
||||||
|
""",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
"unheard",
|
||||||
|
"never played, anywhere in the library; rotates weekly",
|
||||||
|
PLAYED_CTE + SELECT_TRACK + f"""
|
||||||
|
LEFT JOIN played p ON p.track_id = t.id
|
||||||
|
WHERE {PLAYABLE} AND p.track_id IS NULL
|
||||||
|
ORDER BY {SHUFFLE}
|
||||||
|
LIMIT :limit
|
||||||
|
""",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def library_root_of(store):
|
||||||
|
"""Return the directory Lidarr's artist folders sit under.
|
||||||
|
|
||||||
|
Derived rather than configured, because it has to agree with what Lidarr
|
||||||
|
reports and no one wants to keep a second copy of that in step by hand.
|
||||||
|
"""
|
||||||
|
paths = [
|
||||||
|
row["path"]
|
||||||
|
for row in store.connection.execute(
|
||||||
|
"SELECT path FROM lidarr_artist WHERE path IS NOT NULL AND path <> ''"
|
||||||
|
)
|
||||||
|
]
|
||||||
|
if not paths:
|
||||||
|
return None
|
||||||
|
if len(paths) == 1:
|
||||||
|
return str(Path(paths[0]).parent)
|
||||||
|
try:
|
||||||
|
return os.path.commonpath(paths)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def mirror_path_for(source, library_root, mirror_root):
|
||||||
|
"""Return where music-mirror would have put the MP3 for a source file."""
|
||||||
|
try:
|
||||||
|
relative = Path(source).relative_to(library_root)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
return (Path(mirror_root) / relative).with_suffix(MIRROR_SUFFIX)
|
||||||
|
|
||||||
|
|
||||||
|
def write_playlist(path, entries):
|
||||||
|
"""Write one extended M3U, atomically.
|
||||||
|
|
||||||
|
Paths are relative to the playlist file, so the same playlist works from the
|
||||||
|
NAS, from a Mac over SMB and from Linux without rewriting.
|
||||||
|
"""
|
||||||
|
lines = ["#EXTM3U"]
|
||||||
|
for entry in entries:
|
||||||
|
seconds = round((entry["duration"] or 0) / 1000)
|
||||||
|
lines.append(f"#EXTINF:{seconds},{entry['artist']} - {entry['title']}")
|
||||||
|
lines.append(os.path.relpath(entry["mirror"], path.parent))
|
||||||
|
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
handle, temporary = tempfile.mkstemp(dir=path.parent, suffix=".m3u.part")
|
||||||
|
os.close(handle)
|
||||||
|
temporary = Path(temporary)
|
||||||
|
try:
|
||||||
|
temporary.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||||
|
# The mirror is read back by something else; see music-mirror, which had
|
||||||
|
# to learn this the hard way.
|
||||||
|
mode = temporary.stat().st_mode
|
||||||
|
if not mode & GROUP_READ:
|
||||||
|
temporary.chmod(mode | GROUP_READ)
|
||||||
|
os.replace(temporary, path)
|
||||||
|
finally:
|
||||||
|
temporary.unlink(missing_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
def build_playlists(store, mirror_root, library_root, limit, now):
|
||||||
|
"""Write every playlist into the mirror. Returns how many tracks were listed.
|
||||||
|
|
||||||
|
A track is only listed once its mirror file has been confirmed to exist. The
|
||||||
|
index knows what Lidarr holds, which is the lossless source; whether the MP3
|
||||||
|
beside it has been encoded yet is music-mirror's business and is checked
|
||||||
|
rather than assumed.
|
||||||
|
"""
|
||||||
|
directory = Path(mirror_root) / PLAYLIST_DIRECTORY
|
||||||
|
parameters = {
|
||||||
|
"limit": limit,
|
||||||
|
"year_ago": now - 365 * 86400,
|
||||||
|
"week": now // ROTATION_PERIOD_SECONDS,
|
||||||
|
}
|
||||||
|
|
||||||
|
total = 0
|
||||||
|
missing = 0
|
||||||
|
for name, description, sql in PLAYLISTS:
|
||||||
|
entries = []
|
||||||
|
for row in store.connection.execute(sql, parameters):
|
||||||
|
mirror = mirror_path_for(row["path"], library_root, mirror_root)
|
||||||
|
if mirror is None or not mirror.is_file():
|
||||||
|
missing += 1
|
||||||
|
continue
|
||||||
|
entries.append({**dict(row), "mirror": mirror})
|
||||||
|
write_playlist(directory / f"{name}.m3u", entries)
|
||||||
|
total += len(entries)
|
||||||
|
logger.info("playlist %-20s %4d tracks -- %s", name, len(entries), description)
|
||||||
|
|
||||||
|
if missing:
|
||||||
|
logger.warning(
|
||||||
|
"%d selected tracks had no file in the mirror and were left out."
|
||||||
|
" A handful means music-mirror has not encoded them yet; a large"
|
||||||
|
" number means --library-root or --mirror is pointing at the wrong"
|
||||||
|
" place, since the paths are then being mapped to nothing.",
|
||||||
|
missing,
|
||||||
|
)
|
||||||
|
return total
|
||||||
|
|
||||||
|
|
||||||
def coverage_report(store):
|
def coverage_report(store):
|
||||||
"""Log how much of the listening history could be tied to the library.
|
"""Log how much of the listening history could be tied to the library.
|
||||||
|
|
||||||
@@ -1262,7 +1492,10 @@ def report(store, now):
|
|||||||
logger.info(" top track 90d %2d: %-50s %d", position, label[:50], row["plays"])
|
logger.info(" top track 90d %2d: %-50s %d", position, label[:50], row["plays"])
|
||||||
|
|
||||||
|
|
||||||
def run_once(client, store, user, now, backfill_limit, lidarr=None):
|
def run_once(
|
||||||
|
client, store, user, now, backfill_limit, lidarr=None, mirror=None,
|
||||||
|
library_root=None, playlist_limit=100,
|
||||||
|
):
|
||||||
"""Run a single pass. Returns the number of scrobbles added."""
|
"""Run a single pass. Returns the number of scrobbles added."""
|
||||||
started = time.monotonic()
|
started = time.monotonic()
|
||||||
added = catch_up(client, store, user, now)
|
added = catch_up(client, store, user, now)
|
||||||
@@ -1275,6 +1508,15 @@ def run_once(client, store, user, now, backfill_limit, lidarr=None):
|
|||||||
# arrived, and they need a verdict too.
|
# arrived, and they need a verdict too.
|
||||||
if store.scalar("SELECT COUNT(*) FROM lidarr_track"):
|
if store.scalar("SELECT COUNT(*) FROM lidarr_track"):
|
||||||
match_library(store)
|
match_library(store)
|
||||||
|
if mirror is not None:
|
||||||
|
root = library_root or library_root_of(store)
|
||||||
|
if root is None:
|
||||||
|
logger.warning(
|
||||||
|
"cannot work out where Lidarr's music lives, so no playlists were"
|
||||||
|
" written; set --library-root"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
build_playlists(store, mirror, root, playlist_limit, now)
|
||||||
|
|
||||||
logger.info(
|
logger.info(
|
||||||
"pass complete in %.1fs: %d scrobbles added, %d loved",
|
"pass complete in %.1fs: %d scrobbles added, %d loved",
|
||||||
@@ -1347,6 +1589,25 @@ def build_parser():
|
|||||||
default=os.getenv("MUSIC_CURATOR_LIDARR_API_KEY"),
|
default=os.getenv("MUSIC_CURATOR_LIDARR_API_KEY"),
|
||||||
help="Lidarr API key (env MUSIC_CURATOR_LIDARR_API_KEY)",
|
help="Lidarr API key (env MUSIC_CURATOR_LIDARR_API_KEY)",
|
||||||
)
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--mirror",
|
||||||
|
default=os.getenv("MUSIC_CURATOR_MIRROR"),
|
||||||
|
help="root of the MP3 mirror; playlists are written into it"
|
||||||
|
" (env MUSIC_CURATOR_MIRROR)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--library-root",
|
||||||
|
default=os.getenv("MUSIC_CURATOR_LIBRARY_ROOT"),
|
||||||
|
help="prefix to strip from Lidarr's track paths when mapping them into the"
|
||||||
|
" mirror; derived from the indexed artist folders when unset"
|
||||||
|
" (env MUSIC_CURATOR_LIBRARY_ROOT)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--playlist-limit",
|
||||||
|
type=int,
|
||||||
|
default=int(os.getenv("MUSIC_CURATOR_PLAYLIST_LIMIT", "100")),
|
||||||
|
help="most tracks to put in any one playlist (env MUSIC_CURATOR_PLAYLIST_LIMIT)",
|
||||||
|
)
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"--skip-index",
|
"--skip-index",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
@@ -1411,7 +1672,17 @@ def main(argv=None, clock=time.time):
|
|||||||
while True:
|
while True:
|
||||||
now = int(clock())
|
now = int(clock())
|
||||||
try:
|
try:
|
||||||
run_once(client, store, args.user, now, args.backfill_limit, lidarr)
|
run_once(
|
||||||
|
client,
|
||||||
|
store,
|
||||||
|
args.user,
|
||||||
|
now,
|
||||||
|
args.backfill_limit,
|
||||||
|
lidarr,
|
||||||
|
args.mirror,
|
||||||
|
args.library_root,
|
||||||
|
args.playlist_limit,
|
||||||
|
)
|
||||||
except (LastfmError, LidarrError) as error:
|
except (LastfmError, LidarrError) as error:
|
||||||
logger.error("%s", error)
|
logger.error("%s", error)
|
||||||
if interval is None:
|
if interval is None:
|
||||||
|
|||||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "music-curator"
|
name = "music-curator"
|
||||||
version = "0.3.1"
|
version = "0.4.0"
|
||||||
description = "Ingest a Last.fm listening history and curate a music library from it"
|
description = "Ingest a Last.fm listening history and curate a music library from it"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11"
|
requires-python = ">=3.11"
|
||||||
|
|||||||
@@ -1,5 +1,7 @@
|
|||||||
import json
|
import json
|
||||||
|
import stat
|
||||||
import urllib.error
|
import urllib.error
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from conftest import FakeLastfm, FakeLidarr, make_loved, make_tracks
|
from conftest import FakeLastfm, FakeLidarr, make_loved, make_tracks
|
||||||
@@ -825,3 +827,159 @@ def test_the_title_index_is_used_for_the_report_lookup(tmp_path):
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
assert "lidarr_track_title" in plan, plan
|
assert "lidarr_track_title" in plan, plan
|
||||||
|
|
||||||
|
|
||||||
|
def playlist_library(tmp_path):
|
||||||
|
"""A library on disk, with mirror MP3s beside the Lidarr source paths."""
|
||||||
|
source = tmp_path / "music"
|
||||||
|
mirror = tmp_path / "mirror"
|
||||||
|
library = [
|
||||||
|
{
|
||||||
|
"name": "Played Band",
|
||||||
|
"albums": [
|
||||||
|
{
|
||||||
|
"title": "Known",
|
||||||
|
"tracks": [{"title": "Hit"}, {"title": "Album Track"}],
|
||||||
|
}
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Silent Band",
|
||||||
|
"albums": [{"title": "Unknown", "tracks": [{"title": "Never Heard"}]}],
|
||||||
|
},
|
||||||
|
]
|
||||||
|
api = FakeLidarr(library)
|
||||||
|
# FakeLidarr invents /music/<artist>/<album>/<title>.flac; put the mirror
|
||||||
|
# MP3s at the paths music-mirror would have produced from those.
|
||||||
|
for handle in api.files:
|
||||||
|
relative = Path(handle["path"]).relative_to("/music")
|
||||||
|
handle["path"] = str(source / relative)
|
||||||
|
mp3 = (mirror / relative).with_suffix(".mp3")
|
||||||
|
mp3.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
mp3.write_bytes(b"not really an mp3")
|
||||||
|
for artist in api.artists:
|
||||||
|
artist["path"] = str(source / Path(artist["path"]).name)
|
||||||
|
return api, source, mirror
|
||||||
|
|
||||||
|
|
||||||
|
def test_playlists_are_written_into_the_mirror(tmp_path):
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
ingest(store, FakeLastfm([scrobble_of("Played Band", "Hit")] * 1))
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
music_curator.match_library(store)
|
||||||
|
|
||||||
|
music_curator.build_playlists(store, mirror, str(source), 100, NOW)
|
||||||
|
|
||||||
|
written = sorted(p.name for p in (mirror / "_playlists").glob("*.m3u"))
|
||||||
|
assert written == [
|
||||||
|
"all-time.m3u",
|
||||||
|
"deep-cuts.m3u",
|
||||||
|
"heavy-rotation.m3u",
|
||||||
|
"neglected.m3u",
|
||||||
|
"unheard-favourites.m3u",
|
||||||
|
"unheard.m3u",
|
||||||
|
]
|
||||||
|
|
||||||
|
played = (mirror / "_playlists" / "all-time.m3u").read_text().splitlines()
|
||||||
|
assert played[0] == "#EXTM3U"
|
||||||
|
assert played[1].startswith("#EXTINF:")
|
||||||
|
assert "Played Band - Hit" in played[1]
|
||||||
|
# Relative to the playlist file, so the same file works from any mount.
|
||||||
|
assert played[2] == "../Played Band/Known/Hit.mp3"
|
||||||
|
assert (mirror / "_playlists" / "all-time.m3u").parent.joinpath(played[2]).resolve().is_file()
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unplayed_track_lands_in_the_unheard_playlist(tmp_path):
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
ingest(store, FakeLastfm([scrobble_of("Played Band", "Hit")]))
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
music_curator.match_library(store)
|
||||||
|
|
||||||
|
music_curator.build_playlists(store, mirror, str(source), 100, NOW)
|
||||||
|
|
||||||
|
unheard = (mirror / "_playlists" / "unheard.m3u").read_text()
|
||||||
|
assert "Never Heard" in unheard
|
||||||
|
assert "Album Track" in unheard
|
||||||
|
# The one thing that was played must not be in it.
|
||||||
|
assert "Played Band - Hit\n" not in unheard
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_track_with_no_mirror_file_is_left_out(tmp_path):
|
||||||
|
"""The index knows what Lidarr holds; whether music-mirror has encoded the
|
||||||
|
MP3 yet is a different question, and is checked rather than assumed."""
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
for mp3 in mirror.rglob("*.mp3"):
|
||||||
|
mp3.unlink()
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
ingest(store, FakeLastfm([scrobble_of("Played Band", "Hit")]))
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
music_curator.match_library(store)
|
||||||
|
|
||||||
|
total = music_curator.build_playlists(store, mirror, str(source), 100, NOW)
|
||||||
|
|
||||||
|
assert total == 0
|
||||||
|
assert (mirror / "_playlists" / "all-time.m3u").read_text() == "#EXTM3U\n"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_library_root_is_derived_from_the_artist_folders(tmp_path):
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
|
||||||
|
assert music_curator.library_root_of(store) == str(source)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_playlist_limit_is_honoured(tmp_path):
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
music_curator.match_library(store)
|
||||||
|
|
||||||
|
music_curator.build_playlists(store, mirror, str(source), 1, NOW)
|
||||||
|
|
||||||
|
unheard = (mirror / "_playlists" / "unheard.m3u").read_text().splitlines()
|
||||||
|
assert len([line for line in unheard if line.startswith("#EXTINF")]) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_rotation_moves_weekly_not_every_pass(tmp_path):
|
||||||
|
"""A playlist that reorders on every pass is one that has to be re-imported
|
||||||
|
on every pass; the Music app imports a snapshot, it does not track a file."""
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
music_curator.match_library(store)
|
||||||
|
|
||||||
|
def unheard_at(when):
|
||||||
|
music_curator.build_playlists(store, mirror, str(source), 100, when)
|
||||||
|
return (mirror / "_playlists" / "unheard.m3u").read_text()
|
||||||
|
|
||||||
|
same_week = unheard_at(NOW), unheard_at(NOW + 3600)
|
||||||
|
assert same_week[0] == same_week[1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_playlists_are_group_readable(tmp_path):
|
||||||
|
api, source, mirror = playlist_library(tmp_path)
|
||||||
|
store = store_at(tmp_path)
|
||||||
|
music_curator.index_library(
|
||||||
|
music_curator.Lidarr("http://lidarr", "key", transport=api), store
|
||||||
|
)
|
||||||
|
music_curator.match_library(store)
|
||||||
|
|
||||||
|
music_curator.build_playlists(store, mirror, str(source), 100, NOW)
|
||||||
|
|
||||||
|
for playlist in (mirror / "_playlists").glob("*.m3u"):
|
||||||
|
assert playlist.stat().st_mode & stat.S_IRGRP, playlist
|
||||||
|
|||||||
Reference in New Issue
Block a user