19 Commits
Author SHA1 Message Date
lyrathorpe 7b6f6e0016 chore(release): v0.6.0 2026-08-24 17:49:48 +00:00
lyrathorpe 0ae2630a79 Merge pull request 'feat: rebuild the moods from real tags, add exclusions, fix playlist ownership' (#12) from feat/moods-from-real-tags into main
Build and publish container / build (push) Successful in 1m9s
Reviewed-on: #12
2026-08-24 18:48:43 +01:00
Emma Thorpe a7d16ca0b2 fix: give playlists the mirror's ownership rather than root's
Build and publish container / build (pull_request) Successful in 2m40s
Playlists were being written owned by root. The image runs as root by default,
deliberately, so that a bind-mounted dataset of any ownership stays writable --
but everything it writes then comes out root-owned, and a root-owned playlist
inside a mirror owned by the apps account is unreadable to whatever serves it.
The group-read bit does not help when the group is also root.

Copy the ownership of the mirror the playlist is being written into. That needs
no configuration, cannot drift from whatever the mirror actually is, and does
nothing at all when the two already agree -- which is the case whenever the
container is run with an explicit user.

Applied to the temporary file before the rename, so a playlist is never briefly
visible owned by the wrong account, and to the _playlists directory when this is
the run that creates it. A chown that is refused is ignored rather than fatal:
that only happens when not running as root, which is precisely the case where
the ownership was already right.
2026-08-24 18:45:25 +01:00
Emma Thorpe 8c4da6e14e feat: let a mood exclude tags, and widen the eighties one to the canon
Build and publish container / build (pull_request) Successful in 2m52s
The eighties mood selected on synth-specific tags only, on the grounds that the
bare `80s` tag drags in Def Leppard and Bon Jovi. Checking the canon against
live Last.fm pages shows what that costs. Eurythmics is tagged 80s, new wave,
pop, female vocalists, synth pop -- four of those five are tags no mood may use.
Frankie Goes to Hollywood is tagged 80s, new wave, pop, british, dance, and
survives only on new wave. An act tagged purely 80s, pop and dance, which is a
great deal of eighties pop, was missed outright.

Two spellings also matter more than the census suggested. Of Depeche Mode, Duran
Duran, Eurythmics and Frankie Goes to Hollywood, three carry "synth pop" with a
space and only one carries "synthpop" without. The census could not show this
because it only sees artists already in the library; the spelling that was kept
was the rarer one.

Add an `exclude` list to a mood. An excluded tag drops the artist outright
rather than docking their score, which is the only construction that expresses
"the eighties, but not the stadium rock" -- weighting cannot do it, because the
tag it would weight is the one both share. The synth acts do not carry hard rock
or hair metal, and that is the whole of the difference.

A mood may not both select on and exclude the same tag; that is rejected when
the definitions are read rather than silently producing nothing.
2026-08-24 18:39:59 +01:00
Emma Thorpe 41f6b290d8 feat: rebuild the moods from the library's measured tag distribution
Build and publish container / build (pull_request) Successful in 2m40s
The first set of moods was guessed at, and a tag census of the real library
shows how badly. Four tags in the dance mood -- edm, big room, hardstyle,
electronic dance music -- carry nothing whatsoever, while techno, electro, big
beat, rave and progressive house together carry over seven thousand plays and
were all absent. The drum and bass mood missed four of the six spellings the
taggers actually use, plus techstep, darkstep and hospital records.

The eighties mood was the worst of it. It leaned on the bare `80s` tag, which is
the eleventh most-played tag here and sits on Def Leppard, Bon Jovi and AC/DC
rather than on anything with a synthesiser in it. Removing it and adding a
hair-metal mood -- glam rock, hair metal, glam metal, arena rock, AOR, windowed
to 1975-1994 -- puts roughly eight thousand plays somewhere sensible and stops
the synth playlist being a stadium rock playlist.

Whole clusters had no mood at all: metal at around twenty-five thousand plays
across heavy, thrash, speed, power, death and NWOBHM, and pop punk and emo at a
similar figure. Both now exist, along with bass, nu-metal and indie.

Three kinds of tag are excluded on principle, with a test enforcing it, because
each produces a playlist that looks reasonable and is not. Nationality describes
a passport rather than a sound, and `american` alone spans 241 artists. `rock`
and `electronic` span 340 and 275, which is most of the collection. And
Last.fm's most popular tag for an artist is frequently the artist's own name, so
selecting on `green day` yields a Green Day playlist wearing a genre's clothes.
2026-08-24 18:34:34 +01:00
lyrathorpe fbce764dc5 chore(release): v0.5.1 2026-08-24 17:17:33 +00:00
lyrathorpe 71c7115507 Merge pull request 'fix: look tags up by artist name rather than by MusicBrainz id' (#11) from fix/tag-lookup-by-name into main
Build and publish container / build (push) Successful in 2m48s
Reviewed-on: #11
2026-08-24 18:14:41 +01:00
Emma Thorpe 40dfce8a4c fix: look tags up by artist name rather than by MusicBrainz id
Build and publish container / build (pull_request) Successful in 2m53s
Tag lookups asked by MusicBrainz id whenever Lidarr had one, which is always.
The reasoning was that an id cannot be ambiguous the way a name can. In practice
Last.fm's mbid index is stale and partial, and it answered "the artist you
supplied could not be found" for Devo, Escape the Fate, Blasterjaxx, Frank
Carter & the Rattlesnakes and a long tail of others -- artists whose Last.fm
pages plainly exist and carry precisely the tags the moods are built from. Devo
is tagged new wave, post-punk and 80s; Escape the Fate is tagged post-hardcore,
screamo and emocore. Both were skipped.

Ask by name first, which is the index Last.fm's own site runs on, and keep the
id only as a fallback for a name Lidarr spells differently.

Failures were also being dropped without recording the attempt, so every one of
those artists was re-queried on every subsequent pass, indefinitely. They are now
distinguished: an artist that neither key resolves is recorded as fetched with
no tags and not asked about again, while a genuine failure -- a rate limit, a
bad key -- is deliberately left unrecorded so the next pass retries it. Telling
the two apart needed the service's own error number, so LastfmError now carries
it.
2026-08-24 18:13:23 +01:00
lyrathorpe ddd126cc0d chore(release): v0.5.0 2026-08-24 17:04:05 +00:00
lyrathorpe 97b27f8daa Merge pull request 'feat: mood playlists from Last.fm crowd tags' (#10) from feat/tag-vibes into main
Build and publish container / build (push) Successful in 4m22s
Reviewed-on: #10
2026-08-24 17:59:43 +01:00
lyrathorpe 88b96b8159 chore(release): v0.4.0 2026-08-24 16:57:43 +00:00
Emma Thorpe 15e5ee5aea feat: mood playlists from Last.fm crowd tags
Build and publish container / build (pull_request) Successful in 3m50s
A second set of playlists selecting by genre and mood rather than by play
history: eighties synths, high energy rock, screamo, drum and bass, dance,
classic rock.

The tags come from Last.fm rather than MusicBrainz. MusicBrainz genres arrive
free with the Lidarr index, which makes them the obvious choice and the wrong
one: they are sparse and formal, and will not tell you a record is screamo or
synthwave. Crowd tags will, because people typed them.

One request per artist, by MusicBrainz id where Lidarr has one, refreshed every
ninety days. An artist Last.fm has never heard of is recorded as fetched with no
tags rather than left unmarked, so it is not asked about again on every pass
forever. The tag table is keyed on the normalised artist name, not the Lidarr
id, so it survives an artist being removed and re-added there.

Weights are taken from the response's count where it has one. The documented
sample carries only a name and a URL, a live response also carries a 0-100
count, and depending on either alone would be a guess -- so the count is used
when present and the documented ordering by popularity stands in when it is not.

An artist qualifies for a mood when their weights inside it sum to at least
thirty. A single low-weight tag is not a genre, it is somebody's stray opinion.
A mood may also restrict release years, which is what separates eighties synth
records from everything else a synthpop tag drags in.

The built-in set is chosen for this library rather than as a taxonomy, and
--vibes replaces it wholesale with a JSON file so a new mood does not need a new
release. Names are validated when that file is read: an invalid one would
otherwise only surface as a playlist written somewhere unintended.
2026-08-24 17:56:35 +01:00
lyrathorpe 05cca3508c Merge pull request 'feat: write playlists into the mirror from the listening history' (#9) from feat/playlists into main
Build and publish container / build (push) Successful in 4m43s
Reviewed-on: #9
2026-08-24 17:53:03 +01:00
Emma Thorpe cda3d8463b feat: write playlists into the mirror from the listening history
Build and publish container / build (pull_request) Successful in 5m13s
Six rules over the scrobble history and the library index, written as extended
M3U into <mirror>/_playlists/ and rebuilt every pass. Three look at what gets
played -- heavy rotation, all-time, and things played heavily once and silent
for a year. Three look at what does not: album tracks skipped on records
otherwise played constantly, unplayed tracks by the artists played most, and
unplayed tracks from anywhere. Four fifths of the library has never been played,
so both halves are needed.

Ninety days was the obvious window for "recent" and is the wrong one. On a real
history it holds a few hundred plays spread across a twenty-thousand track
rotation, and nothing ranks meaningfully. Twelve months does.

The two rotating playlists are shuffled by week rather than by pass. A pass runs
every few hours; a playlist that reorders itself each time is one that has to be
re-imported each time, because the Music app imports a snapshot of a file rather
than tracking it.

Lidarr knows where the lossless source is, and the playlists have to point at
the MP3s music-mirror produced from it. The library root is derived from the
common parent of the indexed artist folders, so it agrees with Lidarr by
construction instead of being kept in step by hand, and can be overridden.
Entries are relative to the playlist file, so one file works from the NAS, from
a Mac over SMB and from Linux.

A track is listed only once its mirror file has been confirmed to exist: Lidarr
holding the FLAC says nothing about whether the MP3 has been encoded. When many
are missing the run says so, because that is what a misconfigured root looks
like -- every path mapping to nothing -- and silence there would present an
empty playlist as a correct one.
2026-08-24 17:50:58 +01:00
lyrathorpe 08f099aaa9 chore(release): v0.3.1 2026-08-24 16:42:06 +00:00
lyrathorpe 719a6372ad Merge pull request 'fix: distinguish a title collision from an attribution miss, and index for it' (#8) from fix/title-collisions-and-index into main
Build and publish container / build (push) Successful in 4m54s
Reviewed-on: #8
2026-08-24 17:37:12 +01:00
Emma Thorpe 61ce751ac5 fix: distinguish a title collision from an attribution miss, and index for it
Build and publish container / build (pull_request) Successful in 5m16s
Two faults in the split added last change, both visible in the first real run.

It reported 700 pairs as attribution disagreements on the strength of the
library holding the same title under a different artist. The examples show what
that actually caught: "Everyday" matched Def Leppard, "Kaleidoscope" matched
Chappell Roan, "Fight for Your Right" matched Motley Crue. Different songs that
happen to share a name. Across fifty thousand tracks that is not an edge case,
it is the common case, and presenting it as a matcher failure argues for exactly
the title-only matching tier that would produce this rubbish on purpose.

The signal for a real attribution miss is narrower: the library's own title
credits the artist the play is filed under, as in "Voodoo People (Pendulum
Remix)" against a scrobble credited to Pendulum. The normalised title has that
suffix stripped -- which is what let the two meet in the first place -- so the
raw title is searched for the name. Collisions are now counted and named
separately, as what they are.

The same query also took forty-seven seconds. lidarr_track was indexed on
(norm_artist, norm_title), which a lookup by title alone cannot use because its
leading column is the artist, so every unmatched key scanned all eighty-four
thousand tracks. Add the index on the title by itself; the query plan changes
from an automatic partial index to a covering one.
2026-08-24 17:35:32 +01:00
lyrathorpe d1c10d32d9 Merge pull request 'ci: build the image once instead of twice' (#7) from ci/one-build-not-two into main
Reviewed-on: #7
2026-08-24 17:31:47 +01:00
Emma Thorpe 45d99ff039 ci: build the image once instead of twice
Build and publish container / build (pull_request) Successful in 5m40s
A pull request took roughly eleven minutes to go green, and the log shows one
CACHED line in the whole run. The image was being built twice, in full.

The test stage is built by the runner's docker daemon. The runtime stage was
then built by docker/build-push-action, which runs under a buildx builder that
setup-buildx-action creates in its own container with its own cache. The two
share nothing, so the second build pulled the base image again, ran pip install
again, and exported the layers again -- about four and a half minutes, plus
another thirty-five seconds to boot buildkit. The comment above the test step
claimed those layers were shared, which is what made this look reasonable.

buildx earns that overhead when producing several architectures. This produces
linux/amd64 only, by an explicit decision recorded in the workflow, so it earns
nothing here. Use plain docker build against the same daemon that ran the
tests, and push with docker push. The runtime stage is a strict prefix of the
test stage, so every layer is a cache hit: measured at 1.3 seconds locally
against roughly four and a half minutes in CI.

The remaining time is the runner itself, which is slow in absolute terms --
pytest takes four seconds locally and a hundred and two in CI. That is not
something the workflow can fix.
2026-08-24 17:24:56 +01:00
7 changed files with 1544 additions and 59 deletions
+29 -19
View File
@@ -45,7 +45,8 @@ jobs:
# The suite runs inside the image, against the interpreter that ships, # The suite runs inside the image, against the interpreter that ships,
# rather than against whatever the runner happens to provide. A failing # rather than against whatever the runner happens to provide. A failing
# test fails the build. Layers are shared with the push build below. # test fails the build. The runtime stage below is built from the same
# daemon afterwards, so its layers are already in cache.
- name: Run the test suite inside the image - name: Run the test suite inside the image
run: docker build --target test -t music-curator:test . run: docker build --target test -t music-curator:test .
@@ -124,9 +125,6 @@ jobs:
echo "release=${release}" >> "$GITHUB_OUTPUT" echo "release=${release}" >> "$GITHUB_OUTPUT"
echo "Computed bump=${bump}, release=${release}, base=${base}" echo "Computed bump=${bump}, release=${release}, base=${base}"
- name: Set up Buildx
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4
- name: Log in to the Gitea container registry - name: Log in to the Gitea container registry
if: github.event_name != 'pull_request' if: github.event_name != 'pull_request'
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4 uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4
@@ -135,21 +133,33 @@ jobs:
username: ${{ github.repository_owner }} username: ${{ github.repository_owner }}
password: ${{ secrets.PACKAGES_TOKEN }} password: ${{ secrets.PACKAGES_TOKEN }}
- name: Build and push # Plain `docker build` rather than buildx. buildx boots its own buildkit
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7 # in a container with a cache of its own, so it shared nothing with the
with: # test build above and rebuilt the image from the base image up -- two
context: . # full builds per run. It earns that cost when building for several
# Without this the last stage in the Dockerfile -- the test stage -- # platforms; this only ever targets the amd64 NAS, so it does not.
# would be what gets published. #
target: runtime # `--target runtime` is a strict prefix of the test stage, so every layer
# The NAS is the only host this runs on. Building arm64 as well would # is already in the daemon's cache and this resolves in seconds.
# mean emulating it under QEMU for no consumer. - name: Build the runtime image
platforms: linux/amd64 run: |
push: ${{ github.event_name != 'pull_request' }} set -euo pipefail
tags: ${{ steps.version.outputs.tags }} tags=()
labels: | while IFS= read -r tag; do
org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }} [ -n "$tag" ] && tags+=(-t "$tag")
org.opencontainers.image.revision=${{ github.sha }} done <<< "${{ steps.version.outputs.tags }}"
docker build --target runtime \
--label "org.opencontainers.image.source=${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}" \
--label "org.opencontainers.image.revision=${GITHUB_SHA}" \
"${tags[@]}" .
- name: Push
if: github.event_name != 'pull_request'
run: |
set -euo pipefail
while IFS= read -r tag; do
[ -n "$tag" ] && docker push "$tag"
done <<< "${{ steps.version.outputs.tags }}"
# Record the release: write the computed version into pyproject.toml, then # Record the release: write the computed version into pyproject.toml, then
# commit and tag it, so the packaging metadata always matches the release # commit and tag it, so the packaging metadata always matches the release
+173 -13
View File
@@ -7,9 +7,11 @@ which keeps an MP3 copy of a lossless library for an iPod. This one answers the
question that mirror cannot: which of it is worth carrying, and which of it has question that mirror cannot: which of it is worth carrying, and which of it has
not been played in years. not been played in years.
**This is stage two.** It ingests the scrobble history, indexes the library from **This is stage three.** It ingests the scrobble history, indexes the library
Lidarr, and matches one to the other. There are no playlists yet, and it writes from Lidarr, matches one to the other, and writes playlists into the mirror —
nothing back — every Lidarr call is a `GET`. See "Where this is going" below. by listening history and by mood.
Nothing is written back to Lidarr — every call there is a `GET`. See "Where this
is going" below.
## What it does today ## What it does today
@@ -18,6 +20,7 @@ nothing back — every Lidarr call is a `GET`. See "Where this is going" below.
- Indexes every artist, album and track Lidarr knows about, with file paths and - Indexes every artist, album and track Lidarr knows about, with file paths and
the date each file landed. the date each file landed.
- Ties the two together and reports how well it managed. - Ties the two together and reports how well it managed.
- Writes M3U playlists into the mirror, from the listening history.
## Matching ## Matching
@@ -110,17 +113,168 @@ unmatched by an artist the library holds: N pairs, M plays
``` ```
That is a track that was played by an artist the library holds. It is then That is a track that was played by an artist the library holds. It is then
split again, because owning an artist is a weak proxy for owning a track: split three ways, because owning an artist is a weak proxy for owning a track
and a shared title is a weak proxy for a shared song:
- **the title exists under another artist** — an attribution disagreement, a - **the library's own title credits the scrobbled artist** — `Voodoo People
remixer or a guest billed as the artist. These are the genuine misses, and (Pendulum Remix)` against a play credited to Pendulum. Same song, filed
each is a candidate for being wrongly called cold in stage four. The report under the original artist. These are the genuine misses.
names the artist the library files them under. - **the same title under an unrelated artist** — a collision, not a miss.
- **the title is nowhere in the library** — never bought. No amount of matching Across fifty thousand tracks these are constant: `Everyday` is Rusko and
conjures a file that does not exist. also Def Leppard, `Kaleidoscope` is Delta Heavy and also Chappell Roan.
Matching on title alone would be far worse than missing them, which is why
there is no such tier.
- **the title is nowhere in the library** — never bought.
Counting both as matcher failures overstates the problem and would over-block Only the first is worth chasing. Counting all three as matcher failures
the cull. The report lists the worst of each by play count. overstates the problem and would over-block the cull.
## Playlists
Written into `<mirror>/_playlists/` as extended M3U, rebuilt every pass. Six
rules, capped at `--playlist-limit` tracks each:
| Playlist | Rule |
| -------------------- | --------------------------------------------------------- |
| `heavy-rotation` | Most played over the last twelve months |
| `all-time` | Most played ever |
| `neglected` | Played heavily once, silent for twelve months |
| `deep-cuts` | Never played, from albums whose other tracks you play constantly |
| `unheard-favourites` | Never played, by the artists you play most |
| `unheard` | Never played, anywhere in the library |
Ninety days was the obvious window for "recent" and is the wrong one: on a real
history it holds a few hundred plays spread thinly across a twenty-thousand
track rotation, so nothing ranks meaningfully. Twelve months does.
The two `unheard` playlists rotate **weekly**, not per pass. A pass runs every
few hours, and a playlist that reorders itself each time is one that has to be
re-imported each time — the Music app imports a snapshot of a file, it does not
track it.
### Moods
A second set of playlists selects by **Last.fm's crowd tags** rather than by
listening history. MusicBrainz genres arrive free with the Lidarr index and are
no use for this: they are sparse and formal, and will not tell you a record is
screamo or synthwave. People typing tags will.
Tags are fetched once per artist — one `artist.getTopTags` call each — and
refreshed every ninety days. `--tag-limit` spreads the first sweep over several
passes.
Artists are looked up **by name**, not by MusicBrainz id, despite Lidarr having
an id for every one of them. Last.fm's mbid index is stale and partial: it
answers "the artist you supplied could not be found" for Devo, Escape the Fate,
Blasterjaxx and a few hundred others whose pages plainly exist and carry exactly
the tags wanted. Its name index is the one its own site runs on. The id is kept
only as a fallback, for a name Lidarr spells differently.
An artist neither key resolves is recorded as fetched with no tags, so the next
pass does not spend a request on it again. A genuine failure — a rate limit, a
bad key — is *not* recorded, so that one is retried.
The built-in moods are built from the **tag distribution of this library**,
measured, rather than from a general taxonomy:
| Mood | Selected on |
| --------------- | -------------------------------------------------------- |
| `drum-and-bass` | drum and bass and its six spellings, liquid funk, neurofunk, jungle, techstep, hospital records |
| `bass` | dubstep, brostep, grime, trip-hop, big beat |
| `dance` | house and its variants, trance, techno, electro, rave |
| `pop-punk` | pop punk, punk, emo, emocore, easycore, power pop |
| `screamo` | screamo, post-hardcore, metalcore, melodic hardcore, trancecore |
| `heavy-metal` | heavy metal, thrash, speed, power, death, prog, NWOBHM |
| `hair-metal` | hair metal, glam metal, glam rock, arena rock, AOR — 1975-1994 |
| `nu-metal` | nu metal, alternative metal, rapcore, industrial |
| `classic-rock` | classic rock, prog, psychedelic, blues rock, 70s, 60s |
| `80s-synths` | 80s, new wave, synth pop, electropop, post-punk — 1975-1992, rock excluded |
| `indie` | indie, indie rock, indie pop, britpop, singer-songwriter |
Measuring first mattered. `synthwave`, `edm`, `big room` and `hardstyle` are
plausible tags that carry **nothing at all** here, while `techstep`, `easycore`
and `hospital records` carry real weight. Guessing produces the first list.
Three kinds of tag are never used, and there is a test enforcing it:
- **Nationality** — `american` alone spans 241 artists. A passport is not a
sound.
- **`rock` and `electronic`** — 340 and 275 artists, most of the library. A
mood that matches everything is not a mood.
- **Artist names** — Last.fm's most popular tag for an artist is frequently
their own name. `green day`, `paramore` and `queen` are single-artist
playlists waiting to happen.
### Exclusions
A mood may also list `exclude`. An excluded tag drops the artist outright rather
than docking their score, and it exists because `80s-synths` cannot be written
any other way.
`80s` is the eleventh most-played tag here, and it sits on Def Leppard and Bon
Jovi exactly as heavily as on Eurythmics. Weighting cannot separate them,
because the tag it would weight is the one they share. What does separate them
is that the stadium rock also carries `hard rock` and `hair metal`, and the
synth acts do not.
Checked against live Last.fm pages, since the tag census only sees artists
already in the library:
| Artist | Tags |
| --- | --- |
| Eurythmics | `80s`, `new wave`, `pop`, `female vocalists`, `synth pop` |
| Frankie Goes to Hollywood | `80s`, `new wave`, `pop`, `british`, `dance` |
| Depeche Mode | `electronic`, `synthpop`, `new wave`, `80s`, `synth pop` |
| Duran Duran | `new wave`, `80s`, `pop`, `synth pop`, `rock` |
Four of Eurythmics' five tags are ones no mood may use. Three of the four
artists spell it **`synth pop`** with a space; only one spells it `synthpop`.
Guessing one spelling would have missed most of the canon.
An artist qualifies when their tag weights inside a mood sum to at least 30 out
of Last.fm's 0-100 scale. One low-weight tag is not a genre, it is somebody's
stray opinion.
`years` filters on the album's release date, which is what separates eighties
synth records from everything else a synthpop tag drags in.
`--vibes` replaces the whole set with a JSON file of the same shape, so a new
mood does not need a new release:
```json
[{ "name": "shoegaze", "tags": ["shoegaze", "dream pop"], "min_score": 40 }]
```
Names are validated when the file is read, not when the file is written. A bad
one would otherwise surface as a playlist created somewhere unintended.
### Paths
Lidarr knows where the lossless source is; the playlists have to point at the
MP3s music-mirror made from it. The mapping strips a library root from Lidarr's
track paths and re-roots them under the mirror, with the suffix changed.
`--library-root` is derived from the common parent of the indexed artist folders
when unset, so it agrees with Lidarr by construction rather than by being kept
in step by hand. Override it if that guess is wrong.
Entries are written **relative to the playlist file**, so one playlist works
from the NAS, from a Mac over SMB, and from Linux, without rewriting.
Each playlist is given the **owner and group of the mirror** it is written
into. The image runs as root by default so that a bind mount of any ownership
stays writable, and the cost of that is output owned by root — which the account
serving the share cannot read, group bit or no group bit, because the group is
also root. Copying the mirror's own ownership avoids having to be told what it
should be, and does nothing when the two already agree.
A track is only listed once its mirror file has been confirmed to exist. Lidarr
holding the FLAC says nothing about whether the MP3 has been encoded yet. If a
large number are missing, the run says so — that is what a wrong `--library-root`
or `--mirror` looks like, since the paths then map to nothing at all.
`_playlists/` survives music-mirror's prune: it only deletes `*.mp3`, and its
empty-directory sweep skips a directory holding M3Us.
## How the ingest works ## How the ingest works
@@ -176,6 +330,11 @@ music-curator --report-only # report on the store, fetch nothing
| `--backfill-limit` | `MUSIC_CURATOR_BACKFILL_LIMIT` | `0` | Cap backfill requests per pass; 0 for no cap | | `--backfill-limit` | `MUSIC_CURATOR_BACKFILL_LIMIT` | `0` | Cap backfill requests per pass; 0 for no cap |
| `--lidarr-url` | `MUSIC_CURATOR_LIDARR_URL` | unset | Lidarr base URL, e.g. `http://lidarr:8686` | | `--lidarr-url` | `MUSIC_CURATOR_LIDARR_URL` | unset | Lidarr base URL, e.g. `http://lidarr:8686` |
| `--lidarr-api-key` | `MUSIC_CURATOR_LIDARR_API_KEY` | unset | Lidarr API key | | `--lidarr-api-key` | `MUSIC_CURATOR_LIDARR_API_KEY` | unset | Lidarr API key |
| `--mirror` | `MUSIC_CURATOR_MIRROR` | unset | Root of the MP3 mirror; playlists go here |
| `--library-root` | `MUSIC_CURATOR_LIBRARY_ROOT` | derived | Prefix to strip from Lidarr's paths |
| `--playlist-limit` | `MUSIC_CURATOR_PLAYLIST_LIMIT` | `100` | Most tracks in any one playlist |
| `--vibes` | `MUSIC_CURATOR_VIBES` | built-in | JSON file of mood definitions |
| `--tag-limit` | `MUSIC_CURATOR_TAG_LIMIT` | `0` | Cap artist tag lookups per pass |
| `--skip-index` | — | off | Match against the index already held | | `--skip-index` | — | off | Match against the index already held |
| `--report-only` | — | off | Report without fetching | | `--report-only` | — | off | Report without fetching |
@@ -224,7 +383,8 @@ nix shell nixpkgs#python3Packages.pytest -c pytest
| ------------------------------------------------ | ------------ | | ------------------------------------------------ | ------------ |
| Last.fm ingest and store | done | | Last.fm ingest and store | done |
| Lidarr index and the scrobble-to-track matcher | done | | Lidarr index and the scrobble-to-track matcher | done |
| M3U playlists written into the mirror | next | | M3U playlists from the listening history | done |
| Genre and mood playlists from Last.fm tags | done |
| Cold-music report, unmonitoring what is not played | last | | Cold-music report, unmonitoring what is not played | last |
The cull will unmonitor cold albums in Lidarr and tag their artists. It will The cull will unmonitor cold albums in Lidarr and tag their artists. It will
+6
View File
@@ -30,5 +30,11 @@ services:
# Cap the backfill at this many requests per pass. Unlimited by default, # Cap the backfill at this many requests per pass. Unlimited by default,
# which finishes a long history in one go. # which finishes a long history in one go.
# MUSIC_CURATOR_BACKFILL_LIMIT: "0" # MUSIC_CURATOR_BACKFILL_LIMIT: "0"
# The MP3 mirror music-mirror maintains. Playlists are written into
# _playlists/ inside it; leave unset to skip them.
MUSIC_CURATOR_MIRROR: /mirror
# Most tracks in any one playlist.
# MUSIC_CURATOR_PLAYLIST_LIMIT: "100"
volumes: volumes:
- /mnt/tank/apps/music-curator:/data - /mnt/tank/apps/music-curator:/data
- /mnt/tank/media/music-mp3:/mirror
+738 -19
View File
@@ -26,6 +26,7 @@ import re
import signal import signal
import sqlite3 import sqlite3
import sys import sys
import tempfile
import time import time
import unicodedata import unicodedata
import urllib.error import urllib.error
@@ -62,6 +63,28 @@ REQUEST_DELAY_SECONDS = 0.25
BACKOFF_SECONDS = 2.0 BACKOFF_SECONDS = 2.0
BACKOFF_CEILING_SECONDS = 60.0 BACKOFF_CEILING_SECONDS = 60.0
# What music-mirror names its output, and where playlists are written inside the
# mirror. music-mirror's prune only deletes `*.mp3` and only removes directories
# it finds empty, so a directory of M3Us survives it untouched.
MIRROR_SUFFIX = ".mp3"
PLAYLIST_DIRECTORY = "_playlists"
# Tags are re-fetched this often. They move slowly, and the first pass over a
# library already costs one request per artist.
TAG_REFRESH_SECONDS = 90 * 86400
# How much of an artist's tag weight has to fall inside a vibe before their
# tracks are eligible for it. Weights are Last.fm's own 0-100 popularity, summed
# across whichever of the vibe's tags the artist carries.
VIBE_MIN_SCORE = 30
# A vibe writes a file named after itself, so the name has to be a filename.
SAFE_VIBE_NAME = re.compile(r"^[a-z0-9][a-z0-9-]*$")
# Group-readable, for the same reason music-mirror sets it: the mirror is read
# back by whatever serves it, and a playlist nobody can read is not a playlist.
GROUP_READ = 0o040
SCHEMA_VERSION = "2" SCHEMA_VERSION = "2"
# Versions this build upgrades in place. Everything added since version 1 is a # Versions this build upgrades in place. Everything added since version 1 is a
@@ -141,6 +164,11 @@ CREATE TABLE IF NOT EXISTS lidarr_track (
); );
CREATE INDEX IF NOT EXISTS lidarr_track_recording ON lidarr_track (recording_mbid); CREATE INDEX IF NOT EXISTS lidarr_track_recording ON lidarr_track (recording_mbid);
CREATE INDEX IF NOT EXISTS lidarr_track_norm ON lidarr_track (norm_artist, norm_title); CREATE INDEX IF NOT EXISTS lidarr_track_norm ON lidarr_track (norm_artist, norm_title);
-- Separate from the composite above, which a lookup by title alone cannot use:
-- its leading column is the artist. The report searches by title on its own, and
-- without this it scans every track for every unmatched key -- forty-seven
-- seconds on a library of eighty-four thousand.
CREATE INDEX IF NOT EXISTS lidarr_track_title ON lidarr_track (norm_title);
CREATE INDEX IF NOT EXISTS lidarr_track_album ON lidarr_track (album_id); CREATE INDEX IF NOT EXISTS lidarr_track_album ON lidarr_track (album_id);
-- One row per distinct thing listened to, with the verdict on whether it could -- One row per distinct thing listened to, with the verdict on whether it could
@@ -161,6 +189,24 @@ CREATE TABLE IF NOT EXISTS scrobble_key (
PRIMARY KEY (artist, track) PRIMARY KEY (artist, track)
); );
CREATE INDEX IF NOT EXISTS scrobble_key_track ON scrobble_key (track_id); CREATE INDEX IF NOT EXISTS scrobble_key_track ON scrobble_key (track_id);
-- Last.fm's crowd tags for each library artist. MusicBrainz genres come free
-- with the Lidarr index and are useless for this: they are sparse and formal,
-- and will not tell you a record is screamo or synthwave. People typing tags
-- will. Keyed on the normalised name rather than the Lidarr id so it survives
-- an artist being removed and re-added there.
CREATE TABLE IF NOT EXISTS artist_tag (
norm_artist TEXT NOT NULL,
tag TEXT NOT NULL,
weight INTEGER NOT NULL,
PRIMARY KEY (norm_artist, tag)
);
CREATE INDEX IF NOT EXISTS artist_tag_tag ON artist_tag (tag);
CREATE TABLE IF NOT EXISTS artist_tag_fetched (
norm_artist TEXT PRIMARY KEY,
fetched_at INTEGER NOT NULL
);
""" """
@@ -220,7 +266,16 @@ WHITESPACE = re.compile(r"\s+")
class LastfmError(Exception): class LastfmError(Exception):
"""A Last.fm request that failed in a way retrying will not fix.""" """A Last.fm request that failed in a way retrying will not fix.
`code` is the service's own error number where the failure came from the
API rather than the transport. Callers need it to tell "this thing does not
exist", which is final, from "something went wrong", which is not.
"""
def __init__(self, message, code=None):
super().__init__(message)
self.code = code
class LidarrError(Exception): class LidarrError(Exception):
@@ -401,7 +456,7 @@ class Lastfm:
return payload return payload
detail = f"error {code}: {payload.get('message', '')}".strip() detail = f"error {code}: {payload.get('message', '')}".strip()
if code not in RETRYABLE_ERRORS: if code not in RETRYABLE_ERRORS:
raise LastfmError(f"{method}: {detail}") raise LastfmError(f"{method}: {detail}", code=code)
self._retry_or_raise(method, attempt, detail, None) self._retry_or_raise(method, attempt, detail, None)
raise LastfmError(f"{method}: gave up after {self.attempts} attempts") raise LastfmError(f"{method}: gave up after {self.attempts} attempts")
@@ -530,6 +585,24 @@ class Store:
tracks, tracks,
) )
def replace_tags(self, norm_artist, pairs, now):
"""Replace one artist's tags, and record that they were fetched.
The fetch is recorded even when nothing came back, so an artist Last.fm
has never heard of is not asked about again on every pass.
"""
with self.connection:
self.connection.execute("DELETE FROM artist_tag WHERE norm_artist = ?", (norm_artist,))
self.connection.executemany(
"INSERT OR IGNORE INTO artist_tag (norm_artist, tag, weight) VALUES (?, ?, ?)",
[(norm_artist, tag, weight) for tag, weight in pairs],
)
self.connection.execute(
"INSERT INTO artist_tag_fetched (norm_artist, fetched_at) VALUES (?, ?)"
" ON CONFLICT (norm_artist) DO UPDATE SET fetched_at = excluded.fetched_at",
(norm_artist, now),
)
def rebuild_keys(self): def rebuild_keys(self):
"""Collapse the scrobble history into one row per distinct track. """Collapse the scrobble history into one row per distinct track.
@@ -1041,6 +1114,573 @@ def format_time(uts):
return datetime.fromtimestamp(uts, tz=timezone.utc).strftime("%Y-%m-%d %H:%M") return datetime.fromtimestamp(uts, tz=timezone.utc).strftime("%Y-%m-%d %H:%M")
# Playlists written into the mirror. Each is a rule over the listening history
# and the library index; none of them look at the audio.
#
# The rotating ones are shuffled by week rather than by pass. A pass runs every
# few hours, and a playlist that reorders itself every time is one that has to
# be re-imported every time -- the Music app does not track a file, it imports a
# snapshot of one.
ROTATION_PERIOD_SECONDS = 7 * 86400
SHUFFLE_MULTIPLIER = 2654435761
SHUFFLE_MODULUS = 104729
# Common table expressions the rules share. `played` is every library track that
# has ever been matched to a scrobble, with its count and the last time it was
# heard; `recent` is the same restricted to a window.
PLAYED_CTE = """
WITH played AS (
SELECT sk.track_id AS track_id, COUNT(*) AS plays, MAX(s.uts) AS last_uts
FROM scrobble s
JOIN scrobble_key sk ON sk.artist = s.artist AND sk.track = s.track
WHERE sk.track_id IS NOT NULL
GROUP BY sk.track_id
)
"""
RECENT_CTE = """
WITH recent AS (
SELECT sk.track_id AS track_id, COUNT(*) AS plays
FROM scrobble s
JOIN scrobble_key sk ON sk.artist = s.artist AND sk.track = s.track
WHERE sk.track_id IS NOT NULL AND s.uts >= :year_ago
GROUP BY sk.track_id
)
"""
SELECT_TRACK = """
SELECT t.id AS id, a.name AS artist, t.title AS title,
t.duration AS duration, t.path AS path
FROM lidarr_track t
JOIN lidarr_artist a ON a.id = t.artist_id
"""
PLAYABLE = " t.has_file = 1 AND t.path IS NOT NULL AND t.path <> '' "
SHUFFLE = f" ((t.id * {SHUFFLE_MULTIPLIER}) + :week) % {SHUFFLE_MODULUS} "
PLAYLISTS = (
(
"heavy-rotation",
"most played over the last twelve months",
RECENT_CTE + SELECT_TRACK + f"""
JOIN recent r ON r.track_id = t.id
WHERE {PLAYABLE}
ORDER BY r.plays DESC, a.name, t.title
LIMIT :limit
""",
),
(
"all-time",
"most played ever",
PLAYED_CTE + SELECT_TRACK + f"""
JOIN played p ON p.track_id = t.id
WHERE {PLAYABLE}
ORDER BY p.plays DESC, a.name, t.title
LIMIT :limit
""",
),
(
"neglected",
"played heavily once, silent for twelve months",
PLAYED_CTE + SELECT_TRACK + f"""
JOIN played p ON p.track_id = t.id
WHERE {PLAYABLE} AND p.last_uts < :year_ago
ORDER BY p.plays DESC, a.name, t.title
LIMIT :limit
""",
),
(
"deep-cuts",
"never played, from albums whose other tracks you play constantly",
PLAYED_CTE + """,
album_plays AS (
SELECT t.album_id AS album_id, SUM(p.plays) AS plays
FROM played p JOIN lidarr_track t ON t.id = p.track_id
GROUP BY t.album_id
)
""" + SELECT_TRACK + f"""
JOIN album_plays ap ON ap.album_id = t.album_id
LEFT JOIN played p ON p.track_id = t.id
WHERE {PLAYABLE} AND p.track_id IS NULL
ORDER BY ap.plays DESC, t.id
LIMIT :limit
""",
),
(
"unheard-favourites",
"never played, by the artists you play most; rotates weekly",
PLAYED_CTE + """,
artist_plays AS (
SELECT t.artist_id AS artist_id, SUM(p.plays) AS plays
FROM played p JOIN lidarr_track t ON t.id = p.track_id
GROUP BY t.artist_id
)
""" + SELECT_TRACK + f"""
LEFT JOIN played p ON p.track_id = t.id
WHERE {PLAYABLE} AND p.track_id IS NULL
AND t.artist_id IN (SELECT artist_id FROM artist_plays
ORDER BY plays DESC LIMIT 50)
ORDER BY {SHUFFLE}
LIMIT :limit
""",
),
(
"unheard",
"never played, anywhere in the library; rotates weekly",
PLAYED_CTE + SELECT_TRACK + f"""
LEFT JOIN played p ON p.track_id = t.id
WHERE {PLAYABLE} AND p.track_id IS NULL
ORDER BY {SHUFFLE}
LIMIT :limit
""",
),
)
# Moods, as sets of Last.fm tags. Chosen for this library -- drum and bass,
# punk and its descendants, big-room dance, classic rock -- rather than as a
# general taxonomy. `--vibes` replaces the lot with a JSON file of the same
# shape, so a new one does not need a new release.
#
# `years` filters on the album's release date, which is what separates eighties
# synth records from everything a synthpop tag would otherwise drag in.
DEFAULT_VIBES = (
# Built from the tag distribution of this library rather than from a general
# taxonomy, which is why some obvious-looking tags are absent and some
# unobvious ones are here. `synthwave`, `edm`, `big room` and `hardstyle`
# carry nothing at all; `techstep`, `easycore` and `hospital records` carry
# real weight.
#
# Three kinds of tag are deliberately never used. Nationality -- american,
# british, swedish -- describes a passport, not a sound, and `american`
# alone spans 241 artists. `rock` and `electronic` span 340 and 275, which
# is most of the library and therefore no mood at all. And Last.fm's most
# popular tag for an artist is frequently their own name, so `green day`,
# `paramore` and `queen` are single-artist playlists waiting to happen.
{
"name": "drum-and-bass",
"tags": [
"drum and bass", "dnb", "drum n bass", "drum'n'bass", "drum & bass",
"drum 'n' bass", "liquid funk", "neurofunk", "jungle", "techstep",
"darkstep", "drumstep", "breakbeat", "hospital records",
],
},
{
"name": "bass",
"tags": ["dubstep", "brostep", "grime", "trip-hop", "big beat"],
},
{
"name": "dance",
"tags": [
"house", "electro house", "progressive house", "tech house", "trance",
"techno", "electro", "rave", "dance", "minimal",
],
},
{
"name": "pop-punk",
"tags": [
"pop punk", "pop-punk", "punk rock", "punk", "skate punk", "emo",
"emocore", "easycore", "powerpop", "power pop", "post-grunge",
],
},
{
"name": "screamo",
"tags": [
"screamo", "post-hardcore", "metalcore", "melodic metalcore",
"melodic hardcore", "hardcore", "trancecore", "deathcore",
],
},
{
"name": "heavy-metal",
"tags": [
"heavy metal", "metal", "thrash metal", "thrash", "speed metal",
"power metal", "death metal", "progressive metal", "nwobhm",
"classic metal",
],
},
{
"name": "hair-metal",
"tags": [
"hair metal", "glam metal", "glam rock", "arena rock", "aor",
"rock and roll", "rock n roll",
],
"years": [1975, 1994],
},
{
"name": "nu-metal",
"tags": [
"nu metal", "nu-metal", "alternative metal", "rapcore",
"industrial metal", "industrial rock", "industrial",
],
},
{
"name": "classic-rock",
"tags": [
"classic rock", "progressive rock", "psychedelic rock", "psychedelic",
"blues rock", "blues", "southern rock", "art rock", "space rock",
"british invasion", "folk rock", "70s", "60s",
],
},
{
"name": "80s-synths",
# `80s` is included, which on its own would drag in Def Leppard and Bon
# Jovi -- they carry it as heavily as Eurythmics does. The exclusion is
# what separates them: the stadium rock is also tagged hard rock and
# hair metal, and the synth acts are not. Weighting cannot do this,
# because the tag it would weight is the one they share.
#
# Both spellings of synth pop are listed. Of Depeche Mode, Duran Duran,
# Eurythmics and Frankie Goes to Hollywood, three carry "synth pop" with
# a space and only one carries "synthpop" without.
"tags": [
"80s", "new wave", "synth pop", "synthpop", "synth-pop", "synthwave",
"electropop", "new romantic", "post-punk", "post-punk revival",
],
"exclude": [
"hard rock", "hair metal", "glam metal", "glam rock", "heavy metal",
"metal", "arena rock", "aor", "nwobhm", "thrash metal",
"classic rock", "southern rock", "blues rock",
],
"years": [1975, 1992],
},
{
"name": "indie",
"tags": ["indie", "indie rock", "indie pop", "britpop", "singer-songwriter"],
},
)
def load_vibes(path):
"""Return the vibe definitions, from a file when one is given.
Validated up front rather than at write time: a bad name would otherwise
surface as a file created somewhere unintended, which is a poor way to find
out about a typo.
"""
if not path:
return DEFAULT_VIBES
try:
loaded = json.loads(Path(path).read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as error:
raise ValueError(f"cannot read vibes from {path}: {error}") from error
if not isinstance(loaded, list):
raise ValueError(f"{path}: expected a list of vibes")
for vibe in loaded:
name = vibe.get("name") if isinstance(vibe, dict) else None
if not name or not SAFE_VIBE_NAME.fullmatch(str(name)):
raise ValueError(f"{path}: {name!r} is not a usable vibe name (a-z, 0-9, -)")
if not vibe.get("tags"):
raise ValueError(f"{path}: vibe {name!r} lists no tags")
overlap = {str(tag).casefold() for tag in vibe["tags"]} & {
str(tag).casefold() for tag in vibe.get("exclude", [])
}
if overlap:
raise ValueError(
f"{path}: vibe {name!r} both selects on and excludes {sorted(overlap)}"
)
return tuple(loaded)
def parse_tags(payload):
"""Return (tag, weight) pairs from an artist.getTopTags response.
The documented sample carries only a name and a URL per tag; a live response
also carries a 0-100 `count`. Rather than depend on which, the count is used
when present and the documented ordering -- by popularity -- stands in for it
when it is not.
"""
block = payload.get("toptags") or {}
pairs = []
for position, entry in enumerate(as_list(block.get("tag"))):
tag = (entry.get("name") or "").strip().casefold()
if not tag:
continue
weight = int(entry.get("count") or 0) or max(1, 100 - position * 5)
pairs.append((tag, weight))
return pairs
def fetch_tags(client, name, mbid):
"""Return an artist's tags, asking by name first.
By name, not by MusicBrainz id, despite Lidarr having an id for everything.
Last.fm's mbid index is stale and partial -- it answers "the artist you
supplied could not be found" for Devo, Escape the Fate and a few hundred
others whose pages plainly exist -- while the name index is the one its own
site runs on. The id is kept only as a fallback for a name Lidarr spells
differently.
Returns an empty list when the artist is genuinely unknown, which the caller
records so it is not asked again.
"""
attempts = [{"artist": name}] if name else []
if mbid:
attempts.append({"mbid": mbid})
for query in attempts:
try:
return parse_tags(client.call("artist.getTopTags", {**query, "autocorrect": 1}))
except LastfmError as error:
# Error 6 here means "no such artist", which the next key may still
# answer. Anything else is a real failure and belongs to the caller.
if error.code != 6:
raise
return []
def sync_tags(client, store, now, limit=0):
"""Fetch crowd tags for library artists that have none, or stale ones.
One request per artist, once, and then only for artists newly added. An
artist Last.fm has never heard of is recorded as fetched with no tags, so it
is not asked about again every pass.
"""
stale = store.connection.execute(
"SELECT a.norm_name AS norm_name, a.name AS name, a.mbid AS mbid"
" FROM lidarr_artist a"
" LEFT JOIN artist_tag_fetched f ON f.norm_artist = a.norm_name"
" WHERE a.norm_name <> ''"
" AND (f.fetched_at IS NULL OR f.fetched_at < :cutoff)"
" ORDER BY a.name",
{"cutoff": now - TAG_REFRESH_SECONDS},
).fetchall()
if not stale:
return 0
if limit > 0 and len(stale) > limit:
logger.info("tagging %d of %d artists this pass; the rest follow next", limit, len(stale))
stale = stale[:limit]
else:
logger.info("fetching tags for %d artists", len(stale))
resolved = 0
unknown = 0
for artist in stale:
try:
pairs = fetch_tags(client, artist["name"], artist["mbid"])
except LastfmError as error:
# Something went wrong rather than the artist not existing. Left
# unrecorded on purpose, so the next pass tries again.
logger.warning("no tags for %s: %s", artist["name"], error)
continue
store.replace_tags(artist["norm_name"], pairs, now)
resolved += 1
if not pairs:
unknown += 1
logger.info("tagged %d artists (%d with nothing to say about them)", resolved, unknown)
return resolved
def build_vibe_playlists(store, vibes, mirror_root, library_root, limit, now):
"""Write one playlist per mood. Returns how many tracks were listed."""
if not store.scalar("SELECT COUNT(*) FROM artist_tag"):
return 0
directory = Path(mirror_root) / PLAYLIST_DIRECTORY
week = now // ROTATION_PERIOD_SECONDS
total = 0
for vibe in vibes:
tags = [str(tag).strip().casefold() for tag in vibe["tags"]]
placeholders = ",".join("?" * len(tags))
parameters = [*tags, vibe.get("min_score", VIBE_MIN_SCORE)]
# An exclusion drops the artist outright rather than docking their
# score. It is the only way to write "the eighties, but not the stadium
# rock": those artists carry `80s` as heavily as the synth acts do, so
# no amount of weighting separates them -- but they also carry `hard
# rock`, and the synth acts do not.
excluded = [str(tag).strip().casefold() for tag in vibe.get("exclude", [])]
exclude_clause = ""
if excluded:
exclude_clause = (
" AND NOT EXISTS (SELECT 1 FROM artist_tag x"
" WHERE x.norm_artist = a.norm_name"
f" AND x.tag IN ({','.join('?' * len(excluded))}))"
)
parameters += excluded
years = vibe.get("years")
year_clause = ""
if years:
year_clause = (
" AND CAST(substr(al.release_date, 1, 4) AS INTEGER) BETWEEN ? AND ?"
)
parameters += [int(years[0]), int(years[1])]
parameters += [week, limit]
sql = f"""
WITH vibe AS (
SELECT norm_artist, SUM(weight) AS score
FROM artist_tag
WHERE tag IN ({placeholders})
GROUP BY norm_artist
HAVING SUM(weight) >= ?
)
SELECT t.id AS id, a.name AS artist, t.title AS title,
t.duration AS duration, t.path AS path
FROM lidarr_track t
JOIN lidarr_artist a ON a.id = t.artist_id
JOIN vibe v ON v.norm_artist = a.norm_name
LEFT JOIN lidarr_album al ON al.id = t.album_id
WHERE {PLAYABLE}{exclude_clause}{year_clause}
ORDER BY ((t.id * {SHUFFLE_MULTIPLIER}) + ?) % {SHUFFLE_MODULUS}
LIMIT ?
"""
entries = []
for row in store.connection.execute(sql, parameters):
mirror = mirror_path_for(row["path"], library_root, mirror_root)
if mirror is None or not mirror.is_file():
continue
entries.append({**dict(row), "mirror": mirror})
write_playlist(directory / f"{vibe['name']}.m3u", entries, Path(mirror_root))
total += len(entries)
logger.info("playlist %-20s %4d tracks -- by tag", vibe["name"], len(entries))
return total
def library_root_of(store):
"""Return the directory Lidarr's artist folders sit under.
Derived rather than configured, because it has to agree with what Lidarr
reports and no one wants to keep a second copy of that in step by hand.
"""
paths = [
row["path"]
for row in store.connection.execute(
"SELECT path FROM lidarr_artist WHERE path IS NOT NULL AND path <> ''"
)
]
if not paths:
return None
if len(paths) == 1:
return str(Path(paths[0]).parent)
try:
return os.path.commonpath(paths)
except ValueError:
return None
def mirror_path_for(source, library_root, mirror_root):
"""Return where music-mirror would have put the MP3 for a source file."""
try:
relative = Path(source).relative_to(library_root)
except ValueError:
return None
return (Path(mirror_root) / relative).with_suffix(MIRROR_SUFFIX)
def set_ownership(path, uid, gid):
"""Give a path an owner and group. Returns whether anything changed."""
try:
current = path.stat()
if (current.st_uid, current.st_gid) == (uid, gid):
return False
os.chown(path, uid, gid)
except OSError:
# Not permitted unless running as root, which is the case where the
# ownership is already whatever the caller runs as.
return False
return True
def match_ownership(path, reference):
"""Give a path the owner and group of the tree it is joining.
The image runs as root by default, so that a bind mount of any ownership
stays writable. The cost is that everything it writes comes out root-owned,
and a root-owned playlist inside a mirror owned by the apps account is
unreadable to the thing that serves it -- the group bit does not help when
the group is root.
Copying the mirror's own ownership avoids having to be told what it should
be, and is a no-op when the two already agree.
"""
try:
wanted = reference.stat()
except OSError:
return False
return set_ownership(path, wanted.st_uid, wanted.st_gid)
def write_playlist(path, entries, reference=None):
"""Write one extended M3U, atomically.
Paths are relative to the playlist file, so the same playlist works from the
NAS, from a Mac over SMB and from Linux without rewriting.
"""
lines = ["#EXTM3U"]
for entry in entries:
seconds = round((entry["duration"] or 0) / 1000)
lines.append(f"#EXTINF:{seconds},{entry['artist']} - {entry['title']}")
lines.append(os.path.relpath(entry["mirror"], path.parent))
fresh = not path.parent.exists()
path.parent.mkdir(parents=True, exist_ok=True)
if fresh and reference is not None:
match_ownership(path.parent, reference)
handle, temporary = tempfile.mkstemp(dir=path.parent, suffix=".m3u.part")
os.close(handle)
temporary = Path(temporary)
try:
temporary.write_text("\n".join(lines) + "\n", encoding="utf-8")
# The mirror is read back by something else; see music-mirror, which had
# to learn this the hard way.
mode = temporary.stat().st_mode
if not mode & GROUP_READ:
temporary.chmod(mode | GROUP_READ)
# Before the rename, so the playlist is never briefly visible owned by
# the wrong account.
if reference is not None:
match_ownership(temporary, reference)
os.replace(temporary, path)
finally:
temporary.unlink(missing_ok=True)
def build_playlists(store, mirror_root, library_root, limit, now):
"""Write every playlist into the mirror. Returns how many tracks were listed.
A track is only listed once its mirror file has been confirmed to exist. The
index knows what Lidarr holds, which is the lossless source; whether the MP3
beside it has been encoded yet is music-mirror's business and is checked
rather than assumed.
"""
directory = Path(mirror_root) / PLAYLIST_DIRECTORY
parameters = {
"limit": limit,
"year_ago": now - 365 * 86400,
"week": now // ROTATION_PERIOD_SECONDS,
}
total = 0
missing = 0
for name, description, sql in PLAYLISTS:
entries = []
for row in store.connection.execute(sql, parameters):
mirror = mirror_path_for(row["path"], library_root, mirror_root)
if mirror is None or not mirror.is_file():
missing += 1
continue
entries.append({**dict(row), "mirror": mirror})
write_playlist(directory / f"{name}.m3u", entries, Path(mirror_root))
total += len(entries)
logger.info("playlist %-20s %4d tracks -- %s", name, len(entries), description)
if missing:
logger.warning(
"%d selected tracks had no file in the mirror and were left out."
" A handful means music-mirror has not encoded them yet; a large"
" number means --library-root or --mirror is pointing at the wrong"
" place, since the paths are then being mapped to nothing.",
missing,
)
return total
def coverage_report(store): def coverage_report(store):
"""Log how much of the listening history could be tied to the library. """Log how much of the listening history could be tied to the library.
@@ -1125,47 +1765,68 @@ def coverage_report(store):
) )
# Owning an artist is a weak proxy for owning a track, so that figure alone # Owning an artist is a weak proxy for owning a track, so that figure alone
# overstates the matcher's failings. Split it. A title the library holds # overstates the matcher's failings. Split it -- but not on the title alone.
# under some other artist is an attribution disagreement -- a remixer # Across fifty thousand tracks, titles collide constantly: "Everyday" is
# credited as the artist, a guest billed as one -- and is a real miss. A # Rusko and also Def Leppard, "Kaleidoscope" is Delta Heavy and also
# title the library does not hold at all was simply never bought, and no # Chappell Roan. Matching those would be worse than missing them.
# amount of matching will conjure it. #
# The signal for a genuine attribution miss is that the library's own title
# credits the artist the scrobble is filed under -- "Voodoo People (Pendulum
# Remix)" against a play credited to Pendulum. The normalised title has that
# suffix stripped, which is exactly what let them meet, so the raw one has to
# be searched for the name.
attribution = store.connection.execute( attribution = store.connection.execute(
"SELECT COUNT(*) AS pairs, COALESCE(SUM(plays), 0) AS plays FROM scrobble_key k"
" WHERE k.track_id IS NULL"
" AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)"
" AND EXISTS (SELECT 1 FROM lidarr_track t"
" WHERE t.norm_title = k.norm_track"
" AND instr(lower(t.title), lower(k.artist)) > 0)"
).fetchone()
collision = store.connection.execute(
"SELECT COUNT(*) AS pairs, COALESCE(SUM(plays), 0) AS plays FROM scrobble_key k" "SELECT COUNT(*) AS pairs, COALESCE(SUM(plays), 0) AS plays FROM scrobble_key k"
" WHERE k.track_id IS NULL" " WHERE k.track_id IS NULL"
" AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)" " AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)"
" AND EXISTS (SELECT 1 FROM lidarr_track t WHERE t.norm_title = k.norm_track)" " AND EXISTS (SELECT 1 FROM lidarr_track t WHERE t.norm_title = k.norm_track)"
).fetchone() ).fetchone()
logger.info( logger.info(
" of those, the title exists under another artist: %d pairs, %d plays" " the library's title credits the scrobbled artist: %d pairs, %d plays"
" -- attribution disagreements, and the genuine misses", " -- remixes and guest spots, and the genuine misses",
attribution["pairs"], attribution["pairs"],
attribution["plays"], attribution["plays"],
) )
logger.info(
" same title under an unrelated artist: %d pairs, %d plays"
" -- title collisions, not misses; matching these would be a mistake",
collision["pairs"] - attribution["pairs"],
collision["plays"] - attribution["plays"],
)
logger.info( logger.info(
" the rest, %d pairs, %d plays: you own the artist but not the track", " the rest, %d pairs, %d plays: you own the artist but not the track",
suspect["pairs"] - attribution["pairs"], suspect["pairs"] - collision["pairs"],
suspect["plays"] - attribution["plays"], suspect["plays"] - collision["plays"],
) )
mismatched = store.connection.execute( mismatched = store.connection.execute(
"SELECT k.artist, k.track, k.plays," "SELECT k.artist, k.track, k.plays,"
" (SELECT a.name FROM lidarr_track t" " (SELECT t.title FROM lidarr_track t"
" JOIN lidarr_artist a ON a.id = t.artist_id" " WHERE t.norm_title = k.norm_track"
" WHERE t.norm_title = k.norm_track LIMIT 1) AS filed_under" " AND instr(lower(t.title), lower(k.artist)) > 0 LIMIT 1) AS library_title"
" FROM scrobble_key k" " FROM scrobble_key k"
" WHERE k.track_id IS NULL" " WHERE k.track_id IS NULL"
" AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)" " AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)"
" AND EXISTS (SELECT 1 FROM lidarr_track t WHERE t.norm_title = k.norm_track)" " AND EXISTS (SELECT 1 FROM lidarr_track t"
" WHERE t.norm_title = k.norm_track"
" AND instr(lower(t.title), lower(k.artist)) > 0)"
" ORDER BY k.plays DESC, k.artist LIMIT 10" " ORDER BY k.plays DESC, k.artist LIMIT 10"
).fetchall() ).fetchall()
for position, row in enumerate(mismatched, start=1): for position, row in enumerate(mismatched, start=1):
logger.info( logger.info(
" attribution %2d: %-45s %4d plays, filed under %s", " attribution %2d: %-45s %4d plays, library has %r",
position, position,
f"{row['artist']} - {row['track']}"[:45], f"{row['artist']} - {row['track']}"[:45],
row["plays"], row["plays"],
row["filed_under"], row["library_title"],
) )
with_files = store.scalar("SELECT COUNT(*) FROM lidarr_track WHERE has_file = 1") with_files = store.scalar("SELECT COUNT(*) FROM lidarr_track WHERE has_file = 1")
@@ -1236,7 +1897,10 @@ def report(store, now):
logger.info(" top track 90d %2d: %-50s %d", position, label[:50], row["plays"]) logger.info(" top track 90d %2d: %-50s %d", position, label[:50], row["plays"])
def run_once(client, store, user, now, backfill_limit, lidarr=None): def run_once(
client, store, user, now, backfill_limit, lidarr=None, mirror=None,
library_root=None, playlist_limit=100, vibes=DEFAULT_VIBES, tag_limit=0,
):
"""Run a single pass. Returns the number of scrobbles added.""" """Run a single pass. Returns the number of scrobbles added."""
started = time.monotonic() started = time.monotonic()
added = catch_up(client, store, user, now) added = catch_up(client, store, user, now)
@@ -1249,6 +1913,17 @@ def run_once(client, store, user, now, backfill_limit, lidarr=None):
# arrived, and they need a verdict too. # arrived, and they need a verdict too.
if store.scalar("SELECT COUNT(*) FROM lidarr_track"): if store.scalar("SELECT COUNT(*) FROM lidarr_track"):
match_library(store) match_library(store)
if mirror is not None:
root = library_root or library_root_of(store)
if root is None:
logger.warning(
"cannot work out where Lidarr's music lives, so no playlists were"
" written; set --library-root"
)
else:
build_playlists(store, mirror, root, playlist_limit, now)
sync_tags(client, store, now, limit=tag_limit)
build_vibe_playlists(store, vibes, mirror, root, playlist_limit, now)
logger.info( logger.info(
"pass complete in %.1fs: %d scrobbles added, %d loved", "pass complete in %.1fs: %d scrobbles added, %d loved",
@@ -1321,6 +1996,37 @@ def build_parser():
default=os.getenv("MUSIC_CURATOR_LIDARR_API_KEY"), default=os.getenv("MUSIC_CURATOR_LIDARR_API_KEY"),
help="Lidarr API key (env MUSIC_CURATOR_LIDARR_API_KEY)", help="Lidarr API key (env MUSIC_CURATOR_LIDARR_API_KEY)",
) )
parser.add_argument(
"--mirror",
default=os.getenv("MUSIC_CURATOR_MIRROR"),
help="root of the MP3 mirror; playlists are written into it"
" (env MUSIC_CURATOR_MIRROR)",
)
parser.add_argument(
"--library-root",
default=os.getenv("MUSIC_CURATOR_LIBRARY_ROOT"),
help="prefix to strip from Lidarr's track paths when mapping them into the"
" mirror; derived from the indexed artist folders when unset"
" (env MUSIC_CURATOR_LIBRARY_ROOT)",
)
parser.add_argument(
"--playlist-limit",
type=int,
default=int(os.getenv("MUSIC_CURATOR_PLAYLIST_LIMIT", "100")),
help="most tracks to put in any one playlist (env MUSIC_CURATOR_PLAYLIST_LIMIT)",
)
parser.add_argument(
"--vibes",
default=os.getenv("MUSIC_CURATOR_VIBES"),
help="JSON file of mood definitions, replacing the built-in set"
" (env MUSIC_CURATOR_VIBES)",
)
parser.add_argument(
"--tag-limit",
type=int,
default=int(os.getenv("MUSIC_CURATOR_TAG_LIMIT", "0")),
help="cap artist tag lookups per pass; 0 for no cap (env MUSIC_CURATOR_TAG_LIMIT)",
)
parser.add_argument( parser.add_argument(
"--skip-index", "--skip-index",
action="store_true", action="store_true",
@@ -1345,6 +2051,7 @@ def main(argv=None, clock=time.time):
try: try:
interval = parse_interval(args.interval) if args.interval else None interval = parse_interval(args.interval) if args.interval else None
vibes = load_vibes(args.vibes)
except ValueError as error: except ValueError as error:
logger.error("%s", error) logger.error("%s", error)
return 2 return 2
@@ -1385,7 +2092,19 @@ def main(argv=None, clock=time.time):
while True: while True:
now = int(clock()) now = int(clock())
try: try:
run_once(client, store, args.user, now, args.backfill_limit, lidarr) run_once(
client,
store,
args.user,
now,
args.backfill_limit,
lidarr,
args.mirror,
args.library_root,
args.playlist_limit,
vibes,
args.tag_limit,
)
except (LastfmError, LidarrError) as error: except (LastfmError, LidarrError) as error:
logger.error("%s", error) logger.error("%s", error)
if interval is None: if interval is None:
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "music-curator" name = "music-curator"
version = "0.3.0" version = "0.6.0"
description = "Ingest a Last.fm listening history and curate a music library from it" description = "Ingest a Last.fm listening history and curate a music library from it"
readme = "README.md" readme = "README.md"
requires-python = ">=3.11" requires-python = ">=3.11"
+12 -1
View File
@@ -56,7 +56,9 @@ class FakeLastfm:
is prepended to the first page with no `date`. is prepended to the first page with no `date`.
""" """
def __init__(self, tracks=(), loved=(), nowplaying=None, outcomes=()): def __init__(self, tracks=(), loved=(), nowplaying=None, outcomes=(), tags=None):
# Keyed by mbid or by artist name, whichever the caller asked with.
self.tags = tags or {}
self.tracks = sorted(tracks, key=lambda track: int(track["date"]["uts"]), reverse=True) self.tracks = sorted(tracks, key=lambda track: int(track["date"]["uts"]), reverse=True)
self.loved = list(loved) self.loved = list(loved)
self.nowplaying = nowplaying self.nowplaying = nowplaying
@@ -82,6 +84,15 @@ class FakeLastfm:
return json.dumps(self._recent(query)) return json.dumps(self._recent(query))
if method == "user.getlovedtracks": if method == "user.getlovedtracks":
return json.dumps(self._loved(query)) return json.dumps(self._loved(query))
if method == "artist.gettoptags":
key = query.get("mbid") or query.get("artist", "")
if key not in self.tags:
# What the real service says for a key it cannot resolve, which
# for mbids is a great many artists whose pages plainly exist.
return json.dumps(
{"error": 6, "message": "The artist you supplied could not be found"}
)
return json.dumps({"toptags": {"tag": self.tags[key]}})
raise AssertionError(f"unexpected method {method}") raise AssertionError(f"unexpected method {method}")
def _recent(self, query): def _recent(self, query):
+585 -6
View File
@@ -1,5 +1,8 @@
import json import json
import os
import stat
import urllib.error import urllib.error
from pathlib import Path
import pytest import pytest
from conftest import FakeLastfm, FakeLidarr, make_loved, make_tracks from conftest import FakeLastfm, FakeLidarr, make_loved, make_tracks
@@ -55,9 +58,23 @@ LIBRARY = [
{ {
"name": "The Prodigy", "name": "The Prodigy",
"albums": [ "albums": [
{"title": "The Fat of the Land", "tracks": [{"title": "Breathe (Remastered)"}]} {
"title": "The Fat of the Land",
"tracks": [
{"title": "Breathe (Remastered)"},
# Credits its remixer in the title, which is the only signal
# separating a real attribution miss from a title collision.
{"title": "Voodoo People (Pendulum Remix)"},
],
}
], ],
}, },
# Held by the library in its own right, which is what puts its scrobbles
# inside the "artist the library holds" filter at all.
{
"name": "Pendulum",
"albums": [{"title": "Immersion", "tracks": [{"title": "Watercolour"}]}],
},
] ]
@@ -393,7 +410,7 @@ def test_keys_carry_the_play_count_and_the_span(tmp_path):
def test_re_indexing_drops_what_lidarr_no_longer_has(tmp_path): def test_re_indexing_drops_what_lidarr_no_longer_has(tmp_path):
"""The index is Lidarr's mirror, not an accumulation of everything ever seen.""" """The index is Lidarr's mirror, not an accumulation of everything ever seen."""
store = indexed(tmp_path, [scrobble_of("AC/DC", "Hells Bells")]) store = indexed(tmp_path, [scrobble_of("AC/DC", "Hells Bells")])
assert store.scalar("SELECT COUNT(*) FROM lidarr_artist") == 3 assert store.scalar("SELECT COUNT(*) FROM lidarr_artist") == 4
music_curator.index_library( music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=FakeLidarr(LIBRARY[:1])), store music_curator.Lidarr("http://lidarr", "key", transport=FakeLidarr(LIBRARY[:1])), store
@@ -512,7 +529,7 @@ def test_albums_come_from_the_unfiltered_endpoint(tmp_path):
album_calls = [query for path, query in api.calls if path == "album"] album_calls = [query for path, query in api.calls if path == "album"]
assert album_calls == [{}] assert album_calls == [{}]
assert store.scalar("SELECT COUNT(*) FROM lidarr_album") == 3 assert store.scalar("SELECT COUNT(*) FROM lidarr_album") == 4
def test_a_bad_album_falls_back_to_asking_per_artist(tmp_path): def test_a_bad_album_falls_back_to_asking_per_artist(tmp_path):
@@ -526,7 +543,7 @@ def test_a_bad_album_falls_back_to_asking_per_artist(tmp_path):
assert store.scalar("SELECT COUNT(*) FROM lidarr_album WHERE artist_id = 2") == 0 assert store.scalar("SELECT COUNT(*) FROM lidarr_album WHERE artist_id = 2") == 0
# The other two artists keep their albums. # The other two artists keep their albums.
assert store.scalar("SELECT COUNT(*) FROM lidarr_album") == 2 assert store.scalar("SELECT COUNT(*) FROM lidarr_album") == 3
assert store.get_state("index_albums_skipped") == "1" assert store.get_state("index_albums_skipped") == "1"
@@ -549,10 +566,10 @@ def test_an_artist_lidarr_cannot_serve_does_not_kill_the_index(tmp_path):
music_curator.index_library(music_curator.Lidarr("http://lidarr", "key", transport=api), store) music_curator.index_library(music_curator.Lidarr("http://lidarr", "key", transport=api), store)
assert store.scalar("SELECT COUNT(*) FROM lidarr_artist") == 3 assert store.scalar("SELECT COUNT(*) FROM lidarr_artist") == 4
assert store.scalar("SELECT COUNT(*) FROM lidarr_track WHERE artist_id = 2") == 0 assert store.scalar("SELECT COUNT(*) FROM lidarr_track WHERE artist_id = 2") == 0
# The other two artists are indexed in full. # The other two artists are indexed in full.
assert store.scalar("SELECT COUNT(*) FROM lidarr_track") == 3 assert store.scalar("SELECT COUNT(*) FROM lidarr_track") == 5
assert store.get_state("index_skipped") == "1" assert store.get_state("index_skipped") == "1"
@@ -763,3 +780,565 @@ def test_the_report_survives_the_attribution_split(tmp_path):
store = indexed(tmp_path, [scrobble_of("Yellowcard", "Hells Bells")]) store = indexed(tmp_path, [scrobble_of("Yellowcard", "Hells Bells")])
music_curator.report(store, NOW) music_curator.report(store, NOW)
def attribution_pairs(store):
"""Unmatched pairs where the library's own title credits the scrobbled artist."""
return [
row["artist"]
for row in store.connection.execute(
"SELECT k.artist FROM scrobble_key k WHERE k.track_id IS NULL"
" AND EXISTS (SELECT 1 FROM lidarr_artist a WHERE a.norm_name = k.norm_artist)"
" AND EXISTS (SELECT 1 FROM lidarr_track t"
" WHERE t.norm_title = k.norm_track"
" AND instr(lower(t.title), lower(k.artist)) > 0)"
)
]
def test_a_shared_title_is_not_an_attribution_miss(tmp_path):
"""Across fifty thousand tracks, titles collide constantly: "Everyday" is
Rusko and also Def Leppard. Matching those would be worse than missing."""
store = indexed(tmp_path, [scrobble_of("Yellowcard", "Hells Bells")])
# The library holds "Hells Bells", by AC/DC, and its title says nothing
# about Yellowcard. A collision, not a miss.
assert verdict(store, "Yellowcard", "Hells Bells") == ("none", None)
assert attribution_pairs(store) == []
def test_a_remix_credited_in_the_library_title_is_an_attribution_miss(tmp_path):
"""The library has "Voodoo People (Pendulum Remix)" under The Prodigy; the
scrobble credits Pendulum. Same song, different filing."""
store = indexed(tmp_path, [scrobble_of("Pendulum", "Voodoo People")])
assert attribution_pairs(store) == ["Pendulum"]
def test_the_title_index_is_used_for_the_report_lookup(tmp_path):
"""Without it the report scans every track for every unmatched key: forty-
seven seconds on a real library."""
store = indexed(tmp_path, [])
plan = "\n".join(
row[-1]
for row in store.connection.execute(
"EXPLAIN QUERY PLAN SELECT 1 FROM scrobble_key k WHERE k.track_id IS NULL"
" AND EXISTS (SELECT 1 FROM lidarr_track t WHERE t.norm_title = k.norm_track)"
)
)
assert "lidarr_track_title" in plan, plan
def playlist_library(tmp_path):
"""A library on disk, with mirror MP3s beside the Lidarr source paths."""
source = tmp_path / "music"
mirror = tmp_path / "mirror"
library = [
{
"name": "Played Band",
"albums": [
{
"title": "Known",
"tracks": [{"title": "Hit"}, {"title": "Album Track"}],
}
],
},
{
"name": "Silent Band",
"albums": [{"title": "Unknown", "tracks": [{"title": "Never Heard"}]}],
},
]
api = FakeLidarr(library)
# FakeLidarr invents /music/<artist>/<album>/<title>.flac; put the mirror
# MP3s at the paths music-mirror would have produced from those.
for handle in api.files:
relative = Path(handle["path"]).relative_to("/music")
handle["path"] = str(source / relative)
mp3 = (mirror / relative).with_suffix(".mp3")
mp3.parent.mkdir(parents=True, exist_ok=True)
mp3.write_bytes(b"not really an mp3")
for artist in api.artists:
artist["path"] = str(source / Path(artist["path"]).name)
return api, source, mirror
def test_playlists_are_written_into_the_mirror(tmp_path):
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
ingest(store, FakeLastfm([scrobble_of("Played Band", "Hit")] * 1))
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
music_curator.match_library(store)
music_curator.build_playlists(store, mirror, str(source), 100, NOW)
written = sorted(p.name for p in (mirror / "_playlists").glob("*.m3u"))
assert written == [
"all-time.m3u",
"deep-cuts.m3u",
"heavy-rotation.m3u",
"neglected.m3u",
"unheard-favourites.m3u",
"unheard.m3u",
]
played = (mirror / "_playlists" / "all-time.m3u").read_text().splitlines()
assert played[0] == "#EXTM3U"
assert played[1].startswith("#EXTINF:")
assert "Played Band - Hit" in played[1]
# Relative to the playlist file, so the same file works from any mount.
assert played[2] == "../Played Band/Known/Hit.mp3"
assert (mirror / "_playlists" / "all-time.m3u").parent.joinpath(played[2]).resolve().is_file()
def test_an_unplayed_track_lands_in_the_unheard_playlist(tmp_path):
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
ingest(store, FakeLastfm([scrobble_of("Played Band", "Hit")]))
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
music_curator.match_library(store)
music_curator.build_playlists(store, mirror, str(source), 100, NOW)
unheard = (mirror / "_playlists" / "unheard.m3u").read_text()
assert "Never Heard" in unheard
assert "Album Track" in unheard
# The one thing that was played must not be in it.
assert "Played Band - Hit\n" not in unheard
def test_a_track_with_no_mirror_file_is_left_out(tmp_path):
"""The index knows what Lidarr holds; whether music-mirror has encoded the
MP3 yet is a different question, and is checked rather than assumed."""
api, source, mirror = playlist_library(tmp_path)
for mp3 in mirror.rglob("*.mp3"):
mp3.unlink()
store = store_at(tmp_path)
ingest(store, FakeLastfm([scrobble_of("Played Band", "Hit")]))
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
music_curator.match_library(store)
total = music_curator.build_playlists(store, mirror, str(source), 100, NOW)
assert total == 0
assert (mirror / "_playlists" / "all-time.m3u").read_text() == "#EXTM3U\n"
def test_the_library_root_is_derived_from_the_artist_folders(tmp_path):
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
assert music_curator.library_root_of(store) == str(source)
def test_the_playlist_limit_is_honoured(tmp_path):
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
music_curator.match_library(store)
music_curator.build_playlists(store, mirror, str(source), 1, NOW)
unheard = (mirror / "_playlists" / "unheard.m3u").read_text().splitlines()
assert len([line for line in unheard if line.startswith("#EXTINF")]) == 1
def test_the_rotation_moves_weekly_not_every_pass(tmp_path):
"""A playlist that reorders on every pass is one that has to be re-imported
on every pass; the Music app imports a snapshot, it does not track a file."""
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
music_curator.match_library(store)
def unheard_at(when):
music_curator.build_playlists(store, mirror, str(source), 100, when)
return (mirror / "_playlists" / "unheard.m3u").read_text()
same_week = unheard_at(NOW), unheard_at(NOW + 3600)
assert same_week[0] == same_week[1]
def test_playlists_are_group_readable(tmp_path):
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
music_curator.match_library(store)
music_curator.build_playlists(store, mirror, str(source), 100, NOW)
for playlist in (mirror / "_playlists").glob("*.m3u"):
assert playlist.stat().st_mode & stat.S_IRGRP, playlist
def test_tag_weights_fall_back_to_rank_when_no_count_is_sent():
"""The documented sample carries only a name and a URL; a live response also
carries a count. Neither may be relied on alone."""
with_count = music_curator.parse_tags(
{"toptags": {"tag": [{"name": "Screamo", "count": 100}, {"name": "emo", "count": 40}]}}
)
assert with_count == [("screamo", 100), ("emo", 40)]
without = music_curator.parse_tags(
{"toptags": {"tag": [{"name": "screamo"}, {"name": "emo"}]}}
)
assert [tag for tag, _ in without] == ["screamo", "emo"]
assert without[0][1] > without[1][1]
def test_a_lone_tag_is_not_a_list():
assert music_curator.parse_tags({"toptags": {"tag": {"name": "dnb", "count": 90}}}) == [
("dnb", 90)
]
def test_an_artist_lastfm_cannot_answer_for_is_not_asked_again(tmp_path):
"""Recording the fetch even when it returns nothing is what stops a pass
spending a request per unknown artist, forever."""
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
lastfm = FakeLastfm(tags={})
assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 2
before = len(lastfm.calls)
assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 0
assert len(lastfm.calls) == before
def test_tags_are_refetched_once_they_go_stale(tmp_path):
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
lastfm = FakeLastfm(tags={})
music_curator.sync_tags(client_for(lastfm), store, NOW)
later = NOW + music_curator.TAG_REFRESH_SECONDS + 1
assert music_curator.sync_tags(client_for(lastfm), store, later) == 2
def tagged_store(tmp_path, tags):
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
music_curator.sync_tags(client_for(FakeLastfm(tags=tags)), store, NOW)
return store, source, mirror
def test_a_vibe_selects_by_tag(tmp_path):
store, source, mirror = tagged_store(
tmp_path,
{
"Played Band": [{"name": "screamo", "count": 100}],
"Silent Band": [{"name": "classic rock", "count": 100}],
},
)
vibes = [{"name": "screamo", "tags": ["screamo"]}]
music_curator.build_vibe_playlists(store, vibes, mirror, str(source), 100, NOW)
written = (mirror / "_playlists" / "screamo.m3u").read_text()
assert "Played Band" in written
assert "Silent Band" not in written
def test_a_weakly_tagged_artist_is_below_the_threshold(tmp_path):
"""A single low-weight tag is not a genre, it is somebody's stray opinion."""
store, source, mirror = tagged_store(
tmp_path, {"Played Band": [{"name": "screamo", "count": 3}]}
)
vibes = [{"name": "screamo", "tags": ["screamo"]}]
music_curator.build_vibe_playlists(store, vibes, mirror, str(source), 100, NOW)
assert (mirror / "_playlists" / "screamo.m3u").read_text() == "#EXTM3U\n"
def test_a_vibe_can_be_restricted_by_release_year(tmp_path):
"""What separates eighties synth records from everything else a synthpop
tag drags in."""
store, source, mirror = tagged_store(
tmp_path, {"Played Band": [{"name": "synthpop", "count": 100}]}
)
inside = [{"name": "eighties", "tags": ["synthpop"], "years": [1975, 1992]}]
outside = [{"name": "nineties", "tags": ["synthpop"], "years": [1993, 1999]}]
# FakeLidarr dates every album 2019, so neither window should catch it.
music_curator.build_vibe_playlists(store, inside, mirror, str(source), 100, NOW)
music_curator.build_vibe_playlists(store, outside, mirror, str(source), 100, NOW)
assert (mirror / "_playlists" / "eighties.m3u").read_text() == "#EXTM3U\n"
assert (mirror / "_playlists" / "nineties.m3u").read_text() == "#EXTM3U\n"
modern = [{"name": "modern", "tags": ["synthpop"], "years": [2000, 2030]}]
music_curator.build_vibe_playlists(store, modern, mirror, str(source), 100, NOW)
assert "Played Band" in (mirror / "_playlists" / "modern.m3u").read_text()
def test_the_built_in_vibes_are_all_usable_filenames():
for vibe in music_curator.DEFAULT_VIBES:
assert music_curator.SAFE_VIBE_NAME.fullmatch(vibe["name"]), vibe["name"]
assert vibe["tags"]
def test_a_vibes_file_replaces_the_built_in_set(tmp_path):
path = tmp_path / "vibes.json"
path.write_text(json.dumps([{"name": "mine", "tags": ["shoegaze"]}]))
assert music_curator.load_vibes(str(path)) == ({"name": "mine", "tags": ["shoegaze"]},)
assert music_curator.load_vibes(None) is music_curator.DEFAULT_VIBES
@pytest.mark.parametrize(
"content",
[
'{"not": "a list"}',
'[{"tags": ["x"]}]',
'[{"name": "../escape", "tags": ["x"]}]',
'[{"name": "ok"}]',
"not json at all",
],
)
def test_a_bad_vibes_file_is_refused_up_front(tmp_path, content):
"""A bad name would otherwise surface as a file written somewhere
unintended, which is a poor way to learn about a typo."""
path = tmp_path / "vibes.json"
path.write_text(content)
with pytest.raises(ValueError):
music_curator.load_vibes(str(path))
def test_tags_are_looked_up_by_name_not_by_mbid(tmp_path):
"""Last.fm's mbid index is stale: it cannot find Devo or Escape the Fate by
one, though their pages plainly exist. Its name index can."""
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
lastfm = FakeLastfm(tags={"Played Band": [{"name": "screamo", "count": 90}]})
music_curator.sync_tags(client_for(lastfm), store, NOW)
asked = [call for call in lastfm.calls if call.get("method") == "artist.getTopTags"]
# The name is always tried first; the mbid only appears as a fallback for
# the artist that the name could not resolve.
assert "artist" in asked[0]
assert [call for call in asked if call.get("artist") == "Played Band"]
assert not [call for call in asked if call.get("mbid") == "artist-mbid-1"]
assert store.scalar("SELECT COUNT(*) FROM artist_tag WHERE tag = 'screamo'") == 1
def test_the_mbid_is_tried_when_the_name_is_not_found(tmp_path):
"""Kept only for a name Lidarr spells differently to Last.fm."""
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
lastfm = FakeLastfm(tags={"artist-mbid-1": [{"name": "dnb", "count": 80}]})
music_curator.sync_tags(client_for(lastfm), store, NOW)
assert store.scalar("SELECT COUNT(*) FROM artist_tag WHERE tag = 'dnb'") == 1
asked = [call for call in lastfm.calls if call.get("method") == "artist.getTopTags"]
assert any("mbid" in call for call in asked)
def test_an_artist_neither_key_resolves_is_recorded_and_not_retried(tmp_path):
"""Otherwise every pass spends a request on it again, for ever."""
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
lastfm = FakeLastfm(tags={})
assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 2
spent = len(lastfm.calls)
assert music_curator.sync_tags(client_for(lastfm), store, NOW) == 0
assert len(lastfm.calls) == spent
def test_a_real_failure_is_not_recorded_so_the_next_pass_retries(tmp_path):
"""A rate limit is not the same as an artist not existing."""
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
lastfm = FakeLastfm(
tags={"Played Band": [], "Silent Band": []},
outcomes=[{"error": 10, "message": "Invalid API key"}],
)
music_curator.sync_tags(client_for(lastfm), store, NOW)
# One artist failed hard and must still be pending.
assert store.scalar("SELECT COUNT(*) FROM artist_tag_fetched") == 1
# Tags that carry real weight in a real library, against tags that describe a
# passport, span most of the collection, or are somebody's artist name.
USELESS_TAGS = {
"american", "british", "australian", "canadian", "swedish", "dutch", "german",
"scottish", "english", "uk", "usa", "canada",
"rock", "electronic", "pop", "alternative", "metal ", "all", "heavy",
"female vocalists", "male vocalists", "female vocalist",
"my top songs", "cover", "covers", "not emo",
"green day", "paramore", "queen", "bon jovi", "shinedown", "aerosmith",
"journey", "fleetwood mac",
}
def test_no_mood_selects_on_a_useless_tag():
"""Nationality is not a sound; `rock` and `electronic` span most of the
library; and Last.fm's top tag for an artist is often their own name."""
for vibe in music_curator.DEFAULT_VIBES:
overlap = {tag.casefold() for tag in vibe["tags"]} & USELESS_TAGS
assert not overlap, f"{vibe['name']} selects on {overlap}"
def test_mood_names_are_unique():
names = [vibe["name"] for vibe in music_curator.DEFAULT_VIBES]
assert len(names) == len(set(names))
ROCK_TAGS = {
"classic rock", "hard rock", "blues rock", "southern rock", "arena rock",
"glam rock", "hair metal", "glam metal", "heavy metal", "metal", "art rock",
"psychedelic rock", "progressive rock", "rock and roll", "rock n roll",
}
def test_a_decade_tag_in_a_non_rock_mood_must_exclude_the_rock():
"""`80s` sits on Def Leppard and Bon Jovi as heavily as on Eurythmics, so a
mood that reaches for a decade without wanting rock has to say so. A mood
that does want it -- classic-rock reaching for 70s -- is exempt."""
for vibe in music_curator.DEFAULT_VIBES:
tags = {tag.casefold() for tag in vibe["tags"]}
decades = {"60s", "70s", "80s", "90s"} & tags
if not decades or tags & ROCK_TAGS:
continue
excluded = {tag.casefold() for tag in vibe.get("exclude", [])}
missing = {"hard rock", "hair metal"} - excluded
assert not missing, f"{vibe['name']} selects on {decades} without excluding {missing}"
def test_the_eighties_mood_covers_the_canon():
"""Depeche Mode, Duran Duran, Eurythmics and Frankie Goes to Hollywood --
checked against their live Last.fm tags. Three of the four carry "synth pop"
with a space; only one carries it without."""
synths = next(v for v in music_curator.DEFAULT_VIBES if v["name"] == "80s-synths")
tags = {t.casefold() for t in synths["tags"]}
for artist_tags in (
{"80s", "new wave", "pop", "female vocalists", "synth pop"}, # Eurythmics
{"80s", "new wave", "pop", "british", "dance"}, # Frankie
{"electronic", "synthpop", "new wave", "80s", "synth pop"}, # Depeche Mode
{"new wave", "80s", "pop", "synth pop", "rock"}, # Duran Duran
):
assert tags & artist_tags, artist_tags
assert synths["years"] == [1975, 1992]
def test_an_excluded_tag_drops_the_artist(tmp_path):
"""Weighting cannot separate the eighties synth acts from the eighties
stadium rock, because the tag they would be weighted on is the one they
share."""
store, source, mirror = tagged_store(
tmp_path,
{
"Played Band": [{"name": "80s", "count": 100}, {"name": "hard rock", "count": 90}],
"Silent Band": [{"name": "80s", "count": 100}, {"name": "synth pop", "count": 90}],
},
)
vibes = [{"name": "eighties", "tags": ["80s", "synth pop"], "exclude": ["hard rock"]}]
music_curator.build_vibe_playlists(store, vibes, mirror, str(source), 100, NOW)
written = (mirror / "_playlists" / "eighties.m3u").read_text()
assert "Silent Band" in written
assert "Played Band" not in written
def test_a_vibe_cannot_both_select_and_exclude_a_tag(tmp_path):
path = tmp_path / "vibes.json"
path.write_text(json.dumps([{"name": "x", "tags": ["80s"], "exclude": ["80s"]}]))
with pytest.raises(ValueError, match="selects on and excludes"):
music_curator.load_vibes(str(path))
def test_matching_ownership_is_a_no_op_when_it_already_agrees(tmp_path):
target = tmp_path / "file"
target.write_text("x")
assert music_curator.match_ownership(target, tmp_path) is False
def test_ownership_failure_is_tolerated(tmp_path):
"""Not permitted unless running as root -- which is exactly the case where
the ownership is already whatever the caller runs as."""
target = tmp_path / "file"
target.write_text("x")
# uid 0 from a non-root test process: refused, and must not raise.
assert music_curator.set_ownership(target, 0, 0) is False
def test_a_playlist_is_chowned_to_match_the_mirror(tmp_path, monkeypatch):
"""The image runs as root, so its output is root-owned, and a root-owned
playlist in an apps-owned mirror is unreadable to whatever serves it."""
api, source, mirror = playlist_library(tmp_path)
store = store_at(tmp_path)
music_curator.index_library(
music_curator.Lidarr("http://lidarr", "key", transport=api), store
)
music_curator.match_library(store)
attempted = []
real_stat = music_curator.Path.stat
def pretend_mirror_is_owned_by_568(self, *args, **kwargs):
info = real_stat(self, *args, **kwargs)
if self == mirror:
return os.stat_result(
(info.st_mode, info.st_ino, info.st_dev, info.st_nlink, 568, 568,
info.st_size, int(info.st_atime), int(info.st_mtime), int(info.st_ctime))
)
return info
monkeypatch.setattr(music_curator.Path, "stat", pretend_mirror_is_owned_by_568)
monkeypatch.setattr(
music_curator.os, "chown", lambda p, u, g: attempted.append((str(p), u, g))
)
music_curator.build_playlists(store, mirror, str(source), 100, NOW)
assert attempted, "no ownership was applied"
assert all(tuple(owner) == (568, 568) for _, *owner in attempted)
# The temporary file, before the rename, never the finished playlist.
assert all(path.endswith(".part") or path.endswith("_playlists") for path, *_ in attempted)