fix: match bracketed guest credits and hyphenated version suffixes
Build and publish container / build (pull_request) Successful in 10m45s

Two normalisation faults, both found by running a real coverage report's
unmatched list back through the normaliser. Between them they account for five
of the fifteen worst misses by play count.

The guest-credit pattern required whitespace immediately before the word, so it
caught "Yellowcard feat. Tay Jardine" but missed "Self vs Self (feat. In
Flames)" -- and the bracketed form is the more common of the two. An opening
bracket is now allowed in that position.

The trailing-version pattern matched the suffix as a run of non-hyphens, which
cannot cross a hyphen inside the suffix itself: "Gold Dust - Shy FX Re-Edit" and
"Back To Your Roots - Friction & K-Tee Remix" both survived untouched. Matched
lazily instead.

Four version words are added for how drum and bass marks its variants: vip,
bootleg, rework, extended. They only apply inside a bracket or after a trailing
dash, so the exposure is small, and ordinary titles carrying those words --
Editors, Mixed Emotions, Radio Ga Ga, Live and Let Die -- are pinned as tests
against exactly that.

The report also gains the figures that explain why the MBID tier contributes so
little. Two thirds of scrobbles carry a recording id, and only a twentieth of
them join on one: MusicBrainz holds a separate recording per release, and the
two sides rarely choose the same one. Counting the pairs that carried an id and
matched on name anyway measures that disagreement directly, and settles that the
weakness is not a bug in the join.
This commit is contained in:
Emma Thorpe
2026-08-24 16:56:13 +01:00
parent edebecc8ea
commit 7588fee302
3 changed files with 94 additions and 7 deletions
+41 -6
View File
@@ -187,18 +187,28 @@ VERSION_WORDS = (
"anniversary",
"reissue",
"instrumental",
# Drum and bass and its neighbours mark versions their own way.
"vip",
"bootleg",
"rework",
"extended",
)
_VERSIONS = "|".join(VERSION_WORDS)
BRACKETED_VERSION = re.compile(
rf"\s*[\(\[][^\)\]]*\b(?:{_VERSIONS})\b[^\)\]]*[\)\]]\s*$", re.IGNORECASE
)
TRAILING_VERSION = re.compile(rf"\s+-\s+[^-]*\b(?:{_VERSIONS})\b.*$", re.IGNORECASE)
# The suffix is matched lazily rather than as a run of non-hyphens, because the
# thing being stripped frequently contains hyphens of its own -- "Gold Dust -
# Shy FX Re-Edit", "Back To Your Roots - Friction & K-Tee Remix".
TRAILING_VERSION = re.compile(rf"\s+-\s+.*?\b(?:{_VERSIONS})\b.*$", re.IGNORECASE)
# Last.fm routinely carries the guest credit in the artist field where the file
# tag holds only the primary artist -- "Yellowcard feat. Tay Jardine" against a
# tag of "Yellowcard". `with` is deliberately absent: it appears in far too many
# real titles to cut on sight.
GUEST_CREDIT = re.compile(r"\s+(?:feat|ft|featuring)\b.*$", re.IGNORECASE)
# Last.fm routinely carries the guest credit where the file tag holds only the
# primary artist -- "Yellowcard feat. Tay Jardine" against a tag of
# "Yellowcard", or "Self vs Self (feat. In Flames)" against "Self vs Self". The
# opening bracket has to be allowed for: requiring whitespace immediately before
# the word misses every bracketed credit, which is most of them. `with` is
# deliberately absent -- it appears in far too many real titles to cut on sight.
GUEST_CREDIT = re.compile(r"[\s(\[]+(?:feat|ft|featuring)\b.*$", re.IGNORECASE)
LEADING_ARTICLE = re.compile(r"^the\s+")
# Deleted rather than spaced, so "Don't" and "Dont" agree. Every other mark
@@ -1078,6 +1088,31 @@ def coverage_report(store):
row["plays"],
)
# Why the mbid tier performs the way it does. A recording id on both sides
# that still fails to join means the two disagree about which recording the
# song is -- MusicBrainz holds a separate recording per release, and Last.fm
# and Lidarr need not have picked the same one. That is not a fault to fix
# in the matcher; it is the reason the name tier has to carry the load.
library_with_mbid = store.scalar(
"SELECT COUNT(*) FROM lidarr_track WHERE recording_mbid IS NOT NULL"
)
logger.info(
"library tracks carrying a recording MBID: %d of %d (%.1f%%)",
library_with_mbid,
tracks,
100 * library_with_mbid / tracks,
)
disagreed = store.connection.execute(
"SELECT COUNT(*) AS pairs, COALESCE(SUM(plays), 0) AS plays FROM scrobble_key"
" WHERE track_mbid IS NOT NULL AND method = 'name'"
).fetchone()
logger.info(
"carried a recording MBID, joined on name instead: %d pairs, %d plays"
" -- both sides know the song, they disagree on which recording it is",
disagreed["pairs"],
disagreed["plays"],
)
suspect = store.connection.execute(
"SELECT COUNT(*) AS pairs, COALESCE(SUM(plays), 0) AS plays FROM scrobble_key k"
" WHERE k.track_id IS NULL"