fix: hold one connection to Lidarr open, and retry what deserves retrying

Indexing makes two requests per artist, and more when the album fallback fires.
urllib opens a new TCP connection and performs a new DNS lookup for every one of
them, so a large library becomes thousands of lookups inside a few minutes. That
is enough to exhaust a container's resolver, and the result is
"[Errno -3] Try again" on every artist at once -- a failure caused entirely by
how the requests were made rather than by anything wrong with Lidarr.

Add a transport that keeps one connection open per host, so the name is resolved
once and the socket is reused. It retries once on a connection the server has
already closed, since a stale keep-alive announces itself only on use.

Retries were previously declined on the grounds that Lidarr is on the same LAN.
That is not a safe assumption -- it may sit behind a public hostname and a
reverse proxy -- and a transient failure currently costs an artist their entire
entry for that pass. Transient failures are now retried with a backoff. HTTP 500
is deliberately excluded: it is an exception inside Lidarr's serialisation, not
a busy server, and three attempts only delay finding that out.

The same distinction gates the album probe added alongside this. Naming the
offending album costs one request per album of that artist, which is worth it
for a deterministic fault and actively harmful during a network-wide one, where
every artist fails and probing each of them multiplies the load responsible.

The keep-alive transport is tested against a real local HTTP server rather than
a fake, because connection reuse and status mapping are exactly the properties a
fake would assume rather than demonstrate.

music_curator.py

@@ -17,6 +17,8 @@ cursor, so an interrupted run resumes from what it actually has.
 
 import argparse
 import fcntl
+import http.client
+import io
 import json
 import logging
 import os
@@ -48,6 +50,11 @@ PAGE_SIZE = 200
 RETRYABLE_ERRORS = {8, 11, 16, 29}
 RETRYABLE_STATUS = {429, 500, 502, 503, 504}
 
+# Lidarr's own list, and 500 is deliberately absent. A 500 from Lidarr is an
+# unhandled exception inside its serialisation, not a busy server; it will be
+# raised again identically, and retrying only delays finding that out.
+LIDARR_RETRYABLE_STATUS = {429, 502, 503, 504}
+
 # Last.fm asks for no more than five requests a second averaged over five
 # minutes. A full backfill is thousands of requests, so it is worth staying
 # well inside that rather than discovering error 29 halfway through.
@@ -207,7 +214,18 @@ class LastfmError(Exception):
 
 
 class LidarrError(Exception):
-    """A Lidarr request that failed."""
+    """A Lidarr request that failed.
+
+    `transient` separates "the network or the server had a moment" from "this
+    request will fail identically forever". The distinction matters twice: only
+    the first is worth retrying, and only the second is worth investigating,
+    since probing a library-wide outage artist by artist multiplies the load
+    that caused it.
+    """
+
+    def __init__(self, message, transient=False):
+        super().__init__(message)
+        self.transient = transient
 
 
 def normalise(text):
@@ -645,34 +663,118 @@ def sync_loved(client, store, user):
     return len(rows)
 
 
+class KeepAlive:
+    """A transport that holds one connection open per host.
+
+    urllib opens a fresh TCP connection -- and performs a fresh DNS lookup --
+    for every request it makes. Indexing a library is two requests per artist,
+    which on a large collection is thousands of lookups inside a few minutes.
+    That is enough to exhaust a container's resolver, and the failure it
+    produces is `[Errno -3] Try again` on everything at once. Resolving once and
+    reusing the socket removes the cause rather than papering over it, and is
+    considerably faster besides.
+    """
+
+    def __init__(self, timeout=60):
+        self.timeout = timeout
+        self._connections = {}
+
+    def __call__(self, url, timeout=None, headers=None):
+        parsed = urllib.parse.urlparse(url)
+        key = (parsed.scheme, parsed.hostname, parsed.port)
+        target = parsed.path + (f"?{parsed.query}" if parsed.query else "")
+        request_headers = {**(headers or {}), "Accept": "application/json"}
+
+        # Two attempts, because a kept-alive connection the server has since
+        # closed fails on use rather than announcing itself. The second attempt
+        # is on a fresh socket.
+        for attempt in (1, 2):
+            connection = self._connections.get(key)
+            if connection is None:
+                connection = self._connect(parsed, timeout or self.timeout)
+                self._connections[key] = connection
+            try:
+                connection.request("GET", target, headers=request_headers)
+                response = connection.getresponse()
+                body = response.read()
+            except (http.client.HTTPException, OSError) as error:
+                self.close(key)
+                if attempt == 2:
+                    raise urllib.error.URLError(error) from error
+                continue
+
+            if response.status >= 300:
+                # Includes redirects: this client does not follow them, and one
+                # here means the URL is pointing somewhere unintended.
+                raise urllib.error.HTTPError(
+                    url, response.status, response.reason, response.headers, io.BytesIO(body)
+                )
+            return body.decode("utf-8")
+        raise urllib.error.URLError("unreachable")
+
+    @staticmethod
+    def _connect(parsed, timeout):
+        if parsed.scheme == "https":
+            return http.client.HTTPSConnection(parsed.hostname, parsed.port, timeout=timeout)
+        return http.client.HTTPConnection(parsed.hostname, parsed.port, timeout=timeout)
+
+    def close(self, key=None):
+        for handle in [self._connections.pop(key, None)] if key else self._connections.values():
+            if handle is not None:
+                handle.close()
+        if key is None:
+            self._connections.clear()
+
+
 class Lidarr:
     """Minimal read-only Lidarr client.
 
-    No retries: Lidarr is on the same LAN as this, and a failure there means it
-    is down or the key is wrong, neither of which improves on a second attempt.
+    Retries only what is worth retrying. A dropped connection or a resolver
+    hiccup is transient; an HTTP 500 out of Lidarr is an exception in its own
+    serialisation and will be thrown again identically, so spending three
+    attempts on it only slows down finding out.
     """
 
-    def __init__(self, url, api_key, timeout=60, transport=None):
+    def __init__(self, url, api_key, timeout=60, attempts=3, backoff=1.0, transport=None):
         self.root = url.rstrip("/")
         self.api_key = api_key
         self.timeout = timeout
-        self.transport = transport or http_get
+        self.attempts = attempts
+        self.backoff = backoff
+        self.transport = transport or KeepAlive(timeout)
 
     def get(self, path, params=None):
         """Return the decoded response for one API path."""
         query = urllib.parse.urlencode(params or {})
         url = f"{self.root}/api/v1/{path}" + (f"?{query}" if query else "")
-        try:
-            body = self.transport(url, timeout=self.timeout, headers={"X-Api-Key": self.api_key})
-        except urllib.error.HTTPError as error:
-            detail = error_detail(error)
-            raise LidarrError(f"GET {url}: HTTP {error.code}{': ' + detail if detail else ''}")
-        except (urllib.error.URLError, TimeoutError) as error:
-            raise LidarrError(f"GET {url}: {error}") from error
-        try:
-            return json.loads(body)
-        except json.JSONDecodeError as error:
-            raise LidarrError(f"GET {url}: malformed response") from error
+
+        for attempt in range(1, self.attempts + 1):
+            try:
+                body = self.transport(
+                    url, timeout=self.timeout, headers={"X-Api-Key": self.api_key}
+                )
+            except urllib.error.HTTPError as error:
+                detail = error_detail(error)
+                message = f"GET {url}: HTTP {error.code}{': ' + detail if detail else ''}"
+                if error.code not in LIDARR_RETRYABLE_STATUS:
+                    raise LidarrError(message)
+                if attempt >= self.attempts:
+                    raise LidarrError(message, transient=True)
+            except (urllib.error.URLError, TimeoutError) as error:
+                message = f"GET {url}: {error}"
+                if attempt >= self.attempts:
+                    raise LidarrError(message, transient=True) from error
+            else:
+                try:
+                    return json.loads(body)
+                except json.JSONDecodeError as error:
+                    raise LidarrError(f"GET {url}: malformed response") from error
+
+            pause = min(self.backoff * 2 ** (attempt - 1), BACKOFF_CEILING_SECONDS)
+            logger.warning("%s; retrying in %.0fs", message, pause)
+            time.sleep(pause)
+
+        raise LidarrError(f"GET {url}: gave up after {self.attempts} attempts", transient=True)
 
 
 def error_detail(error, limit=300):
@@ -707,6 +809,36 @@ def parse_added(value):
         return None
 
 
+def find_bad_albums(client, artist_id, artist_name):
+    """Name the specific albums Lidarr cannot serialise for one artist.
+
+    Runs only once that artist's album fetch has already failed, so the extra
+    requests are spent on a problem that already exists. Tracks come from an
+    endpoint that still works, and their album ids give a list to probe one at a
+    time; the ones that throw are the culprits. A track title from each is
+    enough to recognise the album in the UI, which the id alone is not.
+    """
+    try:
+        tracks = client.get("track", {"artistId": artist_id})
+    except LidarrError as error:
+        logger.warning("could not probe %s for the offending album: %s", artist_name, error)
+        return []
+
+    sample = {}
+    for track in tracks:
+        sample.setdefault(track.get("albumId"), track.get("title") or "")
+
+    bad = []
+    for album_id, title in sorted(sample.items(), key=lambda item: item[0] or 0):
+        if not album_id:
+            continue
+        try:
+            client.get("album", {"albumIds": album_id})
+        except LidarrError:
+            bad.append((album_id, title))
+    return bad
+
+
 def fetch_albums(client, artists):
     """Return albums grouped by artist id, plus the artists whose albums failed.
 
@@ -738,6 +870,18 @@ def fetch_albums(client, artists):
             name = artist.get("artistName") or str(artist_id)
             logger.warning("could not fetch albums for %s: %s", name, error)
             failed.append(name)
+            # Only worth probing a deterministic failure. When the network or
+            # the resolver is the problem, every artist fails, and probing each
+            # of them album by album multiplies the load that caused it.
+            if not error.transient:
+                for album_id, sample in find_bad_albums(client, artist_id, name):
+                    logger.warning(
+                        "  album id %d is the one Lidarr cannot serialise (it holds the"
+                        " track %r). Open it in Lidarr and leave exactly one release"
+                        " monitored.",
+                        album_id,
+                        sample,
+                    )
     return grouped, failed