A pull request took roughly eleven minutes to go green, and the log shows one
CACHED line in the whole run. The image was being built twice, in full.
The test stage is built by the runner's docker daemon. The runtime stage was
then built by docker/build-push-action, which runs under a buildx builder that
setup-buildx-action creates in its own container with its own cache. The two
share nothing, so the second build pulled the base image again, ran pip install
again, and exported the layers again -- about four and a half minutes, plus
another thirty-five seconds to boot buildkit. The comment above the test step
claimed those layers were shared, which is what made this look reasonable.
buildx earns that overhead when producing several architectures. This produces
linux/amd64 only, by an explicit decision recorded in the workflow, so it earns
nothing here. Use plain docker build against the same daemon that ran the
tests, and push with docker push. The runtime stage is a strict prefix of the
test stage, so every layer is a cache hit: measured at 1.3 seconds locally
against roughly four and a half minutes in CI.
The remaining time is the runner itself, which is slow in absolute terms --
pytest takes four seconds locally and a hundred and two in CI. That is not
something the workflow can fix.
First stage of a curation tool for the music library that music-mirror
mirrors. Before anything can build playlists or decide what has gone cold,
there has to be a local, queryable record of what is actually played; an API
call per question does not scale to a library-sized analysis.
Ingest is in two halves. A catch-up fetches everything scrobbled since the
newest scrobble held, and a backfill walks the history backwards until it runs
out. Both take their bounds from the database rather than from a saved cursor,
so an interrupted run resumes from what it actually has, and both windows are
bounded at each end so paging cannot shift under the fetch while new scrobbles
arrive mid-run.
Scrobbles carry no identifier, so the primary key is timestamp, artist and
track. Two plays of one track in the same second collapse into a single row:
they are indistinguishable in the data, and a surrogate key would make
re-ingest non-idempotent, which is the worse trade.
Three API behaviours are handled explicitly because each fails silently:
the currently-playing track arrives with no timestamp and would be re-ingested
on every pass; a lone result is returned as a bare object rather than a
one-item list; and MBIDs are empty strings rather than absent when unknown,
which would later look like a usable join key.
Retries cover the rate limit and the transient backend errors with an
exponential backoff. An invalid or suspended key fails immediately.
The report exists to surface one number before the next stage is built: the
share of scrobbles carrying a MusicBrainz recording id. Lidarr exposes the same
identifier per track, so those can be joined exactly and the rest must go
through name matching. That percentage bounds how far the matcher can be
trusted.
No runtime dependencies, and the tests run against a fake transport that
reproduces the service's paging and response shapes, so they need neither
network nor credentials.
Emma Thorpe