feat: mirror a lossless library to MP3 for iPod sync (#1)
Build and publish container / build (push) Failing after 51s
Build and publish container / build (push) Failing after 51s
## What A path-for-path MP3 mirror of a lossless library. FLAC in, MP3 out, same relative layout, tags and cover art carried across. Already-MP3 sources are copied rather than re-encoded. Mirror files whose source has gone are deleted, along with any directory they emptied. The source library is never written to — mounted read-only in the compose file, and never opened for writing in code. | Situation | Action | | -------------------------------- | ------------------------------------ | | No mirror file | encode | | Source modified since the mirror | re-encode (a Lidarr quality upgrade) | | Mirror up to date | skip | | Source is already MP3 | copy verbatim | | Source gone | delete, prune empty dirs | ## Design notes - **No database.** Freshness is mtime: an encode is stamped with its source's mtime, so a file is stale exactly when the two differ. Lidarr owns the library; a second tool with its own index would only fall out of step with it. This is also why it is not beets. - **Atomic writes.** Encode to a temp file, rename into place. An interrupted run cannot leave a truncated MP3 that the next run treats as finished. - **A lock file** in the mirror root stops two passes overlapping. - **Refuses a mirror inside the source tree**, which would otherwise recurse. - `--subdir` never prunes: a partial pass cannot distinguish an orphan from a file outside its scope. ## Shipping - Python package with a `music-mirror` console script, no runtime dependencies beyond ffmpeg. - `Dockerfile` plus `compose.yaml` as a TrueNAS Scale Custom App: source dataset read-only, mirror dataset writable, `MUSIC_MIRROR_INTERVAL=6h`. - CI runs the tests **inside the image**, against the ffmpeg that ships, on every pull request, and on merge publishes multi-arch (amd64 + arm64) to this Gitea's registry using the `PACKAGES_SECRET` repository secret. Versioning follows the same conventional-commit scheme as `legacy-email-proxy`, including writing the released version back into `pyproject.toml` so the packaging metadata cannot drift behind the tag. ## Behaviour under Lidarr | Lidarr does this | The mirror does this | | --------------------------------------- | -------------------------------------------------------------------------------- | | Replaces a file with a better rip | Re-encodes in place; same path in, same path out, so no duplicate | | Upgrades MP3 to FLAC | Both map to the same `.mp3` mirror path, so the old one is overwritten | | Renames a track, album or artist folder | Old path pruned, new path encoded — correct, though it re-encodes rather than moving | | Deletes an album or artist | Every orphaned mirror file goes, and the directories they emptied with them | Three defects were found and fixed while writing those tests, each verified to fail against the previous code: - Pruning probed the source tree for a mirror file's original name, lowercase extensions only. A `.FLAC` source was never found, so its mirror file was deleted as an orphan and rebuilt on the next pass, for ever. Pruning now works from the set of paths the pass actually accounted for. - Two sources could claim one mirror path — `01 Song.flac` beside a leftover `01 Song.mp3`. Both encoded to the same destination and each pass found the loser stale. The better format now wins, ties break on path. - The source mtime was read after encoding rather than before, so a file still being written when the pass reached it could be stamped current while holding truncated audio. ## Why there is no flake The deployment target is a container. A flake here would sit on no path between the source and the NAS, and `nix flake check` would test against nixpkgs' ffmpeg while the artefact ships Debian's — precisely the layer these tests exercise. It was removed in favour of running the suite inside the image. The sibling `legacy-email-proxy` keeps its flake because there the flake *is* the deployment mechanism. ## Verification - 21 tests, all real ffmpeg round-trips: layout, tag survival, MP3 output, skip-when-current, re-encode-on-change, orphan pruning, empty-dir removal, `--no-prune`, MP3 passthrough, `--dry-run`, `--subdir`, external cover art embedding, both refusal paths, and the Lidarr lifecycle cases below. - The suite also runs in the multi-stage Docker `test` stage: 21 passed. The published `runtime` stage carries neither the tests nor pytest, verified by inspecting the image. - Container built and run locally against a sample library: correct output path, Cyrillic tags intact. - The release step was extracted from the workflow and run against a scratch repository: it commits and tags when the version changes, and skips the commit but still tags when `pyproject.toml` already carries it. - **Not tested against the real library** — the first run there should be `--dry-run`. ## Follow-ups, not in this PR - Lidarr imports are picked up on the next scheduled pass rather than instantly. A webhook trigger is the obvious next step if six hours feels slow. - `PACKAGES_SECRET` must exist as a repository secret before the first merge, or the login step fails. --------- Co-authored-by: Emma Thorpe <emma.thorpe@citrix.com> Reviewed-on: #1
This commit was merged in pull request #1.
This commit is contained in:
@@ -0,0 +1,80 @@
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
# Ensure the project root is on sys.path when running tests.
|
||||
ROOT = os.path.dirname(os.path.dirname(__file__))
|
||||
if ROOT not in sys.path:
|
||||
sys.path.insert(0, ROOT)
|
||||
|
||||
|
||||
@pytest.fixture(scope="session", autouse=True)
|
||||
def require_ffmpeg():
|
||||
"""The tests exercise real encodes; there is little point faking them."""
|
||||
for tool in ("ffmpeg", "ffprobe"):
|
||||
if shutil.which(tool) is None:
|
||||
pytest.skip(f"{tool} is not on PATH", allow_module_level=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_flac():
|
||||
"""Return a factory writing a short tagged FLAC file."""
|
||||
|
||||
def factory(path, title="Test Title", artist="Test Artist", album="Test Album", seconds=1):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg",
|
||||
"-nostdin",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
f"sine=frequency=440:duration={seconds}",
|
||||
"-metadata",
|
||||
f"title={title}",
|
||||
"-metadata",
|
||||
f"artist={artist}",
|
||||
"-metadata",
|
||||
f"album={album}",
|
||||
"-metadata",
|
||||
"track=3",
|
||||
str(path),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
return path
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def probe_tag():
|
||||
"""Return a helper reading a single metadata tag from a file."""
|
||||
|
||||
def reader(path, tag):
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-show_entries",
|
||||
f"format_tags={tag}",
|
||||
"-of",
|
||||
"default=noprint_wrappers=1:nokey=1",
|
||||
str(path),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return completed.stdout.strip()
|
||||
|
||||
return reader
|
||||
@@ -0,0 +1,323 @@
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
import music_mirror
|
||||
|
||||
|
||||
def run(source, mirror, *extra):
|
||||
return music_mirror.main(["--source", str(source), "--mirror", str(mirror), *extra])
|
||||
|
||||
|
||||
def test_parse_quality_accepts_vbr_and_cbr():
|
||||
assert music_mirror.parse_quality("V0") == ["-q:a", "0"]
|
||||
assert music_mirror.parse_quality("v2") == ["-q:a", "2"]
|
||||
assert music_mirror.parse_quality("256") == ["-b:a", "256k"]
|
||||
|
||||
|
||||
def test_parse_quality_rejects_nonsense():
|
||||
with pytest.raises(ValueError):
|
||||
music_mirror.parse_quality("best")
|
||||
|
||||
|
||||
def test_parse_interval_units():
|
||||
assert music_mirror.parse_interval("90") == 90
|
||||
assert music_mirror.parse_interval("30m") == 1800
|
||||
assert music_mirror.parse_interval("6h") == 21600
|
||||
assert music_mirror.parse_interval("1d") == 86400
|
||||
|
||||
|
||||
def test_encodes_and_preserves_layout_and_tags(tmp_path, make_flac, probe_tag):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Test Artist" / "Test Album" / "03 Song.flac")
|
||||
|
||||
assert run(source, mirror) == 0
|
||||
|
||||
output = mirror / "Test Artist" / "Test Album" / "03 Song.mp3"
|
||||
assert output.is_file()
|
||||
assert probe_tag(output, "title") == "Test Title"
|
||||
assert probe_tag(output, "album") == "Test Album"
|
||||
|
||||
|
||||
def test_output_is_mp3(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
codec = subprocess.run(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"a:0",
|
||||
"-show_entries",
|
||||
"stream=codec_name",
|
||||
"-of",
|
||||
"csv=p=0",
|
||||
str(mirror / "a.mp3"),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout.strip()
|
||||
assert codec == "mp3"
|
||||
|
||||
|
||||
def test_second_pass_skips_unchanged_files(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
first = (mirror / "a.mp3").stat().st_mtime_ns
|
||||
|
||||
run(source, mirror)
|
||||
assert (mirror / "a.mp3").stat().st_mtime_ns == first
|
||||
|
||||
|
||||
def test_changed_source_is_re_encoded(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
track = make_flac(source / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
before = (mirror / "a.mp3").stat().st_mtime
|
||||
|
||||
# A replaced file with a newer mtime is what a Lidarr quality upgrade
|
||||
# looks like on disk.
|
||||
later = time.time() + 120
|
||||
os.utime(track, (later, later))
|
||||
run(source, mirror)
|
||||
|
||||
assert (mirror / "a.mp3").stat().st_mtime > before
|
||||
|
||||
|
||||
def test_deleted_source_is_pruned(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
make_flac(source / "Album" / "b.flac")
|
||||
|
||||
run(source, mirror)
|
||||
(source / "Album" / "b.flac").unlink()
|
||||
run(source, mirror)
|
||||
|
||||
assert (mirror / "Album" / "a.mp3").is_file()
|
||||
assert not (mirror / "Album" / "b.mp3").exists()
|
||||
|
||||
|
||||
def test_emptied_directory_is_removed(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Gone" / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
(source / "Gone" / "a.flac").unlink()
|
||||
run(source, mirror)
|
||||
|
||||
assert not (mirror / "Gone").exists()
|
||||
|
||||
|
||||
def test_no_prune_keeps_orphans(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
(source / "a.flac").unlink()
|
||||
run(source, mirror, "--no-prune")
|
||||
|
||||
assert (mirror / "a.mp3").is_file()
|
||||
|
||||
|
||||
def test_existing_mp3_is_copied_not_re_encoded(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
flac = make_flac(source / "a.flac")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-loglevel", "error", "-y", "-i", str(flac), str(source / "b.mp3")],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
flac.unlink()
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
assert (mirror / "b.mp3").read_bytes() == (source / "b.mp3").read_bytes()
|
||||
|
||||
|
||||
def test_format_upgrade_replaces_rather_than_duplicating(tmp_path, make_flac):
|
||||
"""Lidarr replacing an MP3 with a FLAC must not leave two mirror files."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
flac = make_flac(source / "Album" / "01 Song.flac")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-loglevel", "error", "-y", "-i", str(flac), str(source / "Album" / "01 Song.mp3")],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
flac.unlink()
|
||||
|
||||
run(source, mirror)
|
||||
assert sorted(p.name for p in (mirror / "Album").iterdir()) == ["01 Song.mp3"]
|
||||
|
||||
# The upgrade: the MP3 goes, a FLAC arrives at the same stem.
|
||||
(source / "Album" / "01 Song.mp3").unlink()
|
||||
make_flac(source / "Album" / "01 Song.flac", title="Upgraded")
|
||||
run(source, mirror)
|
||||
|
||||
assert sorted(p.name for p in (mirror / "Album").iterdir()) == ["01 Song.mp3"]
|
||||
|
||||
|
||||
def test_renamed_album_leaves_nothing_behind(tmp_path, make_flac):
|
||||
"""A Lidarr rename is a delete plus an add; the old tree must not linger."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Artist" / "Album (2019)" / "01 Song.flac")
|
||||
|
||||
run(source, mirror)
|
||||
(source / "Artist" / "Album (2019)").rename(source / "Artist" / "Album (2020)")
|
||||
run(source, mirror)
|
||||
|
||||
assert (mirror / "Artist" / "Album (2020)" / "01 Song.mp3").is_file()
|
||||
assert not (mirror / "Artist" / "Album (2019)").exists()
|
||||
|
||||
|
||||
def test_competing_sources_pick_the_lossless_one_and_stay_stable(tmp_path, make_flac, probe_tag):
|
||||
"""Both a FLAC and an MP3 at one stem: the FLAC wins, and stays won."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "a.flac", title="From FLAC")
|
||||
other = make_flac(tmp_path / "scratch" / "other.flac", title="From MP3")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-loglevel", "error", "-y", "-i", str(other), str(source / "a.mp3")],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
# Age the loser well beyond the mtime tolerance, so "is it current?" gives
|
||||
# a definite answer for it rather than one that depends on the clock.
|
||||
older = time.time() - 3600
|
||||
os.utime(source / "a.mp3", (older, older))
|
||||
|
||||
run(source, mirror)
|
||||
assert probe_tag(mirror / "a.mp3", "title") == "From FLAC"
|
||||
|
||||
# The loser must not make the mirror look stale on the next pass, or every
|
||||
# run would re-encode for ever.
|
||||
first = (mirror / "a.mp3").stat().st_mtime_ns
|
||||
run(source, mirror)
|
||||
assert (mirror / "a.mp3").stat().st_mtime_ns == first
|
||||
|
||||
|
||||
def test_uppercase_extension_is_not_pruned_and_re_encoded(tmp_path, make_flac):
|
||||
"""A .FLAC source must not be treated as an orphan on the next pass."""
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
made = make_flac(source / "a.flac")
|
||||
made.rename(source / "a.FLAC")
|
||||
|
||||
run(source, mirror)
|
||||
first = (mirror / "a.mp3").stat().st_mtime_ns
|
||||
|
||||
run(source, mirror)
|
||||
assert (mirror / "a.mp3").is_file()
|
||||
assert (mirror / "a.mp3").stat().st_mtime_ns == first
|
||||
|
||||
|
||||
def test_whole_library_deleted_empties_the_mirror(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "A" / "one.flac")
|
||||
make_flac(source / "B" / "two.flac")
|
||||
|
||||
run(source, mirror)
|
||||
shutil.rmtree(source / "A")
|
||||
shutil.rmtree(source / "B")
|
||||
run(source, mirror)
|
||||
|
||||
assert list(mirror.rglob("*.mp3")) == []
|
||||
assert not (mirror / "A").exists()
|
||||
assert not (mirror / "B").exists()
|
||||
|
||||
|
||||
def test_dry_run_writes_nothing(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "a.flac")
|
||||
|
||||
assert run(source, mirror, "--dry-run") == 0
|
||||
assert not (mirror / "a.mp3").exists()
|
||||
|
||||
|
||||
def test_subdir_limits_the_pass_and_does_not_prune(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "One" / "a.flac")
|
||||
make_flac(source / "Two" / "b.flac")
|
||||
|
||||
assert run(source, mirror, "--subdir", "One") == 0
|
||||
|
||||
assert (mirror / "One" / "a.mp3").is_file()
|
||||
# Outside the requested directory: neither encoded nor treated as an orphan.
|
||||
assert not (mirror / "Two").exists()
|
||||
|
||||
|
||||
def test_external_cover_is_embedded(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=red:s=64x64:d=1",
|
||||
"-frames:v",
|
||||
"1",
|
||||
str(source / "Album" / "cover.jpg"),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
streams = subprocess.run(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"v",
|
||||
"-show_entries",
|
||||
"stream=codec_name",
|
||||
"-of",
|
||||
"csv=p=0",
|
||||
str(mirror / "Album" / "a.mp3"),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout.strip()
|
||||
assert streams # a picture stream is present
|
||||
|
||||
|
||||
def test_mirror_inside_source_is_refused(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
make_flac(source / "a.flac")
|
||||
assert run(source, source / "mp3") == 2
|
||||
|
||||
|
||||
def test_missing_source_is_refused(tmp_path):
|
||||
assert run(tmp_path / "nope", tmp_path / "dst") == 2
|
||||
Reference in New Issue
Block a user