feat: mirror a lossless library to MP3 for iPod sync #1
@@ -0,0 +1,12 @@
|
||||
.git
|
||||
.gitignore
|
||||
tests
|
||||
result
|
||||
result-*
|
||||
__pycache__
|
||||
*.pyc
|
||||
.pytest_cache
|
||||
flake.nix
|
||||
flake.lock
|
||||
package.nix
|
||||
compose.yaml
|
||||
@@ -0,0 +1,41 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: 3.13
|
||||
|
||||
# The test suite runs real encodes, so ffmpeg is a test dependency.
|
||||
- name: Install ffmpeg
|
||||
run: sudo apt-get update && sudo apt-get install --no-install-recommends -y ffmpeg
|
||||
|
||||
- name: Install test dependencies
|
||||
run: python -m pip install --upgrade pip && pip install pytest
|
||||
|
||||
- name: Run unit tests
|
||||
run: python -m pytest
|
||||
|
||||
- name: Build the container image
|
||||
run: docker build -t music-mirror:ci .
|
||||
@@ -0,0 +1,7 @@
|
||||
result
|
||||
result-*
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
.pytest_cache/
|
||||
*.egg-info/
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
FROM python:3.13-slim
|
||||
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
# ffmpeg does the encoding; the application itself has no Python dependencies.
|
||||
RUN apt-get update \
|
||||
&& apt-get install --no-install-recommends -y ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
COPY pyproject.toml README.md ./
|
||||
COPY music_mirror.py ./
|
||||
RUN pip install --no-cache-dir .
|
||||
|
||||
# Runs as root by default so a bind-mounted dataset of any ownership is
|
||||
# writable. Override with `user:` in compose to run as the dataset's owner.
|
||||
ENTRYPOINT ["music-mirror"]
|
||||
@@ -1,3 +1,118 @@
|
||||
# music-mirror
|
||||
|
||||
Maintain a lossy MP3 mirror of a lossless music library
|
||||
Maintain a lossy MP3 mirror of a lossless music library.
|
||||
|
||||
Walks a source library and reproduces it, path for path, as MP3 in a separate
|
||||
tree. Tags and cover art are carried across; sources that are already MP3 are
|
||||
copied rather than re-encoded; mirror files whose source has been deleted are
|
||||
removed. **The source library is never written to** — it is mounted read-only
|
||||
in the supplied compose file, and nothing in the code opens it for writing.
|
||||
|
||||
The intended use is an iPod. Apple's Music app cannot read FLAC at all, so a
|
||||
converted copy has to exist somewhere; this keeps that copy next to the
|
||||
library on a NAS instead of on a laptop, and keeps it current without a human
|
||||
remembering to do anything.
|
||||
|
||||
## How it decides what to do
|
||||
|
||||
| Situation | Action |
|
||||
| -------------------------------- | ---------------------------------------- |
|
||||
| No mirror file | encode |
|
||||
| Source modified since the mirror | re-encode (a Lidarr quality upgrade) |
|
||||
| Mirror up to date | skip |
|
||||
| Source is already MP3 | copy verbatim |
|
||||
| Source gone | delete the mirror file, prune empty dirs |
|
||||
|
||||
Freshness is modification time: an encoded file is stamped with its source's
|
||||
mtime, so a file is stale exactly when the two differ. There is no database to
|
||||
fall out of step with the library, which matters when something else — Lidarr,
|
||||
in this case — is the thing that owns and reorganises it.
|
||||
|
||||
Encodes are written to a temporary file and renamed into place, so an
|
||||
interrupted run cannot leave a truncated MP3 that the next run mistakes for
|
||||
finished work. A lock file in the mirror root stops two passes overlapping.
|
||||
|
||||
## Usage
|
||||
|
||||
```sh
|
||||
music-mirror --source /music --mirror /music-mp3 # one pass
|
||||
music-mirror --source /music --mirror /music-mp3 --interval 6h # keep running
|
||||
music-mirror --source /music --mirror /music-mp3 --dry-run # report only
|
||||
music-mirror --source /music --mirror /music-mp3 --subdir "Artist/Album"
|
||||
```
|
||||
|
||||
| Option | Environment variable | Default | Meaning |
|
||||
| ------------ | ----------------------- | --------- | ---------------------------------------------- |
|
||||
| `--source` | `MUSIC_MIRROR_SOURCE` | — | Root of the lossless library, read-only |
|
||||
| `--mirror` | `MUSIC_MIRROR_MIRROR` | — | Root of the MP3 mirror |
|
||||
| `--quality` | `MUSIC_MIRROR_QUALITY` | `V0` | LAME VBR level `V0`–`V9`, or kbps e.g. `256` |
|
||||
| `--jobs` | `MUSIC_MIRROR_JOBS` | CPU count | Concurrent encodes |
|
||||
| `--interval` | `MUSIC_MIRROR_INTERVAL` | unset | Repeat forever, e.g. `45m`, `6h`, `1d` |
|
||||
| `--subdir` | — | unset | Limit the pass to one directory; skips pruning |
|
||||
| `--no-prune` | — | off | Keep mirror files whose source has gone |
|
||||
| `--dry-run` | — | off | Report what would change, write nothing |
|
||||
|
||||
`--subdir` never prunes: a partial pass cannot tell an orphan from a file
|
||||
outside its own scope.
|
||||
|
||||
Requires `ffmpeg` and `ffprobe` on `PATH`. The container and the Nix package
|
||||
both provide them.
|
||||
|
||||
## Running it on TrueNAS Scale
|
||||
|
||||
`compose.yaml` is a Custom App definition. Build the image on the NAS, adjust
|
||||
the two host paths and the `user:` to match your pool, then add it as a custom
|
||||
app:
|
||||
|
||||
```sh
|
||||
git clone https://code.emmathe.dev/lyrathorpe/music-mirror
|
||||
cd music-mirror && docker build -t music-mirror:latest .
|
||||
```
|
||||
|
||||
Point the mirror at its own dataset rather than a directory inside the music
|
||||
dataset — it is derived data, so it wants its own snapshot policy, its own
|
||||
quota, and its own SMB share. The tool refuses to run with a mirror inside the
|
||||
source tree.
|
||||
|
||||
New Lidarr imports are picked up on the next pass. With `MUSIC_MIRROR_INTERVAL`
|
||||
at `6h` that is the worst case; run `--subdir` by hand if you want an album
|
||||
immediately.
|
||||
|
||||
## Nix
|
||||
|
||||
```sh
|
||||
nix run .#music-mirror -- --source ./flac --mirror ./mp3
|
||||
nix build .#music-mirror # the test suite runs as part of the build
|
||||
nix develop # python, pytest and ffmpeg
|
||||
```
|
||||
|
||||
`overlays.default` provides `pkgs.music-mirror`.
|
||||
|
||||
## Tests
|
||||
|
||||
```sh
|
||||
pytest
|
||||
```
|
||||
|
||||
The tests run real ffmpeg encodes rather than mocking them — the interesting
|
||||
failures are in what ffmpeg actually does with tags, cover art and container
|
||||
formats, and a mock cannot fail that way. They skip if ffmpeg is absent.
|
||||
|
||||
## Getting the result onto an iPod
|
||||
|
||||
The mirror is just a directory of MP3s, so any client will do:
|
||||
|
||||
- **macOS.** Add the mirror's SMB share to the Music app with _Copy files to
|
||||
Music Media folder_ and _Keep Media folder organised_ both **off**. The Mac
|
||||
then stores a library database and nothing else. Keep the share mounted at a
|
||||
stable path — if it is missing when Music opens, every track shows `!`.
|
||||
- **Linux.** Rhythmbox links `libgpod` and handles iPod sync. An iPod Video
|
||||
(5th generation) predates the models whose database has to be signed, so no
|
||||
firmware-hash trickery is needed.
|
||||
|
||||
Neither client transcodes at sync time; they copy finished MP3s.
|
||||
|
||||
Two device-side details worth knowing: the iPod reads cover art from the file's
|
||||
tags and ignores `folder.jpg`, which is why art is embedded here; and volume
|
||||
levelling on the device uses iTunes' Soundcheck tag, not ReplayGain, so
|
||||
ReplayGain tags in the source are not carried over as such.
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# TrueNAS Scale "Custom App" definition.
|
||||
#
|
||||
# Build the image on the NAS first (there is no published image yet):
|
||||
# git clone https://code.emmathe.dev/lyrathorpe/music-mirror
|
||||
# cd music-mirror && docker build -t music-mirror:latest .
|
||||
#
|
||||
# Adjust the two host paths and the user to match your pool. The source is
|
||||
# mounted read-only on purpose: nothing here should ever be able to write to
|
||||
# the lossless library.
|
||||
services:
|
||||
music-mirror:
|
||||
image: music-mirror:latest
|
||||
container_name: music-mirror
|
||||
restart: unless-stopped
|
||||
# The dataset owner, so the mirror is not written as root. `id apps` or the
|
||||
# ownership of the mirror dataset will tell you the right numbers.
|
||||
user: "568:568"
|
||||
environment:
|
||||
MUSIC_MIRROR_SOURCE: /music
|
||||
MUSIC_MIRROR_MIRROR: /mirror
|
||||
# LAME V0 averages about 245 kbps. Use 256 for constant bitrate instead.
|
||||
MUSIC_MIRROR_QUALITY: V0
|
||||
# How long to wait between passes. Lidarr imports are picked up on the
|
||||
# next one.
|
||||
MUSIC_MIRROR_INTERVAL: 6h
|
||||
# Concurrent encodes; defaults to the CPU count. Lower it to leave the
|
||||
# NAS responsive during the first full pass.
|
||||
# MUSIC_MIRROR_JOBS: "4"
|
||||
volumes:
|
||||
- /mnt/tank/media/music:/music:ro
|
||||
- /mnt/tank/media/music-mp3:/mirror
|
||||
Generated
+27
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"nodes": {
|
||||
"nixpkgs": {
|
||||
"locked": {
|
||||
"lastModified": 1787135253,
|
||||
"narHash": "sha256-RD2kNWCG+Bjo6h+JVjWVNntZs2GtRoeY2xHjts/FNkA=",
|
||||
"owner": "nixos",
|
||||
"repo": "nixpkgs",
|
||||
"rev": "ffb3c9b700e759be2ef13237c9d8f953b32a1e46",
|
||||
"type": "github"
|
||||
},
|
||||
"original": {
|
||||
"owner": "nixos",
|
||||
"ref": "nixos-unstable",
|
||||
"repo": "nixpkgs",
|
||||
"type": "github"
|
||||
}
|
||||
},
|
||||
"root": {
|
||||
"inputs": {
|
||||
"nixpkgs": "nixpkgs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"root": "root",
|
||||
"version": 7
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
{
|
||||
description = "Maintain a lossy MP3 mirror of a lossless music library";
|
||||
|
||||
inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
|
||||
|
||||
outputs =
|
||||
{ self, nixpkgs }:
|
||||
let
|
||||
systems = [
|
||||
"x86_64-linux"
|
||||
"aarch64-linux"
|
||||
"x86_64-darwin"
|
||||
"aarch64-darwin"
|
||||
];
|
||||
forAllSystems = fn: nixpkgs.lib.genAttrs systems (system: fn nixpkgs.legacyPackages.${system});
|
||||
in
|
||||
{
|
||||
overlays.default = final: _prev: {
|
||||
music-mirror = final.callPackage ./package.nix { };
|
||||
};
|
||||
|
||||
packages = forAllSystems (pkgs: rec {
|
||||
music-mirror = pkgs.callPackage ./package.nix { };
|
||||
default = music-mirror;
|
||||
});
|
||||
|
||||
# The package builds only if the test suite passes, so this covers both.
|
||||
checks = forAllSystems (pkgs: { inherit (self.packages.${pkgs.system}) music-mirror; });
|
||||
|
||||
devShells = forAllSystems (pkgs: {
|
||||
default = pkgs.mkShellNoCC {
|
||||
packages = [
|
||||
pkgs.python3
|
||||
pkgs.python3Packages.pytest
|
||||
pkgs.ffmpeg
|
||||
];
|
||||
};
|
||||
});
|
||||
|
||||
formatter = forAllSystems (pkgs: pkgs.nixfmt-tree);
|
||||
};
|
||||
}
|
||||
+454
@@ -0,0 +1,454 @@
|
||||
"""Maintain a lossy MP3 mirror of a lossless music library.
|
||||
|
||||
Walks a source library and reproduces it, path for path, as MP3 in a separate
|
||||
directory tree: FLAC in, MP3 out, same relative layout, tags and cover art
|
||||
carried across. Sources that are already MP3 are copied rather than re-encoded.
|
||||
|
||||
The mirror is derived state. It is only ever written to, never read as a
|
||||
source of truth, so it can be deleted and rebuilt at any time. Nothing here
|
||||
writes to the source library.
|
||||
|
||||
Staleness is tracked by modification time: an encoded file is given its
|
||||
source's mtime, so a file is out of date exactly when the two differ. That
|
||||
makes runs idempotent without a database to keep in step.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import fcntl
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger("music-mirror")
|
||||
|
||||
# Sources that are transcoded. Anything ffmpeg can decode works; this list
|
||||
# decides what the walker picks up in the first place.
|
||||
SOURCE_EXTENSIONS = {
|
||||
".flac",
|
||||
".wav",
|
||||
".aif",
|
||||
".aiff",
|
||||
".ape",
|
||||
".wv",
|
||||
".m4a",
|
||||
".alac",
|
||||
".ogg",
|
||||
".opus",
|
||||
".wma",
|
||||
}
|
||||
|
||||
# Already MP3: copied through. Re-encoding lossy audio to lossy audio costs
|
||||
# quality for nothing.
|
||||
COPY_EXTENSIONS = {".mp3"}
|
||||
|
||||
# Looked for in the source directory when a file has no embedded picture.
|
||||
COVER_NAMES = ("cover.jpg", "folder.jpg", "front.jpg", "cover.png", "folder.png")
|
||||
|
||||
# Files the mirror is allowed to contain, and therefore allowed to delete.
|
||||
MIRROR_SUFFIX = ".mp3"
|
||||
|
||||
# Filesystems disagree about mtime precision; SMB in particular rounds.
|
||||
MTIME_TOLERANCE_SECONDS = 2
|
||||
|
||||
|
||||
@dataclass
|
||||
class Result:
|
||||
"""Outcome of processing one file."""
|
||||
|
||||
action: str # encoded | copied | skipped | failed
|
||||
path: Path
|
||||
error: str = ""
|
||||
|
||||
|
||||
def parse_quality(quality):
|
||||
"""Return the ffmpeg arguments for a quality setting.
|
||||
|
||||
Accepts LAME VBR levels (``V0``..``V9``) or a constant bitrate in kbps
|
||||
(``256``). VBR is the better trade at a given average bitrate; CBR is for
|
||||
when a fixed size matters more.
|
||||
"""
|
||||
text = str(quality).strip().lower()
|
||||
if re.fullmatch(r"v[0-9]", text):
|
||||
return ["-q:a", text[1:]]
|
||||
if re.fullmatch(r"[0-9]{2,3}", text):
|
||||
return ["-b:a", f"{text}k"]
|
||||
raise ValueError(f"unrecognised quality {quality!r}: expected V0-V9 or a bitrate like 256")
|
||||
|
||||
|
||||
def parse_interval(interval):
|
||||
"""Return seconds for an interval such as ``30m``, ``6h`` or ``90``."""
|
||||
text = str(interval).strip().lower()
|
||||
match = re.fullmatch(r"([0-9]+)([smhd]?)", text)
|
||||
if not match:
|
||||
raise ValueError(f"unrecognised interval {interval!r}: expected e.g. 45m, 6h, 1d")
|
||||
value = int(match.group(1))
|
||||
return value * {"": 1, "s": 1, "m": 60, "h": 3600, "d": 86400}[match.group(2)]
|
||||
|
||||
|
||||
def mirror_path_for(source, source_root, mirror_root):
|
||||
"""Return the mirror path corresponding to a source file."""
|
||||
return (mirror_root / source.relative_to(source_root)).with_suffix(MIRROR_SUFFIX)
|
||||
|
||||
|
||||
def is_current(source, mirror):
|
||||
"""Return whether the mirror file is up to date with its source."""
|
||||
if not mirror.exists():
|
||||
return False
|
||||
return abs(source.stat().st_mtime - mirror.stat().st_mtime) <= MTIME_TOLERANCE_SECONDS
|
||||
|
||||
|
||||
def find_cover(directory):
|
||||
"""Return an external cover image for a directory, if one is present."""
|
||||
for name in COVER_NAMES:
|
||||
candidate = directory / name
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def has_embedded_picture(source):
|
||||
"""Return whether the source carries its own cover art."""
|
||||
try:
|
||||
probe = subprocess.run(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"v",
|
||||
"-show_entries",
|
||||
"stream=index",
|
||||
"-of",
|
||||
"csv=p=0",
|
||||
str(source),
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=True,
|
||||
)
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
return False
|
||||
return bool(probe.stdout.strip())
|
||||
|
||||
|
||||
def build_command(source, destination, quality_args, cover):
|
||||
"""Return the ffmpeg command that encodes one file."""
|
||||
command = ["ffmpeg", "-nostdin", "-hide_banner", "-loglevel", "error", "-y", "-i", str(source)]
|
||||
|
||||
if cover is not None:
|
||||
command += ["-i", str(cover), "-map", "1:v:0"]
|
||||
else:
|
||||
# Optional: the source may have no picture stream at all.
|
||||
command += ["-map", "0:v:0?"]
|
||||
|
||||
command += [
|
||||
"-map",
|
||||
"0:a:0",
|
||||
"-map_metadata",
|
||||
"0",
|
||||
"-c:a",
|
||||
"libmp3lame",
|
||||
*quality_args,
|
||||
"-c:v",
|
||||
"copy",
|
||||
"-disposition:v",
|
||||
"attached_pic",
|
||||
# ID3v2.3 is the widest-compatibility tag version, and what the iPod
|
||||
# firmware is happiest with; the v1 tag costs 128 bytes.
|
||||
"-id3v2_version",
|
||||
"3",
|
||||
"-write_id3v1",
|
||||
"1",
|
||||
# Stated rather than inferred: the destination is a temporary file
|
||||
# whose suffix ffmpeg would not recognise.
|
||||
"-f",
|
||||
"mp3",
|
||||
str(destination),
|
||||
]
|
||||
return command
|
||||
|
||||
|
||||
def encode(source, mirror, quality_args, dry_run):
|
||||
"""Encode one source file into the mirror, atomically."""
|
||||
if dry_run:
|
||||
logger.info("would encode %s", source)
|
||||
return Result("encoded", mirror)
|
||||
|
||||
mirror.parent.mkdir(parents=True, exist_ok=True)
|
||||
cover = None if has_embedded_picture(source) else find_cover(source.parent)
|
||||
|
||||
# Encode to a temporary file in the destination directory and rename it
|
||||
# into place, so an interrupted run cannot leave a truncated MP3 that the
|
||||
# next run would treat as complete.
|
||||
handle, temporary = tempfile.mkstemp(dir=mirror.parent, suffix=".mp3.part")
|
||||
os.close(handle)
|
||||
temporary = Path(temporary)
|
||||
|
||||
try:
|
||||
command = build_command(source, temporary, quality_args, cover)
|
||||
completed = subprocess.run(command, capture_output=True, text=True)
|
||||
if completed.returncode != 0:
|
||||
lines = completed.stderr.strip().splitlines()
|
||||
return Result("failed", source, lines[-1] if lines else "ffmpeg failed")
|
||||
stat = source.stat()
|
||||
os.utime(temporary, (stat.st_atime, stat.st_mtime))
|
||||
os.replace(temporary, mirror)
|
||||
except Exception as error: # noqa: BLE001 - reported per file, run continues
|
||||
return Result("failed", source, str(error))
|
||||
finally:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
logger.info("encoded %s", source)
|
||||
return Result("encoded", mirror)
|
||||
|
||||
|
||||
def copy(source, mirror, dry_run):
|
||||
"""Copy an already-MP3 source into the mirror."""
|
||||
if dry_run:
|
||||
logger.info("would copy %s", source)
|
||||
return Result("copied", mirror)
|
||||
|
||||
mirror.parent.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
shutil.copy2(source, mirror)
|
||||
except OSError as error:
|
||||
return Result("failed", source, str(error))
|
||||
|
||||
logger.info("copied %s", source)
|
||||
return Result("copied", mirror)
|
||||
|
||||
|
||||
def process(source, source_root, mirror_root, quality_args, dry_run):
|
||||
"""Bring one source file's mirror entry up to date."""
|
||||
mirror = mirror_path_for(source, source_root, mirror_root)
|
||||
if is_current(source, mirror):
|
||||
return Result("skipped", mirror)
|
||||
if source.suffix.lower() in COPY_EXTENSIONS:
|
||||
return copy(source, mirror, dry_run)
|
||||
return encode(source, mirror, quality_args, dry_run)
|
||||
|
||||
|
||||
def find_sources(root):
|
||||
"""Yield every audio file under a root, in a stable order."""
|
||||
extensions = SOURCE_EXTENSIONS | COPY_EXTENSIONS
|
||||
for path in sorted(root.rglob("*")):
|
||||
if path.is_file() and path.suffix.lower() in extensions:
|
||||
yield path
|
||||
|
||||
|
||||
def prune(source_root, mirror_root, dry_run):
|
||||
"""Delete mirror files whose source is gone, and any dirs left empty."""
|
||||
extensions = SOURCE_EXTENSIONS | COPY_EXTENSIONS
|
||||
removed = 0
|
||||
|
||||
for mirror in sorted(mirror_root.rglob(f"*{MIRROR_SUFFIX}")):
|
||||
relative = mirror.relative_to(mirror_root)
|
||||
stem = source_root / relative
|
||||
if any((stem.with_suffix(extension)).exists() for extension in extensions):
|
||||
continue
|
||||
removed += 1
|
||||
if dry_run:
|
||||
logger.info("would remove orphan %s", mirror)
|
||||
continue
|
||||
logger.info("removing orphan %s", mirror)
|
||||
mirror.unlink(missing_ok=True)
|
||||
|
||||
if not dry_run:
|
||||
# Deepest first, so a directory emptied by the loop above is caught.
|
||||
for directory in sorted(mirror_root.rglob("*"), reverse=True):
|
||||
if directory.is_dir() and not any(directory.iterdir()):
|
||||
directory.rmdir()
|
||||
|
||||
return removed
|
||||
|
||||
|
||||
def run_once(scan_root, source_root, mirror_root, quality_args, jobs, dry_run, do_prune):
|
||||
"""Run a single pass. Returns the number of failures.
|
||||
|
||||
`scan_root` is what gets walked and `source_root` is what mirror paths are
|
||||
computed against; they differ only for a partial pass over one directory.
|
||||
"""
|
||||
started = time.monotonic()
|
||||
counts = {"encoded": 0, "copied": 0, "skipped": 0, "failed": 0}
|
||||
failures = []
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool:
|
||||
futures = [
|
||||
pool.submit(process, source, source_root, mirror_root, quality_args, dry_run)
|
||||
for source in find_sources(scan_root)
|
||||
]
|
||||
for future in concurrent.futures.as_completed(futures):
|
||||
result = future.result()
|
||||
counts[result.action] += 1
|
||||
if result.action == "failed":
|
||||
failures.append(result)
|
||||
|
||||
removed = prune(source_root, mirror_root, dry_run) if do_prune else 0
|
||||
|
||||
for failure in failures:
|
||||
logger.error("failed: %s: %s", failure.path, failure.error)
|
||||
|
||||
logger.info(
|
||||
"pass complete in %.1fs: %d encoded, %d copied, %d up to date, %d removed, %d failed",
|
||||
time.monotonic() - started,
|
||||
counts["encoded"],
|
||||
counts["copied"],
|
||||
counts["skipped"],
|
||||
removed,
|
||||
counts["failed"],
|
||||
)
|
||||
return counts["failed"]
|
||||
|
||||
|
||||
def acquire_lock(mirror_root):
|
||||
"""Take an exclusive lock so two passes cannot run over one mirror."""
|
||||
mirror_root.mkdir(parents=True, exist_ok=True)
|
||||
handle = open(mirror_root / ".music-mirror.lock", "w") # noqa: SIM115 - held for the process
|
||||
try:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except OSError:
|
||||
handle.close()
|
||||
return None
|
||||
return handle
|
||||
|
||||
|
||||
def build_parser():
|
||||
"""Return the argument parser. Every option also reads an env var, so the
|
||||
container can be configured without a command line."""
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="music-mirror",
|
||||
description="Maintain a lossy MP3 mirror of a lossless music library.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--source",
|
||||
default=os.getenv("MUSIC_MIRROR_SOURCE"),
|
||||
help="root of the lossless library; never written to (env MUSIC_MIRROR_SOURCE)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mirror",
|
||||
default=os.getenv("MUSIC_MIRROR_MIRROR"),
|
||||
help="root of the MP3 mirror (env MUSIC_MIRROR_MIRROR)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--quality",
|
||||
default=os.getenv("MUSIC_MIRROR_QUALITY", "V0"),
|
||||
help="LAME VBR level (V0-V9) or a CBR bitrate in kbps (env MUSIC_MIRROR_QUALITY)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--jobs",
|
||||
type=int,
|
||||
default=int(os.getenv("MUSIC_MIRROR_JOBS", "0")) or (os.cpu_count() or 4),
|
||||
help="concurrent encodes (env MUSIC_MIRROR_JOBS; default: CPU count)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--interval",
|
||||
default=os.getenv("MUSIC_MIRROR_INTERVAL"),
|
||||
help="repeat forever, waiting this long between passes, e.g. 6h (env MUSIC_MIRROR_INTERVAL)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--subdir",
|
||||
default=None,
|
||||
help="limit the pass to one directory below --source; skips pruning",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-prune",
|
||||
action="store_true",
|
||||
help="keep mirror files whose source has been deleted",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="report what would change without touching the mirror",
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
"""Entry point. Returns a process exit code."""
|
||||
logging.basicConfig(format="%(asctime)s %(levelname)s %(message)s", level=logging.INFO)
|
||||
args = build_parser().parse_args(argv)
|
||||
|
||||
if not args.source or not args.mirror:
|
||||
logger.error("both --source and --mirror are required")
|
||||
return 2
|
||||
|
||||
source_root = Path(args.source).resolve()
|
||||
mirror_root = Path(args.mirror).resolve()
|
||||
|
||||
if not source_root.is_dir():
|
||||
logger.error("source %s is not a directory", source_root)
|
||||
return 2
|
||||
if mirror_root == source_root or mirror_root.is_relative_to(source_root):
|
||||
logger.error("mirror %s must not sit inside the source library", mirror_root)
|
||||
return 2
|
||||
|
||||
try:
|
||||
quality_args = parse_quality(args.quality)
|
||||
interval = parse_interval(args.interval) if args.interval else None
|
||||
except ValueError as error:
|
||||
logger.error("%s", error)
|
||||
return 2
|
||||
|
||||
scan_root = source_root
|
||||
do_prune = not args.no_prune
|
||||
if args.subdir:
|
||||
scan_root = (source_root / args.subdir).resolve()
|
||||
if not scan_root.is_relative_to(source_root) or not scan_root.is_dir():
|
||||
logger.error("--subdir %s is not a directory below the source", args.subdir)
|
||||
return 2
|
||||
# A partial pass cannot tell an orphan from a file outside its scope.
|
||||
do_prune = False
|
||||
|
||||
lock = acquire_lock(mirror_root)
|
||||
if lock is None:
|
||||
logger.error("another pass is already running over %s", mirror_root)
|
||||
return 3
|
||||
|
||||
stopping = False
|
||||
|
||||
def stop(signum, _frame):
|
||||
nonlocal stopping
|
||||
stopping = True
|
||||
logger.info("signal %d received; finishing the current pass", signum)
|
||||
|
||||
signal.signal(signal.SIGTERM, stop)
|
||||
signal.signal(signal.SIGINT, stop)
|
||||
|
||||
try:
|
||||
while True:
|
||||
failures = run_once(
|
||||
scan_root,
|
||||
source_root,
|
||||
mirror_root,
|
||||
quality_args,
|
||||
args.jobs,
|
||||
args.dry_run,
|
||||
do_prune,
|
||||
)
|
||||
if interval is None or stopping:
|
||||
return 1 if failures else 0
|
||||
logger.info("sleeping %ds", interval)
|
||||
for _ in range(interval):
|
||||
if stopping:
|
||||
return 1 if failures else 0
|
||||
time.sleep(1)
|
||||
finally:
|
||||
lock.close()
|
||||
|
||||
|
||||
def run():
|
||||
"""Console-script entry point."""
|
||||
sys.exit(main())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
{
|
||||
lib,
|
||||
python3Packages,
|
||||
makeWrapper,
|
||||
ffmpeg,
|
||||
}:
|
||||
python3Packages.buildPythonApplication {
|
||||
pname = "music-mirror";
|
||||
inherit ((lib.importTOML ./pyproject.toml).project) version;
|
||||
pyproject = true;
|
||||
|
||||
src = lib.fileset.toSource {
|
||||
root = ./.;
|
||||
fileset = lib.fileset.unions [
|
||||
./music_mirror.py
|
||||
./pyproject.toml
|
||||
./pytest.ini
|
||||
./tests
|
||||
];
|
||||
};
|
||||
|
||||
build-system = [ python3Packages.setuptools ];
|
||||
nativeBuildInputs = [ makeWrapper ];
|
||||
|
||||
# ffmpeg and ffprobe are called as subprocesses, so they belong on the
|
||||
# wrapper's PATH rather than in the Python environment.
|
||||
makeWrapperArgs = [
|
||||
"--prefix"
|
||||
"PATH"
|
||||
":"
|
||||
(lib.makeBinPath [ ffmpeg ])
|
||||
];
|
||||
|
||||
nativeCheckInputs = [
|
||||
python3Packages.pytestCheckHook
|
||||
ffmpeg
|
||||
];
|
||||
|
||||
meta = {
|
||||
description = "Maintain a lossy MP3 mirror of a lossless music library";
|
||||
longDescription = ''
|
||||
Reproduces a lossless library, path for path, as MP3 in a separate tree:
|
||||
tags and cover art carried across, sources that are already MP3 copied
|
||||
rather than re-encoded, and mirror files whose source has gone deleted.
|
||||
The source library is never written to.
|
||||
'';
|
||||
homepage = "https://code.emmathe.dev/lyrathorpe/music-mirror";
|
||||
mainProgram = "music-mirror";
|
||||
platforms = lib.platforms.unix;
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=77"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "music-mirror"
|
||||
version = "0.1.0"
|
||||
description = "Maintain a lossy MP3 mirror of a lossless music library"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
# No runtime Python dependencies: the work is done by ffmpeg, which must be on
|
||||
# PATH.
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
music-mirror = "music_mirror:run"
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://code.emmathe.dev/lyrathorpe/music-mirror"
|
||||
|
||||
[tool.setuptools]
|
||||
py-modules = ["music_mirror"]
|
||||
@@ -0,0 +1,5 @@
|
||||
[pytest]
|
||||
minversion = 7.0
|
||||
testpaths = tests
|
||||
python_files = test_*.py
|
||||
addopts = -q
|
||||
@@ -0,0 +1,80 @@
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
# Ensure the project root is on sys.path when running tests.
|
||||
ROOT = os.path.dirname(os.path.dirname(__file__))
|
||||
if ROOT not in sys.path:
|
||||
sys.path.insert(0, ROOT)
|
||||
|
||||
|
||||
@pytest.fixture(scope="session", autouse=True)
|
||||
def require_ffmpeg():
|
||||
"""The tests exercise real encodes; there is little point faking them."""
|
||||
for tool in ("ffmpeg", "ffprobe"):
|
||||
if shutil.which(tool) is None:
|
||||
pytest.skip(f"{tool} is not on PATH", allow_module_level=True)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def make_flac():
|
||||
"""Return a factory writing a short tagged FLAC file."""
|
||||
|
||||
def factory(path, title="Test Title", artist="Test Artist", album="Test Album", seconds=1):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg",
|
||||
"-nostdin",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
f"sine=frequency=440:duration={seconds}",
|
||||
"-metadata",
|
||||
f"title={title}",
|
||||
"-metadata",
|
||||
f"artist={artist}",
|
||||
"-metadata",
|
||||
f"album={album}",
|
||||
"-metadata",
|
||||
"track=3",
|
||||
str(path),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
return path
|
||||
|
||||
return factory
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def probe_tag():
|
||||
"""Return a helper reading a single metadata tag from a file."""
|
||||
|
||||
def reader(path, tag):
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-show_entries",
|
||||
f"format_tags={tag}",
|
||||
"-of",
|
||||
"default=noprint_wrappers=1:nokey=1",
|
||||
str(path),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
return completed.stdout.strip()
|
||||
|
||||
return reader
|
||||
@@ -0,0 +1,228 @@
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
import music_mirror
|
||||
|
||||
|
||||
def run(source, mirror, *extra):
|
||||
return music_mirror.main(["--source", str(source), "--mirror", str(mirror), *extra])
|
||||
|
||||
|
||||
def test_parse_quality_accepts_vbr_and_cbr():
|
||||
assert music_mirror.parse_quality("V0") == ["-q:a", "0"]
|
||||
assert music_mirror.parse_quality("v2") == ["-q:a", "2"]
|
||||
assert music_mirror.parse_quality("256") == ["-b:a", "256k"]
|
||||
|
||||
|
||||
def test_parse_quality_rejects_nonsense():
|
||||
with pytest.raises(ValueError):
|
||||
music_mirror.parse_quality("best")
|
||||
|
||||
|
||||
def test_parse_interval_units():
|
||||
assert music_mirror.parse_interval("90") == 90
|
||||
assert music_mirror.parse_interval("30m") == 1800
|
||||
assert music_mirror.parse_interval("6h") == 21600
|
||||
assert music_mirror.parse_interval("1d") == 86400
|
||||
|
||||
|
||||
def test_encodes_and_preserves_layout_and_tags(tmp_path, make_flac, probe_tag):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Test Artist" / "Test Album" / "03 Song.flac")
|
||||
|
||||
assert run(source, mirror) == 0
|
||||
|
||||
output = mirror / "Test Artist" / "Test Album" / "03 Song.mp3"
|
||||
assert output.is_file()
|
||||
assert probe_tag(output, "title") == "Test Title"
|
||||
assert probe_tag(output, "album") == "Test Album"
|
||||
|
||||
|
||||
def test_output_is_mp3(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
codec = subprocess.run(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"a:0",
|
||||
"-show_entries",
|
||||
"stream=codec_name",
|
||||
"-of",
|
||||
"csv=p=0",
|
||||
str(mirror / "a.mp3"),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout.strip()
|
||||
assert codec == "mp3"
|
||||
|
||||
|
||||
def test_second_pass_skips_unchanged_files(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
first = (mirror / "a.mp3").stat().st_mtime_ns
|
||||
|
||||
run(source, mirror)
|
||||
assert (mirror / "a.mp3").stat().st_mtime_ns == first
|
||||
|
||||
|
||||
def test_changed_source_is_re_encoded(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
track = make_flac(source / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
before = (mirror / "a.mp3").stat().st_mtime
|
||||
|
||||
# A replaced file with a newer mtime is what a Lidarr quality upgrade
|
||||
# looks like on disk.
|
||||
later = time.time() + 120
|
||||
os.utime(track, (later, later))
|
||||
run(source, mirror)
|
||||
|
||||
assert (mirror / "a.mp3").stat().st_mtime > before
|
||||
|
||||
|
||||
def test_deleted_source_is_pruned(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
make_flac(source / "Album" / "b.flac")
|
||||
|
||||
run(source, mirror)
|
||||
(source / "Album" / "b.flac").unlink()
|
||||
run(source, mirror)
|
||||
|
||||
assert (mirror / "Album" / "a.mp3").is_file()
|
||||
assert not (mirror / "Album" / "b.mp3").exists()
|
||||
|
||||
|
||||
def test_emptied_directory_is_removed(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Gone" / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
(source / "Gone" / "a.flac").unlink()
|
||||
run(source, mirror)
|
||||
|
||||
assert not (mirror / "Gone").exists()
|
||||
|
||||
|
||||
def test_no_prune_keeps_orphans(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "a.flac")
|
||||
|
||||
run(source, mirror)
|
||||
(source / "a.flac").unlink()
|
||||
run(source, mirror, "--no-prune")
|
||||
|
||||
assert (mirror / "a.mp3").is_file()
|
||||
|
||||
|
||||
def test_existing_mp3_is_copied_not_re_encoded(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
flac = make_flac(source / "a.flac")
|
||||
subprocess.run(
|
||||
["ffmpeg", "-loglevel", "error", "-y", "-i", str(flac), str(source / "b.mp3")],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
flac.unlink()
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
assert (mirror / "b.mp3").read_bytes() == (source / "b.mp3").read_bytes()
|
||||
|
||||
|
||||
def test_dry_run_writes_nothing(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "a.flac")
|
||||
|
||||
assert run(source, mirror, "--dry-run") == 0
|
||||
assert not (mirror / "a.mp3").exists()
|
||||
|
||||
|
||||
def test_subdir_limits_the_pass_and_does_not_prune(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "One" / "a.flac")
|
||||
make_flac(source / "Two" / "b.flac")
|
||||
|
||||
assert run(source, mirror, "--subdir", "One") == 0
|
||||
|
||||
assert (mirror / "One" / "a.mp3").is_file()
|
||||
# Outside the requested directory: neither encoded nor treated as an orphan.
|
||||
assert not (mirror / "Two").exists()
|
||||
|
||||
|
||||
def test_external_cover_is_embedded(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
mirror = tmp_path / "dst"
|
||||
make_flac(source / "Album" / "a.flac")
|
||||
subprocess.run(
|
||||
[
|
||||
"ffmpeg",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-y",
|
||||
"-f",
|
||||
"lavfi",
|
||||
"-i",
|
||||
"color=c=red:s=64x64:d=1",
|
||||
"-frames:v",
|
||||
"1",
|
||||
str(source / "Album" / "cover.jpg"),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
run(source, mirror)
|
||||
|
||||
streams = subprocess.run(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"v",
|
||||
"-show_entries",
|
||||
"stream=codec_name",
|
||||
"-of",
|
||||
"csv=p=0",
|
||||
str(mirror / "Album" / "a.mp3"),
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout.strip()
|
||||
assert streams # a picture stream is present
|
||||
|
||||
|
||||
def test_mirror_inside_source_is_refused(tmp_path, make_flac):
|
||||
source = tmp_path / "src"
|
||||
make_flac(source / "a.flac")
|
||||
assert run(source, source / "mp3") == 2
|
||||
|
||||
|
||||
def test_missing_source_is_refused(tmp_path):
|
||||
assert run(tmp_path / "nope", tmp_path / "dst") == 2
|
||||
Reference in New Issue
Block a user