Strip old dubtitle at mux + isolate it from pipeline context #4
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "feat/strip-and-isolate-old-dubtitles"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What
The pipeline embedded its output as a subtitle track named
Dubtitlesand then treated that track's presence as "done" — so a regeneration silently no-op'd unless a separate heavy pre-strip pass ran first. Worse, the track iscodec=ass, language=eng, so every stage that reads the fansub as context picked it up as if it were the human fansub and repaired / aligned / mined the new run against the previous run's own mistakes.Context isolation — one shared marker,
common.TRACK_NAME:common.eng_sub_streams()now queriesstream_tags=language,titleand drops any stream titledDubtitles. It is the single chokepoint forrepair.py,tools/timing_compare.pyanddub_signs_merge.py, which inherit the fix with no code change.mine_glossary.eng_sub_text()has its own selector and got the same exclusion.Strip-at-mux:
mux.keep_sub()drops any track namedTRACK_NAMEas its first check, sobuild_cmd's single mkvmerge pass drops the old track and appends the new one — a replace, not a duplicate, and self-healing if a past run left several.mux.process()skips on the stamp alone; the ffprobe "already-muxed" backstop is gone (re-mux is now idempotent).generate.py'sSKIP_IF_MUXEDguard and itshas_dubtitles_track()go with it.Regeneration trigger:
common.PIPELINE_VERSION(1) is recorded in.dubtitles.done;stamp_valid()rejects anything older.GRANDFATHER_VERSION(1) covers pre-versioning stamps, so the rollout regenerates nothing. BumpingPIPELINE_VERSIONis the operator's deliberate signal for a global in-place regeneration.scripts/migrate_write_v1_stamps.pystamps files that carry aDubtitlestrack but no stamp (dry-run by default), so removing the backstop doesn't trigger a mass regeneration on deploy.Review findings fixed in this branch
Two review passes ran against the first cut; both found real defects, each now covered by a test:
-sis a whitelist and mkvmerge's default is copy-every-subtitle-track. Omitting-son an empty keep list copied back the very trackdroppedreported as removed — so every mp4-origin episode, and the "only English sub is ours" case, would have ended up with twoDubtitlestracks.verify()is presence-only, so it would have passed and been stamped. An empty keep list now emits an explicit-S. (The test covering that path had asserted the broken command shape.)generatereturnalready-assforever whilemuxre-embedded the old subtitle and stamped it current. Fixed viacommon.stale_version_stamp()+generate.discard_stale_sidecars(). A second pass caught that the first cut of that fix deleted freshly transcribed sidecars — so a sidecar now counts as a leftover only if it predates the stamp, and.dubtitles.failepisodes are exempt.common.is_our_track()is now the one predicate for both track shapes (ffprobetags.title, mkvmergeproperties.track_name) so the exclude-from-context and drop-at-mux tests can't drift.write_stampreturned a bareerrorafter the remux had already landed — a silent per-sweep full remux, forever. Nowstamp-write-failed, logged loudly, sidecars kept for retry.stamp_valid()no longer raises on a non-intversion(that check runs outsidemux.process()'stry, so one corrupt sidecar aborted a whole sweep);properties: Nonehardening acrossmux.Testing
395 tests pass;
ruffclean on the CI target (now includingcommon.py,mine_glossary.py,scripts/). Every new guard was verified honest by reverting its hunk and watching the test fail.Still pending — needs real media on the server:
mux.pyon an already-dubbed episode; expectdrop-tracks=['sub:Dubtitles(old)'].--apply, confirmmkvmerge -Jshows exactly oneDubtitlestrack. Use an mp4-origin episode — that's the shape the-Sbug hid in.scripts/migrate_write_v1_stamps.py(dry run, then--apply) before deploying, so no-stamp files don't mass-regenerate.PIPELINE_VERSIONon a test show and confirm in-place regeneration.Notes
strip_op.py(referenced in the spec's non-goals) does not exist in this repo and never has — nothing to deprecate.Dubtitlestrack is appended last, so if an old one sat mid-list the dubtitle moves to the end. Players use the default-track flag; index-based scripts may see a different position. Documented in the README.🤖 Generated with Claude Code
repair.llm_llamacpp posts a RAW prompt to /completion, which applies no chat template. Verified against the live Nanbeige 4.2-3B server: that path returns nothing but newlines -- 200 tokens of "\n" -- because the instruct model never sees its template. This is a latent bug in repair.py's llamacpp backend, not just a bake-off gap; it can only work for a base/completion model. --llamacpp-chat routes a candidate through /v1/chat/completions instead, which applies the template, and passes chat_template_kwargs {enable_thinking: false}. Both are required for this fork: with the template but thinking still on, the model fills reasoning_content and returns an empty message (measured: empty after 114s at max_tokens=512; correct output in 4.3s with thinking off). An empty reply is surfaced as "<EMPTY thinking not disabled?>" rather than "" -- an empty string would score as "the model correctly left the line unchanged", turning a broken configuration into a silent perfect no-op. --llamacpp (raw) is kept so the shipping configuration can still be measured as-is. 414 tests pass; ruff clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>Measured on real conf.json targets across three shows, qwen3.5:9b was rewriting a large fraction of lines under the old prompt and pasting glossary names over text that was already correct: This is Border Control. -> This is Cipher Pol. Hey, Sonny, -> Hey, Shanks, Oh, Neptune! -> Oh, Nefertari Vivi! (Neptune was already right) Straw Hat was innocent. -> Straw Hat Pirates were innocent. Uchihime! -> Uchiha! (a name from another franchise) The old prompt already contained "NEVER replace a name in the line with a different name", so restating the rule is not what fixes this. Two alternatives were tested and rejected: stating it more forcefully, and removing the glossary from the prompt entirely (still 38% -- the name list is NOT the trigger, and its removal made the conservative model worse, 8% -> 18%). What worked was structure: * the name list framed as VERIFICATION ONLY, not as material to apply * two worked examples -- one correcting a misspelling, one LEAVING an unlisted name alone, which is the behaviour prose failed to secure * nothing after the ASR line; context and reference now precede it That last point is its own bug: an intermediate version put a trailing "Remember:" reminder after the ASR line and the model echoed the rule text straight into the subtitle output ("...friends in jail We're victims here, Remember: do not introduce or swap names..."). A regression test now pins that nothing follows the ASR line. Edit rate, old -> new (qwen3.5:9b, 40 targets each): One Pace 42% -> 25% Jujutsu Kaisen 28% -> 22% Delicious in Dungeon 22% -> 15% Every glossary-name fabrication above is gone; the good repairs (human -centric -> human-centric, sentence-boundary splits) survive. NOT fixed: the model still invents phonetic names ("Syrahose" -> "Shyarros" on one line and "Shirahoshi" on another -- same token, two answers). Prompt tuning has plateaued there; tracked separately for a deterministic guard. Also carries the bakeoff tooling used to measure this: --conf input, llama.cpp raw and chat backends, and per-model incremental output so a timeout cannot discard completed results. 421 tests pass; ruff clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>Goal: make a 3B model viable so the pipeline consolidates onto one small model that others can actually run. Two things blocked that. 1. THE PROMPT. The previous one balanced badly across models: qwen3.5:9b told only what NOT to do -> rewrote 42% of lines, pasting glossary names over correct text (Neptune -> "Nefertari Vivi", Uchihime -> "Uchiha", a name from a different franchise) nanbeige4.2-3b same prohibitions -> inert. 0 safe fixes across 120 targets, returning input verbatim, losing the real repairs it had made A 6-variant sweep (3 shows x 40 targets, temperature 0, GPU residency verified before each run) found one prompt that fixes both. Measured, 120 targets: qwen3.5:9b safe fixes 6 -> 23 name edits 17 -> 14 nanbeige4.2-3b safe fixes 0 -> 16 name edits 1 -> 2 zero prompt leaks or length blowups for either What did it: - explicit POSITIVE DUTY. Rules phrased only as prohibitions produce a model that does nothing, which is not a repair stage. - REMOVING the worked example that showed a name being left alone. Counter-intuitive, but it over-anchored inaction; removing it was the single biggest gain AND name edits fell for both models. - dropping the blanket "if unsure, return UNCHANGED" escape, which a cautious model takes on every line. Restraint now attaches specifically to proper nouns, where the damage actually was. 2. THE LLAMA.CPP BACKEND, which nanbeige needs, was broken. llm_llamacpp posted a RAW prompt to /completion, applying no chat template; against a live Nanbeige server that returns nothing but newlines (200 tokens of "\n"). It could only ever have worked for a base/completion model. Now uses /v1/chat/completions with chat_template_kwargs.enable_thinking=false -- both required for this fork, which otherwise fills reasoning_content and returns an empty message (empty after 114s at max_tokens=512; correct output in 4.3s with thinking off). An empty reply is treated as "no repair", never as text to embed. REPAIR_LLAMACPP_URL default updated to the chat endpoint. The pre-existing test pinned the raw-/completion body, i.e. pinned the broken shape; updated with the reason. Verified end to end: REPAIR_BACKEND=llamacpp against the live server repairs "it worked Now we run" -> "It worked. Now we run.", fixes "human -centric", and leaves "Sonny" alone. Not fixed: qwen still invents names under this prompt (14 edits/120, incl. one glossary paste, "catch Hirohoshi" -> "catch Crocodile"). nanbeige makes 2 name edits/120. That gap, plus 4.13 GB vs 6.59 GB and ~3.4x faster, is the case for consolidating on nanbeige. 425 tests pass; ruff clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>39 of the library's 79 releases ship more than one English ASS track, and the same signs are typeset into each of them -- a full dialogue track, a signs/songs track, often a CC track. build() read all of them, so every sign rendered two or three times a few pixels apart. One Pace S01E01 carried the episode title three times over. Add common.signs_sub_streams(): when any track names itself as the signs track, that one is the release's curated signs list and the only one read; several rival ones (DAN DA DAN ships MALD's and CR's) collapse to the first. Titles that say nothing useful ("inid4c + SFX") fall back to the historic scan of every track classified per event, which is how a single mixed dialogue+signs track has always worked. SIGNS_TITLE covers the spellings actually present in the library, taken from a probe of all 79 releases: "Signs & Songs", "S&S", "Signs/Songs", "English[Signs]", "Songs + Signs", "Signs and Songs(Hydes)", and "Forced" (a forced track is signs plus foreign-dialogue captions, and releases that ship ['Forced', ''] mean it as the signs track). Not "karaoke" -- FLE's "Karaoke / English / ASS / FLE" is one mixed track, not a signs-only one. eng_sub_streams() keeps its exact contract for repair.py and timing_compare.py, which want the DIALOGUE track; both now share one probe via eng_sub_tracks() so the is_our_track() exclusion cannot drift. Verified on the real One Pace mkv: 80 sign events across three tracks -> 40 from the signs track alone, one title instead of three. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>The ollama/llamacpp dispatch lived only in repair.py, so glossary_verify could reach nothing but Ollama -- the pipeline had to keep a second model resident on a shared 8 GB GPU purely for adjudication. Hoist the client into common.llm_chat(), with the llama.cpp specifics the repair backend had to learn the hard way: /v1/chat/completions rather than /completion (which applies no chat template -- a templated instruct model answers it with nothing but newlines), enable_thinking=false (or the fork spends its whole budget on reasoning_content and returns an empty message), and no model selector (one model per server). Two knobs adjudication needs that repair does not: max_tokens, since a JSON object needs a real budget where a subtitle line needs 80; and first_line=False, since taking the first line of a JSON object truncates it to its opening brace. VERIFY_BACKEND/VERIFY_LLAMACPP_URL default to the REPAIR_* values, so pointing repair at a backend moves adjudication with it unless overridden. Checked against the live servers on four real One Pace adjudications rather than assumed: nanbeige and qwen3:8b agree on all four, and nanbeige answers the hard one in 1.2s against qwen's 16.1s. Small sample, and the one case they differ on ("Zolo" -> "Zolo" vs "Roronoa Zoro") is arguably right both ways, since this stage picks the DUB-preferred spelling. repair.py keeps its own dispatch for now -- it is the quality-critical path and a library-wide regeneration is mid-flight; folding it onto llm_chat() is a follow-up. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>The reference exists to disambiguate a garbled ASR line, not to supply its text. A dubtitle has to match the DUB AUDIO, so a repair that swaps the dub's phrasing for the fansub's yields a subtitle that reads perfectly and is wrong against the sound -- the worst failure available here, because nothing about it looks broken. Measured across every repair summary the library had accumulated: qwen3:8b 2520 repairs -- 84.1% imported words from the reference (29.2% imported 3+) nanbeige 8456 repairs -- 52.5% imported words from the reference (17.1% imported 3+) Both models do it; qwen far worse. Real lines that shipped: "That's enough of that, idiots!" -> "Hold it, you brats!" (the reference, verbatim) "Let's go, Chopper." -> "Well then, shall we go, Chopper?" The only gate was a 0.4-2.5 length band, which a same-length rewrite sails straight through, and which admitted "Huh?" -> "Huh? Help!" at exactly 2.5. accept_repair() now also rejects a repair importing MAX_REF_BORROW (3) or more NEW words present in the reference, and tightens the band to 0.6-1.5. Both env-tunable so they can be adjusted without rebuilding the image. Rejections are counted into the per-episode summary as rejected_guard, so the guard's effect stays visible rather than silently shrinking the numbers. Replayed over the 10971 repairs already on disk: blocks 26.0% of nanbeige's and 37.1% of qwen's, while keeping 2733 of 2736 punctuation-only fixes and name corrections like "Naya Yaskirihara" -> "Kirihara Naoyasu". The known cost is a 3-word proper-noun correction ("Chinlong Tong" -> "Qing Long Tang") now being refused; the deterministic glossary layer is the right place for those anyway. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>Re-consulted the panel after fixing salyut's groq provider (it had been 404ing on a decommissioned model, so the first review was two voices). gpt-oss-120b answered substantively, and while its specific numbers were invented -- it asserted JW("Syrahose","Shirahoshi")=0.92 against a measured 0.755, and built a scenario where R1 admits a non-title -- two of its structural points held up and led somewhere better than either panel got on its own. Measuring every pair the design must match against every near-miss it must refuse shows the two classes overlap completely: every true pair scores LOWER than every false one (Syrahose/Shirahoshi 0.755 vs Warlords/Warlord 0.975). No metric separates them, so the reviewer's "use phoneme-weighted Levenshtein" doesn't help either. Similarity is a recall device; R2 and R3 carry all the safety. That also caught a bug in v2 of this spec: ACQUIRE_MIN_SIM was 0.88, which would have rejected both motivating cases outright. It is 0.72 now. The self-read question that two panels failed to answer turns out to have a concrete answer, and it doesn't run through the canonical string where the wiki gate sits. The glossary is additive and never pruned, and it also seeds Whisper's initial_prompt -- so a wrong entry biases transcription, which raises that spelling's count, which strengthens the very dominance test that admitted it. The gate governs step one; the rest of the loop is closed inside our own output, looking better evidenced each pass. Mitigated by provenance in `acquired`, keeping acquired names out of initial_prompt, and a --revert path.propose() runs best_title + decide() over every harvested token, emitting one proposal per variant that resolves to a wiki title (unmatched tokens are the tier-B queue's business, not a silent drop here). Amendment: the brief's below-floor assertion for "Hirohoshi" was written against an earlier decide() whose floor used the variant count alone; the shipped decide() sums the cluster total (1 + 56 = 57), clearing both the floor and Wilson dominance, so that case now asserts apply/dominant. Amendment: added an english-word gate so an ordinary capitalised English word (e.g. "Name") can never be hard_fixed into a phonetically similar character name (JW("name","nami") = 0.90 clears MIN_SIM and decide() alone would apply it). The gate flags rather than drops, since real characters can also be English words (Brook, Law) and a flag stays recoverable by a human reviewer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>bool(d.get("same_entity")) treated any non-empty string ("no", "false") as truthy, turning a model's textual "no" into a merge -- exactly the false merge tier C exists to prevent. Now only the literal JSON `true` counts; anything else (including 0/1/strings) means "do not merge", since a miss is recoverable but a false merge silently rewrites dialogue. Also corrects escalate()'s docstring: would-expand was retired in an earlier task and no longer exists as a reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>The final-review reproduced this table against real data and found two rows wrong and two right for the wrong reason: - Brooke->Brook now flags (english-word gate: is_english("brooke") is True on cracklib-small/wamerican), not applies. - Warlords is rejected via tier B (never resolves a title at tier 0 - similarity 0.627 < MIN_SIM), not via R2/tier 0 as previously claimed. - Smokey/Smoker now reaches tier C (LLM merge adjudication), so the outcome depends on the LLM, not the 0.409 bound alone. - Surrender/Maybe/Hurry/Listen are not "ignored" - they clear the frequency floor and land in flagged (tier A or tier B), same as any other proposal. Also records that WORDLIST_PATH=/usr/share/cracklib/cracklib-small is required to exercise the english-word gate at all on this laptop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>Pull request closed