Timing Compare (Phase 0): dubtitle-vs-subtitle timing + VAD analysis tool #3

Merged

xenarathon merged 9 commits from feat/timing-compare into main

2026-07-24 21:41:27 -04:00

xenarathon commented

2026-07-24 21:41:25 -04:00

Owner

Summary

Phase-0 analysis tool (tools/timing_compare.py) that compares Whisper dubtitle card timing
against the embedded human-timed dialogue subtitle cues — to measure timing alignment and quantify how
many silence/music hallucinations leak into the output, and (the key part) to distinguish those from
real dub-only lines the subs omit via an independent audio VAD. Read-only, GPU-free, non-invasive —
it changes nothing about how dubtitles are generated. Design brainstormed + panel-reviewed
(specs/timing-compare/).

What's in it

Refactor (behavior-preserving): hoist dialogue_intervals from repair.py → common.py as
dialogue_intervals(video, stream_indices=None); add dialogue_event_count,
dialogue_density_score, is_dialogue_event. repair.py re-imports — pinned identical by tests.
tools/timing_compare.py: CLI; conf.json load + hardening; dialogue-track auto-selection;
RANSAC offset+drift linear fit (handles PAL/framerate drift, not just a constant offset);
slack-aware on-cue/in-gap classification.
tools/vad.py: independent CPU VAD (webrtcvad + ffmpeg-silencedetect fallback, both guarded);
probes the dub audio at each in-gap card's un-aligned Whisper timebase → splits in_gap_silent
(confident hallucination) vs in_gap_speech (real dub-only line a naive gate would drop).
Report (schema_version 2): per-episode/show/overall; applicability_ratio, false_in_gap_rate
(the Phase-1 gate-safety headline), band buckets matching hallucination.py, null-safe aggregates.
webrtcvad added as an optional analysis extra + Dockerfile.builder (won't affect core install).

Notable decisions (from the multi-model panel consult)

RANSAC linear fit over a constant offset — a whole-episode median mis-classifies episode ends on
PAL 25↔23.976 sources.
Independent VAD — a subtitle gap alone can't tell a hallucination from a real dub-only line; the
VAD is what makes the eventual Phase-1 gate safe.
kept_in_gap framed as a "leaked-past-B1" floor (survivorship); Phase-1 trigger recalibrated to
false_in_gap_rate, not %-on-cue.

Test plan

pytest -q → 334 passed (193 → 334; +141 across 4 units, all pure-function/hermetic). Run via
rtk proxy python -m pytest tests/. ruff check . → 0 whole-tree. tools/timing_compare.py --help
imports clean, GPU-free (no faster_whisper).
PENDING (needs the server): the real-show calibration run (T12), real webrtcvad (no py3.14
wheel here), and the ffmpeg audio/sub extraction seams. That run sets the Phase-1
false_in_gap_rate threshold and is where the tool proves out on real data.

🤖 Generated with Claude Code

## Summary Phase-0 **analysis tool** (`tools/timing_compare.py`) that compares Whisper dubtitle card timing against the embedded human-timed dialogue subtitle cues — to measure timing alignment and quantify how many silence/music hallucinations leak into the output, and (the key part) to distinguish those from real dub-only lines the subs omit via an independent audio VAD. **Read-only, GPU-free, non-invasive** — it changes nothing about how dubtitles are generated. Design brainstormed + panel-reviewed (`specs/timing-compare/`). ## What's in it - **Refactor (behavior-preserving):** hoist `dialogue_intervals` from `repair.py` → `common.py` as `dialogue_intervals(video, stream_indices=None)`; add `dialogue_event_count`, `dialogue_density_score`, `is_dialogue_event`. `repair.py` re-imports — pinned identical by tests. - **`tools/timing_compare.py`:** CLI; conf.json load + hardening; dialogue-track auto-selection; **RANSAC offset+drift linear fit** (handles PAL/framerate drift, not just a constant offset); slack-aware on-cue/in-gap classification. - **`tools/vad.py`:** independent CPU VAD (webrtcvad + ffmpeg-silencedetect fallback, both guarded); probes the dub audio at each in-gap card's **un-aligned** Whisper timebase → splits `in_gap_silent` (confident hallucination) vs `in_gap_speech` (real dub-only line a naive gate would drop). - **Report (schema_version 2):** per-episode/show/overall; `applicability_ratio`, `false_in_gap_rate` (the Phase-1 gate-safety headline), band buckets matching `hallucination.py`, null-safe aggregates. - `webrtcvad` added as an optional `analysis` extra + `Dockerfile.builder` (won't affect core install). ## Notable decisions (from the multi-model panel consult) - **RANSAC linear fit** over a constant offset — a whole-episode median mis-classifies episode ends on PAL 25↔23.976 sources. - **Independent VAD** — a subtitle gap alone can't tell a hallucination from a real dub-only line; the VAD is what makes the eventual Phase-1 gate safe. - **`kept_in_gap` framed as a "leaked-past-B1" floor** (survivorship); Phase-1 trigger recalibrated to `false_in_gap_rate`, not %-on-cue. ## Test plan - `pytest -q` → **334 passed** (193 → 334; +141 across 4 units, all pure-function/hermetic). Run via `rtk proxy python -m pytest tests/`. `ruff check .` → 0 whole-tree. `tools/timing_compare.py --help` imports clean, GPU-free (no `faster_whisper`). - **PENDING (needs the server):** the real-show calibration run (T12), real `webrtcvad` (no py3.14 wheel here), and the ffmpeg audio/sub extraction seams. That run sets the Phase-1 `false_in_gap_rate` threshold and is where the tool proves out on real data. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

xenarathon added 9 commits

2026-07-24 21:41:25 -04:00

spec(timing-compare): dubtitle-vs-subtitle timing analysis (Phase 0) a3bb3d470d

GPU-free measurement tool: compares Whisper dubtitle timing against the embedded
human-timed dialogue sub track to validate alignment and quantify silence/music
hallucinations that leak into the output. Analysis only; the gating feature is
Phase 1 (out of scope). Design brainstormed 2026-07-24.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

spec(timing-compare): v3 folds in panel consult (RANSAC drift + VAD probe) 3bf5df4283

Panel consult (salyut, 2026-07-24) found methodology gaps v2 still had:
- constant offset is wrong for PAL/framerate drift -> RANSAC linear offset+drift fit
- a subtitle gap alone can't tell a hallucination from a real dub-only line ->
  independent CPU VAD probe of the dub audio splits in_gap_silent vs in_gap_speech
  (in_gap_speech = false_in_gap_rate = the real speech a naive gate would drop)
- survivorship: kept_in_gap is a leaked-past-B1 floor, reframed
- recalibrated Phase-1 trigger around false_in_gap_rate, not %-on-cue

Adds spec-v3.md (supersedes v1/v2) + updates plan.md/tasks.md. Also captures the
prior minimax-m3 review artifacts (REVIEW.md, spec-v2.md).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

refactor(T1): hoist dialogue_intervals from repair.py into common.py cbb3d9ea7b

Behavior-preserving hoist for the timing-compare tool (T2+): dialogue_intervals()
moves to common.py with a new stream_indices=None default that reproduces
repair.py's exact pre-refactor all-English-stream behavior, plus two new pure
helpers (dialogue_event_count, dialogue_density_score) built on a shared
is_dialogue_event() predicate extracted from the original karaoke/positioned/
style regex checks (unchanged, just relocated). repair.py now imports
dialogue_intervals from common instead of defining it locally.

Locked with hermetic tests: synthetic pysubs2.SSAEvent fixtures pin the
dialogue-vs-sign selection logic, and a monkeypatched extract_sub/
eng_sub_streams pipeline (same pattern test_dub_signs_merge.py uses) exercises
dialogue_intervals' extraction+merge+sort assembly without touching ffmpeg or
real media.

feat(timing-compare/T2-T6): tool core -- CLI, conf.json, track selection, RANSAC fit, classification 9aad3cb128

Builds tools/timing_compare.py's analytical core against specs/timing-compare/spec-v3.md:
argparse CLI + show-dir walk (T2), conf.json load/hardening (T3), English sub-stream
extraction + dialogue-density track selection via common.py's T1 helpers (T4), a
pure-Python (no new numpy dep) RANSAC offset+drift line fit over nearest-onset card/cue
pairs (T5), and slack-aware on-cue/in-gap overlap classification applied through the
fitted model (T6). process_episode() wires T3-T6 into the per-episode data shape U3
(VAD split) and U4 (report/aggregates) will consume; --vad/--vad-aggressiveness are
accepted but not yet wired (U3), and --out isn't yet written (U4) -- both TODO'd inline.

41 new tests cover every pure function (pairing incl. tie-breaks, RANSAC incl. constant-
offset/pure-drift/drift+outliers/null-guard/look_for_drift cases, overlap classification
incl. boundaries) plus conf.json hardening and the walk, hermetically (tmp_path, no
ffmpeg). select_reference_track()'s real ffprobe/ffmpeg calls have no media available in
this environment; process_episode()'s orchestration around it is covered by
monkeypatching select_reference_track, matching tests/test_common.py's convention for
extract_sub -- the ffmpeg subprocess calls themselves are PENDING manual verification.

Bundled as one commit (not one per T2-T6) since the tasks are tightly coupled -- T5/T6
depend on T4's data shapes and were built/tested as a single coherent unit per the U2
work order.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

fix(timing-compare): keep blank lang token in --lang set to match common.SUB_LANGS 70a98b8265

main() built the --lang set with `if s.strip()`, dropping the blank token that
common.SUB_LANGS's default ("eng,en,und,") includes. common.eng_sub_streams
treats an untagged subtitle stream's language as "", so this tool was stricter
than the rest of the pipeline and wrongly marked untagged-only episodes as
no-reference, inflating U4's applicability_ratio no-reference count.

Mirror common.SUB_LANGS's construction (no strip-filter) so the default and any
explicit --lang keep "" the same way, and add regression tests covering both.

feat(timing-compare/T7): VAD probe of in-gap cards -- tools/vad.py + wiring 6558b636c8

Adds the independent acoustic check spec-v3.md's "[v3] VAD probe of in-gap cards"
requires: a kept card sitting in a subtitle gap could be a hallucination over
silence/music, or a real dub-only line the subs omit -- the subtitle track alone
can't tell them apart, only a VAD probe of the dub audio can.

tools/vad.py (new): pure decision core (voiced_ratio_to_verdict, frame_pcm,
parse_silencedetect_output, all unit-tested with synthetic data/text, no I/O) plus
two backends behind vad_probe(wav_path, aggressiveness, backend): "webrtcvad"
(frames a 16k mono window into 10/20/30ms chunks, runs webrtcvad.Vad per frame) and
"ffmpeg-silencedetect" (dep-free energy-based fallback, cruder -- documented). Both
return bool | None; None (never a guessed verdict) on any failure -- missing wav,
unavailable backend, subprocess/parse error.

tools/timing_compare.py: classify_in_gap_cards() extracts each in-gap card's dub-
audio window via a new extract_audio_window() at the card's ORIGINAL Whisper-
timebase [start, end] (never the RANSAC-aligned timestamps -- the offset/drift model
is only for cue comparison) into a per-episode tempdir, probes it, and sets
card["in_gap_vad_verdict"] to in_gap_silent/in_gap_speech/in_gap_vad_error. on-cue
cards are left untouched (no key added) -- the VAD path only adds information, it
can never change T6's classification. select_audio_stream() mirrors generate.py's
eng_audio_index() (same tag preference/REQUIRE_ENG fallback) without importing
generate.py, which pulls in faster_whisper at module scope -- this tool's GPU-free
acceptance criterion forbids that.

*** ENVIRONMENT REALITY *** webrtcvad does not install in this repo's py3.14 dev
venv (no prebuilt wheel; see tools/vad.py's module docstring). The import is
guarded; real webrtcvad decisions and the ffmpeg-window-extract-on-real-video seam
are PENDING manual verification on the server, same status as T4's
select_reference_track. tests/test_vad.py's webrtcvad-backend tests run against a
STUB module (monkeypatched, mirrors test_generate.py's faster_whisper stub) -- they
verify OUR framing/call-shape/aggregation code, not real webrtcvad classification.

45 new tests (37 in tests/test_vad.py, 8 in tests/test_timing_compare.py); all 302
tests pass; ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(timing-compare/T8): add webrtcvad dependency (analysis extra + builder image) 3ab7e85366

pyproject.toml: new [project.optional-dependencies] "analysis" extra
(webrtcvad>=2.0), separate from "dev" so `pip install -e .[dev]` keeps working in
environments (like this repo's py3.14 dev venv) where webrtcvad has no prebuilt
wheel -- tools/vad.py guards the import and falls back to `--vad
ffmpeg-silencedetect` regardless.

Dockerfile.builder: add webrtcvad to the pip-install line alongside pysubs2/
jellyfish. tools/ itself isn't COPY'd into this image (offline analytics, run
standalone -- same as tools/bakeoff.py); installed anyway so the dep is present if
the tool is ever run inside this image/venv, where the Debian python3 has a
prebuilt wheel unlike the dev venv.

Not attempted here: installing/importing webrtcvad in the dev venv (known
unbuildable on py3.14, see tools/vad.py's module docstring) -- this task is deps
declaration only, per the brief.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(timing-compare/T9-T11): schema_version-2 report + aggregates + edge hardening 83eef40cb7

Adds bucket_nsp/bucket_lp (strict cutpoints imported from hallucination.py's
NSP_FLAG/NSP_DROP/LP_FLAG/LP_DROP so the report's bands never drift from B1's
actual thresholds), build_episode_report/aggregate_episodes/build_report (the
full schema_version-2 per-episode/per-show/overall report from spec-v3.md),
atomic --out write, and the printed per-show+overall headline summary.

T10 edge/aggregate hardening: 0-kept-card episodes and all-no-reference shows
null out their ratio fields (pct_cards_on_cue/pct_cues_covered/false_in_gap_rate)
while status counts stay numeric; in_gap_vad_error is counted separately, never
merged into silent/speech; an episode where every in-gap card is vad_error
reports false_in_gap_rate null (unmeasured) rather than a misleading 0.0;
--tolerance out-of-range now logs a clamp warning.

main() now groups episodes by CLI show_dir root (show_name_for_root) instead of
a single flattened walk, since the report needs per-show grouping. Also adds
covered_cue_count()/cues_covered wiring (pct_cues_covered had no existing data
source in U2/U3's process_episode output) and resolve_track_selection_thresholds()
(shared by select_reference_track and the report's config block).

30 new tests in tests/test_timing_compare.py (81 total in-file, 332 overall) --
all pure/hermetic synthetic-data coverage per T11; no media, no GPU, no webrtcvad.
The real-show integration run (T12) remains deferred, needs server media.

Built with help of Claude (Anthropic).

fix(timing-compare): aggregate false_in_gap_rate null on all-vad_error + doc/strip cleanups

CI / test (pull_request) Failing after 1m9s

Details

tests / test (pull_request) Failing after 1m25s

Details

CI / test (push) Failing after 46s

Details

tests / test (push) Failing after 55s

Details

7daee12d9e

aggregate_episodes() pooled false_in_gap_rate as 0.0 whenever every in-gap
card across the pool was in_gap_vad_error (both VAD backends unavailable),
misreporting "no false-in-gap risk" instead of "unmeasured" -- the exact
Phase-1 headline value this bug corrupts. Mirror build_episode_report's
existing null rule at the aggregate level, and surface in_gap_silent/
in_gap_vad_error totals so the unmeasured pool is visible, not just null.

Also: fix common.py's stale "pure stdlib" docstring (it imports pysubs2),
and let dialogue_intervals() reuse a single ev.plaintext.strip() instead of
computing it twice per accepted event.