feat(benchmarks): add reproducible indexing-latency benchmark with real DiDeMo corpus

[Mahnoor-Zaffar]

Summary

Adds vidxp benchmark index-latency, a reproducible indexing-latency benchmark, and extends it with a real-media corpus mode (--corpus didemo) that indexes the real DiDeMo videos prepared by vidxp benchmark prepare.

The benchmark measures per-stage indexing throughput, per-stage wall time, and peak RSS across configurable modalities (scene, actor, dialogue), supports baseline regression detection, and records the exact corpus, git revision, and environment fingerprints so results are reproducible and comparable between builds.

Motivation

VidXP has official quality adapters (DiDeMo, HiREST) but no way to detect performance regressions or measure the latency impact of model/architecture changes. This PR adds that measurement surface in two layers:

1. v1 — synthetic media (deterministic, hermetic). FFmpeg testsrc2 clips generated on the caller's machine, with a seeded synthetic transcript for dialogue. Cheap to run anywhere, fully reproducible, suitable for CI-style regression gates.
2. Real-media corpus (paper-published media). --corpus didemo runs the same pipeline over the real DiDeMo videos (from Localizing Moments in Video with Natural Language, ICCV 2017) already downloaded and verified by vidxp benchmark prepare didemo. Real decode workloads, real faces for actor, and (with --input-mode transcribe) real Whisper transcription for dialogue. Any directory of media files is also accepted.

What's included

• src/vidxp/benchmarks/latency.py — corpus generation/discovery, run orchestrator driving the real run_index/ModelRuntime pipeline, per-stage aggregation, real/synthetic corpus specs, and baseline comparison.
• src/vidxp/benchmarks/cli.py — vidxp benchmark index-latency command with --modalities, --videos, --duration-seconds, --resolution, --repetitions, --input-mode (transcript/transcribe), --audio-mode, --corpus, --reset, --baseline, --baseline-tolerance, --json.
• docs/benchmarking/performance.md — full protocol, report schema, evidence rules, limitations, usage.
• docs/benchmarking/README.md — catalog link.
• tests/test_benchmark_latency.py — 39 unit tests (validation, transcript generation, clip command building, aggregation, baseline comparison, corpus resolution/discovery/overrides, real-vs-synthetic baseline isolation).

Usage

# Synthetic baseline (default: 1 x 8s 320x180 clip, scene-only)
vidxp benchmark index-latency --run-id my-baseline

# Synthetic, multi-modality, 3 reps, regression check
vidxp benchmark index-latency \
  --run-id v2-compare \
  --modalities scene,actor,dialogue \
  --videos 2 --duration-seconds 12 --repetitions 3 \
  --baseline benchmark_runs/latency/synthetic/my-baseline/report.json

# Real DiDeMo media (prepare first, then smoke on one clip)
vidxp benchmark prepare didemo --split validation --annotation-indices 0 --yes
vidxp benchmark index-latency --run-id didemo-smoke \
  --corpus didemo --modalities scene,actor --videos 1

# Real dialogue over real audio (Whisper transcribes)
vidxp benchmark index-latency --run-id didemo-dialogue \
  --corpus didemo --modalities dialogue --input-mode transcribe --videos 2

Key design points

• Baseline isolation: baseline comparison rejects a baseline whose corpus signature differs from the current run, so synthetic and real results never mix. Synthetic signatures include dimensions/duration/seed; real signatures include corpus name, clip count, bytes, duration range, containers, and media-override state.
• Real transcripts: DiDeMo has no released transcripts, so dialogue on a real corpus requires --input-mode transcribe (real Whisper) — synthetic transcripts are never silently substituted for real media.
• Overrides: --corpus didemo honors media-overrides.json written by vidxp benchmark prepare (e.g. the documented Wikimedia replacement clip).
• Stable media IDs: real clips map official DiDeMo video names to stable benchmark media IDs, matching the rest of the benchmark infrastructure.
• Runtime split: real-corpus runs use the real split in the run directory/generation ID, keeping them distinct from synthetic runs.

Validation

• uv run --no-sync ruff check src/vidxp/benchmarks/ tests/test_benchmark_latency.py — clean.
• uv run --no-sync pytest -q tests/test_benchmark_latency.py — 39/39 pass.
• Full suite (with test + benchmarks extras): 685 passed; 22 failures are the documented pre-existing missing-optional-deps (chromadb, cv2) on the local macOS x86_64 host where the actor extra cannot install (onnxruntime has no compatible wheel); the two previously-noted failures (test_benchmark_cli.py, test_benchmarks.py) are among them.
• vidxp benchmark index-latency --help — shows --corpus.
• A live end-to-end index-latency smoke could not be re-run here because the storage dependency chromadb is unavailable on this host; the real-corpus pipeline (discovery → ffprobe validation → source building → corpus spec) was verified against a real generated clip, and the dialogue/transcribe gating was verified to raise before touching storage.

Limitations (documented in performance.md)

• Synthetic testsrc2 has no semantic scene content and no actors; actor measures face-detection overhead with zero detections, and transcript mode measures embedding on a seeded synthetic transcript.
• Real DiDeMo runs depend on what vidxp benchmark prepare didemo downloaded locally and on any recorded media overrides.
• Peak RSS is whole-process (Python + models + Chroma), not a pure indexing-stage measurement.

[tulayha]

With the default reset=False, only the first repetition performs indexing. Every later call uses the same run and generation ID, so run_index() resumes the completed manifest and skips the videos. That means --repetitions 3 measures one indexing run followed by two resume/skip runs. Please ensure every repetition performs a complete fresh indexing cycle and add a test covering the orchestration, not only aggregation.

[tulayha]

This validation also applies to real corpora. As a result, the documented DiDeMo command with --modalities dialogue --input-mode transcribe fails unless --audio-mode flite is added, even though real-corpus audio comes from the video and audio_mode is documented as ignored. Please apply the flite requirement only to generated synthetic media.

[tulayha]

--corpus arrives from Typer as a string, so a directory supplied on the command line never reaches this Path branch. Every string except didemo is treated as an unknown named corpus, making the advertised --corpus <directory> mode unusable. Please recognize directory strings as paths and validate that the resulting directory exists.