Backfill total_s from Cam2V's model_step_wall_s for LingBot benchmark scenarios

[AmirF194]

Refs #564

Cam2V's session.py reports its per-step duration and rate as model_step_wall_s/chunk_fps, and _generated_fps_for_step/_generated_fps_summary only ever look at total_s (falling back to model_step_s), so LingBot's Cam2V scenarios never get a derived generated_fps, and harness.py's total_s highlights come up empty too.

I went with the "report total_s alongside the existing key" option rather than renaming: the runtime-metric-samples ingest path had no key-normalization step at all (unlike the stats-rows path, which already has _KEY_OVERRIDES for this), so I added a small alias step there that backfills total_s from model_step_wall_s when a step doesn't already report total_s. model_step_wall_s stays in the output. Once total_s is present, the existing fallback in _generated_fps_for_step/_generated_fps_summary derives generated_fps on its own, and harness.py's total_s_median/startup_step_total_s highlights and report.py's headline cards pick it up with no changes needed there.

This covers the LingBot scenarios dropping generated_fps and the headline cards, not the regression-gate part of the issue (the metric name lists that skip instead of failing). I couldn't find that gate's config in this repo, so if it lives in an internal tool this half may need a separate change there.

Added test_runtime_benchmark_stats_backfills_total_s_from_model_step_wall_s: feeds a model_step_wall_s/chunk_fps sample through records_from_stats_file and checks both total_s and generated_fps come out. The new test fails without the change. The existing runtime-metric-samples tests (group-by-step, warmup exclusion, the v2 sink one) still pass unchanged.

Couldn't run the full suite locally, this workspace needs CUDA/transformer-engine to build. Checked the change with ruff (check and format, both clean) and by loading tools/benchmarks/metrics.py directly with no other project dependencies, since the module itself only imports the standard library.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

The PR makes Cam2V’s per-step wall duration available under the canonical total_s metric while preserving model_step_wall_s, enabling existing benchmark logic to derive generated FPS and timing highlights.

• Emits total_s directly from the Cam2V model loop.
• Backfills total_s when ingesting older runtime samples that only contain model_step_wall_s.
• Adds application and benchmark-parser coverage for the canonical timing metric and derived FPS.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/cam2v/cam2v/session.py	Emits total_s from the same measured duration as model_step_wall_s, preserving both metric names.
apps/cam2v/tests/test_application.py	Verifies that Cam2V reports identical canonical and integration-specific step durations.
flashdreams/tools/benchmarks/metrics.py	Adds an idempotent runtime-sample alias that backfills total_s without replacing an explicitly reported value.
flashdreams/tests/test_benchmark_harness.py	Covers ingestion of legacy Cam2V-shaped samples and verifies both the alias and derived generated FPS.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Cam2V model step] --> B[model_step_wall_s and total_s]
    C[Older runtime sample with model_step_wall_s] --> D[Canonical alias backfill]
    B --> E[Benchmark record with total_s]
    D --> E
    E --> F[Derived generated_fps]
    E --> G[Timing highlights and report cards]

Loading

_{Reviews (2): Last reviewed commit: "Assign total_s at the source in Cam2VMod..." | Re-trigger Greptile}

[ZenAlexa]

The alias is applied inside records_from_stats_file(), after the runtime stats artifact has already been written. Cam2V's StepResult.metrics therefore continues to contain model_step_wall_s/chunk_fps and omits total_s, so the raw artifact and every direct consumer keep the non-canonical contract described in #564; the first acceptance criterion remains open. I added assert result.metrics["total_s"] == result.metrics["model_step_wall_s"] to test_model_loop_maps_wasd_to_shared_camera_input_and_metrics, and the PR head fails with KeyError: 'total_s'. Assigning total_s beside model_step_wall_s in Cam2VModelLoop.step() gives every consumer the canonical key. This parser alias can also remain for historical artifacts.

[AmirF194]

You're right, the alias only patched the ingest path, so StepResult.metrics itself never got total_s. Pushed 7f1270a: Cam2VModelLoop.step() now sets total_s alongside model_step_wall_s at the source, and the parser alias stays for historical artifacts.

Added your assertion to the existing WASD test: fails with KeyError: 'total_s' on the prior commit, passes now. Ran it CPU-only (torch cpu wheel, no GPU here) along with the rest of test_application.py and test_benchmark_harness.py, 59 passed, 1 skipped, ruff clean on both files.

[ZenAlexa]

Cam2V now emits total_s in the producer, and the parser preserves an explicit canonical value. The runtime-metrics and runtime-records tests pass at 7f1270a8 (59 passed, 1 skipped); restoring the previous producer reproduces KeyError: 'total_s' in the regression. The raw artifact, throughput summary, and report paths now share the metric.