[API] Report FPS in a standardized way

[ArielG-NV]

resolves: #603

Extend MetricsOutputSink to report model step 'FPS' in a standardized way when trying to emit stats via --stats-path.

We will generally try to present 'FPS of something' via model_step FPS since ui_step FPS is quite easy to raise (cheap to run ui_step, expensive to run model_step).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR standardizes model-step FPS reporting in benchmark artifacts.

• Records elapsed time and frame count for each model step.
• Emits a p90 model-step FPS value through MetricsOutputSink.
• Adds output-readiness synchronization and updates runner tests and documentation.
• The latest revision also deletes the shared T2V testing module, leaving existing test consumers broken.

Confidence Score: 4/5

The PR is not safe to merge because deleting t2v.testing prevents multiple supported test suites from collecting.

The FPS reporting implementation is covered by focused runner tests, and deleting the problematic _FrameInspector means the previous inherited-state finding is no longer outstanding. However, the deletion introduces a separate blocking failure: numerous tracked tests still import helpers from t2v.testing, and no replacement module exists.

Files Needing Attention: apps/t2v/t2v/testing.py

Important Files Changed

Filename	Overview
flashdreams/flashdreams/runtime_v2/metrics_output_sink.py	Records model-step timing and frame counts and emits the p90 FPS metric.
flashdreams/flashdreams/runtime_v2/session_runner.py	Delivers each model step's elapsed time and results to the metrics sink.
flashdreams/flashdreams/runtime_v2/step_result.py	Adds an explicit method for waiting until generated output is ready.
apps/t2v/t2v/testing.py	Deletes a documented shared testing module while tracked tests and integrations still import it.
flashdreams/test_v2/test_session_runner.py	Adds coverage for standardized model metrics and FPS output.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Model step completes] --> B[Session runner measures elapsed time]
    B --> C[MetricsOutputSink waits for results]
    C --> D[Record elapsed seconds and frame count]
    D --> E[Compute per-step FPS samples]
    E --> F[Calculate p90 FPS]
    F --> G[Write standardized JSON artifact]

Loading

Comments Outside Diff (1)

1. apps/t2v/t2v/testing.py

   Shared Test Module Deleted

   Deleting this module leaves existing apps/t2v and integration tests importing t2v.testing and its helpers without a replacement. Those suites will fail during test collection with ModuleNotFoundError. The module is also documented as the provider for text-to-video stand-in pipelines, so it must be restored or all consumers must be migrated to a replacement.

_{Reviews (4): Last reviewed commit: "ai test breaks, test is not very correct..." | Re-trigger Greptile}

[greptile-apps]

Metrics state is uninitialized

_FrameInspector now inherits MetricsOutputSink, but its constructor and open() method do not initialize the inherited _model_step_info field. check_t2v_model_impl passes this inspector to run_session, which calls the inherited new_model_step() for the first non-empty model step. That method tries to append to the missing field, raising AttributeError and failing the standard T2V test and real-model check paths. Initialize the inherited state through the base class or implement the new callback explicitly.

[ArielG-NV]

This debug object can implement its own hacks, at the end of the day, it still conforms to our interface.

[greptile-apps]

I agree that _FrameInspector can have its own implementation; the issue is that it currently has no new_model_step implementation, so it inherits MetricsOutputSink.new_model_step. run_session now calls that method before each model result, and the inherited method accesses _model_step_info, which _FrameInspector.__init__ never initializes. It will therefore raise on the first model step. Either override new_model_step with a no-op/custom implementation, or initialize and use the inherited state. As written, it conforms to the old OutputSink interface, but not the new runtime contract introduced by the MetricsOutputSink annotation and callback.

[gtong-nv]

do we have a case where the frame_count in the step_results are different?
Trying to understand why is this needed

[ArielG-NV]

It is not needed, I realized we have this:

        if frame_count <= 0 or any(item.frame_count != frame_count for item in chunk):
            raise ValueError("Every channel in a chunk must have the same frame_count.")
            ```

[gtong-nv]

the new_model_step function is essentially just recording the results with step_elapsed_s, right? The name seems confusing. Maybe name it record_model_step_elapsed_s() ?

[ArielG-NV]

renamed 👍

[ArielG-NV]

Current design does not account for StepResult::_output_ready_event, I need to amend for this

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[ArielG-NV]

added

        for result in step_results:
            result.wait_until_ready()
            ```
            
         This is to ensure we measure model_step fully+correctly.

[ArielG-NV]

/ok to test 7eac488

[gtong-nv]

I understand we need to measure the model step(), but we need to be very careful and avoid this sync point in production (when not measuring the model.step()). This might hurt end to end performance.

[ArielG-NV]

I suppose a more elegant solution exists here, and I should try to implement it.

[ArielG-NV]

we decided to take a different approach, and side-line this idea

[ArielG-NV]

For a clearer justification:

FPS has many different meanings for flashdreams:

• end-to-end fps - Users care about this value since this is the "fluidity of an experience". This includes ui_step, model_step, and presentation overhead
• model_step fps/throughput - Developers care about this value since this is the "upper-bound" in perf. If your model_step is too slow, your ui_step will be bottlenecked by it. This technically is also useful for model developers since a multi-step system can then easily be benchmarked on only the cost of the "model".
• ui_step fps/throughput - Developers care about this value since if you are not hitting ui_step FPS consistently, something may be going very wrong. This number is mostly useless though.

For now we will just stick with .generate metrics since we can refine that system further and have not decided how to move forward with tracking the metrics described above.