feat(rollout): add partial recovery benchmark telemetry

[macandro96]

What does this PR do?

Adds fixed-cadence benchmark telemetry for partial-rollout checkpointing and recovery.

• Adds rollout_checkpointing.telemetry_interval_s, independent of checkpoint frequency.
• Measures checkpoint save/load, native TQ save, total save, barrier wait, exclusive hold, outcomes, and actual successful-checkpoint spacing.
• Exposes checkpoint backpressure separately from storage time.
• Samples canonical and vLLM token throughput, canonical group throughput, replay occupancy, in-flight work, vLLM running/waiting requests, and KV-cache utilization.
• Records group completion and queue-wait distributions.
• Measures recovery progress and throughput.
• Uses deterministic W&B wall-time axes rather than optimizer-step progress.
• Flushes final training metrics before logger shutdown.

This layer is observational and does not change checkpoint or recovery semantics.

Stack

• Layer 4 of 4.
• Previous: feat(rollout): add periodic partial-rollout recovery snapshots #3508.

Configuration

rollout_checkpointing:
  interval_s: 120
  telemetry_interval_s: 30

For an A/B benchmark, keep telemetry_interval_s identical between baseline and checkpoint-enabled runs. Change only checkpoint enablement or frequency.

Primary metrics

• rollout/throughput/vllm_output_tokens_per_second
• rollout/throughput/canonical_output_tokens_per_second
• rollout/throughput/canonical_groups_per_second
• Group completion and queue-wait p50/p95
• vLLM running/waiting requests and KV-cache utilization
• timing/rollout_checkpoint/tq_save_seconds
• timing/rollout_checkpoint/total_save_seconds
• timing/rollout_checkpoint/exclusive_hold_seconds
• timing/rollout_checkpoint/barrier_wait_seconds

Validation

• Focused tests cover timing, throughput sampling, deterministic axes, rollout-manager counters, logger definitions, and final metric flush.
• The PR remains draft while the refreshed focused suite and full CI are rerun.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[macandro96]

Will be superseded by #3925