fix(data-plane): log single-controller metrics before the step is committed - #4044
Merged
terrykong merged 1 commit intoSep 7, 2026
Conversation
`log_metrics(..., step_finished=True)` calls `run.log(..., commit=True)`, which closes that wandb step. wandb accepts a later log against a closed step, returns normally, and discards it -- no exception, no falsy return. `single_controller` called `_log_data_plane_metrics` immediately *after* that commit, so every `data_plane/*` series it produced was computed and dropped. The console still printed "data plane: ...ms" because that print lives inside the discarded call, so the run looked healthy. `grpo_sync` had the identical bug and was fixed when a real run showed 85 logged keys and zero `data_plane/*`. This path was never exercised by a run, so it stayed broken -- its own docstring says it mirrors `grpo_sync`, and it mirrored this too. The test asserts the invariant across every algorithm module rather than one call site, since a third wiring would otherwise repeat it. It compares the *call* (rindex) rather than the first occurrence, which is the method definition and would make the assertion vacuously true -- the first draft of this test passed against the unfixed file. Signed-off-by: Zhiyu Li <zhiyul@oci-aga-slurm-1-dm-02.cm.cluster>
Contributor
Author
|
/ok to test d6dbd00 |
terrykong
approved these changes
Sep 7, 2026
terrykong
merged commit Sep 7, 2026
15a6c26
into
zhiyul/data_plane_observability_metrics
45 of 47 checks passed
ZhiyuLi-Nvidia
added a commit
that referenced
this pull request
Sep 8, 2026
…mitted (#4044) Co-authored-by: Zhiyu Li <zhiyul@oci-aga-slurm-1-dm-02.cm.cluster> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>
ZhiyuLi-Nvidia
added a commit
that referenced
this pull request
Sep 9, 2026
…mitted (#4044) Co-authored-by: Zhiyu Li <zhiyul@oci-aga-slurm-1-dm-02.cm.cluster> Signed-off-by: Zhiyu Li <zhiyul@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a silent metric loss on the async single-controller path, on top of #3616.
The bug
log_metrics(..., step_finished=True)callsrun.log(..., commit=True), whichcloses that wandb step. wandb accepts a later log against a closed step,
returns normally, and discards it — no exception, no falsy return.
single_controller.pycalled the data-plane logger immediately after thatcommit:
So every
data_plane/*series on the async path was computed — snapshotgathered, deltas taken, breakdown table built — and thrown away.
Why it went unnoticed
symptom is an empty chart.
• data plane: …ms, … MB movedprintlives inside the discarded call.
happens inside wandb.
# step_finished=True here since this is the final log of our current stepstopped being true when a log was addedafter it.
grpo_synchad the identical bug. It was caught only because a real GRPO runshowed 85 logged keys and zero
data_plane/*, and was fixed there. This pathwas never exercised by a run — its docstring says it mirrors
grpo_sync, andit mirrored this too.
The fix
Move one statement above the commit.
The test
Asserts the invariant for every algorithm module that logs data-plane
metrics, rather than for one call site — a third wiring would otherwise repeat
it. Source order is the only observable, since the drop happens inside wandb.
Two details worth review:
rindex), not the first occurrence. The firstoccurrence is the method definition, which trivially precedes everything —
my first draft of this test passed against the unfixed file.
step_finished=Truetoo.Verified by running the test's logic against both versions:
single_controllerbefore fixsingle_controllerafter fixgrpo_syncBase
Branched off
zhiyul/data_plane_observability_metricsrather thanmain,because
_log_data_plane_metricsinsingle_controller.pyonly exists in#3616. Against
mainthis would show 51 commits / 66 files; against thefeature branch it is 2 files.