Add P2P IB AbortDevice wait tests (#3637) - #3637
Open
saifhhasan wants to merge 6 commits into
Open
Conversation
meta-cla
Bot
added
the
CLA Signed
Contributor
|
@saifhhasan has exported this pull request. If you are a Meta employee, you can view the originating Diff in D115656603. |
Summary: Add an MCCL create option to select device abort TRAP behavior while keeping SKIP as the default. Initialize the communicator-owned Abort default timeout from init timeout when present, otherwise from MCCL_ABORT_TIMEOUT_MS when positive. This keeps MCCL device waits backed by a default timeout after Timeout becomes an AbortDevice alias, while allowing callers to preserve legacy trap behavior explicitly. Differential Revision: D115814486
saifhhasan
pushed a commit
to saifhhasan/torchcomms-1
that referenced
this pull request
Aug 13, 2026
Summary: Pull Request resolved: meta-pytorch#3637 Adds P2P IB/IBRC `AbortDevice` wait coverage and owns the corresponding test targets. This update moves `p2p_ib_transport_device_abort_test` and its kernel library into this diff, matching the diff that introduces `P2pIbTransportDeviceAbortTest.*` sources. Stack review guide: https://www.internalfb.com/intern/paste/P2459153030/ Differential Revision: D115656603
saifhhasan
force-pushed
the
export-D115656603
branch
from
August 13, 2026 09:44
9e97a76 to
610f1c0
Compare
meta-codesync
Bot
changed the title
added 5 commits
August 13, 2026 08:51
Summary: Fixes the base Prims timeout-to-AbortDevice compatibility layer after the stack rebase. Changes in this update: - Replace stale `//comms/prims:timeout_utils` BUCK deps with `//comms/prims:abort_check` for benchmark, collective benchmark, and AMD transport targets. - Keep P2P IB abort test targets out of the base diff so they live with the test sources in `D115656603`. - Restore standalone parsing of P2P IB LL transport templates by keeping LL capability traits after deleting `LLImpl`. - Propagate the borrowed MPT `AbortDevice` into MCCL AllGather/ReduceScatter launch params so the base alias diff builds standalone. Stack review guide: https://www.internalfb.com/intern/paste/P2459153030/ Differential Revision: D115656601
Summary: Keeps the `Timeout` -> `AbortDevice` migration off the PCIe bus in device spin loops. `Abort` allocates its state with `cudaHostAlloc(..., cudaHostAllocMapped)`, so every `AbortDevice` read of the shared reason is an uncached host round trip. The Prims `Timeout` it replaces compared an on-chip `clock64()` deadline and touched no memory at all, so a direct port turns each wait-loop iteration into a host access. The worst site is the LL small-message path, where the check is per-lane inside the poll loop. Changes: - `checkExpired()` gates the mapped-state read behind the free device clock (`kAbortPollsPerMs`, ~10us), so steady-state polling costs a register compare. Deadline expiry is deliberately *not* throttled, so timeout accuracy is unchanged; only explicit-abort observation is bounded by the poll interval. - Collapse the double `reason()` load on the common path, and cache a observed terminal reason in the handle since it is first-writer-wins and never cleared. - `startTimeout()` uses a host-sampled timeout instead of reading mapped state per thread at kernel entry. A negative cached value falls back to the shared read so CUDA-graph replays still late-bind a `setTimeout()` issued after capture. Behavior notes: - Timeout firing time is unchanged. - Explicit host abort is observed within one poll interval (~10us) rather than on the very next instruction. That is far below any timeout that matters. - Graph-captured kernels that set a timeout after capture keep working via the shared-state fallback. Differential Revision: D115855241
Summary: Migrates NVL wait paths onto `AbortDevice` and keeps the NVL trap-test target matrix aligned with the enum cases that remain in this diff. This update removes stale zero-geometry / bad-signals-per-lane BUCK targets from this layer and renumbers the three remaining trap cases: too many groups, insufficient local signals, and insufficient multimem signals. Stack review guide: https://www.internalfb.com/intern/paste/P2459153030/ Differential Revision: D115656600
Summary: Propagates `AbortDevice` wait/check results through the IB transport paths while preserving the transport-facing API shape. The key behavior is that transport waits can now unwind on SKIP-mode abort instead of only trapping on timeout, and the later P2P IB test diff owns the direct compile/runtime coverage for these paths. Stack review guide: https://www.internalfb.com/intern/paste/P2459153030/ Differential Revision: D115656602
Summary: Pull Request resolved: meta-pytorch#3637 Adds P2P IB/IBRC `AbortDevice` wait coverage and owns the corresponding test targets. This update moves `p2p_ib_transport_device_abort_test` and its kernel library into this diff, matching the diff that introduces `P2pIbTransportDeviceAbortTest.*` sources. Stack review guide: https://www.internalfb.com/intern/paste/P2459153030/ Differential Revision: D115656603
saifhhasan
force-pushed
the
export-D115656603
branch
from
August 13, 2026 16:08
610f1c0 to
4e730a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Adds P2P IB/IBRC
AbortDevicewait coverage and owns the corresponding test targets.This update moves
p2p_ib_transport_device_abort_testand its kernel library into this diff, matching the diff that introducesP2pIbTransportDeviceAbortTest.*sources.Stack review guide: https://www.internalfb.com/intern/paste/P2459153030/
Differential Revision: D115656603