Expose pull-based collective lifecycle events - #3638
Closed
dolpm wants to merge 1 commit into
Closed
Conversation
Summary: Adds an opt-in, pull-based collective lifecycle feed in native CollTrace and exposes copied values through matched NCCLX and `nccl4py` builds. Ownership and configuration - `NCCL_COLLTRACE=lifecycle` creates one `LifecycleEventFeedPlugin` per lifecycle-enabled `ncclComm_t`. Lifecycle-only mode starts CollTrace without also installing CommDump. - `NCCL_COLLTRACE=ALL` and lowercase `all` enable `algostat`, `lifecycle`, and `trace` without implicitly enabling `verbose`; explicit `verbose` composes with either alias. Unknown modes return `ncclInvalidArgument` before partial initialization. - A process-local registry holds weak CollTrace references. The global reader flushes live instances, drains their plugin queues, and stable-sorts copied records by timestamp without extending communicator lifetimes. Public API and identity - `colltrace_get_comm_id(comm)` returns a monotonic process-local communicator ID. - `colltrace_get_latest_coll_id(comm)` returns the latest identity for that communicator. `0` means no collective has been registered; real collective IDs are one-based. Captured collectives retain their capture identity across graph replays. - `colltrace_get_unread_events()` destructively drains unread `enqueue`, `start`, and `end` records across lifecycle-enabled communicators. Each tuple is `(replay_id, comm_id, coll_id, execution_coll_id, event_type, timestamp)`. - Latest-ID state belongs to the communicator plugin, not thread-local storage. Null communicators return `ncclInvalidArgument`; communicators without lifecycle mode return `ncclInvalidUsage`. Lifecycle correctness and build boundary - CPU start/end timestamps are stored before semaphore publication. If a ready wait event unexpectedly reports the epoch sentinel, the poller substitutes its current timestamp so start/end state remains sticky and callbacks are not emitted repeatedly. - The Python bridge releases the GIL around native calls, maps absent replay IDs to `None`, and never exposes pointers into CollTrace state. - NCCLX and `nccl4py` remain a matched-build boundary because the binding links the new native symbols and layouts directly. Reviewed By: pavanbalaji Differential Revision: D114815651
Contributor
|
@dolpm has exported this pull request. If you are a Meta employee, you can view the originating Diff in D114815651. |
Contributor
|
This pull request has been merged in 2504bbe. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Adds an opt-in, pull-based collective lifecycle feed in native CollTrace and exposes copied values through matched NCCLX and
nccl4pybuilds.Ownership and configuration
NCCL_COLLTRACE=lifecyclecreates oneLifecycleEventFeedPluginper lifecycle-enabledncclComm_t. Lifecycle-only mode starts CollTrace without also installing CommDump.NCCL_COLLTRACE=ALLand lowercaseallenablealgostat,lifecycle, andtracewithout implicitly enablingverbose; explicitverbosecomposes with either alias. Unknown modes returnncclInvalidArgumentbefore partial initialization.Public API and identity
colltrace_get_comm_id(comm)returns a monotonic process-local communicator ID.colltrace_get_latest_coll_id(comm)returns the latest identity for that communicator.0means no collective has been registered; real collective IDs are one-based. Captured collectives retain their capture identity across graph replays.colltrace_get_unread_events()destructively drains unreadenqueue,start, andendrecords across lifecycle-enabled communicators. Each tuple is(replay_id, comm_id, coll_id, execution_coll_id, event_type, timestamp).ncclInvalidArgument; communicators without lifecycle mode returnncclInvalidUsage.Lifecycle correctness and build boundary
None, and never exposes pointers into CollTrace state.nccl4pyremain a matched-build boundary because the binding links the new native symbols and layouts directly.Reviewed By: pavanbalaji
Differential Revision: D114815651