Skip to content

Expose pull-based collective lifecycle events - #3638

Closed
dolpm wants to merge 1 commit into
meta-pytorch:mainfrom
dolpm:export-D114815651
Closed

Expose pull-based collective lifecycle events#3638
dolpm wants to merge 1 commit into
meta-pytorch:mainfrom
dolpm:export-D114815651

Conversation

@dolpm

@dolpm dolpm commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary:
Adds an opt-in, pull-based collective lifecycle feed in native CollTrace and exposes copied values through matched NCCLX and nccl4py builds.

Ownership and configuration

  • NCCL_COLLTRACE=lifecycle creates one LifecycleEventFeedPlugin per lifecycle-enabled ncclComm_t. Lifecycle-only mode starts CollTrace without also installing CommDump.
  • NCCL_COLLTRACE=ALL and lowercase all enable algostat, lifecycle, and trace without implicitly enabling verbose; explicit verbose composes with either alias. Unknown modes return ncclInvalidArgument before partial initialization.
  • A process-local registry holds weak CollTrace references. The global reader flushes live instances, drains their plugin queues, and stable-sorts copied records by timestamp without extending communicator lifetimes.

Public API and identity

  • colltrace_get_comm_id(comm) returns a monotonic process-local communicator ID.
  • colltrace_get_latest_coll_id(comm) returns the latest identity for that communicator. 0 means no collective has been registered; real collective IDs are one-based. Captured collectives retain their capture identity across graph replays.
  • colltrace_get_unread_events() destructively drains unread enqueue, start, and end records across lifecycle-enabled communicators. Each tuple is (replay_id, comm_id, coll_id, execution_coll_id, event_type, timestamp).
  • Latest-ID state belongs to the communicator plugin, not thread-local storage. Null communicators return ncclInvalidArgument; communicators without lifecycle mode return ncclInvalidUsage.

Lifecycle correctness and build boundary

  • CPU start/end timestamps are stored before semaphore publication. If a ready wait event unexpectedly reports the epoch sentinel, the poller substitutes its current timestamp so start/end state remains sticky and callbacks are not emitted repeatedly.
  • The Python bridge releases the GIL around native calls, maps absent replay IDs to None, and never exposes pointers into CollTrace state.
  • NCCLX and nccl4py remain a matched-build boundary because the binding links the new native symbols and layouts directly.

Reviewed By: pavanbalaji

Differential Revision: D114815651

Summary:
Adds an opt-in, pull-based collective lifecycle feed in native CollTrace and exposes copied values through matched NCCLX and `nccl4py` builds.

Ownership and configuration

- `NCCL_COLLTRACE=lifecycle` creates one `LifecycleEventFeedPlugin` per lifecycle-enabled `ncclComm_t`. Lifecycle-only mode starts CollTrace without also installing CommDump.
- `NCCL_COLLTRACE=ALL` and lowercase `all` enable `algostat`, `lifecycle`, and `trace` without implicitly enabling `verbose`; explicit `verbose` composes with either alias. Unknown modes return `ncclInvalidArgument` before partial initialization.
- A process-local registry holds weak CollTrace references. The global reader flushes live instances, drains their plugin queues, and stable-sorts copied records by timestamp without extending communicator lifetimes.

Public API and identity

- `colltrace_get_comm_id(comm)` returns a monotonic process-local communicator ID.
- `colltrace_get_latest_coll_id(comm)` returns the latest identity for that communicator. `0` means no collective has been registered; real collective IDs are one-based. Captured collectives retain their capture identity across graph replays.
- `colltrace_get_unread_events()` destructively drains unread `enqueue`, `start`, and `end` records across lifecycle-enabled communicators. Each tuple is `(replay_id, comm_id, coll_id, execution_coll_id, event_type, timestamp)`.
- Latest-ID state belongs to the communicator plugin, not thread-local storage. Null communicators return `ncclInvalidArgument`; communicators without lifecycle mode return `ncclInvalidUsage`.

Lifecycle correctness and build boundary

- CPU start/end timestamps are stored before semaphore publication. If a ready wait event unexpectedly reports the epoch sentinel, the poller substitutes its current timestamp so start/end state remains sticky and callbacks are not emitted repeatedly.
- The Python bridge releases the GIL around native calls, maps absent replay IDs to `None`, and never exposes pointers into CollTrace state.
- NCCLX and `nccl4py` remain a matched-build boundary because the binding links the new native symbols and layouts directly.

Reviewed By: pavanbalaji

Differential Revision: D114815651
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 13, 2026
@meta-codesync

meta-codesync Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

@dolpm has exported this pull request. If you are a Meta employee, you can view the originating Diff in D114815651.

@meta-codesync

meta-codesync Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

This pull request has been merged in 2504bbe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot. Merged meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant