[Roadmap][Feature] Distributed multi-GPU support

[viiccwen]

Summary

This issue tracks the broader roadmap for distributed multi-GPU support in QDP.

The goal is not only to make larger QDP states feasible on one machine, but also to establish a distributed foundation that can grow toward richer placement strategies, stronger communication backends, broader workflow support, multi-node execution, and performance hardening over time.

Roadmap Goal

Build a distributed GPU architecture for QDP that can evolve in stages while remaining understandable, maintainable, and extensible.

At a high level, this roadmap should enable QDP to:

• handle workloads that exceed single-GPU limits
• preserve a simple path for workloads that do not need distribution
• fail clearly when a distributed request is invalid or infeasible
• expose distributed state without forcing immediate gather/readback
• support multiple workflows on top of a shared distributed foundation
• evolve from a first single-node path toward true MPI-backed execution

Current Architecture Direction

The distributed runtime should be rank-local:

• each rank owns a local DeviceMesh
• each rank materializes only its local shards
• scalar global coordination happens through CollectiveCommunicator
• GPU-resident collectives are reserved behind DeviceCollectiveCommunicator
• consumers can use zero-copy local shard views when they can consume GPU pointers directly

The roadmap keeps four concerns separate:

• Planning: global placement, rank-local shard ownership, and memory feasibility.
• Runtime: local shard materialization, normalization, and encoding.
• Transport: host scalar collectives first, future CUDA-aware MPI / NCCL device
  collectives later.
• State: local buffer ownership, zero-copy local shard views, and future
  gather/export contracts.

flowchart LR
    subgraph Rank0[Rank 0]
      R0M[local DeviceMesh] --> R0S[local shards]
      R0S --> R0N[local contribution]
    end

    subgraph Rank1[Rank 1]
      R1M[local DeviceMesh] --> R1S[local shards]
      R1S --> R1N[local contribution]
    end

    R0N --> AR[CollectiveCommunicator]
    R1N --> AR
    AR --> G[global result]

Loading

Roadmap Stages

Stage 1. Rank-local single-node foundation

Add the first distributed amplitude foundation:

• distributed execution context
• shared distributed state planning contract
• rank/world metadata
• rank-local placement planning
• local GPU shard materialization
• single-process fallback collectives
• zero-copy local shard views

Tracked by #1295 and implemented by #1296.

Stage 2. True MPI communicator and launcher path

Add real MPI-backed distributed execution:

• MPI scalar communicator
• mpirun -n N launch path
• rank/world initialization from MPI
• multi-rank smoke tests
• clearer operational documentation

Stage 3. CUDA-aware MPI / NCCL device collectives

Add GPU-resident communication backends where appropriate:

• CUDA-aware MPI backend
• NCCL backend
• backend selection policy
• fallback behavior when GPU collectives are unavailable

Stage 4. Better placement and communication behavior

Improve how distributed execution uses available hardware:

• topology-aware placement policies
• peer-to-peer awareness
• memory-capacity policy improvements
• communication-cost-aware planning

Stage 5. Distributed state usability

Make distributed state easier for the rest of QDP to consume:

• inspection helpers
• gather/export APIs
• downstream consumer contracts
• clearer unsupported-operation errors

Stage 6. Broader workflow support

Expand the distributed foundation beyond the first amplitude path:

• additional encoders where distribution is meaningful
• shared distributed runtime contracts
• workflow-specific adapters that do not duplicate placement/runtime logic

Stage 7. Multi-node evolution and performance hardening

Extend beyond the initial single-node scope when the foundation is ready:

• multi-node deployment model
• production-like launch and diagnostics
• benchmark coverage
• performance regression tests
• scalability and failure-mode documentation

Non-Goals For Early Stages

The early stages should avoid promising:

• immediate full multi-node support
• mandatory MPI/NCCL dependencies
• universal zero-copy across ranks
• complete gather/export semantics
• full performance tuning before the runtime contract stabilizes