[SM90] Validate MegaMoE symmetric buffer peer mappings

[Andrew88Lin]

Problem

sm90_fp8_mega_moe reaches peer ranks through layout::SymBuffer::map(), which is plain
pointer arithmetic over sym_buffer_ptrs:

// deep_gemm/include/deep_gemm/layout/sym_buffer.cuh
constexpr static uint32_t kNumMaxRanks = 72;          // NVL72

offsets[i] = i < size ? (c[i] - base) : 0;
...
int64_t mapped_ptr = offsets[dst_rank_idx] + reinterpret_cast<int64_t>(ptr);

That only holds while every rank's symmetric buffer is mapped into the local GPU address
space, i.e. while all ranks share one NVLink/P2P domain. On SM90 that domain is a single
8-GPU node.

Nothing on the host validates this: num_ranks is only read from sym_buffer_ptrs.size()
for sizing. An EP group spanning two nodes therefore launches happily and dies inside the
kernel with CUDA_ERROR_ILLEGAL_ADDRESS. Because the offending peer access is a
fire-and-forget red in comm/barrier.cuh, the fault is asynchronous — the reported
failure point drifts between identical runs and a core dump blames a later, innocent atomic.

Measured mechanism. On 2 nodes × 8× H200, symm_mem.rendezvous(...).buffer_ptrs returns
a null pointer for every rank outside the local NVLink domain:

rank=0 host=node4 peer= 7 ptr=0x1e030a00c00 rc_ordinal=0 ordinal=7   # same node
rank=0 host=node4 peer= 8 ptr=0x0           rc_ordinal=1 ordinal=None # other node
...
rank=8 host=node5 peer= 0 ptr=0x0           rc_ordinal=1 ordinal=None # mirror image

So offsets[i] = 0 - base = -base, and map(ptr, i) = ptr - base lands near zero. That is
the illegal address.

Change

Check on the host, before launch, that every entry of sym_buffer_ptrs is addressable from
the current device, and fail with a message that names the constraint.

• Scoped to the SM90 path on purpose. A multi-node EP group is legitimate on NVL72, and
  adding the same check to the SM100 path would be a behaviour change for Blackwell users.
  Happy to extend it if you would rather have symmetry.
• Primary predicate is the null pointer above. The cuPointerGetAttribute query is a second
  line of defence for a hypothetical allocator that hands out a real but unmapped remote VA;
  only an unambiguous CUDA_ERROR_INVALID_VALUE is treated as a failure, so a working
  single-domain deployment cannot be rejected by accident.
• Result is cached per (device, pointer set). Symmetric buffers are allocated once at
  rendezvous, so after the first call this costs one std::set lookup per launch.
• DG_SKIP_SYM_BUFFER_CHECK=1 disables it.

Verification

Built from release/v0.1.5 (the branch the shipped wheel comes from; these two files are
byte-identical on dev) via build_sgl_deep_gemm.sh, on 2 nodes × 8× H200 with IB.

Check	Result
Compiles	_C.so + wheel produced, 0 errors
8 ranks / 1 node — no false positive	guard silent, fp8_mega_moe executed
16 ranks / 2 nodes — guard fires	explanatory exception, no illegal address

16-rank output, identical in shape on both nodes:

RuntimeError: Assertion error (csrc/jit_kernels/impls/sm90_fp8_mega_moe.hpp:174):
Symmetric buffer of rank 8 (0x0) is not addressable from rank 3. MegaMoE addresses peers
directly, so all 16 ranks must share a single NVLink/P2P domain, which is a single node on
SM90; it cannot span nodes over IB/RDMA

raised from deep_gemm.fp8_mega_moe(...) before any kernel launch. Rank 8 on the second node
reports the mirror image (rank 0 (0x0) is not addressable from rank 8).

Two caveats, stated plainly:

• The 8-rank run reached bench_kineto and then hit a CUDA OOM allocating the 7.45 GiB L2
  flush buffer, because an unrelated vLLM deployment held ~135 GiB on each GPU. The
  functional run_fused() call before it succeeded, which is what this patch affects; the
  benchmark phase is not.
• Both runs used a reduced shape (--num-experts 32 --hidden 1024 --intermediate-hidden 512 --num-max-tokens-per-rank 512) for the same reason. The guard inspects pointers only, so
  it is shape-independent.

Side note while testing: python setup.py build_ext does not build on either dev or
release/v0.1.5 — csrc/python_api.cpp is still listed as the source but every
register_apis in csrc/apis/*.hpp is now behind #if 0 after the tvm-ffi migration.
Unrelated to this PR, but there is no pull_request-triggered workflow that would have
caught it.

Context

Root cause analysis, the minimal reproducer and the ruled-out hypotheses are in
sgl-project/sglang#36303. Single-node SM90 MegaMoE works, so this is not "SM90 is
unsupported" — it is a missing topology precondition.