ggml adapter: SM120/Qwen3.6-35B target — binding-driven LLM decode windows, MoE catalog family, packless artifacts - #186
Open
LiangSu8899 wants to merge 10 commits into
Open
Conversation
Second (arch, model-family) target for the native ggml adapter: fused-region NVFP4 W4A4 GEMVs (GDN in_proj, attn qkv, lm head), GDN cell span with M<=4 per-token state snapshots and checkpoint replay, format-native MoE expert span over ggml K-quant blocks (bit-exact q8_1 activation clone), out-proj / router / spec-draft-head windows. All launches join the host's PDL chain; verify batches up to M=4 are first-class (MTP speculative decode). Host integration is a CMake option (GGML_CUDA_FLASHRT_SM120 + GGML_CUDA_FLASHRT_PUBLIC_DIR) compiling this translation unit inside the host's ggml-cuda build, with all call sites ifdef-guarded. Binding and qualification gates recorded for the migration baseline.
Region wire buffers (packed e2m1 + atom-layout SF + per-tensor alpha) are now built on first sight of the weight tensors in an evaluated graph, from a pre-capture hook: ggml dequant -> bf16 -> global amax -> ue4m3-ceil block scales -> e2m1 nibbles, reproducing the offline packer byte-for-byte (FRT_REPACK_CHECK memcmp gate, 40/40 regions identical; perplexity receipt unchanged). This removes the side-band region pack dependency. The lm-head keeps its pack: it is quantized from the BF16 checkpoint, which both scores and drafts measurably better than a rebuild from the GGUF's Q6_K copy; the online head build remains as a fallback.
- new family moe_expert_ffn v1: routed expert GLU-FFN with exact in-boundary top-k routing, optional sigmoid-gated shared expert, format-native weight consumption (host K-quant blocks in place, bit-exact activation-quant clone) and first-class token batches; reference implementation replicates the Qwen3.x softmax/top-k/clamp/renorm convention including tie-break. - gated_delta_core v2: snapshot_per_token state-update and replay_in_region checkpoint variants — the rollback discipline speculative verify batches require of stateful regions, with the failure modes documented. - autoregressive_decode_pipeline v3: optional spec_draft stage plus the speculative conformance set (output distribution matches target, explicit draft-state rollback, throughput judged on text). - rewrite the sm120 binding/pins against these families (previous ones were placeholders); qkv_pack/linear_proj variant additions are recorded as proposals instead of version bumps so the pi0.5 binding's pins stay green. Registry loads all 18 families; the existing pi0.5 pins verify green against the updated catalog.
Add the catalog charter (catalog/README.md): a structure entry is a local boundary expression — dimensions, contracts, variant semantics, executable reference — serving context alignment against native pipelines and the torch-side distribution boundary. It adjudicates nothing: performance numbers, verdicts and case histories are conditional on model/hardware/ host/driver and expire silently, so the only arbiter is a test run against the live system; dated results live in campaign records and per-binding qualification gates that are re-established by re-running them. Scrub the recent entries accordingly: drop campaign evidence and throughput numbers from moe_expert_ffn and gated_delta_core, reduce variant comments to pure semantics, drop the judgment-phrased speculative conformance line from autoregressive_decode_pipeline, keep only the parity-vs-reference gate on the new family.
…oldered) The binding yaml becomes the single source for every model-specific constant in the window translation unit: qkv_pack member tables (names, rows, offsets, leaders), region shapes and layer counts, gated-delta cell dims (conv row, state size), out-proj/head names and dims, layer scan bounds. tools/gen_binding_header.py generates a checked-in constants header from the binding's host_params section; the window logic itself is now family-level — a same-family model is a new binding file plus a header regeneration, with no window-code edit. Framework-wide ggml naming conventions (ffn_gate_exps, ffn_moe_out, ...) stay in the window code deliberately: they are host facts shared by every model on this host, not model facts. Also brings the binding up to the strict loader contract (token_select stage, complete_hot_path coverage with stock-served segments declared as host stages); it now validates through flash_rt.structures.binding with pipeline coverage required. Receipts: online repack byte-identity 40/40 vs the offline packs; tg128 and 24-chunk perplexity unchanged; speculative smoke coherent.
Move the sm120 heavy math out of the ggml window file into csrc as the next generation of the warp-split-K family (additive; the earlier bf16out/runtime-M entry is unchanged): compile-time row-count specialization (a runtime M in the MMA hot loop costs measurable time even at M=1), f32 output, and an optional programmatic-dependent-launch join so hosts that overlap every launch keep their chain. The M-rows activation quantizer ships alongside with its device body exposed for fused producers. The ggml adapter now only translates: it includes the csrc device header for its fused GDN producers and calls the csrc entries for standalone quantize/GEMV launches. Receipts unchanged: repack byte-identity 40/40, duplicated-token selftest bit-exact, tg128 and 24-chunk perplexity flat.
Compiled-in windows now run by default: GGML_CUDA_FLASHRT_DISABLE=1 turns the whole layer off, GGML_FLASHRT_NO_<X>=1 disables one window, and the historic FRT_<X>_SWAP switches remain as explicit A/B overrides. The in-process repack and the spec-draft head serving are part of the default set (the draft head builds lazily, only once a loaded draft model's head copy is actually seen, so plain runs spend nothing on it); the full-tier FP4 lm-head keeps its measured quality increment and stays opt-in, as do the archive windows. The draft model's own qkv projections match the kind-1 pack shapes and would be silently swept in; that is a separate acceptance-only judgment, gated off by default (FRT_DRAFT_REGIONS=1), measured flat here. Receipts: same-binary env vs zero-env tg128 identical; zero-env speculative server median matches the env-driven safe tier; zero-env 24-chunk perplexity bit-identical; whole-layer disable reproduces stock.
Offline gates: binding manifest validation, structure-version pins, and binding-header freshness (the generated constants header must match the binding yaml). On-device gates re-establish their numbers by running: duplicated-token bit-exact selftest, online-repack byte-identity against an offline reference pack, 24-chunk perplexity (pinned bit value), and tg128 against the tier's pinned baseline with a tolerance band. First full run: six greens (selftest PASS, repack 40/40 byte-identical, ppl 6.1106 exact, tg128 within band on the full tier).
…e state discipline)
Add the FlashRT-edition GGUF flow: an NVFP4 lm-head quantized from the BF16 checkpoint is spliced into the shipping quantized body (tools/splice_nvfp4_head.py). The stock nvfp4 kernels then serve the head at pack-tier speed by themselves, the adapter's spec-draft head serving wires up from the same tensor by pure shuffle (scale passthrough, alpha = 1), and the last side-band pack file is gone: the artifact judges faster-tier speed with better perplexity than the pack-served full tier on the same body. Judged along the way, gates recorded in the pins file: serving the NVFP4 head through the fp4-activation GEMV adds no speed over stock's q8_1-activation mmvq and costs measurable perplexity (archive switch, opt-in); a whole-model standard requantization loses to the shipping body both at the floor and in FP4-region cost (region quality follows the source tensor's quantization); the draft model's shape-colliding projections stay gated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second (arch, model-family) target for the native ggml adapter, stacked on #185: an LLM decode window set for Qwen3.6-35B-A3B (hybrid GDN + attention, 256-expert MoE, MTP speculative decode) on RTX 5090 (SM120), plus the catalog and tooling that make the next same-family model a data change instead of a code change.
What's in here
Windows (
fr_win_qwen36_sm120.cu) — fused-region NVFP4 W4A4 GEMV packs (GDN in_proj, attention qkv), the GDN cell span with per-token state snapshots and checkpoint replay for speculative verify batches (M ≤ 4), a format-native MoE expert span that consumes GGUF K-quant blocks in place through ggml's own vec_dot device functions (bit-exact q8_1 activation clone, shared expert folded), out-proj / router / lm-head / spec-draft-head windows. All launches join the host's programmatic-dependent-launch chain.Zero-configuration defaults — compiled-in windows run by default;
GGML_CUDA_FLASHRT_DISABLE=1reproduces stock,GGML_FLASHRT_NO_<X>=1disables one window, historicFRT_<X>_SWAPswitches remain as A/B overrides. Weights repack in-process on first sight of the tensors (pre-capture hook), byte-identical to the offline packer, so a stock GGUF runs as-is with no side files.Binding-driven windows — every model-specific constant (pack member tables, region shapes, layer counts, cell dims, head facts) lives in the binding's
host_paramsand reaches the code through a generated constants header (tools/gen_binding_header.py). The window logic is family-level; a same-family model is a new binding plus a header regeneration.Catalog — new
moe_expert_ffnfamily (routed expert GLU-FFN with exact in-boundary top-k, sigmoid-gated shared expert, format-native consumption, token batches as part of the boundary, executable reference);gated_delta_corev2 adds thesnapshot_per_token/replay_in_regionstate semantics speculative rollback requires;autoregressive_decode_pipelinev3 adds the optionalspec_draftstage and speculative conformance. A catalog charter (catalog/README.md) fixes the layer's role: structures express boundaries and never adjudicate — no performance claims or case histories in catalog entries. The pi0.5 binding's pins stay green against all of this.csrc — the warp-split-K GEMV family gains a next-generation entry (
fp4_w4a4_mma_warpsplit_mrows_f32out_sm120): compile-time row-count specialization (a runtime M in the MMA hot loop costs measurable time even at M = 1), f32 output, optional PDL join; an M-rows f32→NVFP4 activation quantizer ships alongside with its device body exposed for fused producers. Existing entries untouched.Qualification — one-command runner (
run_qualification_qwen36_sm120.py): binding manifest, structure pins, binding-header freshness, duplicated-token bit-exact selftest, online-repack byte-identity, pinned perplexity and throughput gates. Six greens on the recorded baseline.Artifact tooling —
tools/splice_nvfp4_head.pybuilds a packless speed-tier GGUF by splicing an NVFP4 lm-head (quantized from the BF16 checkpoint) into the shipping body; the stock nvfp4 kernels then serve the head at swap-tier speed with better perplexity than the side-band pack it replaces.Receipts (RTX 5090, 24-chunk
-ub 1perplexity, tg128 r=5, greedy server medians with text verification)-mddraft +-bs)Negative results are recorded in the pins file rather than the catalog, per the charter.
Notes for the Thor line
The window/matcher code, the binding-driven constants flow, the qualification form and the M ≤ 4 / snapshot / checkpoint disciplines are the reusable parts; the MoE and GDN spans are arch-portable CUDA. The SM120 MMA GEMV needs an SM110 counterpart (or the existing csrc GEMMs) and every judgment re-establishes on Thor's own roofline — in particular the launch-overlap capability differs, so the launch-count verdicts do not transfer.