cuda: add the backend contract and its checker by xiaoyu-xyz · Pull Request #25 · ThinkFlowLab/system1-omni

xiaoyu-xyz · 2026-09-28T09:02:17Z

Purpose

CUDA build paths were diverging: Laya generates CUDA from TileLang (#14), Cua-S1 hand-writes CUDA C++ with cuBLASLt (#19), and the multimodal worker plans Triton (#12). Nothing linked against anything else, so the divergence was invisible until one model reused another's kernels.

This adds the contract and the checker:

contract.md — what backends must agree on: a JSON manifest per backend, four C ABI rules, the numerics that must be declared rather than discovered in a parity failure, and the build-script interface.
check_contract.py — reads the manifests and checks them against that contract.
build_script.py — reads a build script as text and reports what it declares. Never executed: CI must not run repository code to decide whether a manifest is honest.

A backend may be a subdirectory named after itself or sit directly under src/backends/cuda/, which is the layout Laya's kernels/ and tools/ use.

What the checker enforces

The manifest schema, and that every declared source exists.
That build.sh declares the architectures the manifest claims and writes the library the manifest names — from the -gencode flags, resolved against the variable environment in force at each line, so a variable that never reaches nvcc cannot make a target look reachable.
That abi_version matches the <PREFIX>_ABI_VERSION macro in the declared sources. This is the one field whose truth lives in the code, and nothing read it before.
That a validated backend declares both a tolerance and a reference entrypoint.

Test Plan

python3 -m unittest discover -s src/backends/cuda/tests -t .
python3 src/backends/cuda/check_contract.py --repo-root .

Test Result

63 tests. Verified on a clean git archive of the head commit, not only in a working tree.

Against real trees rather than fixtures: #19's merged qwen3_5/ is accepted (0 errors, 0 warnings at its real abi_version 4) and reports a mismatch when the manifest says otherwise; a manifest named typo.backend.json is reported instead of silently exempting the directory; a non-string reference.entrypoint produces a structured issue rather than a TypeError that suppressed the whole report.

On the 500-line guideline

696 lines of core Python (check_contract.py 469, build_script.py 227) — about 39% over, not slightly over. contract.md is documentation and not counted. The two files are one change: the parser is what the checker reads build scripts with, and separating them would ship a checker that cannot run. Splitting further would mean shipping a checker that does not check what this one does.

Status against current `main`

main now has qwen3_5/ from #19 and no manifest for it, so the checker exits 1 there. That is the forcing function working, and it makes the manifest a required follow-up rather than an optional one — it is #77, which also corrects abi_version to the library's real 4.

Deliberately not included

Compile and parity tiers. They need a CUDA container and a GPU runner; contract.md states them as requirements and wiring them is separate, so this needs no CUDA toolkit and no GPU.
Extracting a shared CUDA runtime surface. One backend exists, so the shape would be guesswork.
Reading the built artifact. A build check proves the sources compile and a file of the declared name appears; it does not prove the binary honoured build.architectures, which would need cuobjdump.

Related: #6 (layout), #9 (model roadmap), #12, #14, #19.

Both models that need this readout share the shape of it: Kev (ThinkFlowLab#27) scores scale * dot(k_proj(c_i), q_proj(s)) and softmaxes over a question's candidates; CLM (ThinkFlowLab#9) scores exp(logit_scale) * cos(state_head(s), action_head(c)) and does the same. ThinkFlowLab#9 asks for exactly this fusion, and ThinkFlowLab#19's gemm.cu does not have it. The fusion is more than fewer launches. For a cosine the candidate's norm and its dot with the query need the same elements, so both accumulate in one read of C -- for Kev, 255 rows of 2560 floats read once instead of twice. The softmax then runs in-block over shared memory, so there is no second kernel, no atomics and no global round trip for the similarities. A batch is a grid over questions rather than a loop of launches. candidate_scoring.cu has NEVER BEEN COMPILED. The authoring machine has no CUDA toolkit and no NVIDIA GPU, so nothing here claims it compiles, runs or is fast. What is established is the arithmetic it performs: - reference.py is the float64 oracle, and its invariants are tested: a distribution, 1/K for indistinguishable candidates, no NaN for a zero candidate, no overflow where a naive softmax would produce inf. - kernel_simulation.py reproduces the kernel's algorithm in numpy -- float32 accumulation, the max-subtracted softmax, the norm floored as max(sqrt(sum), eps) and not sqrt(max(sum, eps)) -- and agrees with the oracle to a worst absolute difference of 1.8e-07 over the fixed cases. The simulation rounds twice per multiply-add where the kernel uses fmaf, so that is an upper bound on the kernel's error. - The reduction is a tree, matching __shfl_down_sync, tested structurally and shown to differ from a sequential sum on a concrete input. Tolerance is declared at 1e-4, three orders of magnitude above the simulated worst case, so a hardware failure is a failure rather than tolerance noise. gpu_parity.py checks a compiled library against the reference and skips with a reason when there is no GPU or no library. The inputs are not committed: they are deterministic from a fixed seed and would be about a megabyte of generated floats. 17 tests pass. The backend satisfies the contract from ThinkFlowLab#25, which caught the manifest over-claiming architectures the build script does not reach by default.

Both models that need this readout share the shape of it: Kev (ThinkFlowLab#27) scores scale * dot(k_proj(c_i), q_proj(s)) and softmaxes over a question's candidates; CLM (ThinkFlowLab#9) scores exp(logit_scale) * cos(state_head(s), action_head(c)) and does the same. ThinkFlowLab#9 asks for exactly this fusion and ThinkFlowLab#19's gemm.cu does not have it. The fusion is more than fewer launches. For a cosine the candidate's norm and its dot with the query need the same elements, so both accumulate in one read of C -- for Kev, 255 rows of 2560 floats read once instead of twice. The softmax runs in-block over shared memory, so there is no second kernel, no atomics and no global round trip for the similarities. A batch is a grid over questions. MEASURED, not just intended. RTX 4090 (sm_89), driver 595.58.03, CUDA 13.0 (V13.0.88): worst absolute difference: 1.835e-07 (tolerance 0.0001) all cases within tolerance All 13 fixed cases pass, including a single candidate, identical candidates, a zero candidate, K=255, and logits whose raw exponential overflows. Three defects were found and fixed; the first two only a GPU could show. 1. The parity harness passed host pointers where device pointers were expected, so the kernel dereferenced host memory as device memory. compute-sanitizer located it as an invalid global read with consecutive lanes on consecutive addresses. Fixed by allocating, copying and synchronising through cudart. 2. The query norm was summed across warps that had each computed the same partial: the inner loop already covers all of D within one warp, so the norm came out sqrt(8) too large on a 256-thread block. Every unnormalized case sat at float32 rounding while the normalize cases were off by ~1e-2, which is what pointed at it. Warp 0 now computes it alone. kernel_simulation.py did NOT catch this, because it was written from the intent rather than transcribed from the .cu. A simulation is only as good as its fidelity; hardware is what settles it. 3. The manifest claimed sm_89 while build.sh defaults to sm_80. The contract checker from ThinkFlowLab#25 caught that, as it caught the same class of over-claim in the Laya backend. The manifest is now status=validated with a declared tolerance and a reference entrypoint, which is what the contract requires of a backend making a parity claim. The reference is float64 in reference.py, and gpu_parity.py skips with a reason rather than passing when there is no GPU or no library. Not yet exercised: the batch path beyond questions=1, the K>255 rejection, and any architecture or CUDA version other than sm_89 on 13.0.

hsliuustc0106

Reviewed commit 391cf45e4013b833473e81b71508d547217f1605.

[P2] Include name in the missing-required-field guard. src/backends/cuda/check_contract.py:131 omits name from the early-return condition, then accesses manifest["name"]. A manifest missing only name produces KeyError instead of the intended structured validation report (including --json), aborting checks of other backends. Reproduced directly; all 53 existing tests pass.

xiaoyu-xyz · 2026-09-30T01:12:41Z

Confirmed and fixed in a81a41a — thanks for the precise mechanism.

The required-key list was written twice: once to report the keys missing, and again in the guard that returns before they are read. name was in the first only. A manifest missing just name therefore reported it and then read it anyway, raising out of check_manifest — which also dropped the other backends' results and the --json output, as you saw.

Both now derive from one REQUIRED_KEYS constant. The tests remove each required key in turn and assert a structured error rather than an exception, plus one that reproduces your case directly: a bad manifest beside a good one, asserting the good one is still checked and listed.

Levius-Fubuki

Reviewed a81a41ae2ab8d1b15be868320fb2a85d97b283c7, including the checker, build-script parser, contract and tests.

The original missing-name finding is fixed: the old revision raises KeyError, while this head produces a structured missing-key issue. All 55 existing Python tests passed on Linux; the checker also ran successfully against this branch (which declares no backends).

Changes requested for three independently reproduced cases below. Each uses an isolated fixture and the real checker CLI; the valid control returns zero issues. No CUDA build or GPU parity claim is made.

hsliuustc0106 · 2026-10-03T06:03:46Z

resolve the comments please

All three come from review on ThinkFlowLab#25. 1. reference.entrypoint was used as a path without checking its type. A list reached os.path.isabs, whose TypeError escaped check_manifest: the CLI printed no structured report at all, not even under --json, and every later backend went unchecked. Non-strings are now an Issue and checking continues. 2. A directory holding any *.backend.json was exempt from the undeclared-backend check, but discovery only loads <dir>/<dir>.backend.json. A manifest named typo.backend.json therefore bypassed every check for that backend: exit 0, checked: [], no errors and no warnings. A manifest that is present but not under the expected name is now reported. 3. Architectures read from `arch=` variables and targets written literally in a -gencode flag were combined with `or`, so the literals were discarded whenever a variable existed. A script assigning `arch=${2:-89}` and then compiling `-gencode arch=compute_90,code=sm_90` builds sm_90 only, yet a manifest declaring [89] passed with no findings. build_script.py now reports gencode_architectures: the targets that actually reach nvcc, resolved against the variable environment in force at each line, so a loop that reassigns `arch` still yields both targets. The checker treats those as the authority and reports an assigned value that never reaches a -gencode flag. 62 tests, seven of them new and one per defect. Verified against ThinkFlowLab#19's real build.sh, which still parses to variables [89] / gencode [89] and passes with no findings.

xiaoyu-xyz · 2026-10-03T11:58:05Z

All three fixed in dc3fd16.

reference.entrypoint is type-checked before it is used as a path. A non-string is an Issue and the remaining backends are still checked, so --json always produces a report.
A manifest that is present but not under the name discovery looks for is now reported rather than exempting the directory — has typo.backend.json but not x.backend.json, so it is never discovered.
build_script.py now reports the targets that actually reach nvcc, taken from the -gencode flags and resolved against the variable environment in force at each line, so a loop that reassigns arch still yields both targets. The checker treats those as the authority and reports an assigned value that never reaches a flag.

Your third case now fails as build script assigns [89] but only passes [90] to nvcc.

62 tests, one per defect plus the two guards. Verified against #19's real build.sh, which still parses to variables [89] / gencode [89] and passes with no findings.

hsliuustc0106

Independent review — verdict: ready to approve. All findings are non-blocking. Everything below ran on CPU/macOS (Python 3.9.6); no CUDA involved.

Verified locally (detached worktree at dc3fd162):

python3 -m unittest discover -s src/backends/cuda/tests -t . — 62 tests, OK
python3 src/backends/cuda/check_contract.py --repo-root . — exit 0, no backends declared ✓ (true at this PR's base, which predates qwen3_5/; see finding 3)
The build-script fixture matches the real qwen3_5/build.sh argument handling verbatim (out=${1:?…}, arch=${2:-${CUDA_COMPUTE_CAP:-89}}, -gencode "arch=compute_${arch},…", -o "$out/libqwen3_5_cuda.so"), and the parser reads scripts as text without executing them ✓
The contract.md example manifest, written as qwen3_5/qwen3_5.backend.json against the real qwen3_5/ tree: accepted, 0 errors, 2 warnings (src/models/kev/ and recipe/cua_s1/check_native.py don't exist yet) — both by design

Findings (non-blocking):

The example manifest understates the real ABI. contract.md shows "abi_version": 1 for a manifest presented as qwen3_5's shape (its exact 9 sources, status: validated), but the real library is CS1_ABI_VERSION 3 (src/backends/cuda/qwen3_5/ops.h:16). The checker can't catch that mismatch (it doesn't read headers), so the example is the only guard — please use 3, or state the numbering explicitly.
Wrong file cited for the sm_80 note. §3 says "#19's attention.cu documents sm_80+" — attention.cu has no arch note at main. The notes live in mma.cuh:1 and gdn_prefill.cu:1.
"No backends declared" holds only at this PR's base. Merged into current main, the checker exits 1: [error] qwen3_5: has kernel sources (...) but no qwen3_5.backend.json manifest — verified in a simulated merge. That is the forcing function working as designed, but it makes a qwen3_5.backend.json follow-up required, not optional; worth stating here or landing the manifest in this PR.
Body numbers are stale at the head commit: 7 files / +1630 (not 6 / +1448), 62 tests (not 53), 644 core lines (not 566) — so the overage vs the 500-line guideline is ~29%, not "slightly over".
Nits: the three ParseBuildScriptTest cases are duplicated between test_build_script.py and test_check_contract.py; the "not executable in git" message checks the filesystem bit (os.access), not the index — wording only, and the suggested fix (git update-index --chmod=+x) is the right one.

Not verified here: the tampering matrix and the #16 tools/ rejection (no tools/ on main to point the checker at), and CI coverage — ci.yml runs only the Rust suite and tests/benchmarks, so neither these tests nor the checker run in CI. The PR states that wiring is separate and it stays out of scope; agreed.

All from the independent review on ThinkFlowLab#25. The manifest's `abi_version` was ambiguous and unverifiable. contract.md described a per-library `<prefix>_abi_version()` symbol but never said the manifest field was that number, and its example showed 1 for a manifest presented as qwen3_5's shape — while ThinkFlowLab#19's ops.h says CS1_ABI_VERSION 4. Nothing read the header, so a manifest disagreeing with its own library was accepted and the example was the only guard. `abi_version` is now defined as the library's own, and the checker reads the `#define <PREFIX>_ABI_VERSION N` out of the declared sources and requires the manifest to match. Against the real tree this reports abi_version is 1 but ops.h defines CS1_ABI_VERSION 4 and passes at 4. A backend whose sources define no such macro gets a warning rather than silence, and sources defining two different values are an error. The cross-backend sameness rule is removed. It would have forced unrelated model engines onto one interface version, which the repository layout explicitly does not ask for; each manifest now has to match its own header instead. A test asserts two backends may declare different versions. Two documentation errors the review found: - §3 cited `attention.cu` for the sm_80 note. That file has none; `mma.cuh` and `gdn_prefill.cu` each say "sm_80 and later" on their first line. - The example manifest now uses 4, matching ops.h. Two nits: three parser cases were duplicated between test_build_script.py and test_check_contract.py and are now only in the former, where the parser is tested; and the not-executable message said "in git" while it checks the working tree's bit, which a fresh clone takes from the committed one. 63 tests.

The manifest said abi_version 1. ThinkFlowLab#19's ops.h defines CS1_ABI_VERSION 4, so the declaration disagreed with the library it describes. Nothing caught it when this was written, because the checker did not read headers. ThinkFlowLab#25 now does: it reads the `#define <PREFIX>_ABI_VERSION N` out of the declared sources and requires the manifest to match, which is what surfaced this. abi_version is 1 but ops.h defines CS1_ABI_VERSION 4 also strengthens the description in the same PR.

xiaoyu-xyz · 2026-10-04T14:52:26Z

All five addressed in 19f95a4.

abi_version. You were right that the example was the only guard, and that nothing read the header. The field is now defined as the library's own rather than a version of the document, and the checker reads #define <PREFIX>_ABI_VERSION N out of the declared sources and requires the manifest to match. Against the real tree it reports abi_version is 1 but ops.h defines CS1_ABI_VERSION 4; the value is 4, not 3 — ops.h has been bumped since you read it. No macro in the sources is a warning rather than silence, two different values inside one backend is an error, and the cross-backend sameness rule is gone: it would have forced unrelated engines onto one interface version, which the layout explicitly does not ask for. A test asserts two backends may declare different versions.
Fixed — mma.cuh and gdn_prefill.cu, both of which say "sm_80 and later" on line 1.
Agreed, and cuda: declare the qwen3_5 backend #77 is that follow-up. It also corrects the manifest's abi_version to 4, which your finding 1 is what surfaced. The body now says it is required rather than optional.
Body corrected and stated plainly: 6 files / +1719, 63 tests, 696 lines of core Python, 39% over the guideline.
The three parser cases now live only in test_build_script.py, where the parser is tested; and the message no longer says "in git", since it reads the working tree's execute bit.

The header gap was the useful one — thanks. A convention whose only enforcement is an example in a document is not enforced.

…ed against `cs_score_abi_version()` returned a literal and nothing else in the source stated the number, so `scoring.backend.json`'s `abi_version` had no counterpart to disagree with. ThinkFlowLab#25 now reads `#define <PREFIX>_ABI_VERSION N` out of a backend's declared sources and requires the manifest to match; this backend was the one it reported a warning for. The macro is defined once and the function returns it, so the two cannot drift: the manifest says 1, the macro says 1, and a manifest saying anything else is now reported as `abi_version is 2 but candidate_scoring.cu defines CS_SCORE_ABI_VERSION 1`. No behaviour change: the function body expands to `return 1;` exactly as before, so the compiled library is unchanged apart from line numbers. The file has not been recompiled here -- there is no CUDA toolkit on this machine -- but the preprocessed result is what it was. 17 tests still pass.

Levius-Fubuki · 2026-10-04T15:30:41Z

Rechecked 19f95a4: the original fixes pass, but the new ABI scan crashes on "sources": 42 or true, aborting the entire check without JSON output. Please validate sources before scanning and add a malformed-manifest regression test.

xiaoyu-xyz mentioned this pull request Sep 28, 2026

cuda: add a single build entry point for the Laya backend linear3735/system1-omni#1

Open

xiaoyu-xyz force-pushed the cuda-backend-contract branch 3 times, most recently from 6e25eee to 391cf45 Compare September 28, 2026 09:36

xiaoyu-xyz changed the title ~~cuda: add the backend contract, a contract check, and CUDA CI~~ cuda: add the backend contract and its checker Sep 28, 2026

This was referenced Sep 28, 2026

cuda: add the compile-matrix and parity drivers xiaoyu-xyz/system1-omni#1

Open

ci: run the CUDA checks xiaoyu-xyz/system1-omni#2

Open

[RFC]: Native Laya inference with Rust and CUDA #14

Open

Levius-Fubuki mentioned this pull request Sep 29, 2026

Cua-S1 multimodal stack: validation results and cross-PR integration checklist #36

Open

4 tasks

hsliuustc0106 requested changes Sep 29, 2026

View reviewed changes

xiaoyu-xyz force-pushed the cuda-backend-contract branch 2 times, most recently from 19bdff7 to a81a41a Compare September 30, 2026 01:12

Levius-Fubuki requested changes Oct 1, 2026

View reviewed changes

Comment thread src/backends/cuda/check_contract.py Outdated

Comment thread src/backends/cuda/check_contract.py Outdated

Comment thread src/backends/cuda/check_contract.py Outdated

xiaoyu-xyz force-pushed the cuda-backend-contract branch from a81a41a to dc3fd16 Compare October 3, 2026 11:57

xiaoyu-xyz mentioned this pull request Oct 3, 2026

cuda: add a fused candidate-scoring kernel for CLM and Kev #69

Draft

hsliuustc0106 reviewed Oct 4, 2026

View reviewed changes

xiaoyu-xyz mentioned this pull request Oct 4, 2026

cuda: declare the qwen3_5 backend #77

Draft

4 tasks

Navigation Menu

cuda: add the backend contract and its checker - #25

cuda: add the backend contract and its checker#25
xiaoyu-xyz wants to merge 2 commits into
ThinkFlowLab:mainfrom
xiaoyu-xyz:cuda-backend-contract

xiaoyu-xyz commented Sep 28, 2026 •

edited

Loading

Uh oh!

hsliuustc0106 left a comment

Uh oh!

xiaoyu-xyz commented Sep 30, 2026

Uh oh!

Levius-Fubuki left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hsliuustc0106 commented Oct 3, 2026

Uh oh!

xiaoyu-xyz commented Oct 3, 2026

Uh oh!

hsliuustc0106 left a comment

Uh oh!

xiaoyu-xyz commented Oct 4, 2026

Uh oh!

Levius-Fubuki commented Oct 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

xiaoyu-xyz commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

What the checker enforces

Test Plan

Test Result

On the 500-line guideline

Status against current main

Deliberately not included

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

xiaoyu-xyz commented Sep 30, 2026

Uh oh!

Levius-Fubuki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hsliuustc0106 commented Oct 3, 2026

Uh oh!

xiaoyu-xyz commented Oct 3, 2026

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

xiaoyu-xyz commented Oct 4, 2026

Uh oh!

Levius-Fubuki commented Oct 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xiaoyu-xyz commented Sep 28, 2026 •

edited

Loading

Status against current `main`