Add Cua-S1 0.2 multimodal CUDA worker with upstream parity by Levius-Fubuki · Pull Request #12 · ThinkFlowLab/system1-omni

Levius-Fubuki · 2026-09-27T12:08:45Z

Change

Add screenshot-conditioned Cua-S1 inference with strict inline-image/request validation, pinned weight verification, direct Transformers/PEFT execution, and a loopback HTTP worker. The HTTP adapter lives in src/frontend/cua_s1.py; model execution and protocol handling live in src/models/cua_s1/multimodal/.

Scope

Only runtime Python source is included in the diff against main: 3 files, +571/-0 lines. Tests, recipe/benchmark/profiling tools, documentation, dependency lists, CI and ignore-file changes are excluded. The upstream license notice remains in the source header.

Validation

Head 934e1ad1: 73 passed, using the preserved pre-cleanup tests and recipe helpers outside this checkout, with imports directed at this head's runtime source.
Runtime source is byte-for-byte unchanged from the pre-cleanup head. Ruff lint/format, whitespace checks and the frontend.cua_s1 --help entry point pass.
GPU numerical/performance experiments were not rerun for this scope-only cleanup.

Setup and archived support files

Pinned dependencies, weight downloader and setup guide and the validation suite are retained at the immutable pre-cleanup revision. These helper files are no longer part of the current branch. The worker still requires the verified upstream weights.lock.json beside the base checkpoint directory.

Launch the prepared environment with:

PYTHONPATH=src python -m frontend.cua_s1 \
  --base weights/Qwen3.5-4B \
  --adapter weights/cua-s1-4b-0.2/multimodal

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

twu3202 · 2026-09-27T12:45:11Z

Thanks, this lines up well with #11. I added a small follow-up commit to #11 so both workers answer errors the same way: error bodies are {"detail": "..."} (what the LAYA worker returns), malformed JSON including repeated keys is 400, and a question without instructions is 422. #12 currently returns 422 with "error" for all of these. Could it switch to "detail", use 400 for malformed JSON, and require instructions?

The text worker will sit next to yours in src/models/cua_s1/text/, with tests/cua_s1/test_text_*.py and recipe/cua_s1/text.md, so the two PRs only overlap in one row of the README layout table.

hsliuustc0106 · 2026-09-27T22:53:59Z

fix conflicts

Levius-Fubuki · 2026-09-28T01:09:27Z

Fixed in ee970c2: merged the latest main, resolved conflicts, and aligned the error contract with #11. Local tests and checks pass; CI is awaiting maintainer approval.

Resolve the .gitignore and recipe/README.md conflicts with the frontend merge (ThinkFlowLab#2): use the same Python ignores as ThinkFlowLab#12, and list the Cua-S1 text recipe next to the Laya one. Signed-off-by: Tianyao Wu <rayroy31@gmail.com>

hsliuustc0106

Two actionable findings: unsupported image aspect ratios currently return HTTP 500, and lock-release assertions race with handler cleanup.

Validation: full CPU suite: 61 passed, 1 failed; focused server rerun: 24 passed, 1 failed. Both failures were the lock assertion on different malformed inputs. The reused environment had Pillow 12.3.0 rather than pinned 11.3.0. The aspect-ratio case was reproduced through the real worker handler with a processor stub invoking the smart_resize function extracted from pinned Transformers 5.17.0. Archived reference/candidate reports agree exactly on tensor fingerprints and probabilities for all nine forwards. GPU inference was not rerun.

hsliuustc0106 · 2026-09-29T13:40:43Z

+            if source.format != expected[prefix]:
+                raise InvalidRequest("image format does not match its MIME type")
+            w, h = source.size
+            if (


[P2] Reject unsupported image aspect ratios with 422

A 2048×1 PNG passes these size checks, but the pinned Qwen image processor rejects aspect ratios above 200 with ValueError. Since prepare() does not translate this input error into InvalidRequest, the HTTP handler returns 500 ("inference failed") for a well-formed unsupported request, contrary to the documented 422 contract. Reproduced through the worker handler using the pinned resize function. Validate the aspect ratio before processing and add a regression test for its HTTP rejection.

Fixed in e6cf582: images above 200:1 now return 422 before inference. Added HTTP rejection and boundary tests for both orientations.

hsliuustc0106 · 2026-09-29T13:40:43Z

+    server.engine.predict = unexpected
+    status, body = call(url + "/v1/systemone", raw)
+    assert status == 400 and set(body) == {"detail"}
+    assert not server.inference_lock.locked()


[P2] Synchronize the lock-release assertion with handler cleanup

Receiving the complete HTTP response does not guarantee the server thread has executed its finally block: send_json() writes the response before inference_lock.release(). This immediate assertion therefore races with cleanup. It failed in both the full suite and a focused server rerun, on different malformed inputs. Wait for handler completion or use a bounded lock acquisition (releasing it afterward) to verify eventual cleanup. The assertion in test_model_failure_is_not_reported_as_success at line 88 has the same race.

Fixed in e6cf582 with bounded lock acquisition and deterministic delayed-cleanup tests. All 70 tests pass with Pillow 11.3.0 and 12.3.0; the 14 cleanup cases also passed three extra runs.

hsliuustc0106 · 2026-09-29T16:00:26Z

@@ -0,0 +1,21 @@
+MIT License


remove this

hsliuustc0106 · 2026-09-29T16:01:14Z

@@ -0,0 +1,136 @@
+"""Small loopback HTTP worker; the Rust frontend remains the public serving layer."""


do we need to push server under the model folder?

hsliuustc0106 · 2026-09-29T16:01:28Z

@@ -0,0 +1,102 @@
+{


is this necerssary?

Remove archived experiment outputs and assistant planning notes. Keep runtime code, reproducible tools, and model verification metadata unchanged. Link historical reports to an immutable commit and ignore future local artifacts. Remove only tests tied to the deleted historical evidence bundles.

…fest

Remove supplemental tests, recipe tools, documentation, CI, and configuration changes from the PR diff. Runtime Python source and inline third-party notice are unchanged. Validation uses the pre-cleanup test/tool snapshot outside the checkout.

hsliuustc0106 · 2026-09-30T08:54:42Z

Thanks for the implementation and parity validation. I’d like to clarify the architectural direction before merging.

The current Cua-S1 plan explicitly starts with a Transformers/PEFT worker, so this PR follows that approach. However, I want System1-Omni to prioritize native model execution with CUDA/Metal backends. Expanding the Python worker to multimodal inference doesn’t advance that goal.

Please keep this implementation available as a correctness reference and rework the production contribution toward a minimal native Cua-S1 text/prefill path first. Vision and LoRA support can follow incrementally. We should update the existing Cua-S1 plan to reflect this direction as well.

twu3202 · 2026-09-30T09:10:53Z

For the text path, #19 already runs Cua-S1 natively: Rust and its own CUDA kernels for Qwen3.5-4B, with the text adapter merged into the weights beforehand and no Python or PyTorch at run time. I'm trimming it now to a minimal version (eager execution, cuBLASLt's default GEMM choices, no Python-compatibility layer), with CUDA Graphs and GEMM tuning as separate follow-ups. I'll also update the plan in src/models/cua_s1/README.md to put the native path first, with the Transformers workers kept as the correctness reference it is checked against. Vision could then build on the same native path.

hsliuustc0106 · 2026-09-30T09:46:56Z

@Levius-Fubuki I will merge it for now but please check my comments

Levius-Fubuki · 2026-09-30T12:18:30Z

Thanks for merging
I’ll keep this as a correctness reference and align further contributions with the native CUDA/Metal direction, coordinating with #19 to avoid duplicating the text/prefill work.

Add a /v1/systemone worker for the Cua-S1 4B 0.2 text adapter in src/models/cua_s1/text/, next to the multimodal worker proposed in ThinkFlowLab#12. It loads the base model and the PEFT adapter directly through Transformers and PEFT, follows the contract in src/models/cua_s1/README.md, and answers choice questions only. Add the fixed input set and tests in tests/cua_s1/ (contract and HTTP tests that need no weights, and tokenizer checks), and recipe/cua_s1/text.md with setup, launch, a parity check against upstream FourBModel and a latency script. Ignore the recipe's weights/ and .venv/ with the same .gitignore lines as ThinkFlowLab#12. Part of ThinkFlowLab#10. Signed-off-by: Tianyao Wu <rayroy31@gmail.com>

Add Cua-S1 multimodal CUDA worker and parity recipe

74c95e5

Copilot AI lite review requested due to automatic review settings September 27, 2026 12:08

Copilot AI reviewed Sep 27, 2026

twu3202 mentioned this pull request Sep 27, 2026

docs: add Cua-S1 4B 0.2 inference contract #11

Merged

twu3202 mentioned this pull request Sep 27, 2026

cua_s1: add a text worker that loads Qwen3.5-4B through Transformers #13

Merged

4 tasks done

Merge main and align multimodal worker error contract

ee970c2

twu3202 mentioned this pull request Sep 28, 2026

cua_s1: add a native CUDA text worker #19

Merged

4 tasks done

This was referenced Sep 28, 2026

Measure CUDA Graph replay on Cua-S1 multimodal language forward #20

Closed

Serve Cua-S1 multimodal requests with bounded segmented CUDA Graphs #22

Merged

This was referenced Sep 28, 2026

cuda: add the backend contract and its checker #25

Open

cuda: add the compile-matrix and parity drivers xiaoyu-xyz/system1-omni#1

Open

ci: run the CUDA checks xiaoyu-xyz/system1-omni#2

Open

[RFC]: Native Laya inference with Rust and CUDA #14

Open

Levius-Fubuki mentioned this pull request Sep 29, 2026

Cua-S1 multimodal stack: validation results and cross-PR integration checklist #36

Open

4 tasks

hsliuustc0106 requested changes Sep 29, 2026

View reviewed changes

Merge main and fix multimodal validation review findings

e6cf582

hsliuustc0106 reviewed Sep 29, 2026

View reviewed changes

Levius-Fubuki added 2 commits September 30, 2026 00:35

Move Cua-S1 HTTP serving out of models and reuse upstream weight mani…

9093fd8

…fest

chore: limit PR to multimodal runtime code

934e1ad

Remove supplemental tests, recipe tools, documentation, CI, and configuration changes from the PR diff. Runtime Python source and inline third-party notice are unchanged. Validation uses the pre-cleanup test/tool snapshot outside the checkout.

hsliuustc0106 merged commit b50aa28 into ThinkFlowLab:main Sep 30, 2026

twu3202 mentioned this pull request Sep 30, 2026

docs: build a documentation site with MkDocs and deploy it to Pages #51

Merged

4 tasks done

Levius-Fubuki mentioned this pull request Sep 30, 2026

Add Cua-S1 multimodal reference tensor exports #53

Open

4 tasks

twu3202 mentioned this pull request Oct 1, 2026

docs: add a supported models and hardware page #60

Merged

4 tasks done

		@@ -0,0 +1,136 @@
		"""Small loopback HTTP worker; the Rust frontend remains the public serving layer."""

Conversation

Levius-Fubuki commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change

Scope

Validation

Setup and archived support files

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

twu3202 commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsliuustc0106 commented Sep 27, 2026

Uh oh!

Levius-Fubuki commented Sep 28, 2026

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Levius-Fubuki Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

Levius-Fubuki Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 commented Sep 30, 2026

Uh oh!

twu3202 commented Sep 30, 2026

Uh oh!

hsliuustc0106 commented Sep 30, 2026

Uh oh!

Levius-Fubuki commented Sep 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Levius-Fubuki commented Sep 27, 2026 •

edited

Loading

twu3202 commented Sep 27, 2026 •

edited

Loading