fix(local-runtime): stop silently shipping the CPU engine to GPU hosts - #13
Merged
Conversation
added 6 commits
August 24, 2026 21:41
A box with four H200s installed and ran linux-x86_64-cpu. No error, no warning — a 9B model ran on CPU next to four idle GPUs, and the CPU choice was indistinguishable from a correct one. Root cause: gpu_detect scraped the literal "CUDA Version:" out of nvidia-smi's banner. Driver 610.43.02 prints "CUDA UMD Version: 13.3", so the parse returned None, recommend_backend_for skipped its CUDA branch, and selection fell through to cpu. Everything else worked; only the version parse failed. Rather than add a second string literal to a scrape that vendors keep changing, host capability detection now reads layered, evidence-carrying sources through the shared ziee-hardware parser: nvidia-smi --version, then the banner, then -q (driver-reported), then nvcc and the libcudart soname (toolkit-derived, and only when --query-gpu confirms a real device, because is_cuda_available() is satisfied by a library file with no driver check). Detection now says what it did: - INFO naming the version and which source produced it - WARN when a GPU is present but no version could be read, listing every source tried - WARN once when a GPU is present and cpu was selected anyway, printing the detected versions and the published tags so a parser failure and a genuinely missing artifact are distinguishable The predicate behind that warning is a pure function with its own test — an invariant living only inside a log line is not an invariant. Cross-platform, which is the point of the refactor: - Windows: CUDA was never detected at all, since the resolver had only Unix paths and never appended .exe. It now resolves from OS-set locations, built from environment rather than hardcoded drive letters. CUDA_PATH/HIP_PATH are user-settable so they serve toolkit binaries only — nvidia-smi, the authoritative probe, cannot be redirected by them. Per-binary policy is additive over the generic scan, so no existing name stops resolving. - ROCm: the existing /opt/rocm/.info/version stays first and unchanged; further sources follow. A major is never guessed when all are silent, since selection needs an exact major match and a wrong guess loads a broken build. - Metal: deliberately unchanged. Both macOS arms already return true, so a runtime-arch check would be a nil-gain edit to code no machine here can compile. The reasoning is recorded in-file so it is not re-litigated. Also memoise the version probes. recommend_backend runs once per release row at three call sites with per_page up to 500, and detect_cuda_version was unmemoised, so nvidia-smi was re-spawned once per row per request. Verified on the affected hardware: 4 GPUs, version 13.3 via nvidia-smi --version, selection cuda13.2. Not verified, and stated as such: macOS (no Darwin toolchain), Windows (no host), AMD/ROCm (no hardware).
… resolution
Blind audit findings on the previous commit. The first is a security
regression that commit introduced.
resolve_system_binary's Windows branch only returned on a hit and then fell
through to the POSIX trusted-dir scan. Those paths are not inert on Windows:
PathBuf::from("/usr/bin").join("nvidia-smi.exe") resolves against the current
drive as C:\usr\bin\nvidia-smi.exe, and the default C:\ ACL lets an
unprivileged user create that directory and own what is in it. The server
would execute an attacker-planted binary as its own user and parse its stdout
as the host CUDA version. Adding EXE_SUFFIX is exactly what armed this — the
old resolver joined a bare, unlaunchable name — and it needed no environment
control, bypassing the env split entirely. Gate the POSIX scan off on Windows.
Stop resolving any executable from a user-settable variable. The previous
split protected which ANSWER was trusted, not which BINARY ran: rocm-smi is
spawned unconditionally from detect_all(), so %HIP_PATH%\bin\rocm-smi.exe was
code execution in the server process for anyone who could set one env var —
the same PATH-shadowing class the trusted-dir list exists to close. The cost
is that a Windows nvcc in a custom toolkit dir is no longer found; nvidia-smi
lives in System32 and is the primary source, so CUDA detection is unaffected.
Refuse UNC paths. \\attacker\share is absolute and dot-dot-free, so path
hygiene alone accepted it, yielding a remote binary over SMB with NTLM
authentication to the attacker's host. A test had asserted UNC was accepted,
pinning the worst case rather than guarding it; it now asserts refusal.
Stop the loud warning firing on a false positive and then going silent. A
release with no assets for this platform yields an empty published set, which
was read as a CPU fallback: it warned wrongly (nothing was selected) and spent
the one-shot latch, swallowing the genuine occurrence later in the process.
Since selection runs once per catalogue release, this fired routinely.
Abort the driver-probe chain on the first timeout. These calls are made
synchronously from async fns with no spawn_blocking, so paying PROBE_TIMEOUT
per flag variant meant ~18s of a wedged worker on the very host this change
targets. Retrying the same unusable binary with different flags cannot help.
Read only the ROCM version label from rocm-smi. ROCM-SMI-LIB is that library's
own semver and is decoupled from the release — ROCm 6.x ships
librocm_smi64.so.7 — so it would report major 7 for a ROCm 6 host and, once a
rocm7 artifact exists, install a build that cannot load.
Validate $ROCM_PATH before joining it into a filesystem read, matching the
rule already applied on the Windows path.
…uild Second blind audit round. The main finding is a regression the FIRST round's fix introduced — this feature's own bug, reintroduced by a different cause. Round 1 made the driver-probe loop return None on the first None, to cap the blocking budget, and cuda_evidence memoised that None for the process lifetime. But probe_command_with_timeout returns None for three different reasons: timeout, unresolvable binary, and a spawn io::Error. So momentary fd or memory pressure when the first request lands, or a cold nvidia-smi exceeding PROBE_TIMEOUT — whose own doc notes a cold nvidia-smi can take tens of seconds while the driver initialises — would latch the CPU build until the process restarts, on a host with a perfectly healthy GPU. Before the early-return, the later probes acted as a retry that recovered once the first call warmed the driver. Cache only success permanently; retry failure a bounded three times. A transient failure recovers, and a permanently broken host still cannot pay the probe cost once per release row. Stop widening what counts as CUDA-available. Sharing CUDART_PATHS between the version lookup and the availability check grew the latter from two paths to four, so a RHEL or aarch64 box with the toolkit and no driver would newly report cuda from /detect-gpu and warn about an undeterminable version on a machine with no GPU. Selection was unaffected either way, so the widening bought nothing and cost a false report. Return the trimmed value from the ROCM_PATH check instead of a bool. It trimmed internally while the caller formatted the untrimmed value, so a padded ROCM_PATH passed validation and then read a relative path that never matches — silently disabling the source, with a test pinning the padded value as safe. Mark UNIX_TRUSTED_DIRS dead-code-allowed on Windows, where the cfg gate added last round leaves it unreferenced, and restrict the uname/sleep resolution test to non-Windows where those names cannot resolve. Correct the Windows comment, which understated its own cost: nvcc, rocm-smi and hipconfig now resolve nowhere at all there, not merely outside a custom directory. CUDA detection is unaffected because nvidia-smi lives in System32.
…rate The version parsers this module depends on live in ziee_hardware::gpu_version, shared so the two copies cannot re-diverge. The consequence is easy to miss and was: their fixture suite sits in a different crate, in a different workspace, behind a feature that is off by default, so cargo test --workspace from src-app runs none of it and two of the tests are not even compiled. Record the split in the module header with the command that actually runs them.
Both invariants were proven only in the parser crate. That is the wrong layer for them: INV-3 is about what a pre-R6xx host still SELECTS, and INV-4 is about what a fabricated version would CAUSE — a cuda610 artifact that does not exist, or a major the host cannot run. Assert both through to the selector. A CUDA 12.4 host must get cuda12.9, the newest compatible 12.x build — not cpu and not cuda13.2. A bare driver version must yield no CUDA version and fall to cpu rather than inventing a major. Also enumerate the sdk-resident test ids individually in the module header instead of as a range, so each is traceable to the crate and command that runs it.
added 3 commits
August 25, 2026 12:23
paws PR #12 merges first and moves main's sdk pointer to eed4419d7, so this branch's old pin conflicted on the submodule line. Rebased this feature's three sdk commits onto eed4419d7 rather than onto the sdk paws branch tip. The tip cannot be pinned: 8693247, feature-surface's testid regen, is now an ancestor of every commit on paws, and it drops seven template-assistants-* ids for a page that is deleted only on that branch. paws main still has the page, so pinning any paws-reachable commit fails check:testid-registry until feature-surface lands. eed4419d7 is off the c38e9fc lineage and carries no regen. Verified on the combined tree: check:testid-registry up to date (1799 ids), ziee-hardware 46 passed, gpu_detect 30 passed, and the CORS change is present.
They stay in branch history — commit 9b47494 and earlier carry the full record — they just must not land on main. PR #10 leaked its set, and the consequence is concrete rather than cosmetic: lifecycle-check refuses to resolve a feature directory once .lifecycle holds more than one, so every later worker has to pass --dir, and --dir resolves against the process cwd rather than --repo, so a relative path silently targets the wrong tree. That is the papercut being stopped here. Only this feature's directory is removed. .lifecycle/default-model-onboarding is PR #10's leftover and belongs to the worker clearing it; deleting another feature's audit trail is also what the validator's own A1 gate refuses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
A box with 4x NVIDIA H200 NVL installed and ran
linux-x86_64-cpu. A 9B model ran on CPU next to four idle GPUs — no error, no warning, and the CPU choice was indistinguishable from a correct one.Root cause, re-verified against the live host rather than taken on trust —
gpu_detect.rs:270:Driver 610.43.02 prints
| NVIDIA-SMI 610.43.02 KMD Version: 610.43.02 CUDA UMD Version: 13.3 |. The literalCUDA Version:occurs zero times, so the parse returnedNone,recommend_backend_forskipped its CUDA branch, and selection fell through tocpu. Everything else worked:nvidia-smiresolved and answered in 0.165s. Only the version parse failed.Before / after, on the affected hardware
parse_cuda_smi_version(610 banner)None13.3cpucuda13.2Captured RED before the fix and GREEN after. Zero GPU memory allocated, nothing downloaded.
What changed, and why not just a second string literal
The defect class is broader than the literal, which is the actual scope:
nvidia-smi --version, the banner,-q(driver-reported), thennvccand thelibcudartsoname (toolkit-derived, and only when--query-gpuconfirms a real device —is_cuda_available()is satisfied by a library file with no driver check).cpuwas chosen anyway. The predicate behind that warning is a pure function with its own test — an invariant living only inside a log line is not an invariant..exe), so every NVIDIA Windows user silently got the CPU build. That now works.Also fixes a real perf bug:
recommend_backendruns once per release row at three call sites withper_pageup to 500, anddetect_cuda_versionwas unmemoised — onenvidia-smispawn per row per request.Deliberately unchanged:
recommend_backend_for(it ignores the host CUDA minor, which is correct under CUDA 11+ minor-version compatibility; tightening it would rejectcuda13.2on a 13.0 driver where it works), and Metal (both macOS arms already returntrue, so a runtime-arch check is a nil-gain edit to code no machine here can compile).Blind audit found two HIGH defects I had introduced
Two rounds, independent auditors, diff-only context. Round 1 caught a Windows binary-planting hole this change armed: the resolver fell through to POSIX paths on Windows, and
PathBuf::from("/usr/bin").join("nvidia-smi.exe")resolves asC:\usr\bin\nvidia-smi.exe, which an unprivileged user can create and own. AddingEXE_SUFFIXis exactly what armed it. Also a macOS build break invisible until a release tag.Round 2 caught a regression in round 1's own fix: an early-return plus
OnceLockturned a transient probe failure (spawnEMFILE, or a coldnvidia-smiexceeding the 3s budget — its own doc says cold can take tens of seconds) into a permanent CPU fallback for the process lifetime — this feature's bug, reintroduced by a different cause. Now only success is cached; failure retries a bounded 3 times.Full record in
.lifecycle/gpu-backend-detect/(stripped at merge).Verification
cargo test -p ziee --lib gpu_detect::→ 30 passedcargo test -p ziee-hardware --features gpu-detect --lib→ 46 passedcargo test -p ziee --test integration_tests llm_local_runtime::gpu→ 2 passedcargo check --workspace --all-targets→ exit 0Not verified, stated plainly: macOS (no Darwin toolchain — the build-break fix is verified by construction, not compilation), Windows (no host — the change cannot regress it, since an unresolved binary yields today's
None, but "CUDA now detected" is reasoned not observed), AMD/ROCm (no hardware).⚠ Merge ordering — this PR pins a submodule
Requires ziee-ai/sdk#4 (base
paws) to land first; the shared parser lives there. The gitlink here is3ac7efb, verified remote-to-remote against the published sdk branch.A concurrent sdk PR from the
realtime-sseworker is open againstpawsfrom the same base. File sets don't overlap, but the superproject gitlink does: whichever sdk PR merges intopawsfirst, the other must rebase and re-pin before its paws PR can merge.Pre-existing issues found, not fixed here
credential_is_withheld_from_untrusted_targetsis red onmain(fails onhttp://[::1]:41234). Proven pre-existing by stashing this entire change and re-running.just checkfails before reaching anything:justfile:73grepssrc-app/sandbox-rootfs/compat.toml, deleted when the rootfs build moved out..lifecycle/default-model-onboardingis committed onmain— PR Install a default local model from Onboarding, with no API key #10 bypassed the merge-gate C5 strip. Not removed here; the validator's A1 gate fails any branch that deletes an inherited feature dir.ziee-hardwarehas no subprocess timeout at all, on a path reached synchronously fromGET /hardware.gpu_detectsolved this withprobe_command_with_timeout;detection.rsnever inherited it. Worth its own task.