Editor preview runs the real C — eliminate the Python twin - #2
Open
silviot wants to merge 8 commits into
Open
Conversation
A C module is no longer shadowed by a hand-written Python stand-in that the user had to keep behaviour-identical. The live preview now runs the SAME native/<mod>.c the badge does: the editor compiles it to a WebAssembly CPython extension (against a shim that reimplements MicroPython's py/dynruntime.h on the CPython C API) and imports it into the simulator's Pyodide. On wasm32 mp_int_t is 32-bit — identical to the ESP32 — so integer overflow and the whole dynruntime surface behave exactly as on hardware; a spec-scope module cannot observe it isn't on a badge. Two host feasibility spikes proved this end to end (the shim path, 22/22; and — as the documented future option — real MicroPython in wasm loading a recompiled natmod against the genuine runtime). - boot.py: C modules load as compiled .so injected via install_native_module into /live-native (on sys.path); the per-swap module purge covers them so edits re-import. No twin, no ImportError stub, no parity check — all removed. - sim-worker.js: `native` message writes the compiled .so before the swap that imports it (queued pre-ready like swap/reset). - preview-build.js + preview-worker.js: the in-browser preview compiler — clang.wasm (wasm32) + wasm-ld.wasm link a Pyodide side module against the shim; content-addressed per module, cache -> prebuilt -> browser compiler. Behind a runtimeValidated manifest flag until QA. - app.js: the swap gates on the preview compile — while compiling the preview is "pending"; a compile error pins to the C line and keeps the last good app; compiled modules are re-injected on every worker respawn. Creating a C module makes ONLY native/<mod>.c (+ the sys.path shim); a .py may not share a C module's name (files.moduleNameConflicts). - files.js: no pyFallbackTemplate; badgePyFiles is just app.py + real Python modules. Template + fastcode demo reworded (demo twin deleted). - tests: test-native-preview.mjs loads a REAL compiled .so (the shim spike's fastcode) and proves add(20,22)==42 from C, rebuild re-imports, un-built module rejects the swap keeping last good, removal drops the stale .so. test-runtime multi-file tests moved to Python modules. The preview toolchain bundle (wasm-ld + shim + headers) is building; the sim-side is proven now with real compiled fixtures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TLAG6JndPQewe7w9gVRKgh
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TLAG6JndPQewe7w9gVRKgh
The mpy-preview-v1 bundle froze its arg vectors as top-level cc1/ld arrays
with baked-in /sysroot paths ({IN}/{OUT}/{QSTR} the only placeholders), and
clang.wasm tears down LLVM's option registry after each callMain — so:
- preview-worker.js uses args.cc1/args.ld, mounts the sysroot tar at
/sysroot, works in /work, and creates a FRESH clang instance per
translation unit (the -mllvm wasm-EH/SjLj flags are rejected on a reused
instance); clang is fetched from the badge bundle (clang-xtensa-v1) so the
14 MB download is shared, wasm-ld + sysroot + args from mpy-preview-v1.
- pwa.js: ?nosw disables + tears down the service worker (iterative dev
against a live server); preview-build.js: ?previewforce accepts the
toolchain before its runtimeValidated flag is flipped (QA).
Verified live in a browser (fresh origin): add a C module → only
native/fastcode.c → the preview compiles it client-side (clang.wasm +
wasm-ld.wasm, ~0.2 s warm) and runs the real C: "C says 42", and
add(2e9, 2e9) shows -294967296 — the 32-bit wrap the badge does, which a
Python twin would have gotten wrong (4000000000). A compile error marks
the C line and keeps the last good app running.
native-build/browser-preview/: the agent's bundle reference (cc1 args,
wasm-ld build script, README with the reproducible pipeline + shim role).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLAG6JndPQewe7w9gVRKgh
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TLAG6JndPQewe7w9gVRKgh
The line under the tabs read "C module: this same C runs in the preview
(compiled to WebAssembly) and on the badge…" — a description of what a C
module is, which is exactly what you'd expect ("all our food is edible").
Replaced with just the actionable status for the file you're editing:
✓ live in the preview · ✓ built for badge, or "compiling…", or
"✗ won't compile — line N" (badge-build chip suppressed while the code
doesn't compile). Shown only on a C file; app.py keeps the run-dot. The
what-is-a-C-module explanation lives in the ? panel / c-modules.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLAG6JndPQewe7w9gVRKgh
… download Opening a C demo (or making a C module) used to download the ~15 MB preview toolchain once before it could run the C. Now the demo's C is compiled to its preview side module at SITE-BUILD time and shipped as a content-addressed previews/<key>.json, which the client resolves before it ever touches the compiler — verified live: a clean demo load fetches only the ~16 KB artifact, zero toolchain requests, and renders 42 from the real compiled C. - preview-compile-core.js: extracted the environment-agnostic compile+link core (qstr/glue generation, per-TU fresh-clang, wasm-ld) shared by the browser worker and the build-time prebuild, so their output can't drift. preview-worker.js now just boots (fetch) + base64s; preview-build.js single-sources PREVIEW_TOOLCHAIN_ID from the core and exports previewModuleKeys (the client key logic the prebuild reuses). - native-build/browser-preview/prebuild_previews.mjs: runs the exact browser pipeline (clang.wasm + wasm-ld.wasm from the mpy-preview-v1 bundle) under Node, keyed identically to the client. Network-tolerant: warns + skips if the toolchain can't be fetched (demo then compiles in-browser at runtime). build.sh stage 3d wires it; the artifact is picked up by the SW precache so it also works offline. Toolchain is cached in web/.cache so repeat builds don't re-download. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TLAG6JndPQewe7w9gVRKgh
_sync_live_files and install_native_module write module files that the next swap imports, but Python's import machinery caches each directory's listing — so a module written after being removed (add → remove → re-add, or a rejected swap restoring the accepted project) could fail with ModuleNotFoundError because the finder still saw the stale contents. Made a test flaky (helper module intermittently unimportable); would hit real users editing project files. importlib.invalidate_caches() after touching /live and /live-native fixes it deterministically (5/5). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TLAG6JndPQewe7w9gVRKgh
The starter native/<mod>.c led with mechanism the C author doesn't act on (compiles to .mpy, loads at import, no reflashing, how the preview executes it) and handed over mpy_init as mysterious boilerplate — MP_DYNRUNTIME_INIT_ ENTRY / mp_store_global(...) with no hint which line is theirs. Rewritten around the task: it opens with `import <mod>; <mod>.add(20,22) -> 42`, states the two-step pattern (write the function, register it in mpy_init), comments the mp_obj_t <-> C conversions and the _2 arg-count macro, and — the part that was opaque — labels ENTRY/EXIT as the required frame to leave alone and shows the exact mp_store_global line to copy per exported function. Same treatment for the fastcode demo's C so the two match. (Types/limits and the how-it-runs details live in the ? panel.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TLAG6JndPQewe7w9gVRKgh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The badge editor's live preview now runs the user's real C — the same
native/<mod>.cthat ships to the badge. There is no longer a hand-written Python "twin" the user had to keep behaviour-identical. One implementation, written once.Live: https://protogon.codemyriad.io/editor-c/#fastcode — open it (or ＋ → C module) and the C compiles in your browser and runs;
add(20, 22)shows 42, andadd(2_000_000_000, 2_000_000_000)shows −294967296 — the 32-bit wrap the badge does, which a Python stand-in would have gotten wrong (4000000000).Why this is faithful, not a mock
A native module can only ever touch MicroPython's small
py/dynruntime.hAPI, routed throughmp_fun_table. The editor compiles the module to a WebAssembly CPython extension against a shim that reimplements that one API on Pyodide's CPython C API, andimport <mod>in the sim loads it. Because the module can't observe anything outsidedynruntime.h, and the shim implements it — including 32-bitmp_int_ton wasm32, identical to the ESP32 — integer overflow, floats, str/bytes, buffers and exceptions all behave exactly as on hardware. The only things a shim can't reproduce (defining brand-new MicroPython object types, poking raw tagged-pointer /mp_obj_base_tmemory) are outside the supported API; the editor rejects modules that use them.Two independent host spikes proved this end to end before any editor change:
fastcode/features0/features1compile to a side module, 22/22 proofs incl. the int-overflow fidelity, buffers, exceptions.mp_fun_table— perfect fidelity, but a weeks-long re-host of the whole simulator (scheduler, eventbus, ctx, boot.py) with an open ASYNCIFY question. The shim slots into the existing, mature Pyodide sim with small blast radius, so it ships now; the emulator is a proven future upgrade.In-browser compile pipeline
clang.wasm(wasm32, reused from the badge toolchain so the ~14 MB download is shared) compiles the module + shim runtime + a generatedPyInitglue;wasm-ld.wasm(lld, built for this) links a Pyodide side module (~7 KB, imports CPython symbols dynamically, exports onlyPyInit_<mod>). No emscripten driver, no server — entirely client-side, offline after the one-time toolchain fetch. A freshclanginstance per translation unit is required (clang.wasm tears down LLVM's option registry after each run). Bundle at the immutable, same-origin/toolchains/mpy-preview-v1/, behind aruntimeValidatedkill switch (flipped after the live QA above).Editor changes
native/<mod>.c(plus the badgesys.pathshim inapp.py). No.pytwin; a.pymay not share a C module's name (files.moduleNameConflicts).preview-build.jscontent-addresses per module (cache → prebuilt → in-browser compiler);boot.pyloads the.soviainstall_native_moduleinto/live-nativeand purges it on each edit so changes re-import.pyFallbackTemplate, the fallback ImportError stub, and the API-surface parity warning — all obsolete once the preview runs the real C.c-modules.md,NATIVE-MODULES.md) rewritten to the new model; thefastcodesample lost its twin.Tests
test-native-preview.mjs— bootsboot.pyand loads a real compiled.so(the shim spike'sfastcode), provingadd(20,22)==42from C, rebuild re-imports, an un-built module rejects the swap keeping the last good app, and removal drops the stale.so.test-files.mjs— name-conflict validation;test-runtime.mjsmulti-file tests moved to Python sibling modules. All suites green; verified live on the deployed site.Instant demos (done)
The demo projects' C is compiled to its preview side module at site-build time and shipped as a content-addressed
previews/<key>.json, so opening a C demo runs the real C instantly — verified live: a clean load fetches only the ~16 KB artifact, zero toolchain requests, renders 42. The prebuild runs the exact browser pipeline (clang.wasm + wasm-ld.wasm) under Node via a shared compile core, keyed identically to the client, and the artifact is picked up by the offline precache. (Making a new C module still fetches the ~15 MB toolchain once, then caches it — that's the general path; only the bundled demos are prebuilt.)Also fixed a real correctness bug found while wiring this up:
boot.pynow callsimportlib.invalidate_caches()after writing/removing live module files, so adding → removing → re-adding a module (or a rejected swap restoring the project) can't leave a just-written module unimportable via a stale directory cache.🤖 Generated with Claude Code
https://claude.ai/code/session_01TLAG6JndPQewe7w9gVRKgh