Unify wasm-bindgen output under -sWASM_BINDGEN with marker-based detection - #27208
Conversation
cddd117 to
b7e9291
Compare
b7e9291 to
c10cb62
Compare
c10cb62 to
918ef09
Compare
|
@walkingeyerobot has now also confirmed that this unification approach works in his testing too. |
cc70816 to
cc29009
Compare
|
@sbc100 Sam, this looks good to me, but I'd appreciate it if you could give it a quick look before we hit the merge button. Thanks! |
cc29009 to
b9865b9
Compare
…ttributes (#27436) This implements support for JS libraries declaring their own public exports using per-symbol attributes, split out from #27208 as requested in review there. The attributes separate symbol inclusion from export visibility: * `__export: true` exports the symbol if it is otherwise included. * `__force: true` includes the symbol even when nothing references it, without exporting it, analogous to `DEFAULT_LIBRARY_FUNCS_TO_INCLUDE`. * Using both unconditionally includes and exports the symbol. This allows binding layers such as wasm-bindgen to define a public JS API surface distinct from the wasm export names, including exported classes and namespaces: ```js addToLibrary({ $Counter__export: true, $Counter__force: true, $Counter: class Counter { /* ... */ }, }); ``` The attributes are consumed as JS library compile-time metadata and do not appear in generated output. The JS compiler forwards only the symbols actually emitted through `extraExports`, allowing the `WASM_ESM_INTEGRATION` wrapper to re-export them without mutating or returning the global `EXPORTED_FUNCTIONS` setting. Decorated exports are also retained through meta-DCE. Test coverage verifies the independent attribute semantics under legacy modularized output, `MODULARIZE=instance`, `WASM_ESM_INTEGRATION`, and optimized `-O3` builds, together with decorator validation. _Made with AI assistance under my review_
3aee74b to
2d5101f
Compare
|
This is now ready for final review. |
6f07766 to
e5ccce2
Compare
9baaba5 to
f9b3ee7
Compare
|
Can you remind my we wouldn't want to just make |
|
@sbc100 I've added a further change here to remove |
sbc100
left a comment
There was a problem hiding this comment.
Does the commit message and description now need updating?
| if settings.WASM_BINDGEN and not building.is_wasm_bindgen_module(wasm_target): | ||
| settings.WASM_BINDGEN = 0 | ||
| if bindgen_exports: | ||
| building.link_lld(linker_args, wasm_target, external_symbols=js_syms) |
There was a problem hiding this comment.
What is this block doing? If WASM_BINDGEN is set but is_wasm_bindgen_module return false why do we need to then link again? Are the same wasm bindgen usages for which is_wasm_bindgen_module returns false still?
There was a problem hiding this comment.
Yeah this is a failure of the marker detection - the marker is not a symbol but a custom section, therefore finding it was requiring an initial link first.
This is clearly stupid yes, but was avoiding some of the other unnecessary steps previouslty.
I've added a third approach now using objdump to find the custom section marker instead. PTAL.
wasm-bindgen (wasm-bindgen/wasm-bindgen#5210, updated for the __export/ __force symbol attributes from emscripten-core#27436 in wasm-bindgen/wasm-bindgen#5253) hoists the clean exported API (functions, classes, enums) into top-level library symbols that self-register as exports via its `--js-library`. The user-facing API now comes from wasm-bindgen's library in every flow, so emscripten no longer needs to guess or own the export set. This collapses the previous staticlib-only handling into a single -sWASM_BINDGEN path that works whether cargo/rustc drives the link (bin crate, emcc as the linker, rustc supplies -sEXPORTED_FUNCTIONS) or emcc drives it (staticlib, exports discovered locally): - The exports the wasm-bindgen expansion reaches by name - the supplied EXPORTED_FUNCTIONS (method shims, the __wbindgen_* runtime, the marker, main) plus anything its expansion adds - are internal glue, not a user API. They are captured and kept off every export layer: the ESM wrapper (user_requested_exports), the factory Module attachment (EXPORTED_FUNCTIONS, via should_export), and the keepalive pass in finalize_wasm. `main` still runs automatically on init; `_main` isn't surfaced. - A genuine EMSCRIPTEN_KEEPALIVE C/C++ export is not in that internal set and remains surfaced, so a hand-written native export composes with wasm-bindgen's API in the same module. Human-supplied EXPORTED_FUNCTIONS are not preserved through wasm-bindgen linkage yet (the rustc-supplied set is indistinguishable from glue); that can be revisited later. - Strip the placeholder symbols wasm-bindgen consumes (__wbindgen_describe*, __externref_*, ...) so they aren't reported as undefined exports. - Only run nm-based export discovery for explicit -sWASM_BINDGEN when no driver supplied EXPORTED_FUNCTIONS; the rustc-driven link already lists them exactly. - Wire imported JS: feed library_bindgen.extern-pre.js as extern-pre-js and copy the snippets/ dir next to the output so relative imports resolve. - The WASM_ESM_INTEGRATION wrapper re-exports the JS library symbols that were exported (MODULARIZE=instance), and provides wasmExports via a namespace import of the wasm so by-name export access works. Add -sWASM_BINDGEN=auto: run wasm-bindgen only when the linked wasm carries wasm-bindgen's __wasm_bindgen_emscripten_marker custom section, which is how cargo/rustc opts in when driving emcc as the linker (addressing the request to replace implicit marker detection with an explicit flag); otherwise it is a no-op and wasm-bindgen need not be installed. Both output modes then expose only the clean API (e.g. a `Greeter` class). Add an end-to-end test parameterized over the ESM and factory output modes (built via cargo with -sWASM_BINDGEN=auto), a no-marker test asserting auto is a no-op for an ordinary build, extend the staticlib integration test to assert an EMSCRIPTEN_KEEPALIVE export survives alongside the wasm-bindgen API, and install a pinned wasm-bindgen-cli alongside rust in CI so the flow is always exercised. The wasm-bindgen library and CLI are pinned to the same wasm-bindgen main rev (the attribute support is not yet in a release); they must match exactly, and can move to a version pin once released.
- Drop the greeter test's .cargo/config.toml: -sDEFAULT_TO_CXX and the exceptions/panic flags are no longer needed since emscripten-core#27496, and the link args are now passed via -Clink-arg rustflags in a config written by the test instead of through EMCC_CFLAGS. - Only pass --experimental-wasm-modules to node < 25. - Rename the test parameterizations to esm_integration/es6. - Clarify in the WASM_BINDGEN setting docs when the linked wasm carries the marker section and why the staticlib flow still needs =1. - Merge the wasm-bindgen comments in finalize_wasm. - Note in postamble.js that the wasmExports namespace import is only needed for wasm-bindgen 0.2.127 and can go once the minimum version includes wasm-bindgen/wasm-bindgen#5270.
The wasm-bindgen export detection previously lived inside lld_flags, gated on WASM_BINDGEN == 1 because 'auto' cannot be resolved until a linked module exists. Restructure so that the two concerns are decided in phase_link, on the initial link's output: - WASM_BINDGEN=auto is resolved against the linked wasm's marker section (moved here from phase_post_link). - Then, whenever wasm-bindgen mode is on and the link driver did not supply EXPORTED_FUNCTIONS, the exports wasm-bindgen reaches by name are discovered from the linker inputs and the module is re-linked to retain them. They are passed straight to the linker so they are not mistaken for user-requested exports. This drops the linker_inputs plumbing through link_lld/lld_flags and the WASM_BINDGEN special case in lld_flags.
- Keep USER_EXPORTS truthful after the wasm-bindgen step (only drop the placeholder exports it consumed) instead of clearing it, so the existing unused-main handling in finalize_wasm applies as-is and the WASM_BINDGEN special case there can go. - Validate WASM_BINDGEN values. - Install the prebuilt wasm-bindgen CLI in CI instead of building it. - Trim comments.
ff9412e to
55adaba
Compare
| # Otherwise, discover the symbols directly from the linker inputs for e.g. static | ||
| # linking Rust. | ||
| if settings.WASM_BINDGEN and 'EXPORTED_FUNCTIONS' not in user_settings: | ||
| linker_args += [f'--export={e}' for e in building.get_wasm_bindgen_exported_symbols(linker_inputs)] |
There was a problem hiding this comment.
Do you think one day we could remove get_wasm_bindgen_exported_symbols? Ideally these symbols would self-export? Maybe we could instead force any archive members that has_wasm_bindgen_marker to be included by the linker?
Doesn't need to be part of this PR, just wondering.
There was a problem hiding this comment.
The problem here is these are the core Wasm symbols usually reported by the Rust linker itself, based on visibility rules not present in the archive! Something that should be done better in the static case indeed. I haven't investigated deeply, but having Rust better align with C++ visibility conventions for Wasm sounds sensible to me, I wasn't involved in these original decisions.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
This change seems to be causing some tests to fail in CI. See https://app.circleci.com/pipelines/github/emscripten-core/emscripten/54289/workflows/db8f6174-b54d-4ed8-9ebd-914022760b58/jobs/1295567 : |
#27676) This removes the `WASM_ESM_INTEGRATION && WASM_BINDGEN` namespace import of the wasm module added in #27208, resolves #27658. wasm-bindgen 0.2.128 no longer reads `wasmExports[...]` inline in its emscripten glue; it binds the asmjs-mangled identifiers that `assignWasmExports` already provides for every export (wasm-bindgen/wasm-bindgen#5270), so the aggregate object is not needed. * Remove the conditional `import * as wasmExports` from `src/postamble.js`, leaving the plain `var wasmExports` declaration for all modes. * Bump the pinned wasm-bindgen library and CLI from 0.2.127 to 0.2.128 (test crates, `cargo add` calls, CircleCI install). * `test_wasm_bindgen_tsd_multi_return` now asserts the correct public surface: only wasm-bindgen's unwrapping wrapper `multi_value_return(): number` is typed, and the raw multi-value `Result` ABI export `_multi_value_return` stays internal. With 0.2.127 the raw export leaked into `WasmModule`, which is what the old `[number, number, number]` assertion was matching. All `test_wasm_bindgen_*` tests pass against wasm-bindgen 0.2.128. _Made with AI assistance under my review_
This unifies wasm-bindgen output generation under a single
-sWASM_BINDGENpath, building on the wasm-bindgen side change in wasm-bindgen/wasm-bindgen#5210. It supersedes #27179.Previously
-sWASM_BINDGENwas effectively a staticlib-only flow where emcc had to discover and own the export set itself. With wasm-bindgen/wasm-bindgen#5210, wasm-bindgen hoists its clean exported API (functions, classes, enums) into top-level library symbols that self-register as public exports via the__export/__forcesymbol attributes (#27436, wasm-bindgen/wasm-bindgen#5253) in its--js-library. The user-facing API now comes from wasm-bindgen's own library in every flow, so emscripten no longer needs to guess or own exports — it just surfaces what wasm-bindgen registered. That lets one-sWASM_BINDGENpath serve both whether cargo/rustc drives the link (bin crate, emcc as the linker, rustc supplies-sEXPORTED_FUNCTIONS) or emcc drives it (staticlib, exports discovered locally).The model for a public export is the union across both systems: wasm-bindgen's self-registered API, plus emscripten's own
EMSCRIPTEN_KEEPALIVEexports — minus wasm-bindgen's internal expansion glue, which must not spill into the public surface.What's implemented:
-sWASM_BINDGENis a no-op unless the linker inputs carry wasm-bindgen's__wasm_bindgen_emscripten_markercustom section (emitted by the wasm-bindgen crate). The check runs on the inputs before the link (llvm-objdump --section-headers, so archive members are covered), so there is a single link and no speculative exports. When the marker is absent wasm-bindgen is never invoked and need not be installed, so cargo/rustc can pass the flag unconditionally via-Clink-argwithout affecting ordinary builds.EXPORTED_FUNCTIONS(method shims, the__wbindgen_*runtime, the marker,main) plus anything its expansion adds — are captured as internal glue and kept off every export layer: the ESM wrapper (user_requested_exports), the factoryModuleattachment (EXPORTED_FUNCTIONS, viashould_export), and the keepalive pass infinalize_wasm.mainstill runs automatically on init;_mainisn't surfaced.EMSCRIPTEN_KEEPALIVEC/C++ export is not in that internal set and remains surfaced, so a hand-written native export composes with wasm-bindgen's API in the same module.__wbindgen_describe*,__externref_*, ...) are stripped so they aren't reported as undefined exports.EXPORTED_FUNCTIONS; the rustc-driven link already lists them exactly.library_bindgen.extern-pre.jsis fed as extern-pre-js and thesnippets/dir is copied next to the output so relative imports resolve.WASM_ESM_INTEGRATION,wasmExportsis provided via a namespace import of the wasm so wasm-bindgen's by-name export access works. This is gated behindWASM_BINDGENso plain ESM integration keeps per-symbol named imports and their tree-shakability. (The wrapper re-export of self-registered JS library symbols itself landed in Support JS libraries self-registering their exports with visibility attributes #27436.)Both output modes then expose only the clean API (e.g. a
Greeterclass).Testing:
-sWASM_ESM_INTEGRATION) and factory (-sMODULARIZE -sEXPORT_ES6) output modes, built viacargo buildwith-sWASM_BINDGENpassed as a link arg, asserting the cleanGreeterAPI works,mainruns on init, and none of the raw wasm exports leak.-sWASM_BINDGENis a no-op for an ordinaryhello_world.cbuild (doesn't require wasm-bindgen installed).EMSCRIPTEN_KEEPALIVEnative export survives alongside the wasm-bindgen API (fails under a blanket export wipe).wasm-bindgen-clialongside rust so the flow is always exercised.Not in scope (follow-up): preserving human-supplied
EXPORTED_FUNCTIONSthrough wasm-bindgen linkage (the rustc-supplied set is currently indistinguishable from glue, so explicit exports aren't surfaced).