Skip to content

Unify wasm-bindgen output under -sWASM_BINDGEN with marker-based detection - #27208

Merged
sbc100 merged 14 commits into
emscripten-core:mainfrom
guybedford:wasm-bindgen-unified-flow
Sep 5, 2026
Merged

Unify wasm-bindgen output under -sWASM_BINDGEN with marker-based detection#27208
sbc100 merged 14 commits into
emscripten-core:mainfrom
guybedford:wasm-bindgen-unified-flow

Conversation

@guybedford

@guybedford guybedford commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

This unifies wasm-bindgen output generation under a single -sWASM_BINDGEN path, building on the wasm-bindgen side change in wasm-bindgen/wasm-bindgen#5210. It supersedes #27179.

Previously -sWASM_BINDGEN was effectively a staticlib-only flow where emcc had to discover and own the export set itself. With wasm-bindgen/wasm-bindgen#5210, wasm-bindgen hoists its clean exported API (functions, classes, enums) into top-level library symbols that self-register as public exports via the __export/__force symbol attributes (#27436, wasm-bindgen/wasm-bindgen#5253) in its --js-library. The user-facing API now comes from wasm-bindgen's own library in every flow, so emscripten no longer needs to guess or own exports — it just surfaces what wasm-bindgen registered. That lets one -sWASM_BINDGEN path serve both whether cargo/rustc drives the link (bin crate, emcc as the linker, rustc supplies -sEXPORTED_FUNCTIONS) or emcc drives it (staticlib, exports discovered locally).

The model for a public export is the union across both systems: wasm-bindgen's self-registered API, plus emscripten's own EMSCRIPTEN_KEEPALIVE exports — minus wasm-bindgen's internal expansion glue, which must not spill into the public surface.

What's implemented:

  • -sWASM_BINDGEN is a no-op unless the linker inputs carry wasm-bindgen's __wasm_bindgen_emscripten_marker custom section (emitted by the wasm-bindgen crate). The check runs on the inputs before the link (llvm-objdump --section-headers, so archive members are covered), so there is a single link and no speculative exports. When the marker is absent wasm-bindgen is never invoked and need not be installed, so cargo/rustc can pass the flag unconditionally via -Clink-arg without affecting ordinary builds.
  • The exports the wasm-bindgen expansion reaches by name — the supplied EXPORTED_FUNCTIONS (method shims, the __wbindgen_* runtime, the marker, main) plus anything its expansion adds — are captured as internal glue and kept off every export layer: the ESM wrapper (user_requested_exports), the factory Module attachment (EXPORTED_FUNCTIONS, via should_export), and the keepalive pass in finalize_wasm. main still runs automatically on init; _main isn't surfaced.
  • A genuine EMSCRIPTEN_KEEPALIVE C/C++ export is not in that internal set and remains surfaced, so a hand-written native export composes with wasm-bindgen's API in the same module.
  • Placeholder symbols wasm-bindgen consumes (__wbindgen_describe*, __externref_*, ...) are stripped so they aren't reported as undefined exports.
  • nm-based export discovery only runs when no driver supplied EXPORTED_FUNCTIONS; the rustc-driven link already lists them exactly.
  • Imported JS is wired through: library_bindgen.extern-pre.js is fed as extern-pre-js and the snippets/ dir is copied next to the output so relative imports resolve.
  • Under WASM_ESM_INTEGRATION, wasmExports is provided via a namespace import of the wasm so wasm-bindgen's by-name export access works. This is gated behind WASM_BINDGEN so plain ESM integration keeps per-symbol named imports and their tree-shakability. (The wrapper re-export of self-registered JS library symbols itself landed in Support JS libraries self-registering their exports with visibility attributes #27436.)

Both output modes then expose only the clean API (e.g. a Greeter class).

Testing:

  • End-to-end test parameterized over the ESM (-sWASM_ESM_INTEGRATION) and factory (-sMODULARIZE -sEXPORT_ES6) output modes, built via cargo build with -sWASM_BINDGEN passed as a link arg, asserting the clean Greeter API works, main runs on init, and none of the raw wasm exports leak.
  • A no-marker test asserting -sWASM_BINDGEN is a no-op for an ordinary hello_world.c build (doesn't require wasm-bindgen installed).
  • The staticlib integration test is extended to assert an EMSCRIPTEN_KEEPALIVE native export survives alongside the wasm-bindgen API (fails under a blanket export wipe).
  • CI installs a pinned wasm-bindgen-cli alongside rust so the flow is always exercised.

Not in scope (follow-up): preserving human-supplied EXPORTED_FUNCTIONS through wasm-bindgen linkage (the rustc-supplied set is currently indistinguishable from glue, so explicit exports aren't surfaced).

@guybedford
guybedford force-pushed the wasm-bindgen-unified-flow branch from cddd117 to b7e9291 Compare June 26, 2026 23:24
@guybedford
guybedford force-pushed the wasm-bindgen-unified-flow branch from b7e9291 to c10cb62 Compare June 26, 2026 23:32
@guybedford guybedford changed the title Unify wasm-bindgen output under -sWASM_BINDGEN Unify wasm-bindgen output under -sWASM_BINDGEN, add -sWASM_BINDGEN=auto Jun 26, 2026
@guybedford
guybedford force-pushed the wasm-bindgen-unified-flow branch from c10cb62 to 918ef09 Compare June 29, 2026 23:54
@guybedford

Copy link
Copy Markdown
Collaborator Author

@walkingeyerobot has now also confirmed that this unification approach works in his testing too.

@guybedford
guybedford force-pushed the wasm-bindgen-unified-flow branch 2 times, most recently from cc70816 to cc29009 Compare July 1, 2026 22:48
guybedford added a commit to guybedford/emscripten that referenced this pull request Jul 24, 2026
guybedford added a commit to guybedford/emscripten that referenced this pull request Jul 28, 2026
@walkingeyerobot
walkingeyerobot requested a review from sbc100 July 28, 2026 17:41
@walkingeyerobot

Copy link
Copy Markdown
Collaborator

@sbc100 Sam, this looks good to me, but I'd appreciate it if you could give it a quick look before we hit the merge button. Thanks!

Comment thread src/jsifier.mjs Outdated
Comment thread src/postamble.js
Comment thread tools/emscripten.py Outdated
@guybedford
guybedford force-pushed the wasm-bindgen-unified-flow branch from cc29009 to b9865b9 Compare July 28, 2026 21:16
sbc100 pushed a commit that referenced this pull request Aug 5, 2026
…ttributes (#27436)

This implements support for JS libraries declaring their own public
exports using per-symbol attributes, split out from #27208 as requested
in review there.

The attributes separate symbol inclusion from export visibility:

* `__export: true` exports the symbol if it is otherwise included.
* `__force: true` includes the symbol even when nothing references it,
without exporting it, analogous to `DEFAULT_LIBRARY_FUNCS_TO_INCLUDE`.
* Using both unconditionally includes and exports the symbol.

This allows binding layers such as wasm-bindgen to define a public JS
API surface distinct from the wasm export names, including exported
classes and namespaces:

```js
addToLibrary({
  $Counter__export: true,
  $Counter__force: true,
  $Counter: class Counter { /* ... */ },
});
```

The attributes are consumed as JS library compile-time metadata and do
not appear in generated output. The JS compiler forwards only the
symbols actually emitted through `extraExports`, allowing the
`WASM_ESM_INTEGRATION` wrapper to re-export them without mutating or
returning the global `EXPORTED_FUNCTIONS` setting. Decorated exports are
also retained through meta-DCE.

Test coverage verifies the independent attribute semantics under legacy
modularized output, `MODULARIZE=instance`, `WASM_ESM_INTEGRATION`, and
optimized `-O3` builds, together with decorator validation.

_Made with AI assistance under my review_
@guybedford
guybedford force-pushed the wasm-bindgen-unified-flow branch 3 times, most recently from 3aee74b to 2d5101f Compare August 8, 2026 00:57
@guybedford

Copy link
Copy Markdown
Collaborator Author

This is now ready for final review.

@guybedford
guybedford force-pushed the wasm-bindgen-unified-flow branch 4 times, most recently from 6f07766 to e5ccce2 Compare August 10, 2026 21:34
@guybedford
guybedford force-pushed the wasm-bindgen-unified-flow branch from 9baaba5 to f9b3ee7 Compare August 29, 2026 00:49
Comment thread .circleci/config.yml Outdated
Comment thread src/postamble.js
Comment thread test/rust/bindgen_greeter/.cargo/config.toml Outdated
Comment thread site/source/docs/tools_reference/settings_reference.rst Outdated
Comment thread tools/link.py Outdated
@sbc100

sbc100 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Can you remind my we wouldn't want to just make auto the default? i.e. make -sWASM_BINDGEN do nothing when the section tag is missing?

Comment thread tools/emscripten.py Outdated
@guybedford

Copy link
Copy Markdown
Collaborator Author

@sbc100 I've added a further change here to remove -sWASM_BINDGEN=auto and just unify this as the default -sWASM_BINDGEN behavior, given the marker is always reliably found at this point.

@sbc100 sbc100 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the commit message and description now need updating?

Comment thread tools/link.py Outdated
if settings.WASM_BINDGEN and not building.is_wasm_bindgen_module(wasm_target):
settings.WASM_BINDGEN = 0
if bindgen_exports:
building.link_lld(linker_args, wasm_target, external_symbols=js_syms)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this block doing? If WASM_BINDGEN is set but is_wasm_bindgen_module return false why do we need to then link again? Are the same wasm bindgen usages for which is_wasm_bindgen_module returns false still?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is a failure of the marker detection - the marker is not a symbol but a custom section, therefore finding it was requiring an initial link first.

This is clearly stupid yes, but was avoiding some of the other unnecessary steps previouslty.

I've added a third approach now using objdump to find the custom section marker instead. PTAL.

@guybedford guybedford changed the title Unify wasm-bindgen output under -sWASM_BINDGEN, add -sWASM_BINDGEN=auto Unify wasm-bindgen output under -sWASM_BINDGEN with marker-based detection Sep 4, 2026
guybedford and others added 12 commits September 4, 2026 14:21
wasm-bindgen (wasm-bindgen/wasm-bindgen#5210, updated for the __export/
__force symbol attributes from emscripten-core#27436 in wasm-bindgen/wasm-bindgen#5253)
hoists the clean exported API (functions, classes, enums) into top-level
library symbols that self-register as exports via its `--js-library`.
The user-facing API now comes from wasm-bindgen's library in every flow,
so emscripten no longer needs to guess or own the export set.

This collapses the previous staticlib-only handling into a single
-sWASM_BINDGEN path that works whether cargo/rustc drives the link (bin
crate, emcc as the linker, rustc supplies -sEXPORTED_FUNCTIONS) or emcc
drives it (staticlib, exports discovered locally):

- The exports the wasm-bindgen expansion reaches by name - the supplied
  EXPORTED_FUNCTIONS (method shims, the __wbindgen_* runtime, the marker,
  main) plus anything its expansion adds - are internal glue, not a user
  API. They are captured and kept off every export layer: the ESM wrapper
  (user_requested_exports), the factory Module attachment
  (EXPORTED_FUNCTIONS, via should_export), and the keepalive pass in
  finalize_wasm. `main` still runs automatically on init; `_main` isn't
  surfaced.
- A genuine EMSCRIPTEN_KEEPALIVE C/C++ export is not in that internal set
  and remains surfaced, so a hand-written native export composes with
  wasm-bindgen's API in the same module. Human-supplied EXPORTED_FUNCTIONS
  are not preserved through wasm-bindgen linkage yet (the rustc-supplied
  set is indistinguishable from glue); that can be revisited later.
- Strip the placeholder symbols wasm-bindgen consumes
  (__wbindgen_describe*, __externref_*, ...) so they aren't reported as
  undefined exports.
- Only run nm-based export discovery for explicit -sWASM_BINDGEN when no
  driver supplied EXPORTED_FUNCTIONS; the rustc-driven link already lists
  them exactly.
- Wire imported JS: feed library_bindgen.extern-pre.js as extern-pre-js
  and copy the snippets/ dir next to the output so relative imports
  resolve.
- The WASM_ESM_INTEGRATION wrapper re-exports the JS library symbols that
  were exported (MODULARIZE=instance), and provides wasmExports via a
  namespace import of the wasm so by-name export access works.

Add -sWASM_BINDGEN=auto: run wasm-bindgen only when the linked wasm
carries wasm-bindgen's __wasm_bindgen_emscripten_marker custom section,
which is how cargo/rustc opts in when driving emcc as the linker
(addressing the request to replace implicit marker detection with an
explicit flag); otherwise it is a no-op and wasm-bindgen need not be
installed.

Both output modes then expose only the clean API (e.g. a `Greeter`
class). Add an end-to-end test parameterized over the ESM and factory
output modes (built via cargo with -sWASM_BINDGEN=auto), a no-marker
test asserting auto is a no-op for an ordinary build, extend the
staticlib integration test to assert an EMSCRIPTEN_KEEPALIVE export
survives alongside the wasm-bindgen API, and install a pinned
wasm-bindgen-cli alongside rust in CI so the flow is always exercised.

The wasm-bindgen library and CLI are pinned to the same wasm-bindgen
main rev (the attribute support is not yet in a release); they must
match exactly, and can move to a version pin once released.
- Drop the greeter test's .cargo/config.toml: -sDEFAULT_TO_CXX and the
  exceptions/panic flags are no longer needed since emscripten-core#27496, and the link
  args are now passed via -Clink-arg rustflags in a config written by the
  test instead of through EMCC_CFLAGS.
- Only pass --experimental-wasm-modules to node < 25.
- Rename the test parameterizations to esm_integration/es6.
- Clarify in the WASM_BINDGEN setting docs when the linked wasm carries
  the marker section and why the staticlib flow still needs =1.
- Merge the wasm-bindgen comments in finalize_wasm.
- Note in postamble.js that the wasmExports namespace import is only
  needed for wasm-bindgen 0.2.127 and can go once the minimum version
  includes wasm-bindgen/wasm-bindgen#5270.
The wasm-bindgen export detection previously lived inside lld_flags,
gated on WASM_BINDGEN == 1 because 'auto' cannot be resolved until a
linked module exists. Restructure so that the two concerns are decided
in phase_link, on the initial link's output:

- WASM_BINDGEN=auto is resolved against the linked wasm's marker
  section (moved here from phase_post_link).
- Then, whenever wasm-bindgen mode is on and the link driver did not
  supply EXPORTED_FUNCTIONS, the exports wasm-bindgen reaches by name
  are discovered from the linker inputs and the module is re-linked to
  retain them. They are passed straight to the linker so they are not
  mistaken for user-requested exports.

This drops the linker_inputs plumbing through link_lld/lld_flags and
the WASM_BINDGEN special case in lld_flags.
- Keep USER_EXPORTS truthful after the wasm-bindgen step (only drop the
  placeholder exports it consumed) instead of clearing it, so the
  existing unused-main handling in finalize_wasm applies as-is and the
  WASM_BINDGEN special case there can go.
- Validate WASM_BINDGEN values.
- Install the prebuilt wasm-bindgen CLI in CI instead of building it.
- Trim comments.
@guybedford
guybedford force-pushed the wasm-bindgen-unified-flow branch from ff9412e to 55adaba Compare September 4, 2026 21:27
Comment thread ChangeLog.md Outdated
Comment thread tools/link.py
# Otherwise, discover the symbols directly from the linker inputs for e.g. static
# linking Rust.
if settings.WASM_BINDGEN and 'EXPORTED_FUNCTIONS' not in user_settings:
linker_args += [f'--export={e}' for e in building.get_wasm_bindgen_exported_symbols(linker_inputs)]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think one day we could remove get_wasm_bindgen_exported_symbols? Ideally these symbols would self-export? Maybe we could instead force any archive members that has_wasm_bindgen_marker to be included by the linker?

Doesn't need to be part of this PR, just wondering.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem here is these are the core Wasm symbols usually reported by the Rust linker itself, based on visibility rules not present in the archive! Something that should be done better in the static case indeed. I haven't investigated deeply, but having Rust better align with C++ visibility conventions for Wasm sounds sensible to me, I wasn't involved in these original decisions.

@guybedford

This comment was marked as off-topic.

@sbc100
sbc100 enabled auto-merge (squash) September 4, 2026 22:48
@sbc100
sbc100 merged commit 8de9aa0 into emscripten-core:main Sep 5, 2026
47 checks passed
@guybedford
guybedford deleted the wasm-bindgen-unified-flow branch September 5, 2026 00:13
@sbc100

sbc100 commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

This change seems to be causing some tests to fail in CI. See https://app.circleci.com/pipelines/github/emscripten-core/emscripten/54289/workflows/db8f6174-b54d-4ed8-9ebd-914022760b58/jobs/1295567 :

error: 

it looks like the Rust project used to create this Wasm file was linked against
version of wasm-bindgen that uses a different bindgen format than this binary:

  rust Wasm file schema version: 0.2.128
     this binary schema version: 0.2.127 (a579ee62b)

Currently the bindgen format is unstable enough that these two schema versions
must exactly match. You can accomplish this by either updating this binary or
the wasm-bindgen dependency in the Rust project.

You should be able to update the wasm-bindgen dependency with:

    cargo update -p wasm-bindgen --precise 0.2.127 (a579ee62b)

don't forget to recompile your Wasm file! Alternatively, you can update the
binary with:

    cargo install -f wasm-bindgen-cli --version 0.2.128

if this warning fails to go away though and you're not sure what to do feel free
to open an issue at https://github.com/wasm-bindgen/wasm-bindgen/issues!

emcc: error: '/root/.cargo/bin/wasm-bindgen test_multi.wasm --keep-lld-exports --keep-debug --out-dir /tmp/emtest_l7t4steo/emscripten_temp_1tropu2u/bindgen_out' failed (returned 1)
None
None
  test_wasm_bindgen_tsd_multi_return (test_other.other) ... FAIL (0.001s)

sbc100 pushed a commit that referenced this pull request Sep 5, 2026
#27676)

This removes the `WASM_ESM_INTEGRATION && WASM_BINDGEN` namespace import
of the wasm module added in #27208, resolves #27658.

wasm-bindgen 0.2.128 no longer reads `wasmExports[...]` inline in its
emscripten glue; it binds the asmjs-mangled identifiers that
`assignWasmExports` already provides for every export
(wasm-bindgen/wasm-bindgen#5270), so the aggregate object is not needed.

* Remove the conditional `import * as wasmExports` from
`src/postamble.js`, leaving the plain `var wasmExports` declaration for
all modes.
* Bump the pinned wasm-bindgen library and CLI from 0.2.127 to 0.2.128
(test crates, `cargo add` calls, CircleCI install).
* `test_wasm_bindgen_tsd_multi_return` now asserts the correct public
surface: only wasm-bindgen's unwrapping wrapper `multi_value_return():
number` is typed, and the raw multi-value `Result` ABI export
`_multi_value_return` stays internal. With 0.2.127 the raw export leaked
into `WasmModule`, which is what the old `[number, number, number]`
assertion was matching.

All `test_wasm_bindgen_*` tests pass against wasm-bindgen 0.2.128.

_Made with AI assistance under my review_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants