[nanvix] E: Build array as .so (Phase 0 of .a -> .so migration) - #690
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR advances the Nanvix “static stdlib modules (.a in python.elf) → shared extensions (.so via dlopen)” migration by converting array into a shared extension module and updating the Nanvix build/test harness so the shared-module path is exercised during smoke testing. In the current stacked diff, it also includes supporting toolchain/build changes needed for reliable .so linking/loading (wrapper + link flags), aligning Nanvix’s layout more closely with upstream CPython’s lib-dynload model.
Changes:
- Generate
Modules/Setup.localwith explicit*static*and*shared*groups, addingarray arraymodule.cunder*shared*. - Extend the Nanvix smoke test to import
array, assert it’s not built-in, and run a small runtime check (verifying dlopen path). - Add/propagate toolchain/linker support for shared extensions (cc-wrapper in the toolchain Dockerfile and updated link inputs for
python.elfsymbol export/availability).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
Makefile.nanvix |
Updates Nanvix link inputs and adds an existence guard for libnvx_crt0.a to support dlopen’d extensions resolving against python.elf. |
.nanvix/test.py |
Extends the hello/smoke script to validate array loads as a shared extension and performs a minimal correctness check. |
.nanvix/lxml.py |
Generates Modules/Setup.local with explicit *static*/*shared* groups and adds array as a shared module entry. |
.nanvix/docker/Dockerfile |
Installs host build helpers (pip/ninja/Cython) and installs a gcc/g++ wrapper to strip exe-only flags for -shared links. |
.nanvix/docker/cc-wrapper.sh |
New compiler-driver wrapper to distinguish exe vs shared links and filter toxic LDFLAGS for .so builds. |
.nanvix/docker.py |
Updates Docker-side Setup.local generation to use *static* and add the *shared* array entry. |
.nanvix/config.py |
Keeps the (currently unused) configure_env() helper in sync with updated linker inputs and adds libnvx_crt0 to toolchain paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+137
to
+143
| ifdef CONFIG_NANVIX | ||
| ifneq ($(filter clean distclean,$(MAKECMDGOALS)),$(MAKECMDGOALS)) | ||
| ifeq ($(wildcard $(LIBNVX_CRT0)),) | ||
| $(error libnvx_crt0.a not found at $(LIBNVX_CRT0). Update the Nanvix sysroot to one that ships libnvx_crt0.a (the nvx-crt0 crate must be present and built into the sysroot lib/ directory).) | ||
| endif | ||
| endif | ||
| endif |
Comment on lines
+451
to
+455
| # Phase 0 of the .a -> .so migration: `array` is now a shared | ||
| # extension at lib/python3.12/lib-dynload/array.cpython-312.so | ||
| # (built from `*shared* array arraymodule.c` in Setup.local). | ||
| # Asserting it is NOT in `sys.builtin_module_names` proves the | ||
| # dlopen path is exercised end-to-end; if the .so failed to load, |
This was referenced Jun 3, 2026
This was referenced Jun 12, 2026
ppenna
force-pushed
the
feat/phase0-array-so
branch
from
July 16, 2026 01:23
2b881a1 to
125d9e4
Compare
|
Reworked onto current Local validation: |
This was referenced Jul 16, 2026
Comment on lines
+142
to
+146
| install: build | ||
| $(MAKE) install DESTDIR="$(DESTDIR)" | ||
| mkdir -p "$(DESTDIR)$(INSTALL_PREFIX)/lib" | ||
| cp -f "$(NANVIX_SDK_ROOT)/lib/libc.so" "$(DESTDIR)$(INSTALL_PREFIX)/lib/libc.so" | ||
| cp -f "$(NANVIX_SDK_ROOT)/lib/libm.so" "$(DESTDIR)$(INSTALL_PREFIX)/lib/libm.so" |
| _SETUP_LOCAL_TEMPLATE = """\ | ||
| # Auto-generated by .nanvix/lxml.py -- do not edit manually. | ||
|
|
||
| # Statically-linked extension modules for Nanvix builds. |
| f"printf '%s\\n' " | ||
| f"'# Auto-generated by .nanvix/docker.py -- do not edit manually.' " | ||
| f"'' " | ||
| f"'# Statically-linked extension modules for Nanvix builds.' " |
ppenna
force-pushed
the
feat/phase0-array-so
branch
from
July 16, 2026 16:21
125d9e4 to
f448a3b
Compare
| _SETUP_LOCAL_TEMPLATE = """\ | ||
| # Auto-generated by .nanvix/lxml.py -- do not edit manually. | ||
|
|
||
| # Statically-linked extension modules for Nanvix builds. |
| f"printf '%s\\n' " | ||
| f"'# Auto-generated by .nanvix/docker.py -- do not edit manually.' " | ||
| f"'' " | ||
| f"'# Statically-linked extension modules for Nanvix builds.' " |
Comment on lines
+60
to
+65
| CFLAGS="-O3 -fPIE -fomit-frame-pointer -fno-unwind-tables -fno-asynchronous-unwind-tables" \ | ||
| CFLAGS_NODIST="-fno-semantic-interposition" \ | ||
| LDFLAGS="$(DEPENDENCY_LDFLAGS) -static -Wl,--export-dynamic" \ | ||
| LDFLAGS="$(DEPENDENCY_LDFLAGS) -Wl,--export-dynamic" \ | ||
| LINKFORSHARED="-Wl,-pie,--export-dynamic,--no-dynamic-linker,-z,notext,-z,norelro,--hash-style=sysv" \ | ||
| LDSHARED="$(TARGET_CC) -shared -Wl,--no-as-needed,-z,now,-rpath,/lib,-Bdynamic -L$(NANVIX_SDK_ROOT)/lib -l:libc.so -l:libm.so -Wl,-Bstatic" \ | ||
| LDCXXSHARED="$(TARGET_CXX) -shared -Wl,--no-as-needed,-z,now,-rpath,/lib,-Bdynamic -L$(NANVIX_SDK_ROOT)/lib -l:libc.so -l:libm.so -Wl,-Bstatic" \ |
Comment on lines
+229
to
+233
| def stage( | ||
| args: build_mod.MakeArgs, | ||
| *, | ||
| require_array_so: bool = True, | ||
| ) -> None: |
Migrate the `array` stdlib module from a statically-linked builtin to a shared extension loaded via dlopen (Phase 0 of the .a -> .so migration). Use the SDK static-PIE and dynamic-loader contract instead of the retired GCC wrapper and manual Newlib runtime composition. Shared extensions bind to the SDK libc.so/libm.so through /lib, and the install flow stages those runtime libraries. Build (Makefile.nanvix): - Compile with -fPIE and drop -static from LDFLAGS so the interpreter is a static-PIE that can dlopen extensions. - Add LINKFORSHARED, LDSHARED, and LDCXXSHARED so shared modules link against the SDK libc.so/libm.so with runpath /lib. - Install libc.so and libm.so into $(DESTDIR)/lib so the loader finds them regardless of INSTALL_PREFIX. - Make the configure marker depend on Makefile.nanvix and clean before reconfiguring, so changed compiler/linker flags never relink stale, non-PIE objects. Setup generation (.nanvix/lxml.py, .nanvix/_docker.py): - Emit explicit *static* / *shared* sections in Modules/Setup.local and declare `array` under *shared*, placed before Setup.stdlib's static declaration so makesetup's first-rule-wins picks the shared build. - Keep the native and Docker Setup.local generators in sync. Docker (.nanvix/_docker.py): - Include the effective make invocation and Makefile.nanvix contents in the persistent-volume input hash so switching release/test mode or changing build settings forces a clean rebuild. Tests (.nanvix/_test.py): - Extend the hello-world smoke test to import `array`, exercise a round trip, and assert it loaded via dlopen (CPYTHON_TEST_ARRAY / ARRAY_SO); run_hello now fails if those markers are absent. - Gate the shared-module assertion behind require_array_so so downloaded older release artifacts remain testable via a .downloaded-release marker. - Select the exact linux-x86 runtime release asset, cache it by immutable asset id with an atomic partial-file download, and record the release tag. - Remove any prior ramfs image before rebuilding so it is not embedded into its replacement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ppenna
force-pushed
the
feat/phase0-array-so
branch
from
July 16, 2026 16:51
f448a3b to
9ed60e4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First step of the
.a→.somigration plan documented innanvix-todo/cpython-static-to-shared-migration.md. Convertsarrayfrom a static built-in module into a shared extension loaded viadlopenat runtime, matching upstream CPython's layout for stdlib extension modules.The diff is small (3 files, 32 / -2 lines) and is intentionally a proof-of-concept for one module so the round-trip (configure →
setup.py build_ext→ install → import via dlopen) can be validated end-to-end before the bulk Tier-1 conversion lands.Changes
.nanvix/docker.py+.nanvix/lxml.py—Setup.localgeneration:*static*directive (was an implicit#comment thatmakesetuptolerated).*shared*group witharray arraymodule.cso CPython'ssetup.py build_extproducesarray.cpython-312-i686-nanvix.soatlib/python3.12/lib-dynload/..nanvix/test.py— extend thetest_hello.pysmoke test toimport array, assert it is NOT insys.builtin_module_names, and exercise a tiny array operation. This proves the dlopen path end-to-end on every test run; if the.sofails to load (loader bug, missing PIE relocation, symbol-resolution issue, ...) the smoke test fails immediately rather than silently regressing.Validation
./z buildand./z test:array.cpython-312.sois produced at<sysroot>/lib/python3.12/lib-dynload/array.cpython-312.so.CPYTHON_TEST_ARRAY_SO: array loaded via dlopen from /lib/python3.12/lib-dynload/array.cpython-312.so.regrtest: 39 of 40 batches pass. The remaining failure istest_structwith 32 real subtest failures (not import errors); this is investigated in a follow-up as it appears unrelated to the.a→.somigration itself.Dependencies
This PR depends on the following prereq PRs landing first. Reviewers: please wait for these (and the matching
ghcr.io/nanvix/toolchain-python:latestrebuild) before testing this one.--whole-archive+--export-dynamicinpython.elfLIBS, so dlopen'd.sos can resolve C/C++ runtime symbols against the main executable.ninjaandCythoninto the toolchain image (needed by future meson-based extensions; harmless forarray).libnvx_crt0.aintopython.elf(becomes unnecessary once [nvx] Cut over to nvx-crt0 startup crate nanvix#2453 lands, but harmless meanwhile).i686-nanvix-gccwrapper that strips exe-only LDFLAGS for.solinks. The.solink uses it.STB_WEAKundefined symbols. Without this, any.sowith a weak UND fails to dlopen.libposix.ais the sole owner ofsysallocstate ("Fix 2 v3"). Without this, the firstdlopencollides with the heap.dlfcnkeeps the leading/on absolute paths. Without this,dlopen()ofarray.cpython-312.sofails inside CPython'sregrtest(whichchdirs into a scratch dir).