Skip to content

darwinarm64: native Apple Silicon support — Mach kernel, W^X, Darwin AAPCS64 policy, ObjC bridge, Cocoa IDE - #599

Draft
egao1980 wants to merge 16 commits into
Clozure:arm64from
egao1980:cursor/darwinarm64-port-9af3
Draft

darwinarm64: native Apple Silicon support — Mach kernel, W^X, Darwin AAPCS64 policy, ObjC bridge, Cocoa IDE#599
egao1980 wants to merge 16 commits into
Clozure:arm64from
egao1980:cursor/darwinarm64-port-9af3

Conversation

@egao1980

@egao1980 egao1980 commented Aug 13, 2026

Copy link
Copy Markdown

Summary

Disclaimer: This is AI coded, human guided and tested work.

Native Darwin/arm64 (Apple Silicon) support on top of the current arm64 branch, in seven commits:

  1. native kernellisp-kernel/darwinarm64/ builds darm64cl; Mach exception server (arm64-darwin-mach.c, ported from the darwinx8664 path); MAP_JIT-based W^X (executable lisp code in a MAP_JIT mapping, purify moves pure objects to AREA_READONLY RX, dynamic heap never executable, write-protect toggles only in kernel C); GC corrupt-header diagnostics; 16KiB-page-aware guard stepping.
  2. runtime and FFI — the OS-independent AAPCS64 logic (classification, expand-ff-call, callback glue) moves from lib/ffi-linuxarm64.lisp to a shared lib/ffi-arm64.lisp; both OS files are now thin delegators. Darwin ABI divergences ride on top, keyed on the target OS: variadic arguments are stack-only behind a :variadic sentinel (with the objc_msgSend family excluded — its ... prototype does not use the C variadic convention on arm64), and non-variadic stack overflow packs scalars at natural size/alignment. Also: Darwin syscall convention (x16, svc #0x80, carry→-errno), bcond_ext/ADRP/C() assembler portability for Mach-O, save0–save2 GC-visibility spills in .SPsyscall (same contract as ffcall), %throw/%throwing-through-cleanup-p for :propagate-throw callbacks, and MAP_JIT-safe callback trampoline installation.
  3. ObjC bridge and Cocoa IDE — Apple tagged pointers, no objc_msgSend*_stret on arm64, varargs sends lowered onto the interpreted %ff-call, call-next-method over ObjC methods, untitled Listener at launch, Trace/Source/Inspect selection fixes. (require :cocoa-application) builds Clozure CL64.app end to end.
  4. interface database tooling — scripts to regenerate the darwin-arm64 .cdb databases from the current SDK with ffigen5; the generated databases stay out of the tree.
  5. stock build path(rebuild-ccl :full t) works like other platforms (cross xload → cold load → purify → save-application); dumplisp clears MAP_JIT macptrs before the image is written.
  6. smoke tests and CI harness — FFI/callback/Cocoa smokes, a byte-stable launch-layout regression reproducer for the ffcall GC-visibility bug class, Cocoa mini-apps, and tools/run-darwin-arm64-ci.sh as a single gate.
  7. porting notesdoc/porting/darwin.md (why -pagezero_size/-no_pie don't exist on arm64 Darwin, the W^X policy and how V8/JSC handle MAP_JIT), doc/porting/progress.md.

Where this branch overlapped work that already landed on arm64 (#572's AAPCS64 FFI, the spentry consolidation, the x10 callback-index trampoline, interpreted %ff-call), the in-tree implementation was kept and only the Darwin deltas were added.

Validation (Apple Silicon, macOS)

  • make -C lisp-kernel/darwinarm64 → native darm64cl
  • (rebuild-ccl :full t) end to end, plus a forced full self-host rebuild (:force t) from the produced image
  • ANSI + CCL suites from ccl-tests: all 21920 tests pass (~40s)
  • Full smoke gate (tools/run-darwin-arm64-ci.sh): math, purify save/reload, cocoa CDB/bridge, interpreted ff-call, clean build, (require :cocoa)/Hemlock, AppKit mini-apps
  • FFI/callback smokes: struct return (HFA NSSize/NSPoint/NSRect, GPR NSRange), callback sret/struct/HFA (incl. runtime drawRect:), HFA arguments, variadic (#/stringWithFormat:), natural-size pack overflow, NSRange foreign-type resolution, ffcall stack discipline, ObjC bridge, call-next-method, init keywords
  • Launch-layout GC regression reproducer passes; the stock-built IDE opens its untitled Listener and stays healthy
  • Deprecated AppKit names (NSOffState, NSCommandKeyMask, NSOKButton, NSNotFound, window masks, bezel styles, control sizes, ...) are static const aliases in modern SDKs, so freshly translated interface databases record them as foreign variables that fail to resolve at load time (the NSOffState error seen above). The IDE, Hemlock bindings, and the bundled easygui/progress examples now use fixed-ABI Lisp constants (cocoa-ide/constants.lisp) instead — audited against every #$ name the IDE build references, so bring-up no longer depends on any of those names being present in the .cdbs.
  • Verified end-to-end against databases regenerated with the ccl-ffigen#6 translate.sh flow on the current SDK: with this branch's injectors applied (inject-objc-msgsend-prototypes — modern objc/message.h declares objc_msgSend argless, without it (require :objc-support) fails with "Extra arguments"; inject-objc-bridge-types; inject-objc-protocol-class; and the new inject-libc-computed-constants for sizeof/cast macros the translator cannot reduce: FIONBIO, HOST_BASIC_INFO_COUNT, CPU_TYPE_*), the full rebuild, ANSI+CCL suites (21920/21920), smoke gate, IDE app build, and IDE interaction smokes all pass on both the regenerated and the previously generated databases. The carbon/gl/quartz/quartzcore databases come out byte-identical between the two flows.
  • Cocoa callback error handling: a Lisp error inside an ObjC callback (e.g. a table-view action) previously corrupted the error-return trampoline (it wrote MAP_JIT memory directly) and could wedge the event thread; the trampoline is now installed via the kernel blit path, the xapropos row -1 action is guarded, and a click-regression smoke covers the flow

Linux/arm64 is intended to be unaffected: the shared lib/ffi-arm64.lisp is a file move of the linuxarm64 implementation (the Darwin policies are dormant behind target-OS checks), and the kernel assembly changes are #ifdef-gated or expand to the previous instructions on Linux (bcond_ext, load_addr_of_lisp_nil). I could not run a linuxarm64 build here; happy to fix anything a Linux CI run turns up.

…nd image support

Bring up the native Apple Silicon lisp kernel (darm64cl):

- lisp-kernel/darwinarm64/Makefile, modeled on linuxarm64 plus the
  darwinx8664 Darwin link specifics (no -pagezero_size / -no_pie on
  arm64 Darwin; ASLR tolerated).
- arm64-darwin-mach.c: Mach exception handling ported from the
  darwinx8664 path in x86-exceptions.c to ARM64 thread/neon state and
  EXC_ARM_* codes.  UUOs arrive as EXC_BAD_INSTRUCTION and are resumed
  into signal_handler via a synthetic ucontext, returning through
  pseudo_sigreturn.  Scratch-frame lifecycle is correct across nested
  exceptions; thread_get_state results are checked.
- W^X: executable lisp code lives in a MAP_JIT mapping
  (darwin_arm64_set_code_heap); purify moves pure objects to
  AREA_READONLY as RX; the dynamic heap is never executable.
  Write-protect toggles happen only in kernel C, never per-thread from
  the exception server.  Pure-page dirtying oscillates RW/RX per fault.
  CCL_DEBUG_WX gates fault dumps; non-debug faults are quiet.
- GC: the corrupt-uvector-header check dumps the neighborhood and the
  root origin (mark phase, slot, parent) before aborting.
- 16KiB-page-aware soft-stack guard stepping; ssize_t MapFile read;
  Darwin syscall convention (x16 + svc #0x80, carry flag mapping to
  -errno).
…all, W^X compile-file

Build the Darwin side of the FFI on the shared AAPCS64 implementation.
The OS-independent record classification, ff-call expansion, and
callback glue move from lib/ffi-linuxarm64.lisp to lib/ffi-arm64.lisp;
ffi-linuxarm64.lisp and ffi-darwinarm64.lisp become thin delegators
that (require "FFI-ARM64").

- Apple ABI divergences, keyed on the target OS in the compiler and in
  the interpreted %ff-call: arguments after a :variadic sentinel are
  stack-only in 8-byte slots (%external-call-expander emits the
  sentinel at the CDB :void boundary, but not for objc_msgSend-family
  entry points, whose ... prototype does not use the C variadic
  convention on arm64), and non-variadic stack overflow packs scalars
  at natural size and alignment instead of AAPCS64's uniform 8-byte
  slots.  Composite (C.13) stack copies stay 8-byte aligned on both
  OSes.
- Darwin assembler/linker portability in the kernel .s: bcond_ext for
  conditional branches to external _SP symbols (Mach-O cannot relocate
  them), ADRP for the lisp_nil global (no text relocs), C() on
  kernel-internal call targets, and the Darwin syscall convention
  (number in x16, svc #0x80, carry flag mapping errno to -errno).
- .SPsyscall spills/reloads save0-save2 around the foreign transition,
  matching the ffcall GC-visibility contract (a thread in a syscall has
  no gc_context, so boxed values left in x19-x21 are invisible to a
  concurrent GC and stale after compaction).
- %throw and %throwing-through-cleanup-p on arm64: only THROW (via
  .SPthrow) counts as throwing through an unwind-protect cleanup, so
  :propagate-throw ObjC callbacks do not patch the foreign LR on a
  normal return.
- Callback trampolines: on Darwin the callback page is MAP_JIT, so the
  stub is assembled into heap scratch and blitted into place by kernel
  C (never pthread_jit_write_protect_np from MAP_JIT-resident lisp).
- compile-file on native Darwin always allocates code vectors in
  MAP_JIT: eval-when (:compile-toplevel) must execute code while the
  heap is NX; fasl dump only needs readable bytes.
- Darwin target arch uses its own nil-value (static space lives at
  #x200000000; MAP_FIXED at the Linux static page is impossible).
- Assorted arm64 runtime fixes (macptr init, backtrace robustness,
  typedef/record layout adoption for Apple CDB quirks).
- Apple arm64 tagged pointers (bit 63) recognized in
  tagged-objc-instance-p.
- Message sends use plain APPLY again; objc_msgSend*_stret is never
  used on Apple arm64 (record returns follow the AAPCS64 conventions
  from the FFI layer).
- Darwin method varargs are stack-only, so e.g. #/stringWithFormat:
  works; ns:protocol resolves via %ensure-class-declaration.
- call-next-method over ObjC methods works (rlet of :objc_super moved
  out of the &rest frame that APPLY's stack could clobber).
- IDE fixes: untitled Listener opens at launch; Trace/Source/Inspect on
  a selected form no longer re-enqueue the form into the Listener;
  tip-of-tree cocoa recompiles do not break Listener typing; file
  dialog and xapropos cleanups.
- Lazy callback trampoline creation and objc-propagate-throw for
  exceptions crossing the bridge.

(require :cocoa-application) builds Clozure CL64.app end to end with a
stock IDE image.
@gmpalter

Copy link
Copy Markdown
Member

@xrme is away until next Tuesday, the 18th.

@egao1980

Copy link
Copy Markdown
Author

I'm still working on IDE parts so it definitely can wait.

@egao1980
egao1980 force-pushed the cursor/darwinarm64-port-9af3 branch 2 times, most recently from d1b084a to a4c8b8a Compare August 13, 2026 18:19
@egao1980

Copy link
Copy Markdown
Author

IDE looks fine at this point including inspector.

@gmpalter

Copy link
Copy Markdown
Member

I just built a macOS Arm64 CCL.

I'm not certain that the CDB files are all ok, however. I'm going to submit a PR to the ccl-ffigen repository with what I did based on your work. (I've replaced the XXX-populate.sh scripts with appropriate translate.sh scripts.)

@gmpalter

Copy link
Copy Markdown
Member

Try to bring up the IDE gets the error ...

> Error: Can't resolve foreign symbol "NSOffState"
> While executing: CCL::RESOLVE-FOREIGN-VARIABLE, in process Initial(0).

@gmpalter

Copy link
Copy Markdown
Member

Here's the ccl-ffigen pull request -- Clozure/ccl-ffigen#6

@egao1980
egao1980 force-pushed the cursor/darwinarm64-port-9af3 branch from a4c8b8a to 1ad6550 Compare August 15, 2026 07:13
Scripts under tools/darwin-arm64-cdb/ regenerate the darwin-arm64
interface databases (.cdb) from the current macOS SDK with ffigen5:

- populate scripts per interface directory (libc, cocoa, gl, carbon,
  quartz, quartzcore) plus h-to-ffi.sh / filter-ffi.py to drop
  Availability macro cycles and unmapped clang kinds.
- inject-*.lisp add the ObjC bridge shims the bridge expects
  (BOOL constants, bridge types, objc_msgSend prototypes, Protocol
  class) into the databases.
- parse helpers wrap library/parse-ffi.lisp for Cocoa-scale .ffi input.

The generated databases themselves stay out of the tree (gitignored);
doc/porting/darwin-cdb.md documents the workflow and per-directory
status.
(rebuild-ccl :full t) is the standard rebuild on darwinarm64, like the
other platforms: cross-xload the boot image, cold load, purify, and
save-application in one pass.  Self-hosting verified (a second full
rebuild from the produced image succeeds without any LAP preload).

- tools/xdarwinarm64.lisp: cross-compilation target setup.
- xdump/xfasload, xarm64fasload: darwinarm64 image target.
- lib/compile-ccl.lisp: darwinarm64 in the rebuild/kernel matrix.
- lib/dumplisp.lisp: MAP_JIT macptrs are cleared before the image is
  written and remapped on restart.
- .gitignore covers darm64cl and generated Mach exception stubs.
Non-interactive regression coverage for the port, all runnable from a
fresh checkout via tools/run-darwin-arm64-ci.sh:

- FFI: struct return, callback sret/struct/HFA, HFA arguments,
  variadic, pack overflow, interpreted ff-call, NSRange foreign type,
  ffcall stack discipline.
- Launch-layout reproducer (tools/darwin-launch-layout-repro.lisp) for
  the formerly GC-invisible ff-call state; byte-stable so a regression
  shows up deterministically.
- Cocoa mini-apps 01-17 exercising the bridge, subclass callbacks,
  call-next-method, menus, event loop, drawRect, and IDE-shaped launch.
- purify / clean-build / require-cocoa / math smokes; opt-in IDE
  diagnostic loggers in tools/ide-debug-loggers.lisp (nothing is loaded
  into product images by default).
- tools/with-timeout: portable hard wall-clock timeout wrapper used by
  the harness.
doc/porting/darwin.md: why -pagezero_size and -no_pie do not exist on
arm64 Darwin, the W^X policy and how other runtimes (V8, JSC) handle
MAP_JIT, and the pure-page dirtying scheme.

doc/porting/progress.md: current status — stock (rebuild-ccl :full t)
self-hosts, (require :cocoa-application) builds the IDE, ccl-tests
244/244, ANSI+CCL suites 21920/21920.
@egao1980
egao1980 force-pushed the cursor/darwinarm64-port-9af3 branch from 1ad6550 to 6749f84 Compare August 15, 2026 07:36
@egao1980

Copy link
Copy Markdown
Author

@gmpalter I've pushed few changes and fixes - your PR is looking great, you probably can add ObjC injectors / libc one from this branch too (inject-* files).

@gmpalter

Copy link
Copy Markdown
Member

You don't need inject-libc-computed-constants.lisp.

The trick with parse-standard-ffi-files is to run it twice. The second run will resolve those computed constants.

@gmpalter

Copy link
Copy Markdown
Member

The problem I reported above about NSOffState is either an ffigen5 or parse-ffi problem.

In AppKit.framework/Headers/NSCell.h, we find these two definitions ...

static const NSControlStateValue NSControlStateValueOff = 0;                                   
static const NSControlStateValue NSOffState API_DEPRECATED_WITH_REPLACEMENT("NSControlStateValueOff", macos(10.0,10.14)) = NSControlStateValueOff;                                           

Those should have resulted in a definition in the CDB for NSOffState but didn't.

ffigen records modern SDK `static const` aliases as unlinkable
`(static)` vars, so #$NSOffState dies in resolve-foreign-variable.
Inject the fixed ABI values and skip static vars in parse-ffi.
@gmpalter

Copy link
Copy Markdown
Member

Here's a warning ...

I updated from macOS 26.6.1 to 26.6.2 yesterday. Now, when I try to launch the image I built a few days ago, I get

[palter@miniMe ~/CCL/ccl-macosarm64](15)$ ./darm64cl
Couldn't load lisp heap image from /Users/palter/CCL/ccl-macosarm64/darm64cl.image: Invalid argument

I'm going to try cross compiling and building a new image to see if that'll work.

@gmpalter

Copy link
Copy Markdown
Member

I just did a fresh cross-compile and cross-xload-level-0. I recompiled the kernel. Same error. 26.6.2 has broken this port!

[palter@miniMe ~/CCL/ccl-macosarm64](38)$ ./darm64cl -I arm64-boot.image 
Couldn't load lisp heap image from arm64-boot.image: Invalid argument

@egao1980

Copy link
Copy Markdown
Author

I'll check a bit later today. Thanks for the feedback.

macOS 26.6.2 broke image loading for a tester with only
"Couldn't load lisp heap image ...: Invalid argument" to go on.
Report which section / mmap / lseek / read failed, with target VA,
length, image offset, and errno, preserving errno for the final
message.
@gmpalter

Copy link
Copy Markdown
Member

It looks like the first call to mmap in CommitMemory in images.c is failing. The call is

   215 	    /* W^X: RWX mmap is rejected.  Dynamic heap is RW only; executable
   216 	       code lives in MAP_JIT (AREA_CODE) or AREA_READONLY after purify. */
-> 217 	    addr = mmap(start, len, MEMPROTECT_RW, MAP_PRIVATE|MAP_ANON|MAP_FIXED, -1, 0);

with start = 0x000000020000000 and len = 16384.

I've attached a sample from Activity Monitor which shows where the various libraries are loaded in memory. (I'm not sure why libRosetta is loaded as this is definitely native Arm code.) Although nothing's loaded into those addresses, it does fall between a couple of libraries. I wonder if macOS is now refusing to map anything between libraries.

Sample of darm64cl.txt

@gmpalter

Copy link
Copy Markdown
Member

For the record, the Symbolics Virtual Lisp Machine (VLM) which I maintain also maps its data into fixed locations in the address space and it runs without problems. The address ranges it uses are 0x10000000000 - 0x100FFFFFFFF and 0x40000000000 - 0x4003FFFFFFF.

Either you'll need to change where areas are mapped into memory or devise a way to make areas relocatable. I suspect the former is easier but has the problem that future macOS updates could subsume the address range requiring finding a new range. The latter is more difficult as I presume there are pointers inside the areas that would need to be relocated. (Perhaps some of the GC's mechanisms could be used to do that.)

Re-signing darm64cl with hardened-process entitlements on 26.6.2/M4
reproduces the image-load failure class: dyld-ro owns 0x200000000, so
MAP_FIXED RW at STATIC_BASE_ADDRESS fails and AREA_STATIC cannot map.
hardened-heap/guard objects boot and GC clean.  Durable fix is
relocatable statics; document the audit trail and add the probe.
tools/darwin-image-load-diag.sh — one-shot report for the
"Couldn't load lisp heap image" class: OS/chip/pagesize, kernel
binary hardened-process entitlements, fixed-VA mmap + VM-region probe
at the CCL bases, and the loader's own failure lines.  Output carries
no usernames, hostnames, or absolute paths.  Probe now also reports
what occupies each VA (protection + user tag).
Hardened VM policy (dyld-ro; reported on macOS 26.6.2) rejects
MAP_FIXED at the canonical STATIC_BASE_ADDRESS — dyld's read-only
state region starts one page above it — killing every image load.

Probe the canonical base at reserve time and fall back to an
OS-chosen address; rebase saved static references by
static_space_bias at load.  The saved base travels in the
AREA_STATIC section header's static_dnodes slot (old images imply
the canonical base), the nilreg area is rebased word-wise (the raw
globals page misparses the object walker), and the relocation mask
gains arm64's fulltag_symbol.  Kernel-global access now compiles
rnil-relative (%get-kernel-global* -> *-from-offset primitives,
resolving the arch-macro xxx markers); kernel-global-address uses
the live nil.  CCL_FORCE_STATIC_RELOC forces the fallback (smoke in
the CI gate), CCL_STATIC_RELOC_AUDIT reports unrebased leftovers,
and CCL_DEBUG_WX now dumps the faulting instruction and GPRs.

Verified: full smoke gate plain and under forced relocation;
hardened (dyld-ro and full Enhanced Security set) boots, GC,
static conses; save/reload round-trips relocated->canonical and
relocated->relocated.  Requires (rebuild-ccl :full t).
@egao1980

Copy link
Copy Markdown
Author

@gmpalter I think I've found (and fixed) your issue. Please check.

@gmpalter

Copy link
Copy Markdown
Member

I just tried your latest commits and was able to build and run CCL on macOS 26.6.2, thank you.

I presume you'll remove the message

note: static space at 0x102628000 (canonical 0x200000000 unavailable: Permission denied)

before this is merged into the arm64 branch.

Callers compute 4KiB-masked addresses (x86 heritage); 16KiB-page
Darwin rejects unaligned MAP_FIXED with EINVAL, and these commits
had been failing silently since bring-up.  A fresh MAP_FIXED of the
containing pages would zero live neighbors, so mprotect(RW) the
span instead.  (Meant to be part of the relocatable-statics commit.)
Relocation is normal operation under hardened VM policy; the note
confused a tester into thinking something was wrong (PR Clozure#599).
…-port-9af3

# Conflicts:
#	compiler/ARM64/arm64-disassemble.lisp
#	lisp-kernel/arm64-spentry.s
Upstream removed the lisp-side syscall layer (SYSCALLS package,
define-syscall, tables); all OS work is ff-calls into libc.  Raw
syscall numbers are not a stable ABI on Darwin, so the spentry stays
unconsumed there by design.
@gmpalter

Copy link
Copy Markdown
Member

@xrme has merged my PR for generating the darwin-arm64-headers into https://github.com/Clozure/ccl-ffigen. So, you no longer need your tools/darwin-arm64-cdb directory.

xrme added a commit that referenced this pull request Aug 26, 2026
Following darwinx8664, define and use mig-generated mach_exc server
thread to catch EXC_BAD_ACCESS, EXC_BAD_INSTRUCTION, EXC_ARITHMETIC
and EXC_SOFTWARE for lisp threads.  Hand-roll a signal context from the
Mach thread state and proceed via the existing signal_handler and
pseudo_sigreturn path.

  arm64-darwin-mach.c: exception handler, synthetic signal context,
                       readonly_area W^X fixups (RX for an NX fetch,
                       RW for a write)
  arm64-exceptions.c:  darwinarm64 specifics, including is_write_fault
                       via ESR, cold-load debugging
  platform-darwinarm64.h: mcontext accessors, opcode/pc, sigreturn
                          config
  darwinarm64/Makefile: new for arm64

Adapted from #599 (egao1980).
xrme added a commit that referenced this pull request Aug 26, 2026
Executable lisp code resides in one of two regions under W^X: code
purified into the saved image's read-only section is mapped RX at
load; code compiled (or fasloaded) since the image started goes into a
MAP_JIT region.  When saving an image, the code from the MAP_JIT
region migrates into the read-only area.  The dynamic heap is never
executable.

The MAP_JIT code area is enough to get started with, but isn't a
normal CCL memory area.  It's tracked with ad-hoc base/active bounds.
The GC knows the bounds, but it doesn't collect or compact the MAP_JIT
region.

The page size on darwinarm64 is 16K (not 4K).  Derive log2_page_size
from the real page size at startup.

The image file is laid out at 4K (0x1000) page granularity even
though the runtime page size is 16K.  Maintain this arrangement.

  memory.c: page-size aware CommitMemory; MAP_JIT region bounds
  image.c:  4K image granularity even though native page size is 16K
  memprotect.h: MAP_JIT region bounds declarations
  pmcl-kernel.c: derive log2_page_size; round stack-guard sizes to
                 real page size

Adapted from #599 (egao1980).
xrme added a commit that referenced this pull request Aug 26, 2026
Extend the "is this a purify source?" test to accept pointers
in the MAP_JIT region.

Forwarding a purified object writes a marker into the old object.
When the old object is in MAP_JIT memory, bracket that store with
pthread_jit_write_protect_np.

In principle, these are all #if defined(DARWIN) && defined(ARM64),
with one exception: the create_thread_context_frame extern in
thread_manager.c (which is #ifdef DARWIN only, and hence shared by all
Darwin ports) was corrected to read siginfo_t **; harmless because the
sole caller passes NULL.

Adapted from #599 (egao1980).
xrme added a commit that referenced this pull request Aug 26, 2026
Add minor portability fixes: asm macro for loading lisp_nil
in both Mach-O and ELF ways, use C(get_tcr) so that Mach-O symbols
get the leading underscore and ELF symbols don't.

Implement the Darwin sigreturn asm: pseudo_sigreturn (used via the
Mach exception handler path) and darwin_sigreturn (Unix signal
fallback).  On non-Darwin, leave harmless stubs.

The bcond_ext workaround is no longer needed now that arm64-spentry.s
is a single file: every _SP branch is intra-object now.

Adapted from #599 (egao1980).
xrme added a commit that referenced this pull request Aug 26, 2026
Factor out the arm64/AAPCS64 FFI support into ffi-arm64.lisp; the
individual ffi-{linux,darwin}arm64.lisp files both use it.

Apple divergences from AAPCS64 are gates on the target OS:
via #+darwinarm64-target for runtime %ff-call, backend-target-os in the
compiler, or via the :variadic keyword (in ffi-arm64.lisp) that
is emitted only on Darwin.

New %throwing-through-cleanup-p workaround for defcallback with
:propagate-throw.

Add an ugly hack where local-git-revision retries after an odd fork
failure.

Update parse-ffi to avoid choking on some incorrect ccl-ffigen output,
and to better process huge numbers of macros as seen in Cocoa.

Adapted from #599 (egao1980).
xrme added a commit that referenced this pull request Aug 26, 2026
- tools/xdarwinarm64.lisp: cross-compilation target setup.
- xdump/xarm64fasload: darwinarm64 image target.
- lib/compile-ccl.lisp: darwinarm64 in the rebuild/kernel matrix.
- lib/dumplisp.lisp: MAP_JIT macptrs are cleared before the image is
   written, and remapped on restart.
- .gitignore: generated Mach exception stubs, interface dirs,
   build/test output

Adapted from #599 (egao1980).
xrme added a commit that referenced this pull request Aug 26, 2026
As long known, we can't rely on a fixed STATIC_BASE_ADDRESS on Darwin,
so support relocating it.

Try the default address at reserve time; if it fails, rebase static
references by static_space_bias (actual - saved).  If we can put
static space where we want it, the bias is 0, and relocation is a
no-op.

Access kernel globals rnil-relative at run time via the
%get-kernel-global-from-offset primitives instead of hard-wiring
target-nil-value into compiled code.  This makes global access work
wherever static space ends up.  This is an arm64-wide change, but is
functionally identical on linuxarm64.

There's a debug environment variable CCL_FORCE_STATIC_RELOC to
force static area relocation; CCL_STATIC_RELOC_AUDIT verifies
the rebasing.  These are subject to deletion at any time.

Adapted from #599 (egao1980).
xrme pushed a commit that referenced this pull request Aug 26, 2026
Relocation is normal operation under hardened VM policy; the note
confused a tester into thinking something was wrong (PR #599).
@xrme

xrme commented Aug 26, 2026

Copy link
Copy Markdown
Member

I've incorporated the changes that are needed to make the tty lisp work. Thank you very much.

They've landed on the arm64 branch. I'll next work on reviewing and bringing over the Cocoa-related changes.

One thing I think that is worth noting: the MAP_JIT region as currently implemented is fine as an interim mechanism to get the ball rolling, but for the darwinarm64 port to be real, a more robust scheme will be needed. (I don't think I'm saying anything you don't already know.). But having a native, self-hosting CCL on darwinarm64 is a win, even if there's more to do. Again, thank you.

xrme added a commit that referenced this pull request Aug 26, 2026
Adapted from #599 (commits 43ca48b + d83c7b6, squashed so the
superseded hand-rolled varargs packer is dropped in favor of the
%ff-call-based marshalling).

Recognize Apple arm64 tagged pointers (bit 63); drop
objc_msgSend*_stret (record returns go through the normal FFI layer);
Darwin stack-only method varargs via %ff-call; call-next-method over
ObjC via heap objc_super record; NXArgv char** type fix. Loads
via (require 'objc-support) and passes smoke tests (msgSend, varargs
int/object/float, defmethod callback,
call-next-method/objc_msgSendSuper) on darm64cl.

This is all Darwin-only; nothing on Linux should be affected.
xrme added a commit that referenced this pull request Aug 27, 2026
Adapted from #599 (43ca48b, d83c7b6).  Left out inject-*.lisp
files and c9b128f (although we might need to grab the static const
special case handling from the parse-ffi.lisp changes).

Speaking of static const, modern SDKs define the pre-10.12 AppKit
names like NSOffState, NSCommandKeyMask, ..., as static const rather
than as macros/enums, so #$ no longer sees them.  Define lisp
constants for them.

Made some modern-SDK fixes across cocoa-ide/ and examples/cocoa.

The implementation of -[FontToNameTransformer transformedValue:] use
the old interleaved type/value varargs notation; the %ff-call on arm64
rejects this (:id read as a value).  This accidentally ended up
working on x86 because it silently drops unrecognized varargs.

The long and short: (require 'cocoa) starts the IDE; the Listener
evaluates forms, the settings window opens.  This is all Darwin-only.
@xrme

xrme commented Aug 27, 2026

Copy link
Copy Markdown
Member

I brought in (to the arm64 branch) most of the objc bridge and cocoa IDE files now, too.

The next thing to verify is to freshly rebuild the interfaces with ccl-ffigen and process them with the parse-ffi as found on the arm64 branch.

I omitted your special-casing of static const that you have in c9b128f; it may be that I need to pick that up when I get around to trying a fresh rebuild of the interfaces.

Like I mentioned in the commit messages, it should now work to do (require 'cocoa) from the arm64 branch.

@no-defun-allowed

no-defun-allowed commented Sep 2, 2026

Copy link
Copy Markdown

MAP_JIT-based W^X (executable lisp code in a MAP_JIT mapping, purify moves pure objects to AREA_READONLY RX, dynamic heap never executable, write-protect toggles only in kernel C)

I did most of this already in my Rosetta-friendly x86-64 port https://github.com/no-defun-allowed/ccl/blob/master/lisp-kernel/code-gc.c and https://github.com/no-defun-allowed/ccl/blob/master/lisp-kernel/image.c#L336-L350. Thank you for nothing, you sloppers.

@xrme

xrme commented Sep 2, 2026

Copy link
Copy Markdown
Member

I did most of this already in my Rosetta-friendly x86-64 port https://github.com/no-defun-allowed/ccl/blob/master/lisp-kernel/code-gc.c and https://github.com/no-defun-allowed/ccl/blob/master/lisp-kernel/image.c#L336-L350. Thank you for nothing, you sloppers.

@no-defun-allowed I expect that the interim implementation currently here will be superseded by something based on what you already did. I know not everyone is going to think it's a good idea to accept contributions like this, but I believe that having a working native CCL on macOS/arm64, even if it is hackish in ways, is worthwhile and will enable faster future progress.

@egao1980

egao1980 commented Sep 2, 2026

Copy link
Copy Markdown
Author

MAP_JIT-based W^X (executable lisp code in a MAP_JIT mapping, purify moves pure objects to AREA_READONLY RX, dynamic heap never executable, write-protect toggles only in kernel C)

I did most of this already in my Rosetta-friendly x86-64 port https://github.com/no-defun-allowed/ccl/blob/master/lisp-kernel/code-gc.c and https://github.com/no-defun-allowed/ccl/blob/master/lisp-kernel/image.c#L336-L350. Thank you for nothing, you sloppers.

Hi @no-defun-allowed, I would also prefer human written / reviewed code. That's why this PR is marked as AI generated human tested draft. This is an expensive web search / prior art study / make it work effort. I've created quite a few smoke tests to help porting CCL to M chips in case the changes won't end up in the main project.
Common Lisp is currently a niche language and PRs can sit in OSS project repos for years (ask me how I know that :). CCL port is a tough challenge - you need either a team or a senior developer with CL + arm64 assembly + MacOS system internals + Cocoa skill set and lots of free time.
Please treat this code as a kickstart to get a working version with IDE until we have the proper release. Please please keep working on CL projects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants