Skip to content

Let a dynamic-extent declaration reach the closure extent analysis - #35

Open
dg1sbg wants to merge 2 commits into
s-expressionists:mainfrom
dg1sbg:feat/dx-declaration-local-calls
Open

Let a dynamic-extent declaration reach the closure extent analysis#35
dg1sbg wants to merge 2 commits into
s-expressionists:mainfrom
dg1sbg:feat/dx-declaration-local-calls

Conversation

@dg1sbg

@dg1sbg dg1sbg commented Aug 3, 2026

Copy link
Copy Markdown

Stacked. The first commit here is the soundness fix submitted separately; GitHub cannot base a PR on a branch that exists only in a fork, so it appears in both. Once that one merges this reduces to the single feature commit. Review the second commit.

DETERMINE-CLOSURE-EXTENT proves non-escape for exactly one shape: every use of the closure is a call carrying :DX-CALL, in the function that encloses it. A closure handed to an FLET or LABELS function therefore always goes on the heap, even when the callee does nothing with it but call it.

Meanwhile a CL:DYNAMIC-EXTENT declaration is parsed into the environment and then dropped — nothing outside Environment/ and CST-to-AST/ reads it, and BIR:DYNAMIC-LETI, which exists and is exported for exactly this purpose, is never constructed anywhere.

This carries the declaration on the binder, which is what BIR:VARIABLE's own docstring says should happen: the declaration constrains the extent of the value bound rather than of the variable.

Permission, not proof

With the declaration, the analysis may walk into a statically known callee and check that the matching parameter does not escape either. Without it, that walk is not attempted. If the walk fails, the declaration is ignored. So a declaration that is wrong costs the optimisation and never memory safety — which seemed the only defensible reading, given that the failure mode of trusting it is heap corruption.

Two generalisations that fall out

  • A callee is not only known via BIR:LOCAL-CALL. A front end that builds BIR from an already compiled representation can leave the call as a plain CALL whose callee is still statically an ENCLOSE, so this resolves the callee datum rather than requiring the instruction class.
  • A parameter is not always bound to a variable. A front end need not emit a LETI for one that is read once, in which case the parameter is used directly by the consuming instruction. USE-RETAINS-NOTHING-P handles both shapes.

Both were found the hard way: the analysis silently never fired on the client I was testing against, for each of these reasons in turn.

Independent of the declaration

Occupying the callee position of a call is now recognised as a non-escape on its own, needing no declaration: invoking a function cannot retain it.

Scope limits

Only required parameters passed exactly once, with nothing but required parameters ahead of them, are followed — so the position-to-parameter mapping is never ambiguous. Recursion through mutually recursive callees terminates on a visited set; a cycle contributes no use that was not already checked on the way in.

Verification

Through Clasp. A closure passed to a capturing FLET called from three sites, 100000 calls: 544 → 504 bytes per call, results unchanged. Two deliberately-wrong declarations — a callee that returns the closure, and one that stores it in a global — both correctly forfeit the optimisation and both closures still work afterwards. Full regression suite on both GC variants plus ANSI: no unexpected failures.

One caveat I would rather state than leave implicit: the CST-to-AST and Abstract-syntax-tree changes are compile-verified only. Clasp does not load those systems — it builds BIR from bytecode — so nothing I ran exercises that path. It is written to mirror how %IGNORE already rides through the same three places, but it deserves a look from someone whose client uses the CST front end.

dg1sbg added 2 commits August 2, 2026 21:00
DETERMINE-CLOSURE-EXTENT set the ENCLOSE's extent to :dynamic from inside
the loop over the variable's readers, so the mutation happened as soon as
the first reader turned out to be a DX call. A later reader that escapes
bails out with RETURN-FROM, which does not undo it, and the closure stays
marked :dynamic. The backend then builds it with cc_stack_enclose over an
alloca in the enclosing frame -- and it escapes anyway. Using it once that
frame is gone is a use-after-return; under Clasp it surfaces as
EXT:BUS-ERROR.

Which reader is seen first decides whether this fires, and that order is
unspecified: Set/set.lisp picks the hashset representation, because the
#+(or) guard on :listset is always false, so DOSET walks an EQ hash table
and the order follows object addresses. Compiling

  (lambda (x) (let ((g (lambda (y) (+ x y)))) (mapcar g '(1 2 3)) g))

300 times with Clasp's native compiler stack-allocated the returned
closure 154 times. MAPCAR carries the :dx-call attribute, and the RETURNI
is the escape.

Move the assignment after the loop, where the conclusion it records has
actually been established. A variable with no readers now qualifies too,
which is correct -- a value that is never read cannot escape; previously
the loop body simply never ran.

Introduced in 3884e93.
DETERMINE-CLOSURE-EXTENT proved non-escape for exactly one shape: every
use of the closure is a call carrying the :DX-CALL attribute, in the
function that encloses it. A closure handed to an FLET or LABELS function
therefore always went on the heap, even when the callee does nothing with
it but call it.

A CL:DYNAMIC-EXTENT declaration was parsed into the environment and then
dropped -- nothing outside Environment/ and CST-to-AST/ ever read it, and
BIR:DYNAMIC-LETI, which exists and is exported for the purpose, is never
constructed anywhere. Carry the declaration on the binder instead, which
is what BIR:VARIABLE's own docstring says should happen, the declaration
constraining the extent of the value bound rather than of the variable.

The declaration is permission, not proof. With it the analysis may walk
into a statically known callee and check that the matching parameter does
not escape either; without it that walk is not attempted, and if it fails
the declaration is ignored. A wrong declaration costs the optimisation
and never memory safety.

Two things fall out of doing that walk properly:

- A callee is not only known via BIR:LOCAL-CALL. A front end building BIR
  from an already compiled representation can leave the call as a plain
  CALL whose callee is still statically an ENCLOSE, so resolve the callee
  datum rather than requiring the instruction class.
- A parameter is not always bound to a variable, since a front end need
  not emit a LETI for one that is read once. USE-RETAINS-NOTHING-P
  handles both shapes.

Occupying the callee position of a call is now recognised as a non-escape
on its own, needing no declaration: invoking a function cannot retain it.

Only required parameters passed exactly once, with nothing but required
parameters ahead of them, are followed, so the position-to-parameter
mapping is never ambiguous. Recursion terminates on a visited set; a
cycle contributes no use not already checked on the way in.
@dg1sbg

dg1sbg commented Aug 3, 2026

Copy link
Copy Markdown
Author

Stacked on #34 — that PR's commit is the first of the two here. Merging #34 first will reduce this to the single feature commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant