Let a dynamic-extent declaration reach the closure extent analysis - #35
Open
dg1sbg wants to merge 2 commits into
Open
Let a dynamic-extent declaration reach the closure extent analysis#35dg1sbg wants to merge 2 commits into
dg1sbg wants to merge 2 commits into
Conversation
DETERMINE-CLOSURE-EXTENT set the ENCLOSE's extent to :dynamic from inside the loop over the variable's readers, so the mutation happened as soon as the first reader turned out to be a DX call. A later reader that escapes bails out with RETURN-FROM, which does not undo it, and the closure stays marked :dynamic. The backend then builds it with cc_stack_enclose over an alloca in the enclosing frame -- and it escapes anyway. Using it once that frame is gone is a use-after-return; under Clasp it surfaces as EXT:BUS-ERROR. Which reader is seen first decides whether this fires, and that order is unspecified: Set/set.lisp picks the hashset representation, because the #+(or) guard on :listset is always false, so DOSET walks an EQ hash table and the order follows object addresses. Compiling (lambda (x) (let ((g (lambda (y) (+ x y)))) (mapcar g '(1 2 3)) g)) 300 times with Clasp's native compiler stack-allocated the returned closure 154 times. MAPCAR carries the :dx-call attribute, and the RETURNI is the escape. Move the assignment after the loop, where the conclusion it records has actually been established. A variable with no readers now qualifies too, which is correct -- a value that is never read cannot escape; previously the loop body simply never ran. Introduced in 3884e93.
DETERMINE-CLOSURE-EXTENT proved non-escape for exactly one shape: every use of the closure is a call carrying the :DX-CALL attribute, in the function that encloses it. A closure handed to an FLET or LABELS function therefore always went on the heap, even when the callee does nothing with it but call it. A CL:DYNAMIC-EXTENT declaration was parsed into the environment and then dropped -- nothing outside Environment/ and CST-to-AST/ ever read it, and BIR:DYNAMIC-LETI, which exists and is exported for the purpose, is never constructed anywhere. Carry the declaration on the binder instead, which is what BIR:VARIABLE's own docstring says should happen, the declaration constraining the extent of the value bound rather than of the variable. The declaration is permission, not proof. With it the analysis may walk into a statically known callee and check that the matching parameter does not escape either; without it that walk is not attempted, and if it fails the declaration is ignored. A wrong declaration costs the optimisation and never memory safety. Two things fall out of doing that walk properly: - A callee is not only known via BIR:LOCAL-CALL. A front end building BIR from an already compiled representation can leave the call as a plain CALL whose callee is still statically an ENCLOSE, so resolve the callee datum rather than requiring the instruction class. - A parameter is not always bound to a variable, since a front end need not emit a LETI for one that is read once. USE-RETAINS-NOTHING-P handles both shapes. Occupying the callee position of a call is now recognised as a non-escape on its own, needing no declaration: invoking a function cannot retain it. Only required parameters passed exactly once, with nothing but required parameters ahead of them, are followed, so the position-to-parameter mapping is never ambiguous. Recursion terminates on a visited set; a cycle contributes no use not already checked on the way in.
Author
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DETERMINE-CLOSURE-EXTENTproves non-escape for exactly one shape: every use of the closure is a call carrying:DX-CALL, in the function that encloses it. A closure handed to anFLETorLABELSfunction therefore always goes on the heap, even when the callee does nothing with it but call it.Meanwhile a
CL:DYNAMIC-EXTENTdeclaration is parsed into the environment and then dropped — nothing outsideEnvironment/andCST-to-AST/reads it, andBIR:DYNAMIC-LETI, which exists and is exported for exactly this purpose, is never constructed anywhere.This carries the declaration on the binder, which is what
BIR:VARIABLE's own docstring says should happen: the declaration constrains the extent of the value bound rather than of the variable.Permission, not proof
With the declaration, the analysis may walk into a statically known callee and check that the matching parameter does not escape either. Without it, that walk is not attempted. If the walk fails, the declaration is ignored. So a declaration that is wrong costs the optimisation and never memory safety — which seemed the only defensible reading, given that the failure mode of trusting it is heap corruption.
Two generalisations that fall out
BIR:LOCAL-CALL. A front end that builds BIR from an already compiled representation can leave the call as a plainCALLwhose callee is still statically anENCLOSE, so this resolves the callee datum rather than requiring the instruction class.LETIfor one that is read once, in which case the parameter is used directly by the consuming instruction.USE-RETAINS-NOTHING-Phandles both shapes.Both were found the hard way: the analysis silently never fired on the client I was testing against, for each of these reasons in turn.
Independent of the declaration
Occupying the callee position of a call is now recognised as a non-escape on its own, needing no declaration: invoking a function cannot retain it.
Scope limits
Only required parameters passed exactly once, with nothing but required parameters ahead of them, are followed — so the position-to-parameter mapping is never ambiguous. Recursion through mutually recursive callees terminates on a visited set; a cycle contributes no use that was not already checked on the way in.
Verification
Through Clasp. A closure passed to a capturing
FLETcalled from three sites, 100000 calls: 544 → 504 bytes per call, results unchanged. Two deliberately-wrong declarations — a callee that returns the closure, and one that stores it in a global — both correctly forfeit the optimisation and both closures still work afterwards. Full regression suite on both GC variants plus ANSI: no unexpected failures.One caveat I would rather state than leave implicit: the
CST-to-ASTandAbstract-syntax-treechanges are compile-verified only. Clasp does not load those systems — it builds BIR from bytecode — so nothing I ran exercises that path. It is written to mirror how%IGNOREalready rides through the same three places, but it deserves a look from someone whose client uses the CST front end.