Skip to content

Implement cached (§15) - #18

Merged
janstrakowski merged 7 commits into
mainfrom
cached-values
Aug 31, 2026
Merged

Implement cached (§15)#18
janstrakowski merged 7 commits into
mainfrom
cached-values

Conversation

@janstrakowski

@janstrakowski janstrakowski commented Aug 31, 2026

Copy link
Copy Markdown
Owner

cached <expr> evaluates an expression once and reads the answer back on every
later run. §15 had pinned the cache key — the expression treated as a
function, hashed as one — but left > TODO: Where does the cache actually live
open. This closes it.

Rebased onto #17. This branch originally carried its own directory and
closure hashing, built as scaffolding. #17 landed the same ground while it
was in flight, and better — decisively, it handles cyclic values, which #16
made constructible and which the version here would have recursed on forever.
All of that is dropped in favour of #17's; what remains is cached and the
pieces #17 does not have.

The key

main hashes a closure as its body's shape plus its captures, which is exactly
right for a Function value — but #arg/#self are dynamic lookups into the
interpreter's stacks, captured by no closure and invisible to free_names.
Left there, let f func (cached (#arg + 1)) gives f 10 the answer 2 — a
wrong value, not a stale cache. cached mixes in the stack entries the
expression can reach, bounded statically by the largest N written in it
(hash_implicit.odin). There is a regression test.

The layout

In ctx.cache's directory (--cache-dir, else the XDG/per-user default), one
entry per key:

<cache>/sha256-<key>                  a File value, stored as itself
<cache>/sha256-<key>.hb/value.hb      anything else, as HashedBuild text
<cache>/sha256-<key>.hb/sha256-<h>    each File inside it, by content hash

A file stays a file and a directory stays a directory, so what a build produced
is still something you can open, diff or copy out. Anything else is written
as HashedBuild's own value syntax, read back by a literals-only parser rather
than import — a hand-edited entry is a parse failure, not code that runs.
Entries commit by rename, which is also how two runs racing on one key settle
it without locking.

Cyclic values are cacheable. A Table reached more than once is written
node "N" { … } at its first occurrence and ref "N" after that:

node "1" { .name = "alice", .friend = node "2" { .name = "bob", .friend = ref "1" } }

The reader creates each Table before reading its entries — exactly as §10's
evaluation order does — so a definition always precedes its references and
nothing needs patching up afterwards. A restored cycle is bisimulation-equal to
the stored one, which is what §6 requires of it. Labels go on merely shared
Tables too: not needed for correctness, since §6 compares structurally, but it
stops a shared value expanding exponentially on the way out.

Also here

  • Four fs write operationsmkdir/rename/unlink/rmdir, on all three
    backends. Finish hashing: directories, closures, cycles, and ctx.cache #17 added reading; the store needs writing.
  • The executable bit is dropped from §3's directory hash, reversing the
    resolution Finish hashing: directories, closures, cycles, and ctx.cache #17 recorded. Finish hashing: directories, closures, cycles, and ctx.cache #17 cited core.filemode as agreeing; git actually
    does the opposite — it never re-derives the bit on a target that cannot report
    one, because core.fileMode=false carries the mode from the index, so a
    committed 100755 round-trips without ever existing in a Windows working
    tree. Hashing it made one tree two values depending on where it was checked
    out: this repo has four such files, so sha256 loadfile "scripts" answered
    two ways. Remembering the bit instead isn't available to a File — a live
    handle with no index beside it. The cache still restores the bit when copying
    a tree: fidelity in the store, not identity in the language.

Testing

230 tests pass, repeatedly, on Windows; all three targets typecheck.
examples/cached.hb is asserted by the suite like every other example.

Two things I could not run locally, both covered by CI:

  • the WASI artifacts — no wasm-ld on this machine, so the backends are
    typechecked for wasi_wasm32 but not linked;
  • hash_linux_test.odin, whose assertion inverts with the exec-bit change:
    it now sets the bit, confirms it landed, and asserts the digest is unchanged.

🤖 Generated with Claude Code

janstrakowski and others added 5 commits August 31, 2026 14:01
`cached <expr>` evaluates an expression once and reads the answer back on
every later run. §15 had pinned the cache *key* — the expression treated as a
function, hashed as one — but left "where does the cache actually live" as an
open TODO, and two of the digests it rests on did not exist.

**The key.** A closure now hashes as its code (the AST subtree, structurally,
so reformatting or commenting it changes nothing), its captured `ctx`, and the
values of the free names it uses. `#arg`/`#self` are dynamic lookups that no
closure captures, so `cached` mixes in the stack entries the expression can
reach, bounded statically by the largest N written in it — without that,
`let f func (cached (#arg + 1))` answers `f 10` with `f 1`'s result.

**The layout**, resolving §15's TODO. One entry per key, in ctx.cache's
directory:

    <cache>/sha256-<key>                  a File value, stored as itself
    <cache>/sha256-<key>.hb/value.hb      anything else, as HashedBuild text
    <cache>/sha256-<key>.hb/sha256-<h>    each File inside it, by content hash

A file stays a file and a directory stays a directory, so what a build produced
is still something you can open. The text format is HashedBuild's own value
syntax read by a separate literals-only parser, not `import`, so a hand-edited
entry is a parse failure rather than code that runs. Entries commit by rename,
which is also how two runs racing on one key settle it.

**Two digests that had to exist first**, both settled with the language's
owner rather than assumed:

- §3's directory hash, over entries sorted by name. Only Linux can report an
  executable bit; WASI and Windows hash every entry as non-executable, so a
  tree containing an executable hashes differently there. SPEC.md §3 now says
  so, and says why the alternatives were worse. This was the open question
  that kept directory hashing unbuilt.
- `ctx.cache`'s, as a tagged constant rather than its path — it is in every
  key now, and baking the path in would invalidate a cache that was moved.

Six operations join the fs layer for this (listing by descriptor with
no-follow classification, mkdir, rename, unlink, rmdir, and the exec bit),
implemented on all three targets.

LANGUAGE.md gains a Caching section and loses `cached` and all three
unhashable kinds from "what isn't built yet"; examples/cached.hb is asserted
by the suite like every other example.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A directory entry now hashes as name and content, with no permission input on
any target. §3 previously kept a single executable flag — the same reduction
git makes (`100644` vs `100755`) — and hashed it where it could be read.

Only Linux can read it. WASI's filestat carries no permission bits and Windows
has no POSIX execute bit, and neither has a stand-in worth using: an `.exe`
extension is a different question, and what MSYS guesses from a shebang is a
guess. So hashing it made one tree two values depending on where it was
checked out — and not hypothetically. Git sets `core.fileMode=false` on
Windows, so the bit round-trips through the repository without ever existing
in the working tree; the same commit checks out executable on Linux and not on
Windows. This repository has four such files, so `sha256 loadfile "scripts"`
would have answered two ways.

Git's own way out is to *remember* the bit rather than re-derive it, which a
`File` cannot do — it is a handle onto a live directory with no index beside
it. Between a digest that disagrees across targets and one that ignores a bit
two of the three cannot see, this takes the second: two trees differing only
in an executable bit are one value, and the platform a build runs on no longer
changes what its inputs hash to.

The bit is still read and still preserved when `cached` copies a tree, so
caching a build output doesn't silently strip it. That is fidelity in the
store, not identity in the language, and the comments now say so.

Covered by a Linux-only test that sets the bit and asserts the digest is
unchanged — Linux-only because elsewhere fs_set_executable_at is a no-op, and
the test would pass without establishing anything.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`sha256 <function>` and `sha256 <directory>` both work as of the `cached`
work, and LANGUAGE.md described them in prose — but nothing in examples/
exercised either, and the Hashing section's snippet block still only showed a
string, a file and a Table. A feature nobody outside this repo can run is not
finished.

examples/hashing.hb gains three entries, each asserting a property rather than
a digest so they stay meaningful:

  - whitespace and comments are not part of a function's code;
  - the values it captures *are* — the property `cached` rests on, since
    without it one entry would serve every argument;
  - a directory is its own kind of value, not its contents run together.

LANGUAGE.md's snippet block gains the two new forms, and the Function bullet
gains the pair of one-liners above. Every snippet in it was run before it was
written down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#17 landed directory, closure and ctx.cache hashing while this branch was
building the same three as scaffolding for `cached`. Its versions win — they
are merged, they are better factored, and decisively they handle **cyclic
values**, which #16 made constructible and which mine would have recursed on
forever.

Dropped: hash_directory.odin, my hash_function.odin, my hash.odin changes, my
Fs_Entry extension and fs_list_dir_at, my hash tests, and my additions to
examples/hashing.hb (three dedicated hashing examples cover it better).
cache_store.odin now reads directories through main's fs_list_entries_at,
whose getdents walk is better than the /proc/self/fd one it replaces, and
refuses a fifo/socket/device the same way §3's hash does.

Kept, because main has none of it:

  - `cached` itself, rebound to value_digest(v, interp) and Hash_Fail.
  - Four fs write operations — mkdir/rename/unlink/rmdir, on all three
    backends. #17 added reading; the store needs writing.
  - The `#arg`/`#self` coverage, now hash_implicit.odin. main's free_names
    collects identifiers and a uses_ctx flag, but nothing looks at
    Implicit_Name or Hole — they are dynamic lookups no closure captures, so
    `let f func (cached (#arg + 1))` would answer `f 10` with 2.

**Cyclic values are cacheable**, which the format could not previously
express. A Table reached more than once is written `node "N" { … }` at its
first occurrence and `ref "N"` after that; the reader creates each Table
before reading its entries, exactly as §10's evaluation order does, so a
definition always precedes its references and nothing needs patching up. A
restored cycle is bisimulation-equal to the stored one, which is what §6
requires. Labels go on merely shared Tables too — not needed for correctness,
since §6 compares structurally, but it stops a shared value expanding
exponentially on the way out.

230 tests pass; all three targets typecheck.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reverses the resolution #17 recorded earlier the same day. A directory entry
now hashes as name and content, with no permission input on any target.

#17 kept a single executable flag — the same reduction git makes — read where
a target can see one and false elsewhere, and cited `core.filemode` as taking
the same position. Looking at what git actually *does* rather than what it
records reverses that. Git never re-derives the bit on a target that cannot
report one: `core.fileMode=false` on Windows means it carries the mode from
the index, so a committed `100755` round-trips through the repository without
ever existing in the working tree. The same commit is executable on Linux and
not on Windows.

So hashing the bit made one tree two values depending on where it was checked
out — and not hypothetically. This repository has four `100755` files, so
`sha256 loadfile "scripts"` would have answered two ways. Remembering the bit
instead, as git does, is not available to a `File`: it is a handle onto a live
directory with no index beside it. Between a digest that disagrees across
targets and one that ignores a bit two of the three cannot see, §3 now takes
the second.

The bit is still read and still restored when `cached` copies a tree, so
caching a build output doesn't silently strip it — fidelity in the store, not
identity in the language, and the comments say so.

hash_linux_test.odin's assertion inverts: it now sets the bit, confirms it
actually landed (otherwise the check would hold for the wrong reason), and
asserts the digest is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@janstrakowski janstrakowski changed the title Implement cached (§15), and the hashing it rests on Implement cached (§15) Aug 31, 2026
janstrakowski and others added 2 commits August 31, 2026 14:41
Caching a value that reaches itself is a user-visible capability and
LANGUAGE.md describes it, but nothing in examples/ ran one. cached.hb now
caches a self-referential Table and asserts the digest survives the round
trip - which is the whole test, since §6 compares cyclic values by
bisimulation, so a back-edge that came back as an unfolding of the wrong
depth would not compare equal to what was stored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two failures CI caught and this machine cannot, both from the four fs write
operations the cache needs.

**wasi-threads**: `cached.hb` died with "could not publish cache entry
(Access)" under iwasm. preview1 checks a rename against two separate rights —
PATH_RENAME_SOURCE on the source descriptor and PATH_RENAME_TARGET on the
destination's — and DIR_RIGHTS asked for neither, so every commit-by-rename
came back ENOTCAPABLE, which fs.odin folds into a bare .Access.

**playground**: the browser refused to instantiate the module at all —
`LinkError: "path_rename": function import requires a callable`. docs/wasi.js
implements preview1 for the playground, and nothing had needed to remove or
rename anything before, so three imports simply weren't there. Added
path_rename, path_unlink_file and path_remove_directory, plus the FileSystem
`remove`/`rename` they sit on, plus the arity entries a spawned thread
marshals them through. `rename` refuses an existing destination, which is what
makes it the cache's commit rather than an overwrite. ENOTEMPTY (55) joins the
ERRNO table, since an rmdir of a non-empty directory now has a way to say so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@janstrakowski
janstrakowski merged commit 0ee8f58 into main Aug 31, 2026
5 checks passed
@janstrakowski
janstrakowski deleted the cached-values branch August 31, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant