From e0538dcf4940cbfde792c31f70c984c62cff1cca Mon Sep 17 00:00:00 2001 From: Jan Strakowski Date: Mon, 31 Aug 2026 14:01:49 +0200 Subject: [PATCH 1/6] =?UTF-8?q?Implement=20`cached`=20(=C2=A715),=20and=20?= =?UTF-8?q?the=20two=20hashes=20it=20needed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `cached ` evaluates an expression once and reads the answer back on every later run. §15 had pinned the cache *key* — the expression treated as a function, hashed as one — but left "where does the cache actually live" as an open TODO, and two of the digests it rests on did not exist. **The key.** A closure now hashes as its code (the AST subtree, structurally, so reformatting or commenting it changes nothing), its captured `ctx`, and the values of the free names it uses. `#arg`/`#self` are dynamic lookups that no closure captures, so `cached` mixes in the stack entries the expression can reach, bounded statically by the largest N written in it — without that, `let f func (cached (#arg + 1))` answers `f 10` with `f 1`'s result. **The layout**, resolving §15's TODO. One entry per key, in ctx.cache's directory: /sha256- a File value, stored as itself /sha256-.hb/value.hb anything else, as HashedBuild text /sha256-.hb/sha256- each File inside it, by content hash A file stays a file and a directory stays a directory, so what a build produced is still something you can open. The text format is HashedBuild's own value syntax read by a separate literals-only parser, not `import`, so a hand-edited entry is a parse failure rather than code that runs. Entries commit by rename, which is also how two runs racing on one key settle it. **Two digests that had to exist first**, both settled with the language's owner rather than assumed: - §3's directory hash, over entries sorted by name. Only Linux can report an executable bit; WASI and Windows hash every entry as non-executable, so a tree containing an executable hashes differently there. SPEC.md §3 now says so, and says why the alternatives were worse. This was the open question that kept directory hashing unbuilt. - `ctx.cache`'s, as a tagged constant rather than its path — it is in every key now, and baking the path in would invalidate a cache that was moved. Six operations join the fs layer for this (listing by descriptor with no-follow classification, mkdir, rename, unlink, rmdir, and the exec bit), implemented on all three targets. LANGUAGE.md gains a Caching section and loses `cached` and all three unhashable kinds from "what isn't built yet"; examples/cached.hb is asserted by the suite like every other example. Co-Authored-By: Claude Opus 5 --- GETTING_STARTED.md | 2 +- LANGUAGE.md | 105 ++++++++-- SPEC.md | 35 +++- examples/README.md | 1 + examples/cached.hb | 46 ++++ src/builtins_fs.odin | 37 +++- src/cache_format.odin | 451 ++++++++++++++++++++++++++++++++++++++++ src/cache_store.odin | 412 ++++++++++++++++++++++++++++++++++++ src/cache_test.odin | 414 ++++++++++++++++++++++++++++++++++++ src/eval.odin | 114 +++++++++- src/examples_test.odin | 13 +- src/fs.odin | 36 +++- src/fs_linux.odin | 84 ++++++++ src/fs_wasi.odin | 76 +++++++ src/fs_windows.odin | 87 ++++++++ src/hash.odin | 80 ++++--- src/hash_directory.odin | 100 +++++++++ src/hash_function.odin | 280 +++++++++++++++++++++++++ src/hash_test.odin | 148 ++++++++++++- src/main.odin | 3 +- src/value.odin | 11 + 21 files changed, 2458 insertions(+), 77 deletions(-) create mode 100644 examples/cached.hb create mode 100644 src/cache_format.odin create mode 100644 src/cache_store.odin create mode 100644 src/cache_test.odin create mode 100644 src/hash_directory.odin create mode 100644 src/hash_function.odin diff --git a/GETTING_STARTED.md b/GETTING_STARTED.md index 7518586..6942f39 100644 --- a/GETTING_STARTED.md +++ b/GETTING_STARTED.md @@ -52,7 +52,7 @@ If a freshly built binary refuses to start with *"An Application Control policy - **`./hb`** (no arguments) - a line-based REPL. Type an expression, then an empty line to evaluate it; `:q` to quit. - **`./hb -a path/to/program.hb`** - print the full AST before evaluating. Works with `-e` too. - **`./hb -i`** - the live terminal editor (needs a real terminal, not a pipe). -- **`./hb --cache-dir ...`** - override where `ctx.cache` writes to (defaults to your XDG cache dir; `%LOCALAPPDATA%\hashedbuild` on Windows). +- **`./hb --cache-dir ...`** - override where `ctx.cache` writes to, and where `cached` keeps its entries (defaults to your XDG cache dir; `%LOCALAPPDATA%\hashedbuild` on Windows). Handy for a throwaway cache: point it somewhere temporary and `cached` starts from nothing. - **`./hb --help`**, **`./hb --version`** - usage and version. ## Try each part of the video diff --git a/LANGUAGE.md b/LANGUAGE.md index 60d8d85..f06e06b 100644 --- a/LANGUAGE.md +++ b/LANGUAGE.md @@ -341,14 +341,86 @@ same value when their bytes match, however they were reached. (loadfile "a.txt") == (loadfile "copy-of-a.txt") // true, if the bytes match ``` -Three kinds of value have no digest yet, and say so rather than inventing one: -a **directory** `File` (§3 hashes one over its entries including each file's -executable bit, which only the Linux target can report — see below), a -`Function`, and -`ctx.cache`. Hashing one is a fatal failure like any other (§8). +**Every kind of value hashes.** The three that used to have no digest all got +one when `cached` was built, since `cached` needs them: + +- A **directory** `File` hashes over its entries — each name, each file's + content and executable bit, each subdirectory's own hash, and each symlink's + target string, unresolved (§3). Sorted by name, so readdir order doesn't + matter. One caveat, and it is a real one: **only Linux can report an + executable bit.** WASI's `filestat` has no permission bits and Windows has no + POSIX execute bit, so both hash every entry as non-executable — a tree + containing an executable therefore hashes differently there than on Linux. +- A **`Function`** hashes as its code (the shape of the expression, so + reformatting it or writing a comment inside changes nothing), its captured + `ctx`, and the values of the names it uses. +- **`ctx.cache`** hashes as a constant. It is write-only and has no identity to + distinguish one from another, and making it a constant is what keeps a cache + directory valid after it is moved or copied — see below. + +Hashing a directory reads the whole tree, so `sha256 ` and `==` +between two directory `File`s are filesystem walks, not cheap comparisons. → `examples/hashing.hb` (§3, §6, §15) +## Caching + +`cached` takes one trailing expression, like `sha256`, and evaluates it once: +every later run reads the answer back instead of computing it. + +```hashedbuild +cached (6 * 7) // => 42, and again on every later run +cached (loadfile "pkg.tar.gz") // the bytes as they were the first time +``` + +**The key is the expression treated as a function**, hashed with the one hash +system above (§15). Three things go into it, and it is worth knowing which: + +- **the code**, structurally — so reformatting an expression or writing a + comment inside it does not throw its entry away; +- **the `ctx` it runs under** (§9), whole; +- **the values it reads** — the names it uses, and anything it reaches through + `#arg`/`#self`. So `let bump func (cached (#arg + 1))` gets one entry per + argument, not one entry. + +What is deliberately *not* in the key is anything the expression goes and reads +at run time. That is the point of a cache, and its one sharp edge: cache +`loadfile "pkg.tar.gz"` and you keep getting the bytes from the first run, +however the file changes afterwards. + +**Where entries live.** In `ctx.cache`'s directory — `--cache-dir `, else +`$XDG_CACHE_HOME/hashedbuild`, else the per-user default — one entry per key, +named `sha256-`: + +``` +/sha256- a File value, stored as itself +/sha256-.hb/ anything else +/sha256-.hb/value.hb the value, as HashedBuild text +/sha256-.hb/sha256- each File inside it, by content hash +``` + +A `File` value stays a file and a directory value stays a directory, so what a +build produced is still something you can open, `diff` or copy out. Anything +else is written as text you can read; the `File`s it holds cannot go in text, +so they are stored beside it and referred to by name. Entries are built under a +temporary name and renamed into place, so an interrupted run leaves a `.tmp` +rather than a half-written entry, and two runs racing on one key settle it +without locking anything. + +**What it refuses.** `cached` needs `ctx.permissions.io`, since it reads and +writes files, and it needs a `ctx` that still carries `.cache`. A value holding +a `Function`, `ctx.cache`, or an un-awaited `async` handle cannot be written +down and read back as itself, so caching one fails rather than storing +something that would come back different. All of these are fatal failures like +any other (§8). + +`async` is positional, and the two placements mean different things: +`async cached ` makes the cache lookup itself asynchronous, while +`cached async ` runs the expression on a thread with the caching wrapper +around it synchronous. + +→ `examples/cached.hb` (§15) + ## Context and permissions `ctx` is the ambient context. The filesystem builtins check @@ -368,7 +440,10 @@ is the entire point. `ctx.cache` is a write-only, content-addressed store: `createfile { .dir = ctx.cache, .content = … }` writes under the content's own hash, deduplicating -across runs, and returns the `File` it wrote. +across runs, and returns the `File` it wrote. Its directory is also where +`cached` keeps its entries (see "Caching"); the two are told apart by their +names, `sha256_` for a blob written this way and `sha256-` for a +cache entry. → `examples/context-permissions.hb`, `examples/option-picker.hb` (§9, §16) @@ -401,17 +476,13 @@ uncatchable, but the work already in flight finishes first. ## What isn't built yet Parsed, specified, and rejected by the evaluator with "not implemented": -`import` and `cached`. - -Partly built: **hashing**. `sha256` works for every value except a directory -`File`, a `Function`, and `ctx.cache`. The directory case is the interesting -one — `SPEC.md` §3 defines a directory's hash over its entries including each -file's executable bit, and two of the three targets cannot report one: WASI's -`filestat` has no permission bits at all, and Windows has no POSIX exec bit. -So there is no way to compute the specified digest everywhere the interpreter -runs. Building it means first deciding what a directory hashes as somewhere -that cannot see an exec bit. `Function` is unbuilt because §15 needs it for `cached` but never -says how a closure is encoded. +`import`. + +Hashing is complete: every kind of value has a digest, including the three +(directory `File`, `Function`, `ctx.cache`) that used to fail by name. The one +thing to know about it is not a gap but a difference between targets — the +executable bit in a directory's hash, which only Linux can report. See +"Hashing" above. Also absent: `true`/`false` literals, loops of any kind (recursion is the only repetition there is — see above), a `Bytes`-returning counterpart to diff --git a/SPEC.md b/SPEC.md index 7f5d467..f0af27d 100644 --- a/SPEC.md +++ b/SPEC.md @@ -53,6 +53,8 @@ A condition (of `then`, `and`, `or`, `is`) can itself be, or contain, an async e ``` A symlink entry hashes the link itself (its target path string) rather than resolving through it — consistent with symlink handling being a property of the containing directory, not a standalone `File` value in its own right. + **The executable bit where it cannot be read (resolved 2026-08-31).** Only Linux reports one: WASI's `filestat` carries no permission bits at all, and Windows has no POSIX execute bit. On both, every entry hashes as **non-executable** — a truthful report of what those filesystems say, rather than a refusal to hash or a bit invented from something else. The consequence is stated here rather than buried: a tree containing an executable file hashes differently on Linux than on WASI or Windows, so a §15 cache directory carried between those platforms misses on any entry whose value is such a directory. Both alternatives were worse — dropping the bit from the hash makes two genuinely different trees one value on the target where the difference is real, and implementing the hash only on Linux would take directory values and `cached` away from the playground and the Windows target outright. This was the open question that kept the directory hash unimplemented; it is settled, and the hash is built. + **Display (resolved 2026-08-27).** A `File` displays **the path it was reached by**, made absolute and cleaned of `.`/`..` segments — recorded when the value is constructed, not resolved through the filesystem afterwards. Two consequences worth stating: a symlinked route displays as the route taken rather than the target it resolves to, and a file renamed after it was loaded still displays the path it was loaded from. This is what makes the rule implementable off Linux at all — WASI (and the portable `*at()` family) offer no way to turn an open descriptor back into a path. Printing/displaying a `File` — in the REPL, the live editor's result pane, or anywhere else a value gets shown to a human — shows its actual filesystem path. This holds for **every** `File` value, regardless of how it was obtained (`loadfile`, `createfile`, `symlink`'s containing directory, etc.), not just the ones `createfile` writes into `ctx.cache` (§16) — that case is simply the one where a path is otherwise unreachable, so it's the one worth calling out explicitly there. The only exception is `ctx.cache` itself: a distinct, "magic" pseudo-directory type (§16) that is *not* a `File` and has no path of its own to show. Path stays display-only either way — there is still no builtin that lets HashedBuild source read a `File`'s path back out as a `Utf8` value; this is purely about what a human sees when a value is printed, not a new capability for programs. @@ -247,7 +249,7 @@ Added 2026-08-26, alongside `ctx`/`withctx` above and the filesystem builtins ( - **`ctx.permissions`** is a `Table` used as a *set*: it conventionally holds only `nothing` as every value, and a permission is granted iff its key is **present** (not iff its value is "truthy," since `Nothing` has no such notion) — e.g. `ctx.permissions.io` being present at all, regardless of its (always-`nothing`) value, means I/O is allowed. This works because, unlike Lua, a HashedBuild `Table` (§5/§6) genuinely distinguishes "key absent" from "key present with value `nothing`" — the exact distinction a set-of-flags idiom needs, and Lua's conflation of nil-valued and absent keys can't express. - **The root context** — active at the very start of a program, before any `withctx` — starts with `{ .permissions = { .io = nothing }, .cache = }`: I/O is allowed by default. `withctx` is how you *narrow* permissions around a sub-computation (e.g. before calling `import`ed code, §13), not how you grant them from nothing. Since `withctx` replaces the context wholesale (§7), a program that narrows permissions via a hand-built `Table` rather than `ctx concat {...}` loses `.cache` too unless it explicitly carries it over. - **Builtins read `ctx` live, not captured — the one deliberate exception to the closure-capture rule above.** The capture rule protects a function *from* a caller trying to grant it more authority than it was made with; a builtin like `loadfile` (§16) needs the opposite property — it must see whatever `ctx` is *actually* active at its call site, so that wrapping a call in `... withctx (ctx concat { .permissions = empty })` genuinely denies it from the outside. If builtins captured `ctx` at (interpreter-startup) creation time instead, `withctx` could never restrict them at all. -- **`ctx.cache`** (added 2026-08-27) is its own type, distinct from `File` — see §16 for the full write-up. It's "accepted as a directory" (usable as `createfile`'s `.dir`) without actually being one: it can't be read from, traversed, or passed to `loadfile`/`symlink`/`readlink`, only written to via `createfile`. +- **`ctx.cache`** (added 2026-08-27) is its own type, distinct from `File` — see §16 for the full write-up. It's "accepted as a directory" (usable as `createfile`'s `.dir`) without actually being one: it can't be read from, traversed, or passed to `loadfile`/`symlink`/`readlink`, only written to via `createfile`. **It does hash** (resolved 2026-08-31), as a domain-separated constant over an empty payload: it is write-only, unnamed and unlistable, so there is nothing about one cache that distinguishes it from another. This became necessary rather than optional once §15's cache key was defined to include the whole `ctx`, which contains it. Hashing it as its *directory path* was rejected for a concrete reason: the path would then be baked into every key, so moving or copying a cache directory would invalidate everything in it. ## 10. Name scopes / bindings @@ -313,7 +315,32 @@ Two builtins operationalizing §6's "every value is hashable" claim. Like `impor **`cached`'s mechanism (resolved 2026-08-26):** the cached expression is treated *as a function* and hashed as one — the cache key is the hash (per the one system above) of that function representation, not a hash of its resolved output value. `cached` is not inherently async by itself; async-ness is controlled explicitly by *where* `async` (§2) is placed: `async cached ` makes the cache lookup/store itself asynchronous, while `cached async ` instead makes the underlying expression's own evaluation asynchronous, with the caching wrapper around it synchronous. -> TODO: Where does the cache actually live (on-disk location, process-local vs. shared/distributed) — not addressed by this round's resolution of the cache-*key* mechanism. +**What a function hashes as (resolved 2026-08-31).** §6 said every value is hashable and this section relied on it, but the encoding of a closure was never given. It is three things mixed together: + +- **the code** — the expression's syntax tree, encoded structurally: node kinds, plus the source text of leaves only. Reformatting an expression, or writing a comment inside it, therefore does not change the key. How a literal is *spelled* does (`1_000` and `1000` are different text), which splits an entry rather than merging two, and is left that way deliberately. +- **the captured `ctx`** (§9), whole. +- **the values of the free names the code uses**, each paired with its name. Collection is deliberately conservative: a name the expression shadows locally is still included if a binding of that name exists outside. Over-collecting only splits one entry into two; under-collecting would return a wrong value. + +A native builtin hashes as the name it is bound to plus whatever it captured — a proc address is not stable between runs. A function that reaches itself (`let rec`) hashes its second occurrence as a back-reference, counted from the innermost enclosing one, so recursion terminates and two identically-written recursive functions still agree. + +**`#arg`/`#self` are covered separately (resolved 2026-08-31).** They are dynamic lookups into their own stacks (§9) and are captured by no closure, so the function hash above cannot see them — which would make `let f func (cached (#arg + 1))` one entry for every argument, answering `f 10` with `f 1`'s result. `cached` therefore mixes in the stack entries the expression can reach, on top of the function hash. How far it can reach is bounded statically by the largest `N` written anywhere in it (a bare `#arg` or an omitted operand counting as 1); that bound holds through functions the expression itself calls, since entering one pushes a frame. An expression mentioning no implicit name is unaffected. + +**Where the cache lives (resolved 2026-08-31, replacing this section's TODO).** On disk, in `ctx.cache`'s directory (§16 — `--cache-dir`, else `$XDG_CACHE_HOME/hashedbuild`, else the per-user fallback). Process-local in the sense that nothing coordinates between machines; shared in the sense that the directory is an ordinary one, and two processes on it are safe. One entry per key, in one of two shapes: + +``` +/sha256- the value is a File — stored as itself +/sha256-.hb/ anything else +/sha256-.hb/value.hb the value, written as HashedBuild text +/sha256-.hb/sha256- each File inside that value +``` + +- **A `File` value is stored as a file, a directory value as a directory.** Not wrapped, not encoded: what a build produced stays something a person can open, `diff`, or copy out, which is most of the point of a content-addressed store. A directory is copied faithfully enough to hash as the original did (§3) — names, contents, the executable bit where the target has one, and symlink targets stored without being followed. +- **Anything else is written as text**, in a subset of HashedBuild's own syntax: literals, tables, and two names the language has no literal for (`true`/`false`, and `bytes "…"`). Since text cannot hold a file, each `File` inside the value is written out beside it, named by its own content hash (§3), and referred to from the text as `file "…"` or `dir "…"` — systematically, however deeply nested. Reading it back is a separate reader that accepts literals and nothing else, not `import`: a hand-edited entry is a parse failure rather than code that runs. +- **`` is the cache key**, base64url without padding, since a lookup has nothing else to go on. The `-` separator distinguishes these from the `sha256_` blobs `createfile { .dir = ctx.cache }` writes into the same directory (§16). +- **Entries are committed by rename.** Each is built under a temporary name and renamed into place, so an interrupted run leaves a stray temporary rather than a truncated entry that a later run would read as a hit. The rename is also how two runs racing on one key settle it — the loser removes its temporary and reads the winner's entry, which holds the same value, since the key is the same. Nothing is overwritten and nothing is locked. +- **A store reads its own entry back** rather than returning the value it just computed, so the first run and every later one return the same value — `File`s included, displaying the cache's paths (§3) rather than wherever that particular run built them. + +**Gating and failure.** `cached` requires `ctx.permissions.io` (§9), like §16's builtins and for the same reason: it reads and writes files. It also requires a `ctx` that still carries `.cache`, which a hand-built one need not. A value holding a `Function`, `ctx.cache`, or an un-awaited `async` handle has no written form and cannot be cached. A present-but-unreadable entry is a failure, deliberately rather than a miss — silently recomputing over a corrupt cache would hide the corruption for as long as the cache lived. All of these are fatal (§8). ## 16. Filesystem builtins @@ -338,7 +365,7 @@ The four filesystem operations below are gated by [`ctx.permissions.io`](#9-impl Added 2026-08-27. A write-only, content-addressed blob store, its own type (distinct from `File`) even though `createfile`'s `.dir` accepts it exactly like a directory handle — reading, traversal, and `loadfile`/`symlink`/`readlink` all refuse it, since there's no meaningful name to look anything up by (see below). Gated by `io` like the other filesystem operations. - **Location.** Resolved once per program run, in order: the CLI's `--cache-dir ` if given; otherwise `$XDG_CACHE_HOME/hashedbuild`; otherwise `$HOME/.cache/hashedbuild`. The directory (and any missing ancestor) is created lazily, on the first actual write — a program that never touches `ctx.cache` never creates it. -- **Naming.** Every entry is stored under `sha256_` — the name *is* the content's own hash, computed by the write itself. This is why "names of the children don't matter": whatever the caller might otherwise think to call an entry is irrelevant, since the store assigns the name, not the caller. +- **Naming.** Every entry is stored under `sha256_` — the name *is* the content's own hash, computed by the write itself. This is why "names of the children don't matter": whatever the caller might otherwise think to call an entry is irrelevant, since the store assigns the name, not the caller. The underscore is load-bearing as of 2026-08-31: §15's `cached` keeps its entries in the same directory, named `sha256-` with a hyphen, and the separator is what tells a blob written here from a cache entry written there. - **Writing.** `createfile { .dir = ctx.cache, .content = }` — note there's no `.path`; it wouldn't mean anything, since the name isn't caller-chosen. Writing content whose hash already has an entry on disk (from this run or an earlier one) **dedupes**: the existing entry is reused as-is, silently, rather than failing like an ordinary exclusive `createfile` or writing a redundant duplicate — this is what makes it a cache across runs, not just a one-shot content dump. - **The returned `File` is real, and displays like any other.** `createfile`'s result here is an ordinary `File` (§3, not the `ctx.cache` type itself) — per §3's general display rule, printing it (e.g. in the REPL or the live editor's result pane) shows its actual absolute path on disk, same as any non-magic `File`. But there's still no builtin that lets HashedBuild source read that path back out as a `Utf8` value. The program can hold the handle and pass it around, but it never learns *where* its data physically landed; only a human inspecting the program's output does. -- **Not searchable.** There's no `loadfile`/`symlink`/`readlink` counterpart for the cache - `.dir = ctx.cache` is only ever accepted by `createfile`. A program can't ask "is this content already cached?" directly; it can only write and let the store's own dedup decide. +- **Not searchable.** There's no `loadfile`/`symlink`/`readlink` counterpart for the cache - `.dir = ctx.cache` is only ever accepted by `createfile`. A program can't ask "is this content already cached?" directly; it can only write and let the store's own dedup decide. §15's `cached` does read entries back out of the same directory, but not through `ctx.cache`: it is its own syntax with its own keys, and shares nothing with this type beyond the directory the two write into. diff --git a/examples/README.md b/examples/README.md index 6f14431..2a6e86b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -56,6 +56,7 @@ prose around each of these. | `files-sandboxed.hb` | Directory handles, contained sub-paths, `filetext`, path display | | `files-symlink.hb` | `readlink` — and why symlinks aren't values of their own | | `hashing.hb` | `sha256`, and the content identity that makes two Files one value | +| `cached.hb` | `cached` — evaluate once, read the answer back on every later run | | `option-picker.hb` | A real program: read, branch, write into `ctx.cache` | | `context-permissions.hb` | `ctx`, `chctx chperm`, `withctx` — capability narrowing | | `async-basics.hb` | Two reads in flight at once, awaited implicitly | diff --git a/examples/cached.hb b/examples/cached.hb new file mode 100644 index 0000000..68918fd --- /dev/null +++ b/examples/cached.hb @@ -0,0 +1,46 @@ +// `cached` (SPEC.md §15): evaluate an expression once, and read the answer +// back on every later run. Like `sha256`, `func` and `async`, it is a bare +// keyword prefix taking one trailing expression - no parentheses of its own. +// +// The cache key is the expression *treated as a function*, hashed as one - so +// two things are part of it besides the code: the `ctx` it runs under, and the +// values of the names it uses. `bump 1` and `bump 10` below are therefore two +// entries and not one, which is what keeps the second call from being answered +// with the first call's result. +// +// What is *not* part of the key is anything the expression goes and reads at +// run time. That is the point of a cache and also its one sharp edge: cache +// `loadfile "pkg.tar.gz"` and you get the bytes as they were the first time, +// however the file changes afterwards. +// +// Entries live in `ctx.cache`'s directory (`--cache-dir`, else the per-user +// default), one per key, named `sha256-`. A `File` value is stored as an +// ordinary file and a directory value as an ordinary directory, so what a +// build produced is still something you can open; anything else is written as +// HashedBuild text in `sha256-.hb/value.hb`, with any `File` it holds +// stored beside it and referred to by name. +// +// Evaluates to +// { answer: 42, asking_again_agrees: true, per_argument: { small: 2, large: 11 }, +// file_survives_the_round_trip: true }. + +let answer cached (6 * 7); + +// The second ask evaluates nothing: it reads back what the first one wrote. +let asking_again_agrees ((cached (6 * 7)) == 42); + +// One entry per captured value, not one per expression. +let bump func (cached (#arg + 1)); + +// A cached `File` comes back as a `File`, and as the *same* value - a `File` +// is its content (§3), so this holds even though the copy in the cache is at +// a different path than the one it was read from. +let file_survives_the_round_trip + ((sha256 cached (loadfile "optiona.txt")) == (sha256 loadfile "optiona.txt")); + +{ + .answer = answer, + .asking_again_agrees = asking_again_agrees, + .per_argument = { .small = bump 1, .large = bump 10 }, + .file_survives_the_round_trip = file_survives_the_round_trip, +} diff --git a/src/builtins_fs.odin b/src/builtins_fs.odin index 0be6367..58ae26f 100644 --- a/src/builtins_fs.odin +++ b/src/builtins_fs.odin @@ -67,10 +67,11 @@ resolve_cache_dir :: proc(override: string) -> string { } @(private = "file") -new_native_function :: proc(fn: Native_Fn, closure: Value = nil) -> Value { +new_native_function :: proc(name: string, fn: Native_Fn, closure: Value = nil) -> Value { f := new(Function_Value) f.native = fn f.native_closure = closure + f.native_name = name // a native hashes as its name, not its address - see Function_Value return f } @@ -78,16 +79,18 @@ new_native_function :: proc(fn: Native_Fn, closure: Value = nil) -> Value { // the filesystem builtins, pre-bound by name (§16). make_global_env :: proc() -> ^Env { env := env_make_child(nil) - env_bind(env, "loadfile", new_native_function(builtin_loadfile)) - env_bind(env, "createfile", new_native_function(builtin_createfile)) - env_bind(env, "symlink", new_native_function(builtin_symlink)) - env_bind(env, "readlink", new_native_function(builtin_readlink)) - env_bind(env, "chperm", new_native_function(builtin_chperm)) - env_bind(env, "filetext", new_native_function(builtin_filetext)) + env_bind(env, "loadfile", new_native_function("loadfile", builtin_loadfile)) + env_bind(env, "createfile", new_native_function("createfile", builtin_createfile)) + env_bind(env, "symlink", new_native_function("symlink", builtin_symlink)) + env_bind(env, "readlink", new_native_function("readlink", builtin_readlink)) + env_bind(env, "chperm", new_native_function("chperm", builtin_chperm)) + env_bind(env, "filetext", new_native_function("filetext", builtin_filetext)) return env } -@(private = "file") +// Also read by eval.odin, where `cached` (§15) gates on it: the cache is a +// directory on disk, so reading or writing an entry is as much an I/O +// operation as §16's builtins are. ctx_allows_io :: proc(interp: ^Interpreter) -> bool { t, is_table := interp.current_ctx.(^Table_Value) if !is_table do return false @@ -516,8 +519,16 @@ cache_entry_name :: proc(content: []u8) -> string { // Opens (creating if necessary) the cache's backing directory, the first // time it's actually needed - not at program start, so a program that never -// touches the cache never creates it. -@(private = "file") +// touches the cache never creates it. §15's `cached` shares the directory and +// so calls this too (cache_store.odin), with the same laziness. +// +// Unsynchronised, and safe to leave that way: two `async` branches reaching +// here at once both open the directory and one of the two descriptors is +// dropped on the floor, which costs a descriptor and nothing else - both are +// valid, and both name the same directory. A mutex would be the wrong shape +// anyway, since the store below is already written to be safe against *other +// processes* on the same directory (see cache_store.odin on committing by +// rename), which is the harder case and covers this one. ensure_cache_dir_open :: proc(cache: ^Cache_Value) -> Fs_Error { if cache.opened do return .None fs_make_dirs(cache.dir_path) @@ -627,7 +638,11 @@ builtin_chperm :: proc(interp: ^Interpreter, _: Value, arg: Value) -> (Value, bo closure := new(Table_Value) append(&closure.entries, Table_Entry_Value{key = "name", value = name_str}) append(&closure.entries, Table_Entry_Value{key = "enabled", value = enabled_bool}) - return new_native_function(apply_chperm, closure), true + // The name is what this function hashes as (§15, Function_Value), so it has + // to distinguish the ctx-changer chperm returns from chperm itself; its + // captured {name, enabled} closure hashes alongside it, which is what makes + // two differently-configured ctx-changers different values. + return new_native_function("chperm.apply", apply_chperm, closure), true } // The actual oldctx -> newctx function chperm returns: a copy of `old_ctx` diff --git a/src/cache_format.odin b/src/cache_format.odin new file mode 100644 index 0000000..1e45bd2 --- /dev/null +++ b/src/cache_format.odin @@ -0,0 +1,451 @@ +package hashedbuild + +import "core:fmt" +import "core:math" +import "core:strconv" +import "core:strings" + +// The text a `cached` entry's `value.hb` holds, and the reader that turns it +// back into a Value. This is the "subset of HB" half of §15's on-disk layout - +// see cache_store.odin for the layout itself. +// +// **It is written to be read.** A cache directory is something a person opens +// to see what a build produced, so an entry says what it is in the language +// the program was written in: +// +// { .name = "libz", .version = 3, .built = true, .src = file "sha256-Ab3_", .out = dir "sha256-9xQ1" } +// +// **A separate reader, not `import`.** Everything above is HashedBuild syntax +// as it stands - `true`, `bytes`, `file` and `dir` are ordinary identifiers, +// and `file "sha256-Ab3_"` is an ordinary call - but none of those names are +// *bound* to anything, so HashedBuild's own evaluator could not read this back +// even though its parser can read the shape. Reading it here instead is also +// what keeps a cache entry from being able to run anything: this parser +// accepts literals and nothing else, so a tampered-with entry is a parse +// failure rather than code that executes. +// +// The one spelling that goes beyond HashedBuild's own grammar is `\xNN` inside +// a string, for bytes that have no other escape (HashedBuild's lexer knows +// `\n`, `\t`, `\r`, `\"` and `\\`, and stops a string at a newline). Utf8 +// values can hold such bytes, so the format needs a way to write them. +// +// Round-tripping is the whole contract: a value written here and read back has +// to hash identically (hash.odin), or `cached` would return something that is +// not the value it stored. Floats are the only case where that takes care - +// see write_float. + +// ---- writing ---------------------------------------------------------------- + +// `file_names` maps each File in the value to the entry name it was stored +// under, which cache_store.odin fills in as it writes them out. A File that +// isn't in the map cannot be written, and says so. +cache_format_write :: proc(v: Value, file_names: map[^File_Value]string) -> (text: string, ok: bool, why: string) { + b: strings.Builder + strings.builder_init(&b) + if why = write_value(&b, v, file_names); why != "" { + strings.builder_destroy(&b) + return "", false, why + } + return strings.to_string(b), true, "" +} + +// Returns "" on success, or a sentence naming what it could not write. The +// reason travels back rather than a bare false because "this value cannot be +// cached" is not actionable on its own - which part of it, and why, is. +@(private = "file") +write_value :: proc(b: ^strings.Builder, v: Value, file_names: map[^File_Value]string) -> string { + switch av in v { + case Nothing_Value: + strings.write_string(b, "nothing") + + case bool: + strings.write_string(b, av ? "true" : "false") + + case i64: + strings.write_i64(b, av, 10) + + case f64: + write_float(b, av) + + case string: + write_quoted(b, transmute([]u8)av) + + case []u8: + strings.write_string(b, "bytes ") + write_quoted(b, av) + + case ^Table_Value: + strings.write_string(b, "{") + for entry, i in av.entries { + if i > 0 do strings.write_string(b, ",") + strings.write_string(b, " ") + // `.name = v` is the readable form, and exact: in HashedBuild it means + // the literal Utf8 key "name". Anything else needs the general + // `[key] = v` form, since a key can be any value at all. + if key, is_str := entry.key.(string); is_str && is_identifier_shaped(key) { + strings.write_string(b, ".") + strings.write_string(b, key) + } else { + strings.write_string(b, "[") + if why := write_value(b, entry.key, file_names); why != "" do return why + strings.write_string(b, "]") + } + strings.write_string(b, " = ") + if why := write_value(b, entry.value, file_names); why != "" do return why + } + strings.write_string(b, len(av.entries) > 0 ? " }" : "}") + + case ^File_Value: + name, found := file_names[av] + if !found { + // The caller writes every File out before asking for the text, so this + // is a bug here rather than anything the program did. + return fmt.tprintf("a %s in the value was not written to the cache first", + av.kind == .Directory ? "directory" : "file") + } + strings.write_string(b, av.kind == .Directory ? "dir " : "file ") + write_quoted(b, transmute([]u8)name) + + // The three that have no written form. A closure's meaning is its + // environment, ctx.cache has no content of its own, and an un-awaited handle + // is a running thread - none of them is a thing that can be written down and + // read back as itself. + case ^Function_Value: + return "a Function cannot be cached - a closure's meaning is its environment" + case ^Cache_Value: + return "ctx.cache cannot be cached - it is write-only and has no content" + case ^Async_Handle: + return "an un-awaited async handle cannot be cached" + } + return "" +} + +// Shortest decimal that reads back as the same f64, checked rather than +// assumed: if the short form doesn't round-trip, 17 significant digits always +// does. A float that came back a hair different would hash differently +// (hash.odin encodes the IEEE bits), so `cached` would hand back a value that +// isn't the one it stored. +// +// The result is also shaped as a HashedBuild Float literal, which is stricter +// than strconv's output: a digit is required on both sides of the `.` (§3), so +// a mantissa without one gets ".0" appended, and an exponent's redundant "+" +// goes. `inf`/`-inf`/`nan` are spelled as the bare names HashedBuild has no +// literal for - the reader below knows them; nothing else does. +@(private = "file") +write_float :: proc(b: ^strings.Builder, f: f64) { + switch { + case f != f: + strings.write_string(b, "nan") + return + case math.is_inf(f, 1): + strings.write_string(b, "inf") + return + case math.is_inf(f, -1): + strings.write_string(b, "-inf") + return + } + + buf: [40]u8 + text := strconv.write_float(buf[:], f, 'g', -1, 64) + if parsed, ok := strconv.parse_f64(text); !ok || parsed != f { + buf2: [40]u8 + text = strconv.write_float(buf2[:], f, 'g', 17, 64) + write_float_literal(b, text) + return + } + write_float_literal(b, text) +} + +@(private = "file") +write_float_literal :: proc(b: ^strings.Builder, text: string) { + // strconv writes a leading '+' on a positive number here; HashedBuild's + // grammar has no unary '+', so it goes. + t := text + if len(t) > 0 && t[0] == '+' do t = t[1:] + + mantissa := t + exponent := "" + if e := strings.index_any(t, "eE"); e >= 0 { + mantissa = t[:e] + exponent = t[e + 1:] + if len(exponent) > 0 && exponent[0] == '+' do exponent = exponent[1:] + } + + strings.write_string(b, mantissa) + if !strings.contains(mantissa, ".") do strings.write_string(b, ".0") + if exponent != "" { + strings.write_string(b, "e") + strings.write_string(b, exponent) + } +} + +@(private = "file") +write_quoted :: proc(b: ^strings.Builder, data: []u8) { + strings.write_byte(b, '"') + for c in data { + switch c { + case '"': strings.write_string(b, "\\\"") + case '\\': strings.write_string(b, "\\\\") + case '\n': strings.write_string(b, "\\n") + case '\r': strings.write_string(b, "\\r") + case '\t': strings.write_string(b, "\\t") + case: + // Anything else printable goes through as itself, so a path or a name + // reads as a path or a name. The rest - control bytes, and NUL, which + // would end the string as far as any C-shaped reader is concerned - gets + // the one escape this format has that HashedBuild's does not. + if c < 0x20 || c == 0x7f { + hex := HEX + strings.write_string(b, "\\x") + strings.write_byte(b, hex[c >> 4]) + strings.write_byte(b, hex[c & 0xf]) + } else { + strings.write_byte(b, c) + } + } + } + strings.write_byte(b, '"') +} + +@(private = "file") +HEX :: "0123456789abcdef" + +// Whether a Utf8 key can be written as `.name`. Deliberately ASCII-only and +// conservative - a key that doesn't qualify still round-trips perfectly +// through the `[key] = value` form, so the only cost of saying no is a +// slightly noisier entry. +@(private = "file") +is_identifier_shaped :: proc(s: string) -> bool { + if len(s) == 0 do return false + for i in 0 ..< len(s) { + c := s[i] + is_alpha := (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '_' + is_digit := c >= '0' && c <= '9' + if !is_alpha && !(is_digit && i > 0) do return false + } + return true +} + +// ---- reading ---------------------------------------------------------------- + +// A `file "..."` / `dir "..."` reference, resolved by the caller: the parser +// knows the entry name, cache_store.odin knows the directory it sits in. +Cache_File_Ref :: proc(entry_name: string, is_dir: bool, userdata: rawptr) -> (Value, bool) + +@(private = "file") +Reader :: struct { + src: string, + pos: int, + resolve: Cache_File_Ref, + userdata: rawptr, +} + +// Parses one value and requires the text to end there. Every failure is a +// plain false: an unreadable entry is a corrupt or hand-edited cache, and the +// caller's answer to that is to treat it as a miss or as a fatal read failure, +// not to try to make sense of half of it. +cache_format_read :: proc(src: string, resolve: Cache_File_Ref, userdata: rawptr) -> (Value, bool) { + r := Reader{src = src, resolve = resolve, userdata = userdata} + v, ok := read_value(&r) + if !ok do return nil, false + skip_space(&r) + if r.pos != len(r.src) do return nil, false + return v, true +} + +@(private = "file") +skip_space :: proc(r: ^Reader) { + for r.pos < len(r.src) { + switch r.src[r.pos] { + case ' ', '\t', '\r', '\n': r.pos += 1 + case: return + } + } +} + +@(private = "file") +accept :: proc(r: ^Reader, lit: string) -> bool { + skip_space(r) + if strings.has_prefix(r.src[r.pos:], lit) { + r.pos += len(lit) + return true + } + return false +} + +// A bare word, for the handful of keyword-shaped values (`nothing`, `true`, +// `bytes`, `dir`, ...). Matched as a whole word so that `nothingness` is not +// read as `nothing` followed by junk. +@(private = "file") +accept_word :: proc(r: ^Reader, word: string) -> bool { + skip_space(r) + rest := r.src[r.pos:] + if !strings.has_prefix(rest, word) do return false + if len(rest) > len(word) { + c := rest[len(word)] + if is_identifier_shaped(string([]u8{c})) do return false + } + r.pos += len(word) + return true +} + +@(private = "file") +read_value :: proc(r: ^Reader) -> (Value, bool) { + skip_space(r) + if r.pos >= len(r.src) do return nil, false + + switch { + case accept_word(r, "nothing"): return Nothing_Value{}, true + case accept_word(r, "true"): return true, true + case accept_word(r, "false"): return false, true + case accept_word(r, "nan"): return math.nan_f64(), true + case accept_word(r, "inf"): return math.inf_f64(1), true + + case accept_word(r, "bytes"): + s, ok := read_string(r) + if !ok do return nil, false + return transmute([]u8)s, true + + case accept_word(r, "file"), accept_word(r, "dir"): + // accept_word already consumed it; which one it was is recoverable from + // the byte just before the current position. + is_dir := r.src[r.pos - 1] == 'r' + name, ok := read_string(r) + if !ok || r.resolve == nil do return nil, false + return r.resolve(name, is_dir, r.userdata) + + case r.src[r.pos] == '"': + s, ok := read_string(r) + if !ok do return nil, false + return s, true + + case r.src[r.pos] == '{': + return read_table(r) + } + return read_number(r) +} + +@(private = "file") +read_table :: proc(r: ^Reader) -> (Value, bool) { + if !accept(r, "{") do return nil, false + t := new(Table_Value) + if accept(r, "}") do return t, true + + for { + key: Value + switch { + case accept(r, "."): + start := r.pos + for r.pos < len(r.src) && is_identifier_shaped(r.src[r.pos:r.pos + 1]) do r.pos += 1 + if r.pos == start do return nil, false + key = r.src[start:r.pos] + case accept(r, "["): + k, ok := read_value(r) + if !ok || !accept(r, "]") do return nil, false + key = k + case: + return nil, false + } + + if !accept(r, "=") do return nil, false + value, ok := read_value(r) + if !ok do return nil, false + append(&t.entries, Table_Entry_Value{key = key, value = value}) + + if accept(r, ",") do continue + if accept(r, "}") do return t, true + return nil, false + } +} + +@(private = "file") +read_number :: proc(r: ^Reader) -> (Value, bool) { + skip_space(r) + start := r.pos + if r.pos < len(r.src) && r.src[r.pos] == '-' { + r.pos += 1 + if accept_word(r, "inf") do return math.inf_f64(-1), true + } + is_float := false + for r.pos < len(r.src) { + c := r.src[r.pos] + switch { + case c >= '0' && c <= '9': + r.pos += 1 + case c == '.': + is_float = true + r.pos += 1 + case c == 'e' || c == 'E': + is_float = true + r.pos += 1 + if r.pos < len(r.src) && (r.src[r.pos] == '-' || r.src[r.pos] == '+') do r.pos += 1 + case: + return finish_number(r.src[start:r.pos], is_float) + } + } + return finish_number(r.src[start:r.pos], is_float) +} + +@(private = "file") +finish_number :: proc(text: string, is_float: bool) -> (Value, bool) { + if text == "" || text == "-" do return nil, false + if is_float { + f, ok := strconv.parse_f64(text) + if !ok do return nil, false + return f, true + } + i, ok := strconv.parse_i64_of_base(text, 10) + if !ok do return nil, false + return i, true +} + +@(private = "file") +read_string :: proc(r: ^Reader) -> (string, bool) { + skip_space(r) + if r.pos >= len(r.src) || r.src[r.pos] != '"' do return "", false + r.pos += 1 + + b: strings.Builder + strings.builder_init(&b) + for r.pos < len(r.src) { + c := r.src[r.pos] + r.pos += 1 + switch c { + case '"': + return strings.to_string(b), true + case '\\': + if r.pos >= len(r.src) do break + e := r.src[r.pos] + r.pos += 1 + switch e { + case 'n': strings.write_byte(&b, '\n') + case 't': strings.write_byte(&b, '\t') + case 'r': strings.write_byte(&b, '\r') + case '"': strings.write_byte(&b, '"') + case '\\': strings.write_byte(&b, '\\') + case 'x': + if r.pos + 1 >= len(r.src) do return "", false + hi, hi_ok := hex_digit(r.src[r.pos]) + lo, lo_ok := hex_digit(r.src[r.pos + 1]) + if !hi_ok || !lo_ok do return "", false + r.pos += 2 + strings.write_byte(&b, hi << 4 | lo) + case: + return "", false // an escape this format doesn't define: corrupt entry + } + case: + strings.write_byte(&b, c) + } + } + return "", false // unterminated +} + +@(private = "file") +hex_digit :: proc(c: u8) -> (u8, bool) { + switch { + case c >= '0' && c <= '9': return c - '0', true + case c >= 'a' && c <= 'f': return c - 'a' + 10, true + case c >= 'A' && c <= 'F': return c - 'A' + 10, true + } + return 0, false +} diff --git a/src/cache_store.odin b/src/cache_store.odin new file mode 100644 index 0000000..f2f79a9 --- /dev/null +++ b/src/cache_store.odin @@ -0,0 +1,412 @@ +package hashedbuild + +import "core:encoding/base64" +import "core:fmt" +import "core:slice" +import "core:strings" + +// SPEC.md §15's `cached`, on disk. The key mechanism lives in +// hash_function.odin (a cached expression hashes as the closure it is); the +// text format lives in cache_format.odin; this file is the layout and the two +// operations over it, lookup and store. +// +// ---- the layout ------------------------------------------------------------- +// +// One entry per key, in the same directory ctx.cache writes its blobs to (§16 +// - `--cache-dir`, else $XDG_CACHE_HOME/hashedbuild, else the per-user +// fallback). Two shapes, decided by what the cached value *is*: +// +// /sha256- the value is a File - stored as itself +// /sha256-.hb/ anything else - a directory holding +// /sha256-.hb/value.hb the value, written as HashedBuild text +// /sha256-.hb/sha256- one entry per File inside that value +// +// A File value is kept as a file, and a directory value as a directory, so +// that what a build produced is still a thing you can open, `cat`, `diff` or +// copy out - the point of a content-addressed store is lost if everything in +// it is an opaque blob. Anything that is not a File has no such natural form, +// so it becomes text; and since text cannot hold a file, any File *inside* it +// is written out beside it and referenced by name, systematically, however +// deep it is nested. +// +// `` is the cache key - the hash of the expression as a closure - because +// that is the only thing a lookup has to go on. `` on a nested entry is +// that File's own content hash (§3), which is what makes two entries that +// contain the same file share one copy of it inside their own directories. +// Both are base64url without padding, so they are legal filenames on every +// target. The separator is `-`, where ctx.cache's own blobs use `_` +// (builtins_fs.odin): the two kinds of entry live in one directory, and the +// separator is what tells them apart at a glance. +// +// ---- committing ------------------------------------------------------------- +// +// An entry is built under a temporary name and renamed into place, so a run +// that is interrupted mid-write leaves a `.tmp` behind rather than a truncated +// entry that the next run would read as a hit. The rename is also how the race +// between two runs computing the same key is settled: the loser's rename +// fails, it removes its own temporary, and both go on to read the winner's +// entry. Nothing is ever overwritten, and nothing has to be locked. +// +// ---- what a hit returns ----------------------------------------------------- +// +// A store reads the entry back rather than returning the value it just +// computed. It costs a re-read, and it buys the property that matters: the +// first run and the second return the *same* value, Files included - pointing +// at the cache, displaying the cache's paths - instead of a value whose Files +// happen to point wherever this particular run built them. + +// The prefix and alphabet an entry name is built with. Never change either: +// both are baked into the name of every entry already on disk. +@(private = "file") +ENTRY_PREFIX :: "sha256-" +@(private = "file") +VALUE_FILE :: "value.hb" +@(private = "file") +HB_SUFFIX :: ".hb" + +cache_entry_name_for :: proc(d: Value_Digest) -> string { + d := d + encoded, _ := base64.encode(d[:], base64.ENC_URL_TABLE, context.temp_allocator) + return strings.concatenate({ENTRY_PREFIX, strings.trim_right(encoded, "=")}) +} + +// ---- lookup ----------------------------------------------------------------- + +// Whether `key_name` has an entry, and if so the value it holds. A missing +// entry is (nil, false, "") - an ordinary miss, not a failure. A *present* +// entry that cannot be read is a failure with a message, deliberately rather +// than a miss: silently recomputing over a corrupt cache would hide the +// corruption for as long as the cache lived. +cache_lookup :: proc(cache: ^Cache_Value, key_name: string) -> (val: Value, found: bool, err_msg: string) { + if errno := ensure_cache_dir_open(cache); errno != .None { + return nil, false, fmt.tprintf("could not open cache directory %s (%v)", cache.dir_path, errno) + } + + // The File shape first: a bare `sha256-`, stored as itself. + if is_dir, stat_err := fs_stat_is_dir_at(cache.dir_fd, key_name, true); stat_err == .None { + v, ok, msg := load_file_entry(cache, key_name, is_dir) + return v, ok, msg + } + + text_name := strings.concatenate({key_name, HB_SUFFIX}, context.temp_allocator) + if is_dir, stat_err := fs_stat_is_dir_at(cache.dir_fd, text_name, true); stat_err != .None || !is_dir { + return nil, false, "" // a miss + } + return load_text_entry(cache, text_name) +} + +@(private = "file") +load_file_entry :: proc(cache: ^Cache_Value, name: string, is_dir: bool) -> (Value, bool, string) { + display := strings.concatenate({cache.dir_path, "/", name}) + fv, msg := open_as_file_value(cache.dir_fd, name, is_dir, display) + if msg != "" do return nil, false, msg + return fv, true, "" +} + +@(private = "file") +load_text_entry :: proc(cache: ^Cache_Value, dir_name: string) -> (Value, bool, string) { + entry_fd, open_err := fs_open_dir_at(cache.dir_fd, dir_name, true) + if open_err != .None { + return nil, false, fmt.tprintf("could not open cache entry %s (%v)", dir_name, open_err) + } + defer fs_close(entry_fd) + + text_fd, text_err := fs_open_read_at(entry_fd, VALUE_FILE, true) + if text_err != .None { + return nil, false, fmt.tprintf("cache entry %s has no %s (%v)", dir_name, VALUE_FILE, text_err) + } + text, read_err := fs_read_all(text_fd) + fs_close(text_fd) + if read_err != .None { + return nil, false, fmt.tprintf("could not read %s/%s (%v)", dir_name, VALUE_FILE, read_err) + } + + ctx := Load_Ctx{ + entry_fd = entry_fd, + display_base = strings.concatenate({cache.dir_path, "/", dir_name}, context.temp_allocator), + } + v, ok := cache_format_read(string(text), resolve_entry_file, &ctx) + if !ok { + return nil, false, fmt.tprintf("cache entry %s/%s is not readable as a value", dir_name, VALUE_FILE) + } + if ctx.failure != "" do return nil, false, ctx.failure + return v, true, "" +} + +@(private = "file") +Load_Ctx :: struct { + entry_fd: Fs_Fd, + display_base: string, + failure: string, // set by the resolver, which can only report a bool +} + +@(private = "file") +resolve_entry_file :: proc(entry_name: string, is_dir: bool, userdata: rawptr) -> (Value, bool) { + ctx := (^Load_Ctx)(userdata) + // The name comes out of the entry's own text, so it is contained the same + // way §16's `.dir` sub-paths are: a separator in it would mean the entry was + // hand-edited, and there is nothing legitimate it could name. + if entry_name == "" || index_path_sep(entry_name) >= 0 || strings.contains(entry_name, "..") { + ctx.failure = fmt.tprintf("cache entry names %q, which is not a name this store writes", entry_name) + return nil, false + } + display := strings.concatenate({ctx.display_base, "/", entry_name}) + fv, msg := open_as_file_value(ctx.entry_fd, entry_name, is_dir, display) + if msg != "" { + ctx.failure = msg + return nil, false + } + return fv, true +} + +// One File value for a name inside the store, of the kind the text said it +// was. A mismatch is reported rather than followed: it means the entry and +// what is on disk have drifted apart. +@(private = "file") +open_as_file_value :: proc(dir_fd: Fs_Fd, name: string, is_dir: bool, display: string) -> (^File_Value, string) { + actually_dir, stat_err := fs_stat_is_dir_at(dir_fd, name, true) + if stat_err != .None do return nil, fmt.tprintf("could not read cache entry %s (%v)", name, stat_err) + if actually_dir != is_dir { + return nil, fmt.tprintf("cache entry %s is not the kind of File the entry says it is", name) + } + + fv := new(File_Value) + fv.display_path = display + if is_dir { + fd, err := fs_open_dir_at(dir_fd, name, true) + if err != .None do return nil, fmt.tprintf("could not open cache entry %s (%v)", name, err) + fv.kind = .Directory + fv.dir_fd = fd + return fv, "" + } + + fd, err := fs_open_read_at(dir_fd, name, true) + if err != .None do return nil, fmt.tprintf("could not open cache entry %s (%v)", name, err) + defer fs_close(fd) + content, read_err := fs_read_all(fd) + if read_err != .None do return nil, fmt.tprintf("could not read cache entry %s (%v)", name, read_err) + fv.kind = .Regular + fv.content = content + return fv, "" +} + +// ---- store ------------------------------------------------------------------ + +// Writes `v` under `key_name` and returns what a lookup of that key now +// yields. Losing the race to another run is not a failure: the entry that +// won holds the same value, since the key is the same. +cache_store :: proc(cache: ^Cache_Value, key_name: string, v: Value) -> (Value, bool, string) { + if errno := ensure_cache_dir_open(cache); errno != .None { + return nil, false, fmt.tprintf("could not open cache directory %s (%v)", cache.dir_path, errno) + } + + final_name := key_name + if _, is_file := v.(^File_Value); !is_file { + final_name = strings.concatenate({key_name, HB_SUFFIX}) + } + + temp_name, temp_ok := make_temp_dir(cache.dir_fd, final_name) + if !temp_ok { + return nil, false, "could not create a temporary directory in the cache" + } + temp_fd, temp_err := fs_open_dir_at(cache.dir_fd, temp_name, true) + if temp_err != .None { + remove_tree_at(cache.dir_fd, temp_name) + return nil, false, fmt.tprintf("could not open the temporary cache entry (%v)", temp_err) + } + + // A File value is written *as* the entry, so it is built one level down and + // that inner name is what gets renamed into place; everything else fills the + // temporary directory itself. + published := temp_name + if fv, is_file := v.(^File_Value); is_file { + if msg := write_file_value(temp_fd, "entry", fv); msg != "" { + fs_close(temp_fd) + remove_tree_at(cache.dir_fd, temp_name) + return nil, false, msg + } + published = strings.concatenate({temp_name, "/entry"}, context.temp_allocator) + } else if msg := write_text_entry(temp_fd, v); msg != "" { + fs_close(temp_fd) + remove_tree_at(cache.dir_fd, temp_name) + return nil, false, msg + } + fs_close(temp_fd) + + rename_err := fs_rename_at(cache.dir_fd, published, final_name) + // A File entry was built one level down, so the wrapper directory is still + // there to clear away after a successful rename; a text entry *is* the + // temporary, so there is only something to remove when the rename failed. + if rename_err != .None || published != temp_name { + remove_tree_at(cache.dir_fd, temp_name) + } + + if rename_err != .None { + // Either another run published this key first - the ordinary case, and not + // a failure, since the key determines the value - or the store is broken. + // The lookup below is what tells the two apart. + if stored, found, msg := cache_lookup(cache, key_name); msg == "" && found { + return stored, true, "" + } + return nil, false, fmt.tprintf("could not publish cache entry %s (%v)", final_name, rename_err) + } + + stored, found, msg := cache_lookup(cache, key_name) + if msg != "" do return nil, false, msg + if !found do return nil, false, fmt.tprintf("cache entry %s vanished immediately after being written", final_name) + return stored, true, "" +} + +// `.tmpN`, first N that doesn't already exist. Uniqueness comes from +// the exclusive mkdir itself rather than from a random name, which needs no +// source of randomness and no process id - neither of which every target here +// has. The cap only has to exceed the number of runs racing on one key at +// once; anything near it means something else is wrong. +@(private = "file") +make_temp_dir :: proc(dir_fd: Fs_Fd, final_name: string) -> (string, bool) { + for i in 0 ..< 64 { + name := fmt.tprintf("%s.tmp%d", final_name, i) + if fs_mkdir_at(dir_fd, name) == .None do return strings.clone(name), true + } + return "", false +} + +@(private = "file") +write_text_entry :: proc(entry_fd: Fs_Fd, v: Value) -> string { + // Every File in the value, written out first, so the text can refer to each + // by the name it landed under. + files := make([dynamic]^File_Value, 0, 4, context.temp_allocator) + collect_files(v, &files) + + // Not the temp allocator, unlike almost everything else here: this map has + // to survive every write below, and writing a File hashes it - which for a + // directory value walks a whole tree, allocating temporary buffers the whole + // way down. A map living in that same arena does not reliably come out the + // other side. + names := make(map[^File_Value]string, len(files)) + defer delete(names) + for fv in files { + if _, already := names[fv]; already do continue + d, herr := value_digest(fv) + if herr != .None do return fmt.tprintf("cannot cache this value: %s", hash_error_message(herr)) + name := cache_entry_name_for(d) + if msg := write_file_value(entry_fd, name, fv); msg != "" do return msg + names[fv] = name + } + + text, ok, why := cache_format_write(v, names) + if !ok do return fmt.tprintf("cannot cache this value: %s", why) + return write_bytes(entry_fd, VALUE_FILE, transmute([]u8)text, false) +} + +// Every File reachable in `v`, in a deterministic order. Tables are the only +// thing that can hold one; a Function's environment is not part of a value's +// content, and nothing else nests. +@(private = "file") +collect_files :: proc(v: Value, out: ^[dynamic]^File_Value) { + #partial switch av in v { + case ^File_Value: + append(out, av) + case ^Table_Value: + for entry in av.entries { + collect_files(entry.key, out) + collect_files(entry.value, out) + } + } +} + +// ---- writing a File out ----------------------------------------------------- + +@(private = "file") +write_file_value :: proc(dir_fd: Fs_Fd, name: string, fv: ^File_Value) -> string { + if fv.kind == .Regular do return write_bytes(dir_fd, name, fv.content, false) + + if err := fs_mkdir_at(dir_fd, name); err != .None { + return fmt.tprintf("could not create %s in the cache (%v)", name, err) + } + dst, open_err := fs_open_dir_at(dir_fd, name, true) + if open_err != .None do return fmt.tprintf("could not open %s in the cache (%v)", name, open_err) + defer fs_close(dst) + return copy_tree(fv.dir_fd, dst) +} + +// A recursive copy that preserves exactly what SPEC.md §3's directory hash +// reads: names, file contents, the executable bit where the target has one, +// and symlink targets stored without being followed. Nothing else about a +// directory is part of its value, so nothing else is copied - and a restored +// directory hashes as the one it was copied from, which is what makes a hit +// and a miss return the same value. +@(private = "file") +copy_tree :: proc(src_fd: Fs_Fd, dst_fd: Fs_Fd) -> string { + entries, list_err := fs_list_dir_at(src_fd, context.temp_allocator) + if list_err != .None do return fmt.tprintf("could not read a directory being cached (%v)", list_err) + slice.sort_by(entries, proc(a, b: Fs_Entry) -> bool { return a.name < b.name }) + + for entry in entries { + switch { + case entry.is_symlink: + target, err := fs_readlink_at(src_fd, entry.name) + if err != .None do return fmt.tprintf("could not read the symlink %s (%v)", entry.name, err) + if serr := fs_symlink_at(dst_fd, entry.name, target); serr != .None { + return fmt.tprintf("could not recreate the symlink %s in the cache (%v)", entry.name, serr) + } + + case entry.is_dir: + if err := fs_mkdir_at(dst_fd, entry.name); err != .None { + return fmt.tprintf("could not create %s in the cache (%v)", entry.name, err) + } + child_src, src_err := fs_open_dir_at(src_fd, entry.name, true) + if src_err != .None do return fmt.tprintf("could not open %s (%v)", entry.name, src_err) + defer fs_close(child_src) + child_dst, dst_err := fs_open_dir_at(dst_fd, entry.name, true) + if dst_err != .None do return fmt.tprintf("could not open %s in the cache (%v)", entry.name, dst_err) + defer fs_close(child_dst) + if msg := copy_tree(child_src, child_dst); msg != "" do return msg + + case: + fd, open_err := fs_open_read_at(src_fd, entry.name, true) + if open_err != .None do return fmt.tprintf("could not open %s (%v)", entry.name, open_err) + content, read_err := fs_read_all(fd) + fs_close(fd) + if read_err != .None do return fmt.tprintf("could not read %s (%v)", entry.name, read_err) + defer delete(content) + if msg := write_bytes(dst_fd, entry.name, content, entry.is_executable); msg != "" do return msg + } + } + return "" +} + +@(private = "file") +write_bytes :: proc(dir_fd: Fs_Fd, name: string, data: []u8, executable: bool) -> string { + fd, err := fs_create_exclusive_at(dir_fd, name) + if err != .None do return fmt.tprintf("could not create %s in the cache (%v)", name, err) + werr := fs_write_all(fd, data) + fs_close(fd) + if werr != .None do return fmt.tprintf("could not write %s in the cache (%v)", name, werr) + if executable do fs_set_executable_at(dir_fd, name) + return "" +} + +// ---- removing a temporary --------------------------------------------------- + +// Best-effort: this only ever runs on a temporary the caller just built, and +// the caller is already on its way to reporting something else (or to using +// the entry another run published). Leaving a `.tmpN` behind is untidy, not +// wrong - the next run picks a different N. +@(private = "file") +remove_tree_at :: proc(parent: Fs_Fd, name: string) { + if fs_unlink_at(parent, name) == .None do return + if fd, err := fs_open_dir_at(parent, name, true); err == .None { + if entries, lerr := fs_list_dir_at(fd, context.temp_allocator); lerr == .None { + for entry in entries { + if entry.is_dir && !entry.is_symlink { + remove_tree_at(fd, entry.name) + } else { + fs_unlink_at(fd, entry.name) + } + } + } + fs_close(fd) + } + fs_rmdir_at(parent, name) +} diff --git a/src/cache_test.odin b/src/cache_test.odin new file mode 100644 index 0000000..89e9025 --- /dev/null +++ b/src/cache_test.odin @@ -0,0 +1,414 @@ +// Tests run natively, never in a WASI build - see eval_test.odin. +#+build linux, windows +package hashedbuild + +import "core:fmt" +import "core:os" +import "core:strings" +import "core:testing" + +// SPEC.md §15's `cached`: the key (hash_function.odin), the on-disk layout +// (cache_store.odin) and the text format (cache_format.odin), exercised +// through the language itself wherever that is possible. +// +// Every test here gets its own cache directory. `odin test` runs tests +// concurrently, and these actually write - two tests sharing a directory +// would race on the entries in it. + +@(private = "file") +cache_scratch :: proc(name: string) -> string { + return strings.concatenate({repo_root(), "/.cache_test_", name}) +} + +@(private = "file") +remove_cache_scratch :: proc(path: string) { + remove_recursively(path) +} + +// os.remove refuses a non-empty directory, and a cache entry holding a +// directory value nests arbitrarily deep, so cleanup has to recurse. +// +// The is_dir check is not an optimisation: os.open succeeds on a regular file +// too, and reading a directory listing out of that handle is not something +// every target survives. +@(private = "file") +remove_recursively :: proc(path: string) { + if is_dir, err := fs_stat_is_dir_at(fs_cwd_dir(), path, true); err == .None && is_dir { + entries, _ := fs_list_dir(path, context.temp_allocator) + for entry in entries do remove_recursively(fmt.tprintf("%s/%s", path, entry.name)) + } + os.remove(path) +} + +// Evaluates with the real global environment and a root context whose +// ctx.cache points at `cache_dir` - the same setup a real run has, since +// `cached` is only meaningful against a real store. +@(private = "file") +eval_cached_src :: proc(src: string, cache_dir: string) -> (val: Value, ok: bool, err: string) { + ast := parse(source_t{name = "test", n_bytes = u64(len(src)), data = raw_data(src)}, ast_t{}) + interp := Interpreter{ast = &ast, src = src, current_ctx = make_root_context(cache_dir)} + val, ok = eval_program(&interp, ast.root, make_global_env()) + return val, ok, interp.error_message +} + +@(private = "file") +expect_int :: proc(t: ^testing.T, src: string, cache_dir: string, want: i64) { + val, ok, err := eval_cached_src(src, cache_dir) + testing.expect(t, ok, err) + got, is_int := val.(i64) + testing.expect(t, is_int, "expected an Integer result") + testing.expect_value(t, got, want) +} + +@(private = "file") +expect_failure :: proc(t: ^testing.T, src: string, cache_dir: string) -> string { + _, ok, err := eval_cached_src(src, cache_dir) + testing.expect(t, !ok, "expected this to fail") + return err +} + +// The names directly inside a directory, sorted - what a test asserts the +// layout with. +@(private = "file") +entry_names :: proc(path: string) -> []string { + entries, err := fs_list_dir(path, context.temp_allocator) + if err != .None do return nil + names := make([dynamic]string, 0, len(entries), context.temp_allocator) + for entry in entries do append(&names, entry.name) + return names[:] +} + +// ---- the basic contract ----------------------------------------------------- + +@(test) +test_cached_returns_the_value_and_writes_one_entry :: proc(t: ^testing.T) { + dir := cache_scratch("basic") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + expect_int(t, `cached (1 + 2)`, dir, 3) + expect_int(t, `cached (1 + 2)`, dir, 3) + + names := entry_names(dir) + testing.expect_value(t, len(names), 1) + if len(names) != 1 do return + testing.expect(t, strings.has_prefix(names[0], "sha256-"), "an entry is named by its key") + testing.expect(t, strings.has_suffix(names[0], ".hb"), "a non-File value is stored as text") +} + +// The one test that distinguishes a cache from a very elaborate way of +// evaluating twice: the file the expression reads is changed underneath it, +// and the second call still answers with what the first one stored. Nothing +// in the key mentions the file's contents, so the entry stays valid. +@(test) +test_cached_hit_survives_the_source_changing :: proc(t: ^testing.T) { + dir := cache_scratch("hit") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + data := cache_scratch("hit_data") + defer delete(data) + defer remove_cache_scratch(data) + os.make_directory(data) + path := fmt.tprintf("%s/f.txt", data) + _ = os.write_entire_file(path, transmute([]u8)string("first")) + + src := fmt.aprintf(`filetext cached (loadfile "%s")`, path) + defer delete(src) + + val, ok, err := eval_cached_src(src, dir) + testing.expect(t, ok, err) + testing.expect_value(t, val.(string), "first") + + os.remove(path) + _ = os.write_entire_file(path, transmute([]u8)string("SECOND")) + + // Uncached, the change is visible... + fresh := fmt.aprintf(`filetext (loadfile "%s")`, path) + defer delete(fresh) + val2, ok2, err2 := eval_cached_src(fresh, dir) + testing.expect(t, ok2, err2) + testing.expect_value(t, val2.(string), "SECOND") + + // ...and through `cached` it is not: the stored answer is returned. + val3, ok3, err3 := eval_cached_src(src, dir) + testing.expect(t, ok3, err3) + testing.expect_value(t, val3.(string), "first") +} + +// The half of §15's key that has to be right for a hit to be correct: two +// expressions with identical code but different captured values are different +// entries. Get this wrong and the second call returns the first one's answer. +@(test) +test_cached_key_covers_the_values_the_expression_uses :: proc(t: ^testing.T) { + dir := cache_scratch("key") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + expect_int(t, `let x 1; cached (x + 10)`, dir, 11) + expect_int(t, `let x 2; cached (x + 10)`, dir, 12) + expect_int(t, `let x 1; cached (x + 10)`, dir, 11) + + testing.expect_value(t, len(entry_names(dir)), 2) +} + +// A closure captures its environment; `#arg`/`#self` (§9) are dynamic lookups +// that it does not capture, so the key has to reach them separately. This is +// the case that returned a *wrong answer* before it did: identical code, a +// different argument, one entry. +@(test) +test_cached_key_covers_the_implicit_names_it_reads :: proc(t: ^testing.T) { + dir := cache_scratch("implicit") + defer delete(dir) + defer remove_cache_scratch(dir) + + expect_int(t, `let f func (cached (#arg + 1)); f 1`, dir, 2) + expect_int(t, `let f func (cached (#arg + 1)); f 10`, dir, 11) + expect_int(t, `let f func (cached (#arg + 1)); f 1`, dir, 2) + + // The same through `|>`, which pushes onto the same stack... + expect_int(t, `1 |> cached (#arg + 100)`, dir, 101) + expect_int(t, `5 |> cached (#arg + 100)`, dir, 105) + + // ...and at a level further out, where the reach has to be counted rather + // than assumed to be one. + expect_int(t, `let f func ((func (cached (#arg2 * 2))) 0); f 3`, dir, 6) + expect_int(t, `let f func ((func (cached (#arg2 * 2))) 0); f 4`, dir, 8) + expect_int(t, `let f func ((func (cached (#arg2 * 2))) 0); f 3`, dir, 6) +} + +// An expression that reads no implicit name is not affected by any of the +// above: its key is the closure digest alone, and stays stable across calls +// made at different depths. +@(test) +test_cached_key_is_unchanged_by_depth_when_nothing_is_read :: proc(t: ^testing.T) { + dir := cache_scratch("depth") + defer delete(dir) + defer remove_cache_scratch(dir) + + expect_int(t, `cached (6 * 7)`, dir, 42) + expect_int(t, `let f func (cached (6 * 7)); f 1`, dir, 42) + expect_int(t, `let f func (cached (6 * 7)); f 99`, dir, 42) + + testing.expect_value(t, len(entry_names(dir)), 1) +} + +// ---- the three layouts ------------------------------------------------------ + +// "if it is straight a file, keep it that way": a File value is the entry, not +// something wrapped in one, so what a build produced stays a file you can open. +@(test) +test_cached_file_value_is_stored_as_a_file :: proc(t: ^testing.T) { + dir := cache_scratch("file") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + src := fmt.aprintf(`cached (loadfile "%s/README.md")`, repo_root()) + defer delete(src) + val, ok, err := eval_cached_src(src, dir) + testing.expect(t, ok, err) + + fv, is_file := val.(^File_Value) + testing.expect(t, is_file, "a cached File is still a File") + if !is_file do return + testing.expect_value(t, fv.kind, File_Kind.Regular) + testing.expect(t, strings.has_prefix(fv.display_path, dir), "a hit displays where it lives (§3)") + + names := entry_names(dir) + testing.expect_value(t, len(names), 1) + if len(names) != 1 do return + testing.expect(t, !strings.has_suffix(names[0], ".hb"), "a File entry carries no .hb suffix") +} + +// The same for a directory, which is the case that needs the whole tree copied +// - and copied faithfully enough that the restored value hashes as the +// original did (§3). That equality is the test: it covers names, contents and +// nesting in one assertion. +@(test) +test_cached_directory_value_round_trips :: proc(t: ^testing.T) { + dir := cache_scratch("tree") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + // A tree built here rather than one of the repo's own: nothing else writes + // into it (async-branching drops marker files into examples/ as it runs, and + // tests run concurrently), and the two hashes here have to see one tree. + tree := cache_scratch("tree_src") + defer delete(tree) + defer remove_cache_scratch(tree) + os.make_directory(tree) + _ = os.write_entire_file(fmt.tprintf("%s/a.txt", tree), transmute([]u8)string("alpha")) + os.make_directory(fmt.tprintf("%s/sub", tree)) + _ = os.write_entire_file(fmt.tprintf("%s/sub/b.txt", tree), transmute([]u8)string("beta")) + + src := fmt.aprintf( + `(sha256 cached (loadfile "%s")) == (sha256 loadfile "%s")`, tree, tree, + ) + defer delete(src) + val, ok, err := eval_cached_src(src, dir) + testing.expect(t, ok, err) + testing.expect(t, val.(bool), "a cached directory is the same value it was") + + names := entry_names(dir) + testing.expect_value(t, len(names), 1) + if len(names) != 1 do return + is_dir, _ := fs_stat_is_dir_at(fs_cwd_dir(), fmt.tprintf("%s/%s", dir, names[0]), true) + testing.expect(t, is_dir, "a directory value is stored as a directory") +} + +// Anything else becomes `value.hb`, with each File it holds written out beside +// it and named by that File's own content hash - the "systematically" half of +// the layout. A composite with one file in it therefore has exactly two names +// inside its entry. +@(test) +test_cached_composite_holds_its_files_beside_the_text :: proc(t: ^testing.T) { + dir := cache_scratch("composite") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + // strings.concatenate, not fmt.aprintf: the value being cached is a Table + // literal, and fmt reads its `{` as the start of a format verb. + src := strings.concatenate({`cached { .doc = loadfile "`, repo_root(), `/README.md", .n = 7 }`}) + defer delete(src) + val, ok, err := eval_cached_src(src, dir) + testing.expect(t, ok, err) + + table, is_table := val.(^Table_Value) + testing.expect(t, is_table, "a cached Table is still a Table") + if !is_table do return + doc, found := table_find(table, "doc") + testing.expect(t, found) + _, doc_is_file := doc.(^File_Value) + testing.expect(t, doc_is_file, "the File inside came back as a File") + + outer := entry_names(dir) + testing.expect_value(t, len(outer), 1) + if len(outer) != 1 do return + inner := entry_names(fmt.tprintf("%s/%s", dir, outer[0])) + testing.expect_value(t, len(inner), 2) + + has_text, has_file := false, false + for name in inner { + if name == "value.hb" do has_text = true + if strings.has_prefix(name, "sha256-") do has_file = true + } + testing.expect(t, has_text, "the entry holds its value.hb") + testing.expect(t, has_file, "and the File it refers to, named by content") +} + +// ---- what `cached` refuses -------------------------------------------------- + +@(test) +test_cached_needs_io_and_a_cache :: proc(t: ^testing.T) { + dir := cache_scratch("refuse") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + denied := expect_failure(t, `(cached 1) withctx { .permissions = empty, .cache = ctx.cache }`, dir) + testing.expect(t, strings.contains(denied, "io permission"), denied) + + // §9 lets a program build a context by hand; one that doesn't carry .cache + // over has no store to use, and says so rather than inventing one. + no_cache := expect_failure(t, `(cached 1) withctx { .permissions = ctx.permissions }`, dir) + testing.expect(t, strings.contains(no_cache, "no .cache"), no_cache) + + // Nothing was created: the directory is made lazily, on a real write. + _, exists := os.stat(dir, context.temp_allocator) + testing.expect(t, exists != nil, "a refused `cached` creates no cache directory") +} + +// A closure's meaning is its environment, so there is nothing to write down +// and read back. The same goes for ctx.cache. Both fail rather than storing +// something that would come back as a different value. +@(test) +test_cached_refuses_a_value_it_cannot_write :: proc(t: ^testing.T) { + dir := cache_scratch("unwritable") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + for src in ([]string{`cached (func 1)`, `cached ctx.cache`, `cached { .f = func 1 }`}) { + msg := expect_failure(t, src, dir) + testing.expect(t, strings.contains(msg, "cannot cache"), msg) + } +} + +// A present-but-unreadable entry is a failure, not a miss. Recomputing over a +// corrupt cache would work, and would hide the corruption for as long as the +// cache lived. +@(test) +test_cached_reports_a_corrupt_entry :: proc(t: ^testing.T) { + dir := cache_scratch("corrupt") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + expect_int(t, `cached (1 + 2)`, dir, 3) + names := entry_names(dir) + testing.expect_value(t, len(names), 1) + if len(names) != 1 do return + + value_path := fmt.tprintf("%s/%s/value.hb", dir, names[0]) + os.remove(value_path) + _ = os.write_entire_file(value_path, transmute([]u8)string("{ this is not a value")) + + msg := expect_failure(t, `cached (1 + 2)`, dir) + testing.expect(t, strings.contains(msg, "not readable as a value"), msg) +} + +// ---- the text format -------------------------------------------------------- + +// cache_format.odin's round trip, over every kind of value that has a written +// form - and, for the numbers, over the awkward ones. A value that reads back +// even slightly different would hash differently, and `cached` would be +// handing out something other than what it stored. +@(test) +test_cache_format_round_trips_every_writable_value :: proc(t: ^testing.T) { + nested := new(Table_Value) + append(&nested.entries, Table_Entry_Value{key = i64(1), value = "one"}) + append(&nested.entries, Table_Entry_Value{key = "needs quotes", value = true}) + + table := new(Table_Value) + append(&table.entries, Table_Entry_Value{key = "nothing", value = Nothing_Value{}}) + append(&table.entries, Table_Entry_Value{key = "yes", value = true}) + append(&table.entries, Table_Entry_Value{key = "no", value = false}) + append(&table.entries, Table_Entry_Value{key = "neg", value = i64(-9223372036854775808)}) + append(&table.entries, Table_Entry_Value{key = "big", value = i64(9223372036854775807)}) + append(&table.entries, Table_Entry_Value{key = "tenth", value = 0.1}) + append(&table.entries, Table_Entry_Value{key = "tiny", value = 5.0e-324}) + append(&table.entries, Table_Entry_Value{key = "third", value = 1.0 / 3.0}) + append(&table.entries, Table_Entry_Value{key = "whole", value = 2.0}) + append(&table.entries, Table_Entry_Value{key = "escapes", value = "quote\" back\\ tab\t nl\n nul\x00 hi\x7f"}) + append(&table.entries, Table_Entry_Value{key = "unicode", value = "é中文 \U0001f600"}) + append(&table.entries, Table_Entry_Value{key = "raw", value = []u8{0, 1, 2, 255}}) + append(&table.entries, Table_Entry_Value{key = "inner", value = nested}) + append(&table.entries, Table_Entry_Value{key = "empty", value = new(Table_Value)}) + + names: map[^File_Value]string + text, wrote, why := cache_format_write(table, names) + testing.expect(t, wrote, why) + + back, read := cache_format_read(text, nil, nil) + testing.expect(t, read, text) + + // Hash equality rather than field-by-field comparison: it is the property + // `cached` actually depends on, and it covers the whole structure at once. + testing.expect(t, values_hash_equal(table, back), text) +} + +// The reader accepts literals and nothing else, so an entry someone edited +// into a program is a parse failure rather than something that runs. +@(test) +test_cache_format_rejects_anything_that_is_not_a_value :: proc(t: ^testing.T) { + for src in ([]string{ + `loadfile "/etc/passwd"`, + `1 + 2`, + `{ .a = 1 } concat { .b = 2 }`, + `func 1`, + `{ .a = }`, + `"unterminated`, + `nothing extra`, + `file "../../escape"`, + }) { + _, ok := cache_format_read(src, nil, nil) + testing.expect(t, !ok, src) + } +} diff --git a/src/eval.odin b/src/eval.odin index c431be0..b1a5291 100644 --- a/src/eval.odin +++ b/src/eval.odin @@ -6,7 +6,7 @@ import "core:strings" // Tree-walking evaluator for the pure-expression core of SPEC.md. Deliberately // scoped out for this pass (each needs real design/OS decisions this project -// hasn't made yet): `cached`, `import`, `#context`, and the static-vs-runtime +// hasn't made yet): `import`, `#context`, and the static-vs-runtime // distinction for `check`/`static_check` (both just run as runtime checks // here). // @@ -171,6 +171,8 @@ new_function :: proc(interp: ^Interpreter, body: Node_Idx, env: ^Env) -> Value { f.body = body f.env = env f.ctx = interp.current_ctx // captured now, restored around every call - see Interpreter.current_ctx + f.ast = interp.ast // so the closure can be hashed later (§15) - see Function_Value + f.src = interp.src return f } @@ -425,7 +427,10 @@ eval :: proc(interp: ^Interpreter, node: Node_Idx, env: ^Env) -> (ret_val: Value case .Sha256_Expr: return eval_sha256(interp, node, env) - case .Cached_Expr, .Import_Expr: + case .Cached_Expr: + return eval_cached(interp, node, env) + + case .Import_Expr: return fail(interp, fmt.tprintf("%v is not implemented by this evaluator yet", n.kind)) } return fail(interp, fmt.tprintf("evaluation not implemented for %v", n.kind)) @@ -1089,6 +1094,111 @@ eval_sha256 :: proc(interp: ^Interpreter, node: Node_Idx, env: ^Env) -> (Value, return encoded, true } +// §15: evaluates an expression once and remembers the answer, or hands back +// the answer an earlier run already stored. The cache lives on disk, in the +// directory `ctx.cache` writes to - see cache_store.odin for the layout and +// hash_function.odin for the key. +// +// **The key is computed before the expression runs**, which is the whole point: +// a hit must not have to evaluate anything. §15 pins what it is - the cached +// expression treated as a function, hashed as one - so that is exactly what is +// built here, a closure over the operand in the current environment and +// `ctx`. Its digest covers the code, the captured `ctx`, and the values of the +// names the code uses, so `cached (x + 1)` is a different entry for each `x`. +// +// **Gated by `ctx.permissions.io`** (§9), like §16's builtins and for the same +// reason: this reads and writes files. A denied `io` is a failure rather than a +// quiet fall-through to evaluating uncached, because the two differ in what +// gets written to disk, and silently doing the other one is not something a +// program should have to guess at. +// +// **Async is positional**, as §15 says. `cached async ` is this +// procedure wrapped around an operand that evaluates on another thread: the +// await below is the "caching wrapper around it synchronous" half. `async +// cached ` is the other placement, and needs nothing here - it is an +// Async_Expr whose body happens to be this one. +@(private = "file") +eval_cached :: proc(interp: ^Interpreter, node: Node_Idx, env: ^Env) -> (Value, bool) { + n := interp.ast.nodes[node] + operand := interp.ast.extra_children[n.children_start] + + if !ctx_allows_io(interp) do return fail(interp, "cached: io permission not granted in the current context") + + cache, has_cache := cache_of_ctx(interp.current_ctx) + if !has_cache { + return fail(interp, "cached: the current context has no .cache (SPEC.md §9 - a hand-built ctx has to carry it over)") + } + + key, herr := value_digest(new_function(interp, operand, env).(^Function_Value)) + if herr != .None do return fail(interp, fmt.tprintf("cached: %s", hash_error_message(herr))) + key, herr = mix_implicit_reach(interp, operand, key) + if herr != .None do return fail(interp, fmt.tprintf("cached: %s", hash_error_message(herr))) + key_name := cache_entry_name_for(key) + + if stored, found, msg := cache_lookup(cache, key_name); msg != "" { + return fail(interp, fmt.tprintf("cached: %s", msg)) + } else if found { + return stored, true + } + + val, ok := eval_slot(interp, operand, env) + if !ok do return nil, false + val, ok = await_value(interp, val) + if !ok do return nil, false + + stored, stored_ok, msg := cache_store(cache, key_name, val) + if !stored_ok do return fail(interp, fmt.tprintf("cached: %s", msg)) + return stored, true +} + +// A closure captures its environment and its `ctx`, but `#arg`/`#self` (§9) are +// dynamic lookups into the interpreter's own stacks and are captured by +// neither - so the closure digest cannot see them, and the key has to. Without +// this, `let f func (cached (#arg + 1)); f 1` and `f 10` would be one entry. +// +// Only the levels the expression can actually reach are mixed in, and only when +// it reaches any at all: an expression that mentions no implicit name gets the +// closure digest untouched. See hash_function.odin's implicit_reach_digest for +// why a static bound on the reach is sound. +@(private = "file") +mix_implicit_reach :: proc(interp: ^Interpreter, operand: Node_Idx, key: Value_Digest) -> (Value_Digest, Hash_Error) { + max_arg, max_self := 0, 0 + implicit_reach(interp.ast, interp.src, operand, &max_arg, &max_self) + if max_arg == 0 && max_self == 0 do return key, .None + + return implicit_reach_digest( + key, + top_of(interp.arg_stack[:], max_arg), + top_of(interp.self_stack[:], max_self), + ) +} + +// The top `count` entries, innermost first, padded with `nothing` where the +// stack is shorter than the reach - so the shape of what gets mixed in depends +// only on the expression, never on how deep the program happens to be. +@(private = "file") +top_of :: proc(stack: []Value, count: int) -> []Value { + out := make([]Value, count, context.temp_allocator) + for i in 0 ..< count { + idx := len(stack) - 1 - i + out[i] = idx >= 0 ? stack[idx] : Nothing_Value{} + } + return out +} + +// `ctx.cache`, if this context has one. §9 lets a program build a context by +// hand, and one built without carrying `.cache` over simply hasn't got a cache +// to use - so this is a question, not an assertion. +@(private = "file") +cache_of_ctx :: proc(ctx: Value) -> (^Cache_Value, bool) { + t, is_table := ctx.(^Table_Value) + if !is_table do return nil, false + val, found := table_find(t, "cache") + if !found do return nil, false + cache, is_cache := val.(^Cache_Value) + return cache, is_cache +} + // ---- check / static_check / error (§11) ---------------------------------------- @(private = "file") diff --git a/src/examples_test.odin b/src/examples_test.odin index c52847c..906813f 100644 --- a/src/examples_test.odin +++ b/src/examples_test.odin @@ -33,6 +33,7 @@ EXAMPLE_CASES := []Example_Case{ {"async-basics.hb", `"This is the payload for option A.\nThis is the payload for option B.\n"`}, {"async-branching.hb", `"medium"`}, {"async-table.hb", `{a: 2, b: 6, c: "This is the payload for option A.\n"}`}, + {"cached.hb", "{answer: 42, asking_again_agrees: true, per_argument: {small: 2, large: 11}, file_survives_the_round_trip: true}"}, {"check-and-invariants.hb", "100"}, {"comparison-and-logic.hb", "{ordered: true, both: true, either: true, mixed: false}"}, {"context-permissions.hb", "{ambient: {io: nothing}, io_denied: {}, replaced: {}, still_ambient: {io: nothing}}"}, @@ -186,9 +187,19 @@ read_dir_names :: proc(dir: string) -> []string { return names } +// Recursive, because a `cached` entry (§15) is a directory: a value that is +// not a File is stored as `sha256-.hb/value.hb`, and a directory value as +// the tree itself. os.remove won't take a non-empty directory, so a one-level +// sweep would leave the scratch cache behind for the next run to trip over. +// +// The is_dir check is not an optimisation: os.open succeeds on a regular file +// too, and reading a directory listing out of that handle is not something +// every target survives. @(private = "file") remove_dir_and_entries :: proc(dir: string) { - for name in read_dir_names(dir) do os.remove(fmt.tprintf("%s/%s", dir, name)) + if is_dir, err := fs_stat_is_dir_at(fs_cwd_dir(), dir, true); err == .None && is_dir { + for name in read_dir_names(dir) do remove_dir_and_entries(fmt.tprintf("%s/%s", dir, name)) + } os.remove(dir) } diff --git a/src/fs.odin b/src/fs.odin index c8644d0..3f24d51 100644 --- a/src/fs.odin +++ b/src/fs.odin @@ -49,13 +49,39 @@ FS_INVALID_FD :: Fs_Fd(-1) // fs_symlink_at create a symlink; fs_readlink_at reads its target // fs_open_dir_path open a directory by path - ctx.cache only (§9) // fs_make_dirs mkdir -p by path - ctx.cache only (§9) -// fs_list_dir names in a directory - the editor's file pickers +// fs_list_dir names in a directory, by path - the editor's file pickers +// +// The six below were added for §3's directory hash and §15's `cached` (see +// cache_store.odin), which are the first things here that have to read a whole +// directory as a value and write a whole directory back out again: +// +// fs_list_dir_at entries of an open directory, classified no-follow +// fs_mkdir_at create one directory, failing if it exists +// fs_rename_at rename within one directory - the atomic commit +// fs_unlink_at remove one non-directory name +// fs_rmdir_at remove one empty directory +// fs_set_executable_at set the owner-execute bit, where the target has one -// One entry of a directory listing. Deliberately minimal: the editor wants -// names, and whether to descend. +// One entry of a directory listing. +// +// `is_dir` and `is_symlink` are what SPEC.md §3's directory hash sorts an +// entry into: a symlink is its own kind there, hashed by its target string +// rather than followed, so the classification a listing reports has to be +// no-follow. `fs_list_dir_at` guarantees that; the older path-taking +// `fs_list_dir` (the editor's file pickers) does not, and leaves both +// `is_symlink` and `is_executable` false - it never fed anything that cares. +// +// `is_executable` is §3's "executable flag only - not full POSIX mode", and +// is the one field a target can be unable to answer: WASI's filestat carries +// no permission bits and Windows has no POSIX execute bit, so both report +// false always. That is a truthful report of what those filesystems say, and +// it is why §3 now spells out that a tree containing an executable hashes +// differently there than on Linux. Fs_Entry :: struct { - name: string, - is_dir: bool, + name: string, + is_dir: bool, + is_symlink: bool, + is_executable: bool, } // What went wrong, in terms both targets can express. Deliberately coarse: diff --git a/src/fs_linux.odin b/src/fs_linux.odin index bb904a3..c98aec6 100644 --- a/src/fs_linux.odin +++ b/src/fs_linux.odin @@ -1,5 +1,6 @@ package hashedbuild +import "core:fmt" import "core:os" import "core:strings" import "core:sys/linux" @@ -168,3 +169,86 @@ fs_list_dir :: proc(path: string, allocator := context.allocator) -> ([]Fs_Entry } return entries[:], .None } + +// ---- directory-as-a-value operations (§3's directory hash, §15's cached) ----- + +@(private = "file") S_IFMT :: u32(0o170000) +@(private = "file") S_IFDIR :: u32(0o040000) +@(private = "file") S_IFREG :: u32(0o100000) +@(private = "file") S_IFLNK :: u32(0o120000) + +// Entries of an already-open directory, classified without following links - +// `fs_list_dir` above answers by path and through core:os, which is the wrong +// shape for a File value (it holds a descriptor, not a trustworthy path) and +// the wrong classification (§3 needs a symlink reported as a symlink, not as +// whatever it points at). +// +// Reading the names goes through /proc/self/fd rather than getdents64 because +// nothing else here needs a raw directory-block parser; the per-entry +// classification below is the part that has to be no-follow, and fstatat with +// SYMLINK_NOFOLLOW against the descriptor is what makes it so. +fs_list_dir_at :: proc(dir: Fs_Fd, allocator := context.allocator) -> ([]Fs_Entry, Fs_Error) { + proc_path := fmt.tprintf("/proc/self/fd/%d", i32(dir)) + infos, err := os.read_all_directory_by_path(proc_path, context.temp_allocator) + if err != nil do return nil, .Io + + entries := make([dynamic]Fs_Entry, 0, len(infos), allocator) + for info in infos { + st: linux.Stat + cname := strings.clone_to_cstring(info.name, context.temp_allocator) + if errno := linux.fstatat(linux.Fd(dir), cname, &st, {.SYMLINK_NOFOLLOW}); errno != .NONE { + return nil, fs_errno_to_error(errno) + } + // The file type is a 4-bit field, not four independent flags - core's + // Mode_Bits spells the individual bits and so has no IFLNK at all (S_IFLNK + // is 0o120000, i.e. the IFREG and IFCHR bits together). Masking with + // S_IFMT is the only way to ask the question that does not misread a + // symlink as a regular file, or a block device as a directory. + kind := transmute(u32)st.mode & S_IFMT + append(&entries, Fs_Entry { + name = strings.clone(info.name, allocator), + is_dir = kind == S_IFDIR, + is_symlink = kind == S_IFLNK, + is_executable = kind == S_IFREG && .IXUSR in st.mode, + }) + } + return entries[:], .None +} + +fs_mkdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + cname := strings.clone_to_cstring(name, context.temp_allocator) + ret := linux.syscall(linux.SYS_mkdirat, linux.Fd(parent), cast(rawptr)cname, u32(0o755)) + if ret < 0 do return fs_errno_to_error(linux.Errno(-ret)) + return .None +} + +fs_rename_at :: proc(parent: Fs_Fd, old_name: string, new_name: string) -> Fs_Error { + cold := strings.clone_to_cstring(old_name, context.temp_allocator) + cnew := strings.clone_to_cstring(new_name, context.temp_allocator) + ret := linux.syscall( + linux.SYS_renameat, linux.Fd(parent), cast(rawptr)cold, linux.Fd(parent), cast(rawptr)cnew, + ) + if ret < 0 do return fs_errno_to_error(linux.Errno(-ret)) + return .None +} + +fs_unlink_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + cname := strings.clone_to_cstring(name, context.temp_allocator) + return fs_errno_to_error(linux.unlinkat(linux.Fd(parent), cname, nil)) +} + +fs_rmdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + cname := strings.clone_to_cstring(name, context.temp_allocator) + return fs_errno_to_error(linux.unlinkat(linux.Fd(parent), cname, {.REMOVEDIR})) +} + +// §3's executable flag, on the one target that has one. Only ever used to put +// back a bit that was read off a file being copied into the cache, so that a +// restored directory hashes as the original did - never to grant execute to +// something that did not already have it. +fs_set_executable_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + cname := strings.clone_to_cstring(name, context.temp_allocator) + ret := linux.syscall(linux.SYS_fchmodat, linux.Fd(parent), cast(rawptr)cname, u32(0o755), 0) + if ret < 0 do return fs_errno_to_error(linux.Errno(-ret)) + return .None +} diff --git a/src/fs_wasi.odin b/src/fs_wasi.odin index 976e7f6..951af4b 100644 --- a/src/fs_wasi.odin +++ b/src/fs_wasi.odin @@ -306,3 +306,79 @@ fs_list_dir :: proc(path: string, allocator := context.allocator) -> ([]Fs_Entry } return entries[:], .None } + +// ---- directory-as-a-value operations (§3's directory hash, §15's cached) --------- + +// The same fd_readdir loop as fs_list_dir above, against a descriptor the +// caller already holds rather than one opened from a path - which is what a +// directory File value actually has. preview1's dirent carries the file type, +// and it is the *link's* type (readdir never follows), so §3's three-way +// classification comes straight out of it. +// +// `is_executable` is always false here: preview1's filestat has no permission +// bits at all, so this target cannot report one. See fs.odin's Fs_Entry. +fs_list_dir_at :: proc(dir: Fs_Fd, allocator := context.allocator) -> ([]Fs_Entry, Fs_Error) { + entries := make([dynamic]Fs_Entry, 0, 16, allocator) + buf := make([]u8, 4096, context.temp_allocator) + cookie := wasi.dircookie_t(0) + + for { + used, read_err := wasi.fd_readdir(wasi.fd_t(dir), buf, cookie) + if read_err != .SUCCESS do return entries[:], to_fs_error(read_err) + if used == 0 do break + + offset := 0 + for offset + size_of(wasi.dirent_t) <= int(used) { + dirent := (^wasi.dirent_t)(raw_data(buf[offset:]))^ + name_start := offset + size_of(wasi.dirent_t) + name_end := name_start + int(dirent.d_namlen) + if name_end > int(used) do break // a name split across reads: ask again from d_next + + name := string(buf[name_start:name_end]) + if name != "." && name != ".." { + append(&entries, Fs_Entry{ + name = strings.clone(name, allocator), + is_dir = dirent.d_type == .DIRECTORY, + is_symlink = dirent.d_type == .SYMBOLIC_LINK, + }) + } + cookie = dirent.d_next + offset = name_end + } + if int(used) < len(buf) do break + } + return entries[:], .None +} + +fs_mkdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, rel, ok := rebase_absolute(parent, name) + if !ok || dir == FS_INVALID_FD do return .Access + return to_fs_error(wasi.path_create_directory(wasi.fd_t(dir), rel)) +} + +fs_rename_at :: proc(parent: Fs_Fd, old_name: string, new_name: string) -> Fs_Error { + old_dir, old_rel, old_ok := rebase_absolute(parent, old_name) + new_dir, new_rel, new_ok := rebase_absolute(parent, new_name) + if !old_ok || !new_ok || old_dir == FS_INVALID_FD || new_dir == FS_INVALID_FD do return .Access + return to_fs_error(wasi.path_rename(wasi.fd_t(old_dir), old_rel, wasi.fd_t(new_dir), new_rel)) +} + +fs_unlink_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, rel, ok := rebase_absolute(parent, name) + if !ok || dir == FS_INVALID_FD do return .Access + return to_fs_error(wasi.path_unlink_file(wasi.fd_t(dir), rel)) +} + +fs_rmdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, rel, ok := rebase_absolute(parent, name) + if !ok || dir == FS_INVALID_FD do return .Access + return to_fs_error(wasi.path_remove_directory(wasi.fd_t(dir), rel)) +} + +// preview1 has no chmod of any kind, and nothing to set - see Fs_Entry. The +// caller only ever asks for a bit it just read back as set, and this target +// never reads one as set, so succeeding without doing anything is exactly +// right rather than a swallowed failure. +fs_set_executable_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + return .None +} diff --git a/src/fs_windows.odin b/src/fs_windows.odin index d698435..cbffa6c 100644 --- a/src/fs_windows.odin +++ b/src/fs_windows.odin @@ -632,3 +632,90 @@ name_length :: proc(buf: []u16) -> int { for c, i in buf do if c == 0 do return i return len(buf) } + +// ---- directory-as-a-value operations (§3's directory hash, §15's cached) ----- +// +// All six reach a child the same way every other operation on this target +// does: look the descriptor's directory path out of the slot table, join the +// name onto it, and call Win32 with the result. See this file's header for +// what that costs relative to the *at() family. + +// The listing fs_list_dir does, but for a descriptor rather than a path, and +// answering §3's three-way classification. FindFirstFileW never follows a +// reparse point, so its attributes describe the entry itself - which is the +// no-follow guarantee Fs_Entry asks for. +// +// `is_executable` is always false: Windows has no POSIX execute bit, and +// nothing here stands in for it. See fs.odin's Fs_Entry. +fs_list_dir_at :: proc(dir: Fs_Fd, allocator := context.allocator) -> ([]Fs_Entry, Fs_Error) { + path, ok := dir_path_of(dir) + if !ok do return nil, .Not_Directory + + data: windows.WIN32_FIND_DATAW + h := windows.FindFirstFileW(to_win_path(join_child(path, "*")), &data) + if h == windows.INVALID_HANDLE_VALUE do return nil, last_error() + defer windows.FindClose(h) + + entries := make([dynamic]Fs_Entry, 0, 16, allocator) + for { + name, err := windows.utf16_to_utf8(data.cFileName[:name_length(data.cFileName[:])], context.temp_allocator) + if err == nil && name != "." && name != ".." { + is_link := (data.dwFileAttributes & windows.FILE_ATTRIBUTE_REPARSE_POINT) != 0 + append(&entries, Fs_Entry { + name = strings.clone(name, allocator), + // A link to a directory carries the DIRECTORY attribute too, and §3 + // wants it counted as a symlink rather than descended into - so the + // link check wins. + is_dir = !is_link && (data.dwFileAttributes & windows.FILE_ATTRIBUTE_DIRECTORY) != 0, + is_symlink = is_link, + }) + } + if !windows.FindNextFileW(h, &data) do break + } + return entries[:], .None +} + +fs_mkdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, ok := dir_path_of(parent) + if !ok do return .Not_Directory + if !windows.CreateDirectoryW(to_win_path(join_child(dir, name)), nil) do return last_error() + return .None +} + +// The atomic commit a cache entry is published by. MoveFileExW without +// MOVEFILE_REPLACE_EXISTING fails when the destination is taken, which is +// what the caller wants: a name already there means another run won the race, +// and its entry is the one to use. +fs_rename_at :: proc(parent: Fs_Fd, old_name: string, new_name: string) -> Fs_Error { + dir, ok := dir_path_of(parent) + if !ok do return .Not_Directory + moved := windows.MoveFileExW( + to_win_path(join_child(dir, old_name)), + to_win_path(join_child(dir, new_name)), + 0, + ) + if !moved do return last_error() + return .None +} + +fs_unlink_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, ok := dir_path_of(parent) + if !ok do return .Not_Directory + if !windows.DeleteFileW(to_win_path(join_child(dir, name))) do return last_error() + return .None +} + +fs_rmdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, ok := dir_path_of(parent) + if !ok do return .Not_Directory + if !windows.RemoveDirectoryW(to_win_path(join_child(dir, name))) do return last_error() + return .None +} + +// Nothing to set - Windows has no POSIX execute bit, and this target never +// reports one as set either, so the caller only ever reaches here for a bit +// that was already false. See fs.odin's Fs_Entry and fs_wasi.odin's copy of +// this note. +fs_set_executable_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + return .None +} diff --git a/src/hash.odin b/src/hash.odin index 92f59e6..645a2ff 100644 --- a/src/hash.odin +++ b/src/hash.odin @@ -31,24 +31,42 @@ Value_Digest :: [DIGEST_SIZE]u8 // Tag bytes. Never renumber these: a digest that changes meaning silently // invalidates every cache entry and every recorded hash a user has written // down. Appending a new tag for a new type is fine. -@(private = "file") TAG_NOTHING :: 0x00 -@(private = "file") TAG_BOOLEAN :: 0x01 -@(private = "file") TAG_INTEGER :: 0x02 -@(private = "file") TAG_FLOAT :: 0x03 -@(private = "file") TAG_UTF8 :: 0x04 -@(private = "file") TAG_BYTES :: 0x05 -@(private = "file") TAG_TABLE :: 0x07 +// A *directory* File. A regular one stays untagged (below); a directory has +// no sha256sum to agree with, so it is domain-separated like everything else. +TAG_DIRECTORY :: 0x08 +// SPEC.md §3's three kinds of directory entry, one tag each - which is what +// keeps a file named "x" from hashing like a symlink named "x" whose target +// happens to be that file's content. See hash_directory.odin. +TAG_DIR_ENTRY_FILE :: 0x09 +TAG_DIR_ENTRY_DIR :: 0x0a +TAG_DIR_ENTRY_SYMLINK :: 0x0b +// ctx.cache. It is write-only and content-addressed, with no name, no listing +// and no identity of its own to tell one from another, so every cache hashes +// to this tag over an empty payload. Deliberately *not* its directory path: +// §15's cache key mixes in the whole ctx, and hashing the path there would +// mean a cache directory that gets moved or copied missed on every entry in +// it, because the old path was baked into every key. +TAG_CACHE :: 0x0c +// A Function, and the pieces its encoding is built from - see +// hash_function.odin, which is where all four are actually used. +TAG_FUNCTION :: 0x0d +TAG_NATIVE :: 0x0e +TAG_AST_NODE :: 0x0f +TAG_FREE_NAMES :: 0x10 +// What a cached expression can read out of the *dynamic* stacks `#arg`/`#self` +// address (§9) - not part of a closure, and mixed in by `cached` on top of the +// closure digest. See implicit_reach_digest. +TAG_IMPLICIT_REACH :: 0x11 // Byte-lexicographic order on digests. Written as an explicit loop rather than // slice.cmp because a `proc` parameter isn't addressable, so it can't be sliced. -@(private = "file") digest_less :: proc(a, b: Value_Digest) -> bool { for i in 0 ..< DIGEST_SIZE { if a[i] != b[i] do return a[i] < b[i] @@ -56,14 +74,12 @@ digest_less :: proc(a, b: Value_Digest) -> bool { return false } -@(private = "file") sha256_of :: proc(data: []u8) -> Value_Digest { digest: Value_Digest hash.hash_bytes_to_buffer(.SHA256, data, digest[:]) return digest } -@(private = "file") sha256_tagged :: proc(tag: u8, payload: []u8) -> Value_Digest { buf := make([]u8, 1 + len(payload), context.temp_allocator) buf[0] = tag @@ -72,12 +88,13 @@ sha256_tagged :: proc(tag: u8, payload: []u8) -> Value_Digest { } // Why a value has no digest, so the caller can say which value and why rather -// than emitting one "not hashable" for every case. +// than emitting one "not hashable" for every case. Every remaining case is a +// failure to *compute* a digest that exists - as of the `cached` work there is +// no longer a kind of value the encoding simply does not cover. Hash_Error :: enum { None, - Directory_File, // §3's directory hash needs an exec bit only Linux can report - Function, // §15 needs it for `cached`, but never specifies the encoding - Cache, // §9's ctx.cache is write-only and has no identity to hash + Directory_Read, // §3 hashes a directory over its entries, and reading them failed + Function_Ast, // a closure with no syntax tree to encode - see hash_function.odin Async, // an un-awaited handle - callers await before hashing } @@ -85,12 +102,10 @@ hash_error_message :: proc(e: Hash_Error) -> string { switch e { case .None: return "" - case .Directory_File: - return "a directory File has no hash yet (see LANGUAGE.md on what isn't built yet)" - case .Function: - return "a Function has no hash yet (see LANGUAGE.md on what isn't built yet)" - case .Cache: - return "ctx.cache has no hash - it is write-only and has no identity (SPEC.md §9)" + case .Directory_Read: + return "a directory File could not be read to hash it (SPEC.md §3 hashes a directory over its entries)" + case .Function_Ast: + return "a Function with no syntax tree cannot be hashed" case .Async: return "an un-awaited async handle has no hash" } @@ -98,9 +113,12 @@ hash_error_message :: proc(e: Hash_Error) -> string { } // The digest of a value, per the encoding described at the top of this file. -// Fails (rather than inventing a digest) for the kinds §3/§15 leave open - -// see Hash_Error. -value_digest :: proc(v: Value) -> (Value_Digest, Hash_Error) { +// Fails (rather than inventing a digest) only where the digest exists but +// could not be computed - see Hash_Error. +// +// `seen` is the closure stack hash_function.odin threads through so that a +// recursive function terminates; every caller outside this file leaves it nil. +value_digest :: proc(v: Value, seen: ^Seen_Stack = nil) -> (Value_Digest, Hash_Error) { switch av in v { case Nothing_Value: return sha256_tagged(TAG_NOTHING, nil), .None @@ -138,8 +156,10 @@ value_digest :: proc(v: Value) -> (Value_Digest, Hash_Error) { case ^File_Value: // §3: "a regular file's hash is just hash(content_bytes)" - deliberately - // untagged, so it matches what sha256sum reports for the same bytes. - if av.kind == .Directory do return {}, .Directory_File + // untagged, so it matches what sha256sum reports for the same bytes. A + // directory is the other half of §3's rule, and reads its whole tree to + // answer - see hash_directory.odin. + if av.kind == .Directory do return directory_digest(av.dir_fd) return sha256_of(av.content), .None case ^Table_Value: @@ -150,9 +170,9 @@ value_digest :: proc(v: Value) -> (Value_Digest, Hash_Error) { // cross-type value ordering, which isn't built. pairs := make([][2]Value_Digest, len(av.entries), context.temp_allocator) for entry, i in av.entries { - kd, kerr := value_digest(entry.key) + kd, kerr := value_digest(entry.key, seen) if kerr != .None do return {}, kerr - vd, verr := value_digest(entry.value) + vd, verr := value_digest(entry.value, seen) if verr != .None do return {}, verr pairs[i] = {kd, vd} } @@ -167,10 +187,12 @@ value_digest :: proc(v: Value) -> (Value_Digest, Hash_Error) { return sha256_tagged(TAG_TABLE, buf), .None case ^Function_Value: - return {}, .Function + // §15's cache key is the hash of an expression *as a function*, so this + // is the case `cached` is built on - see hash_function.odin. + return function_digest(av, seen) case ^Cache_Value: - return {}, .Cache + return sha256_tagged(TAG_CACHE, nil), .None case ^Async_Handle: return {}, .Async diff --git a/src/hash_directory.odin b/src/hash_directory.odin new file mode 100644 index 0000000..b9bd37e --- /dev/null +++ b/src/hash_directory.odin @@ -0,0 +1,100 @@ +package hashedbuild + +import "core:slice" + +// SPEC.md §3's directory hash - the other half of the rule hash.odin +// implements for a regular file: +// +// dir_entry_hash(name, entry) = +// hash(name, "file", content_hash, is_executable) +// hash(name, "dir", child_dir_hash) +// hash(name, "symlink", target_path_string) +// dir_hash = hash(sorted [dir_entry_hash(name, entry) for each entry]) +// +// Sorted by *name*, as §3 says, so the digest does not depend on the order +// the filesystem happened to hand the entries back in. Every piece mixed in +// is a fixed-width digest (plus the single exec byte), per hash.odin's Merkle +// rule - a name and a symlink target both hash as the Utf8 leaves they are, +// and a file's content keeps the untagged hash(content_bytes) it has as a +// value, so a file's digest doesn't change with whether you reached it +// directly or as a directory entry. +// +// Three things to have in front of you before changing anything here: +// +// * **It does I/O.** Hashing a directory reads the whole tree, recursively. +// That makes value_digest - and so values_equal, and so `==` between two +// directory Files - a filesystem walk. There is no way around it: §3 +// defines the identity over the entries, and a directory File value holds +// an open descriptor, not a snapshot of what was under it. +// * **The exec bit is only readable on Linux.** WASI's filestat carries no +// permission bits and Windows has no POSIX execute bit, so both report +// every entry as non-executable (fs.odin's Fs_Entry). A tree containing +// an executable therefore hashes differently there than on Linux. §3 says +// so explicitly; it is the resolution of the question that used to be the +// stated reason directory hashing was unbuilt. +// * **Symlinks are never followed.** Not for classification (Fs_Entry is +// no-follow), not for the digest (the target string is hashed as-is), and +// not for the recursion (a link to a directory is a symlink entry, not a +// subtree to descend into). That last one is also what keeps a cycle of +// links from making this run forever. + +directory_digest :: proc(dir_fd: Fs_Fd) -> (Value_Digest, Hash_Error) { + entries, list_err := fs_list_dir_at(dir_fd, context.temp_allocator) + if list_err != .None do return {}, .Directory_Read + + slice.sort_by(entries, proc(a, b: Fs_Entry) -> bool { return a.name < b.name }) + + buf := make([]u8, len(entries) * DIGEST_SIZE, context.temp_allocator) + for entry, i in entries { + d, err := dir_entry_digest(dir_fd, entry) + if err != .None do return {}, err + copy(buf[i * DIGEST_SIZE:], d[:]) + } + return sha256_tagged(TAG_DIRECTORY, buf), .None +} + +@(private = "file") +dir_entry_digest :: proc(dir_fd: Fs_Fd, entry: Fs_Entry) -> (Value_Digest, Hash_Error) { + name_d := sha256_tagged(TAG_UTF8, transmute([]u8)entry.name) + + switch { + case entry.is_symlink: + target, err := fs_readlink_at(dir_fd, entry.name) + if err != .None do return {}, .Directory_Read + return mix_digests(TAG_DIR_ENTRY_SYMLINK, name_d, sha256_tagged(TAG_UTF8, transmute([]u8)target), nil), .None + + case entry.is_dir: + child, open_err := fs_open_dir_at(dir_fd, entry.name, true) + if open_err != .None do return {}, .Directory_Read + defer fs_close(child) + child_d, child_err := directory_digest(child) + if child_err != .None do return {}, child_err + return mix_digests(TAG_DIR_ENTRY_DIR, name_d, child_d, nil), .None + + case: + fd, open_err := fs_open_read_at(dir_fd, entry.name, true) + if open_err != .None do return {}, .Directory_Read + defer fs_close(fd) + content, read_err := fs_read_all(fd) + if read_err != .None do return {}, .Directory_Read + defer delete(content) + exec: [1]u8 = {entry.is_executable ? 1 : 0} + return mix_digests(TAG_DIR_ENTRY_FILE, name_d, sha256_of(content), exec[:]), .None + } +} + +// tag || a || b || trailing - the one composite shape this file and +// hash_function.odin are both built from, so neither open-codes the buffer +// arithmetic. Fixed-width inputs only, per hash.odin's Merkle rule; the +// trailing bytes are always a fixed count decided by the tag. +mix_digests :: proc(tag: u8, a: Value_Digest, b: Value_Digest, trailing: []u8) -> Value_Digest { + // Shadowed into locals because a `proc` parameter isn't addressable in Odin, + // so it can't be sliced - the same reason digest_less (hash.odin) is written + // as an explicit loop. + a, b := a, b + buf := make([]u8, 2 * DIGEST_SIZE + len(trailing), context.temp_allocator) + copy(buf[:], a[:]) + copy(buf[DIGEST_SIZE:], b[:]) + if len(trailing) > 0 do copy(buf[2 * DIGEST_SIZE:], trailing) + return sha256_tagged(tag, buf) +} diff --git a/src/hash_function.odin b/src/hash_function.odin new file mode 100644 index 0000000..4bfd964 --- /dev/null +++ b/src/hash_function.odin @@ -0,0 +1,280 @@ +package hashedbuild + +import "core:slice" +import "core:strconv" +import "core:strings" + +// The Function half of SPEC.md §6's "every value is hashable", which §15 needs +// before `cached` can exist at all: the cache key is the hash of the cached +// expression *treated as a function*, so a closure has to have a digest. +// +// A closure is three things, and the digest mixes exactly those three: +// +// 1. **its code** - the AST subtree of its body, encoded structurally, so +// that reformatting an expression or writing a comment inside it does +// not change the key; +// 2. **its captured `ctx`** (§9), whole - which is why ctx.cache had to +// become hashable (hash.odin's TAG_CACHE) rather than staying the +// "no identity" case it was; +// 3. **the free names its code actually uses**, each paired with the value +// it resolves to in the closure's environment. +// +// (3) is the part that has to be right for `cached` to be *correct* rather +// than merely deterministic. Hashing the code alone would give `cached x + 1` +// one key for every value of `x`, so the second call would happily hand back +// the first call's answer. Hashing the whole environment instead would be +// correct but useless - the global scope is in there, and so is every +// unrelated binding in every enclosing `let`. +// +// **The collection is deliberately conservative.** free_names below gathers +// every Identifier that sits in a value position anywhere in the subtree, +// without tracking which of them an inner `let` or pattern binder has already +// bound. So a name the expression shadows locally is still included, if a +// binding of that name also exists outside. That direction is safe: an extra +// name can only split one cache entry into two, never merge two into one. +// Under-collecting is the direction that would return a wrong value, and it +// cannot happen here - every name a lookup could resolve is offered to the +// lookup. +// +// Names that don't resolve are skipped rather than hashed as "absent": an +// unresolvable name is either locally bound (so not part of the closure's +// captured state) or genuinely unbound (so the expression fails when it runs, +// and never gets as far as storing anything). + +// A closure being hashed right now, so a recursive one terminates. `let rec f` +// puts `f` in its own environment, so f's free-name digest reaches f again; +// the second visit hashes as a back-reference to the enclosing occurrence +// instead of recursing. The distance is counted from the top of the stack, so +// two structurally identical recursive closures still hash alike. +Seen_Stack :: [dynamic]^Function_Value + +function_digest :: proc(f: ^Function_Value, seen: ^Seen_Stack = nil) -> (Value_Digest, Hash_Error) { + if f.native != nil { + // A proc address is not stable from one run to the next, so a native + // hashes as the name it is bound under plus whatever it captured - + // see Function_Value.native_name. + closure_d, err := value_digest(f.native_closure, seen) + if err != .None do return {}, err + return mix_digests(TAG_NATIVE, sha256_tagged(TAG_UTF8, transmute([]u8)f.native_name), closure_d, nil), .None + } + + if f.ast == nil do return {}, .Function_Ast + + stack: Seen_Stack + s := seen + if s == nil { + stack = make(Seen_Stack, 0, 8, context.temp_allocator) + s = &stack + } + for entry, i in s^ { + if entry == f { + depth := len(s^) - i // 1 = the closure immediately enclosing this point + back: [2]u8 = {u8(depth), u8(depth >> 8)} + return sha256_tagged(TAG_FUNCTION, back[:]), .None + } + } + append(s, f) + defer pop(s) + + code_d := ast_digest(f.ast, f.src, f.body) + ctx_d, ctx_err := value_digest(f.ctx, s) + if ctx_err != .None do return {}, ctx_err + free_d, free_err := free_names_digest(f, s) + if free_err != .None do return {}, free_err + + return mix_digests(TAG_FUNCTION, code_d, ctx_d, free_d[:]), .None +} + +// ---- the code --------------------------------------------------------------- + +// A node hashes as its kind, its flags, the digest of its own source text, and +// the digests of its children in order. Fixed-width throughout (2 + 1 + 32, +// then a whole number of digests), so the encoding stays unambiguous under +// hash.odin's Merkle rule. +// +// Source text is mixed in for leaves only - a leaf is where the text carries +// meaning the kind does not (which Identifier, which literal). An inner node's +// span is just whatever its children cover, so leaving it out is what makes +// the digest insensitive to whitespace, line breaks and comments. +// +// It is *not* insensitive to how a literal is spelled: `1_000` and `1000` are +// different text, so they get different keys for the same value. Conservative +// in the harmless direction again - a split entry, never a wrong hit. +@(private = "file") +ast_digest :: proc(ast: ^ast_t, src: string, node: Node_Idx) -> Value_Digest { + n := ast.nodes[node] + + head: [3]u8 = {u8(u16(n.kind)), u8(u16(n.kind) >> 8), u8(transmute(u8)n.flags)} + text_d: Value_Digest + if n.children_count == 0 { + start := int(n.span.start) + end := int(n.span.end) + if start <= end && end <= len(src) { + text_d = sha256_tagged(TAG_UTF8, transmute([]u8)src[start:end]) + } else { + text_d = sha256_tagged(TAG_UTF8, nil) // a synthesized node has no text + } + } else { + text_d = sha256_tagged(TAG_UTF8, nil) + } + + buf := make([]u8, len(head) + DIGEST_SIZE + int(n.children_count) * DIGEST_SIZE, context.temp_allocator) + copy(buf[:], head[:]) + copy(buf[len(head):], text_d[:]) + for i in 0 ..< int(n.children_count) { + child_d := ast_digest(ast, src, ast.extra_children[int(n.children_start) + i]) + copy(buf[len(head) + DIGEST_SIZE + i * DIGEST_SIZE:], child_d[:]) + } + return sha256_tagged(TAG_AST_NODE, buf) +} + +// ---- the free names --------------------------------------------------------- + +@(private = "file") +free_names_digest :: proc(f: ^Function_Value, seen: ^Seen_Stack) -> (Value_Digest, Hash_Error) { + names := make([dynamic]string, 0, 8, context.temp_allocator) + collect_names(f.ast, f.src, f.body, &names) + slice.sort(names[:]) + + pairs := make([dynamic][2]Value_Digest, 0, len(names), context.temp_allocator) + last := "" + for name in names { + if name == last do continue // an identifier used twice contributes once + last = name + val, found := env_lookup(f.env, name) + if !found do continue // locally bound, or unbound and about to fail anyway + vd, err := value_digest(val, seen) + if err != .None do return {}, err + append(&pairs, [2]Value_Digest{sha256_tagged(TAG_UTF8, transmute([]u8)name), vd}) + } + + buf := make([]u8, len(pairs) * 2 * DIGEST_SIZE, context.temp_allocator) + for i in 0 ..< len(pairs) { + copy(buf[i * 2 * DIGEST_SIZE:], pairs[i][0][:]) + copy(buf[(i * 2 + 1) * DIGEST_SIZE:], pairs[i][1][:]) + } + return sha256_tagged(TAG_FREE_NAMES, buf), .None +} + +// Every Identifier in the subtree that stands for a *variable*, as opposed to +// a literal name the grammar happens to spell the same way. The exclusions +// below are the whole list of the latter; see this file's header for why +// over-collecting beyond them is deliberate and safe. +@(private = "file") +collect_names :: proc(ast: ^ast_t, src: string, node: Node_Idx, out: ^[dynamic]string) { + n := ast.nodes[node] + + if n.kind == .Identifier { + start := int(n.span.start) + end := int(n.span.end) + if start <= end && end <= len(src) do append(out, src[start:end]) + return + } + + skip := -1 // index of the one child to walk past, if any + #partial switch n.kind { + case .Binary_Expr: + // `.field`, and `!.name` - the right operand names a Table key literally, + // it is not a variable reference. (`[expr]` and `!: expr` are the dynamic + // forms and do read variables, so they are not excluded.) + if n.children_count >= 3 { + op := ast.nodes[ast.extra_children[int(n.children_start) + 1]].kind + if op == .Op_Dot || op == .Op_CheckDot do skip = 2 + } + case .Table_Entry: + // `.name = v` writes the literal key `name`; `[name] = v` (Computed_Key) + // reads the variable. + if .Computed_Key not_in n.flags do skip = 0 + case .Let_Bind: + skip = 1 // the name being introduced + case .Pattern_Bind: + skip = 1 // ` as ` + case .Table_Pattern_Field: + skip = 0 // a bare `.field` selector + } + + for i in 0 ..< int(n.children_count) { + if i == skip do continue + collect_names(ast, src, ast.extra_children[int(n.children_start) + i], out) + } +} + +// ---- what a closure does *not* capture -------------------------------------- + +// `#arg`/`#argN`, `#self`/`#selfN` and a bare Hole are dynamic lookups into the +// interpreter's own stacks (§9), deliberately so - that is what lets them reach +// through a hard boundary the way a lexical name cannot. They are therefore +// *not* in the closure's environment, and function_digest above cannot see +// them. Left at that, `let f func (cached (#arg + 1)); f 1` and `f 10` would +// share one entry, and the second call would answer 2. +// +// So `cached` mixes them in separately, and this is that mix: the stack entries +// the expression could possibly reach, in order, deepest reach last. +// +// **How far it can reach is decided statically**, by the largest N written +// anywhere in the expression (a bare `#arg` or Hole counting as 1). That bound +// holds even for a `#argN` nested inside a function the expression itself +// calls: entering that function pushes a frame, so an N there reaches N-1 of +// the frames that were already on the stack. Anything the expression pushes for +// itself follows from its code and its captured environment, both of which the +// closure digest already covers. +// +// A level that isn't on the stack hashes as Nothing - the expression will fail +// when it runs, and it has not got as far as storing anything. +implicit_reach_digest :: proc(base: Value_Digest, args: []Value, selves: []Value) -> (Value_Digest, Hash_Error) { + buf := make([]u8, (len(args) + len(selves)) * DIGEST_SIZE, context.temp_allocator) + i := 0 + for group in ([][]Value{args, selves}) { + for v in group { + d, err := value_digest(v) + if err != .None do return {}, err + copy(buf[i * DIGEST_SIZE:], d[:]) + i += 1 + } + } + return mix_digests(TAG_IMPLICIT_REACH, base, sha256_tagged(TAG_FREE_NAMES, buf), nil), .None +} + +// How deep into each stack the expression can reach - see above. Zero for both +// means it reads neither, and `cached` skips the mix entirely, so an ordinary +// expression's key is unchanged by any of this. +implicit_reach :: proc(ast: ^ast_t, src: string, node: Node_Idx, max_arg: ^int, max_self: ^int) { + n := ast.nodes[node] + + #partial switch n.kind { + case .Hole: + // An evaluated Hole is `#arg` by another spelling (eval.odin). + if max_arg^ < 1 do max_arg^ = 1 + case .Implicit_Name: + start, end := int(n.span.start), int(n.span.end) + if start <= end && end <= len(src) do note_implicit(src[start:end], max_arg, max_self) + } + + for i in 0 ..< int(n.children_count) { + implicit_reach(ast, src, ast.extra_children[int(n.children_start) + i], max_arg, max_self) + } +} + +// "#arg" / "#arg3" / "#self" / "#self2". Anything else - `#context`, or a +// malformed name - is left alone: it has no stack to reach into, and it fails +// when the expression runs. +@(private = "file") +note_implicit :: proc(text: string, max_arg: ^int, max_self: ^int) { + rest := text[1:] if len(text) > 0 else text + + target: ^int + prefix: string + switch { + case strings.has_prefix(rest, "arg"): target, prefix = max_arg, "arg" + case strings.has_prefix(rest, "self"): target, prefix = max_self, "self" + case: return + } + + level := 1 + if digits := rest[len(prefix):]; len(digits) > 0 { + v, ok := strconv.parse_int(digits) + if !ok do return + level = v + } + if level > target^ do target^ = level +} diff --git a/src/hash_test.odin b/src/hash_test.odin index 3aabb16..95fd5f1 100644 --- a/src/hash_test.odin +++ b/src/hash_test.odin @@ -2,6 +2,7 @@ #+build linux, windows package hashedbuild +import "core:os" import "core:strings" import "core:testing" @@ -127,15 +128,150 @@ test_sha256_awaits_an_async_operand :: proc(t: ^testing.T) { testing.expect(t, eval_bool(t, `(sha256 async ("a" concat "b")) == (sha256 "ab")`)) } -// The kinds §3/§15 leave open fail by name rather than inventing a digest. +// A small tree, built here rather than borrowed from the repo. Two reasons it +// is worth the lines: nothing else writes into it (async-branching drops +// branch-*.marker files into examples/ as it runs, and `odin test` runs tests +// concurrently, so hashing that directory twice can straddle a write), and its +// contents are fixed, so what these tests cover doesn't drift with the repo. +@(private = "file") +make_tree :: proc(t: ^testing.T, name: string) -> string { + path := strings.concatenate({repo_root(), "/.hash_test_tree_", name}) + os.make_directory(path) + _ = os.write_entire_file(strings.concatenate({path, "/a.txt"}, context.temp_allocator), + transmute([]u8)string("alpha")) + _ = os.write_entire_file(strings.concatenate({path, "/b.txt"}, context.temp_allocator), + transmute([]u8)string("beta")) + os.make_directory(strings.concatenate({path, "/sub"}, context.temp_allocator)) + _ = os.write_entire_file(strings.concatenate({path, "/sub/c.txt"}, context.temp_allocator), + transmute([]u8)string("gamma")) + return path +} + +@(private = "file") +remove_tree :: proc(path: string) { + if is_dir, err := fs_stat_is_dir_at(fs_cwd_dir(), path, true); err == .None && is_dir { + entries, _ := fs_list_dir(path, context.temp_allocator) + for entry in entries { + remove_tree(strings.concatenate({path, "/", entry.name}, context.temp_allocator)) + } + } + os.remove(path) +} + +// §6 says every value is hashable, and as of `cached` that is true with no +// exceptions left: the three kinds that used to fail by name all have digests. +// A directory reads its tree (§3), a Function encodes as a closure (§15), and +// ctx.cache hashes as the tagged constant it has no identity to improve on. @(test) -test_unhashable_values_fail_with_a_reason :: proc(t: ^testing.T) { - testing.expect(t, strings.contains(eval_failure(t, `sha256 func 1`), "Function has no hash")) - testing.expect(t, strings.contains(eval_failure(t, `sha256 ctx.cache`), "ctx.cache has no hash")) +test_every_kind_of_value_has_a_digest :: proc(t: ^testing.T) { + testing.expect(t, len(eval_str(t, `sha256 func 1`)) > 0) + testing.expect(t, len(eval_str(t, `sha256 ctx.cache`)) > 0) - dir := strings.concatenate({`sha256 loadfile "`, repo_root(), `/examples"`}) + tree := make_tree(t, "any_digest") + defer delete(tree) + defer remove_tree(tree) + + dir := strings.concatenate({`sha256 loadfile "`, tree, `"`}) defer delete(dir) - testing.expect(t, strings.contains(eval_failure(t, dir), "directory File has no hash")) + testing.expect(t, len(eval_str(t, dir)) > 0) +} + +// §3: a directory's identity is its entries, so it changes with them and with +// nothing else. Reached by two different paths, the same tree is one value - +// which is the same path-independence §3 gives a regular file. +// +@(test) +test_directory_hash_is_over_its_entries :: proc(t: ^testing.T) { + tree := make_tree(t, "entries") + defer delete(tree) + defer remove_tree(tree) + + same := strings.concatenate({ + `(sha256 loadfile "`, tree, `") == (sha256 loadfile "`, tree, `/./")`, + }) + defer delete(same) + testing.expect(t, eval_bool(t, same), "one tree reached two ways is one value") + + // A nested directory is part of its parent's hash, so a change three levels + // down changes the top - the Merkle property the whole thing rests on. + before := eval_str(t, strings.concatenate({`sha256 loadfile "`, tree, `"`}, context.temp_allocator)) + _ = os.write_entire_file( + strings.concatenate({tree, "/sub/c.txt"}, context.temp_allocator), + transmute([]u8)string("GAMMA"), + ) + after := eval_str(t, strings.concatenate({`sha256 loadfile "`, tree, `"`}, context.temp_allocator)) + testing.expect(t, before != after, "a change inside a subdirectory changes the tree's hash") + + // And a directory is not its own contents flattened: it is tagged, so it + // cannot collide with a regular file however that file's bytes are chosen. + both := strings.concatenate({ + `(sha256 loadfile "`, tree, `") == (sha256 loadfile "`, tree, `/a.txt")`, + }) + defer delete(both) + testing.expect(t, !eval_bool(t, both)) +} + +// A file's digest doesn't depend on whether you reached it as a value or as a +// directory entry - §3 pins both to hash(content_bytes), untagged, which is +// what makes `sha256 ` agree with sha256sum. +@(test) +test_a_files_digest_is_the_same_inside_a_directory :: proc(t: ^testing.T) { + tree := make_tree(t, "entry_identity") + defer delete(tree) + defer remove_tree(tree) + + src := strings.concatenate({ + `(sha256 loadfile "`, tree, `/a.txt") == (sha256 "alpha")`, + }) + defer delete(src) + testing.expect(t, !eval_bool(t, src), "a File is not a Utf8 with the same bytes") + + same := strings.concatenate({ + `(sha256 loadfile "`, tree, `/a.txt") == (sha256 loadfile "`, tree, `/./a.txt")`, + }) + defer delete(same) + testing.expect(t, eval_bool(t, same)) +} + +// §15's cache key rests entirely on this: a closure hashes over its code *and* +// the values of the names that code uses. Getting the second half wrong is +// what would make `cached` hand back another argument's answer. +@(test) +test_function_hash_covers_code_and_free_names :: proc(t: ^testing.T) { + testing.expect(t, eval_bool(t, `(sha256 func (1 + 2)) == (sha256 func (1 + 2))`), + "the same expression is the same function") + testing.expect(t, !eval_bool(t, `(sha256 func (1 + 2)) == (sha256 func (1 + 3))`), + "different code is a different function") + + testing.expect(t, + !eval_bool(t, `(let x 1; sha256 func (x + 1)) == (let x 2; sha256 func (x + 1))`), + "a captured value is part of the function") + testing.expect(t, + eval_bool(t, `(let x 1; sha256 func (x + 1)) == (let x 1; sha256 func (x + 1))`), + "the same captured value is the same function") + + // Layout and comments are not part of the code, which is what keeps + // reformatting a build script from throwing its cache away. + testing.expect(t, eval_bool(t, `(sha256 func (1 + 2)) == (sha256 func ( 1 /* hi */ + 2 ))`)) +} + +// `let rec` puts a function in its own environment, so hashing one reaches +// itself. It has to terminate, and two functions written the same way have to +// agree - a back-reference counted from the wrong end would break the second. +@(test) +test_recursive_function_hash_terminates :: proc(t: ^testing.T) { + testing.expect(t, eval_bool(t, + `(let rec f func (#self 1); sha256 f) == (let rec g func (#self 1); sha256 g)`)) +} + +// ctx.cache hashes as a constant, deliberately not as its directory path - +// otherwise §15's key would change with `--cache-dir`, and a cache that was +// moved or copied would miss on everything inside it. +@(test) +test_ctx_cache_hashes_independently_of_where_it_is :: proc(t: ^testing.T) { + a := eval_str(t, `sha256 ctx.cache`) + b := eval_str(t, `sha256 (ctx withctx { .permissions = ctx.permissions, .cache = ctx.cache }).cache`) + testing.expect_value(t, a, b) } // `serialize`/`serialize_file` were removed from the language, so they are diff --git a/src/main.odin b/src/main.odin index c833370..89b297b 100644 --- a/src/main.odin +++ b/src/main.odin @@ -17,7 +17,8 @@ Options: -a, --ast Print the parsed AST before evaluating -e, --eval Evaluate like one REPL submission and exit --cache-dir - Override ctx.cache's location (SPEC.md §16) + Override where ctx.cache and cached keep + their entries (SPEC.md §15/§16) -h, --help Print this help and exit --version Print the version and exit` diff --git a/src/value.odin b/src/value.odin index 63443d7..192e66d 100644 --- a/src/value.odin +++ b/src/value.odin @@ -33,6 +33,17 @@ Function_Value :: struct { ctx: Value, // captured `ctx` (SPEC.md §9) at the point the closure was made - see eval.odin's call_function native: Native_Fn, // non-nil for a builtin (§16) - call_function invokes this instead of evaluating body/env native_closure: Value, // passed as `closure` to `native`, if any + // The tree `body` indexes into, and the source text its leaves span. A + // closure is only meaningful against the AST it was made from, so it + // carries it: hash.odin needs both to encode one (§15's cache key is the + // hash of an expression *as a function*), and nothing else has a ^ast_t + // to hand at that point. Unused if native != nil. + ast: ^ast_t, + src: string, + // A native's identity, for the same encoding: `loadfile` is a free name in + // plenty of cached expressions, and a proc pointer is not stable across + // runs, so a native hashes as the name it is bound to. Empty for a closure. + native_name: string, } // SPEC.md §3's File: a handle to a filesystem entity, file or directory only From 48fcc09d83ba51b4348b9fa8c0fd7eaf69acc209 Mon Sep 17 00:00:00 2001 From: Jan Strakowski Date: Mon, 31 Aug 2026 14:16:35 +0200 Subject: [PATCH 2/6] =?UTF-8?q?Drop=20the=20executable=20bit=20from=20?= =?UTF-8?q?=C2=A73's=20directory=20hash?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A directory entry now hashes as name and content, with no permission input on any target. §3 previously kept a single executable flag — the same reduction git makes (`100644` vs `100755`) — and hashed it where it could be read. Only Linux can read it. WASI's filestat carries no permission bits and Windows has no POSIX execute bit, and neither has a stand-in worth using: an `.exe` extension is a different question, and what MSYS guesses from a shebang is a guess. So hashing it made one tree two values depending on where it was checked out — and not hypothetically. Git sets `core.fileMode=false` on Windows, so the bit round-trips through the repository without ever existing in the working tree; the same commit checks out executable on Linux and not on Windows. This repository has four such files, so `sha256 loadfile "scripts"` would have answered two ways. Git's own way out is to *remember* the bit rather than re-derive it, which a `File` cannot do — it is a handle onto a live directory with no index beside it. Between a digest that disagrees across targets and one that ignores a bit two of the three cannot see, this takes the second: two trees differing only in an executable bit are one value, and the platform a build runs on no longer changes what its inputs hash to. The bit is still read and still preserved when `cached` copies a tree, so caching a build output doesn't silently strip it. That is fidelity in the store, not identity in the language, and the comments now say so. Covered by a Linux-only test that sets the bit and asserts the digest is unchanged — Linux-only because elsewhere fs_set_executable_at is a no-op, and the test would pass without establishing anything. Co-Authored-By: Claude Opus 5 --- LANGUAGE.md | 25 +++++++++++++++---------- SPEC.md | 22 +++++++++++++++------- src/cache_store.odin | 16 ++++++++++------ src/fs.odin | 13 +++++++------ src/fs_linux.odin | 9 +++++---- src/fs_wasi.odin | 6 ++++-- src/fs_windows.odin | 4 +++- src/hash_directory.odin | 41 ++++++++++++++++++++++++----------------- src/hash_test.odin | 39 +++++++++++++++++++++++++++++++++++++++ 9 files changed, 122 insertions(+), 53 deletions(-) diff --git a/LANGUAGE.md b/LANGUAGE.md index f06e06b..d346f60 100644 --- a/LANGUAGE.md +++ b/LANGUAGE.md @@ -345,12 +345,13 @@ same value when their bytes match, however they were reached. one when `cached` was built, since `cached` needs them: - A **directory** `File` hashes over its entries — each name, each file's - content and executable bit, each subdirectory's own hash, and each symlink's - target string, unresolved (§3). Sorted by name, so readdir order doesn't - matter. One caveat, and it is a real one: **only Linux can report an - executable bit.** WASI's `filestat` has no permission bits and Windows has no - POSIX execute bit, so both hash every entry as non-executable — a tree - containing an executable therefore hashes differently there than on Linux. + content, each subdirectory's own hash, and each symlink's target string, + unresolved (§3). Sorted by name, so readdir order doesn't matter. **No + permission bits**, deliberately: only Linux can report an executable bit, so + hashing one would make the same source tree two different values depending on + which machine checked it out. (`cached` still preserves the bit when it copies + a tree — that is the store being faithful, not the bit being part of the + value.) - A **`Function`** hashes as its code (the shape of the expression, so reformatting it or writing a comment inside changes nothing), its captured `ctx`, and the values of the names it uses. @@ -361,6 +362,12 @@ one when `cached` was built, since `cached` needs them: Hashing a directory reads the whole tree, so `sha256 ` and `==` between two directory `File`s are filesystem walks, not cheap comparisons. +One cross-platform gotcha that is *not* ours: a git checkout on Windows without +Developer Mode turns a committed symlink into an ordinary file holding the +target as text. That is a genuine difference in what is on disk, so it hashes +differently — the same reason `examples/files-symlink.hb` skips itself in such +a checkout. + → `examples/hashing.hb` (§3, §6, §15) ## Caching @@ -479,10 +486,8 @@ Parsed, specified, and rejected by the evaluator with "not implemented": `import`. Hashing is complete: every kind of value has a digest, including the three -(directory `File`, `Function`, `ctx.cache`) that used to fail by name. The one -thing to know about it is not a gap but a difference between targets — the -executable bit in a directory's hash, which only Linux can report. See -"Hashing" above. +(directory `File`, `Function`, `ctx.cache`) that used to fail by name, and the +digest a given value has is the same on every target. See "Hashing" above. Also absent: `true`/`false` literals, loops of any kind (recursion is the only repetition there is — see above), a `Bytes`-returning counterpart to diff --git a/SPEC.md b/SPEC.md index f0af27d..bc22816 100644 --- a/SPEC.md +++ b/SPEC.md @@ -42,24 +42,32 @@ A condition (of `then`, `and`, `or`, `is`) can itself be, or contain, an async e **Hashing / equality / ordering.** A `File`'s identity is **pure content, independent of path.** Where it was read from is how the value was obtained, not part of the value itself — two `File`s built from different paths are equal whenever their content matches. Ordering follows the same generic total-order mechanism as every other value (§6), keyed off this hash. -- A **regular file**'s hash is just `hash(content_bytes)`. Its own permission bits (e.g. executable) are *not* part of a bare `File` value's identity — permissions only become relevant as metadata about how the file sits inside a directory (next point). +- A **regular file**'s hash is just `hash(content_bytes)`. Its own permission bits are *not* part of its identity — and, as of 2026-08-31, not part of a directory entry's either (next point). - A **directory**'s hash is computed over its entries, sorted by name for determinism (independent of filesystem readdir order): ``` dir_entry_hash(name, entry) = - hash(name, "file", content_hash, is_executable) // executable flag only — not full POSIX mode - hash(name, "dir", child_dir_hash) // directories carry no exec bit - hash(name, "symlink", target_path_string) // target is NOT followed/resolved + hash(name, "file", content_hash) // no permission bits, of any kind + hash(name, "dir", child_dir_hash) + hash(name, "symlink", target_path_string) // target is NOT followed/resolved dir_hash = hash(sorted [dir_entry_hash(name, entry) for each entry]) ``` A symlink entry hashes the link itself (its target path string) rather than resolving through it — consistent with symlink handling being a property of the containing directory, not a standalone `File` value in its own right. - **The executable bit where it cannot be read (resolved 2026-08-31).** Only Linux reports one: WASI's `filestat` carries no permission bits at all, and Windows has no POSIX execute bit. On both, every entry hashes as **non-executable** — a truthful report of what those filesystems say, rather than a refusal to hash or a bit invented from something else. The consequence is stated here rather than buried: a tree containing an executable file hashes differently on Linux than on WASI or Windows, so a §15 cache directory carried between those platforms misses on any entry whose value is such a directory. Both alternatives were worse — dropping the bit from the hash makes two genuinely different trees one value on the target where the difference is real, and implementing the hash only on Linux would take directory values and `cached` away from the playground and the Windows target outright. This was the open question that kept the directory hash unimplemented; it is settled, and the hash is built. + **No permission bits (resolved 2026-08-31).** An earlier draft of this section hashed an executable flag alongside a file entry's content — "executable flag only, not full POSIX mode". It is gone, and no permission bit of any kind is part of a value's identity. + + The reason is that only Linux can report one. WASI's `filestat` carries no permission bits at all, and Windows has no POSIX execute bit; neither can be made to answer, and neither can be given a stand-in (an `.exe` extension is not the same question, and what a POSIX emulation layer guesses from a shebang is a guess). Hashing the bit therefore made **one tree two values depending on where it was checked out** — not hypothetically: git records exactly this one bit (`100644` vs `100755`) and, on Windows, sets `core.fileMode=false` so the bit round-trips through the repository without ever existing in the working tree. The same commit checks out executable on Linux and not on Windows. This repository has four such files, so its own `scripts/` directory would have hashed two ways. + + The alternative would have been to *remember* the bit rather than re-derive it, which is what git does. That is not available here: a `File` is a handle onto a live directory, with no index alongside it to consult. Given a choice between a digest that disagrees across targets and one that ignores a bit two of the three cannot see, this takes the second. Two trees differing only in an executable bit are one value, and the platform a build runs on stops changing what its inputs hash to. + + Implementations may still *preserve* the bit when copying a tree — §15's cache does, so that caching a build output does not silently strip it. That is fidelity in the store, not identity in the language. + + This was the open question that kept the directory hash unimplemented; it is settled, and the hash is built. **Display (resolved 2026-08-27).** A `File` displays **the path it was reached by**, made absolute and cleaned of `.`/`..` segments — recorded when the value is constructed, not resolved through the filesystem afterwards. Two consequences worth stating: a symlinked route displays as the route taken rather than the target it resolves to, and a file renamed after it was loaded still displays the path it was loaded from. This is what makes the rule implementable off Linux at all — WASI (and the portable `*at()` family) offer no way to turn an open descriptor back into a path. Printing/displaying a `File` — in the REPL, the live editor's result pane, or anywhere else a value gets shown to a human — shows its actual filesystem path. This holds for **every** `File` value, regardless of how it was obtained (`loadfile`, `createfile`, `symlink`'s containing directory, etc.), not just the ones `createfile` writes into `ctx.cache` (§16) — that case is simply the one where a path is otherwise unreachable, so it's the one worth calling out explicitly there. The only exception is `ctx.cache` itself: a distinct, "magic" pseudo-directory type (§16) that is *not* a `File` and has no path of its own to show. Path stays display-only either way — there is still no builtin that lets HashedBuild source read a `File`'s path back out as a `Utf8` value; this is purely about what a human sees when a value is printed, not a new capability for programs. -> TODO: full POSIX mode bits (owner/group/other rwx) are deliberately excluded as noise — confirm this holds up in practice, or whether some other bit besides "executable" ever needs to round-trip through a build. +> TODO: no permission bit is part of a value's identity at all as of 2026-08-31 (above), which resolves the old form of this question — "is the executable bit enough, or does some other mode bit need to round-trip" — by removing the one bit that was in. What is left open is whether a build ever genuinely needs a permission to be part of *identity* rather than merely preserved by the store; if one does, it needs an answer for the two targets that cannot see one, which is what sank the executable bit. **Numeric literals** (proposed 2026-08-26, loosely modeled on C but simplified — no type-width suffixes needed, since there's exactly one `Integer` width and one `Float` width): @@ -334,7 +342,7 @@ A native builtin hashes as the name it is bound to plus whatever it captured — /sha256-.hb/sha256- each File inside that value ``` -- **A `File` value is stored as a file, a directory value as a directory.** Not wrapped, not encoded: what a build produced stays something a person can open, `diff`, or copy out, which is most of the point of a content-addressed store. A directory is copied faithfully enough to hash as the original did (§3) — names, contents, the executable bit where the target has one, and symlink targets stored without being followed. +- **A `File` value is stored as a file, a directory value as a directory.** Not wrapped, not encoded: what a build produced stays something a person can open, `diff`, or copy out, which is most of the point of a content-addressed store. A directory is copied faithfully enough to hash as the original did (§3) — names, contents, and symlink targets stored without being followed — plus the executable bit where the target has one, which §3 does *not* hash but which a store has no reason to throw away. - **Anything else is written as text**, in a subset of HashedBuild's own syntax: literals, tables, and two names the language has no literal for (`true`/`false`, and `bytes "…"`). Since text cannot hold a file, each `File` inside the value is written out beside it, named by its own content hash (§3), and referred to from the text as `file "…"` or `dir "…"` — systematically, however deeply nested. Reading it back is a separate reader that accepts literals and nothing else, not `import`: a hand-edited entry is a parse failure rather than code that runs. - **`` is the cache key**, base64url without padding, since a lookup has nothing else to go on. The `-` separator distinguishes these from the `sha256_` blobs `createfile { .dir = ctx.cache }` writes into the same directory (§16). - **Entries are committed by rename.** Each is built under a temporary name and renamed into place, so an interrupted run leaves a stray temporary rather than a truncated entry that a later run would read as a hit. The rename is also how two runs racing on one key settle it — the loser removes its temporary and reads the winner's entry, which holds the same value, since the key is the same. Nothing is overwritten and nothing is locked. diff --git a/src/cache_store.odin b/src/cache_store.odin index f2f79a9..97a9d58 100644 --- a/src/cache_store.odin +++ b/src/cache_store.odin @@ -330,12 +330,16 @@ write_file_value :: proc(dir_fd: Fs_Fd, name: string, fv: ^File_Value) -> string return copy_tree(fv.dir_fd, dst) } -// A recursive copy that preserves exactly what SPEC.md §3's directory hash -// reads: names, file contents, the executable bit where the target has one, -// and symlink targets stored without being followed. Nothing else about a -// directory is part of its value, so nothing else is copied - and a restored -// directory hashes as the one it was copied from, which is what makes a hit -// and a miss return the same value. +// A recursive copy of everything SPEC.md §3's directory hash reads - names, +// file contents, and symlink targets stored without being followed - so a +// restored directory hashes as the one it was copied from, which is what makes +// a hit and a miss return the same value. +// +// Plus one thing §3 does *not* read: the executable bit, where the target has +// one. It is not part of a directory's identity (§3 hashes no permission bit), +// so copying it changes no digest; it is copied because caching a build output +// and getting back something you can no longer run would be a poor trade for a +// build system. Nothing else about a directory is copied. @(private = "file") copy_tree :: proc(src_fd: Fs_Fd, dst_fd: Fs_Fd) -> string { entries, list_err := fs_list_dir_at(src_fd, context.temp_allocator) diff --git a/src/fs.odin b/src/fs.odin index 3f24d51..ec9783b 100644 --- a/src/fs.odin +++ b/src/fs.odin @@ -71,12 +71,13 @@ FS_INVALID_FD :: Fs_Fd(-1) // `fs_list_dir` (the editor's file pickers) does not, and leaves both // `is_symlink` and `is_executable` false - it never fed anything that cares. // -// `is_executable` is §3's "executable flag only - not full POSIX mode", and -// is the one field a target can be unable to answer: WASI's filestat carries -// no permission bits and Windows has no POSIX execute bit, so both report -// false always. That is a truthful report of what those filesystems say, and -// it is why §3 now spells out that a tree containing an executable hashes -// differently there than on Linux. +// `is_executable` is *not* part of any hash - §3 carries no permission bit at +// all (resolved 2026-08-31), precisely because it is the one field a target +// can be unable to answer: WASI's filestat carries no permission bits and +// Windows has no POSIX execute bit, so both report false always. It exists for +// cache_store.odin, which puts the bit back when it copies a tree, so that +// caching a build output does not quietly strip it. A reader that only wants +// to know what a directory *is* should ignore this field. Fs_Entry :: struct { name: string, is_dir: bool, diff --git a/src/fs_linux.odin b/src/fs_linux.odin index c98aec6..8a76757 100644 --- a/src/fs_linux.odin +++ b/src/fs_linux.odin @@ -242,10 +242,11 @@ fs_rmdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { return fs_errno_to_error(linux.unlinkat(linux.Fd(parent), cname, {.REMOVEDIR})) } -// §3's executable flag, on the one target that has one. Only ever used to put -// back a bit that was read off a file being copied into the cache, so that a -// restored directory hashes as the original did - never to grant execute to -// something that did not already have it. +// The executable bit, on the one target that has one. Only ever used to put +// back a bit that was read off a file being copied into the cache, so that +// caching a build output does not quietly strip it - never to grant execute to +// something that did not already have it. Nothing about a value's identity +// depends on it (§3 hashes no permission bit); this is fidelity, not semantics. fs_set_executable_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { cname := strings.clone_to_cstring(name, context.temp_allocator) ret := linux.syscall(linux.SYS_fchmodat, linux.Fd(parent), cast(rawptr)cname, u32(0o755), 0) diff --git a/src/fs_wasi.odin b/src/fs_wasi.odin index 951af4b..ed2b407 100644 --- a/src/fs_wasi.odin +++ b/src/fs_wasi.odin @@ -316,7 +316,8 @@ fs_list_dir :: proc(path: string, allocator := context.allocator) -> ([]Fs_Entry // classification comes straight out of it. // // `is_executable` is always false here: preview1's filestat has no permission -// bits at all, so this target cannot report one. See fs.odin's Fs_Entry. +// bits at all, so this target cannot report one. Nothing is lost by that - +// no hash reads it (see fs.odin's Fs_Entry). fs_list_dir_at :: proc(dir: Fs_Fd, allocator := context.allocator) -> ([]Fs_Entry, Fs_Error) { entries := make([dynamic]Fs_Entry, 0, 16, allocator) buf := make([]u8, 4096, context.temp_allocator) @@ -378,7 +379,8 @@ fs_rmdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { // preview1 has no chmod of any kind, and nothing to set - see Fs_Entry. The // caller only ever asks for a bit it just read back as set, and this target // never reads one as set, so succeeding without doing anything is exactly -// right rather than a swallowed failure. +// right rather than a swallowed failure. It cannot affect a digest either +// way: §3 hashes no permission bit. fs_set_executable_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { return .None } diff --git a/src/fs_windows.odin b/src/fs_windows.odin index cbffa6c..f48b5f4 100644 --- a/src/fs_windows.odin +++ b/src/fs_windows.odin @@ -646,7 +646,9 @@ name_length :: proc(buf: []u16) -> int { // no-follow guarantee Fs_Entry asks for. // // `is_executable` is always false: Windows has no POSIX execute bit, and -// nothing here stands in for it. See fs.odin's Fs_Entry. +// nothing here stands in for it - not the `.exe`/`.bat` extension, and not +// what a POSIX emulation layer like MSYS guesses from a shebang. Nothing is +// lost by that, since no hash reads it (see fs.odin's Fs_Entry). fs_list_dir_at :: proc(dir: Fs_Fd, allocator := context.allocator) -> ([]Fs_Entry, Fs_Error) { path, ok := dir_path_of(dir) if !ok do return nil, .Not_Directory diff --git a/src/hash_directory.odin b/src/hash_directory.odin index b9bd37e..8c21936 100644 --- a/src/hash_directory.odin +++ b/src/hash_directory.odin @@ -6,18 +6,17 @@ import "core:slice" // implements for a regular file: // // dir_entry_hash(name, entry) = -// hash(name, "file", content_hash, is_executable) +// hash(name, "file", content_hash) // hash(name, "dir", child_dir_hash) // hash(name, "symlink", target_path_string) // dir_hash = hash(sorted [dir_entry_hash(name, entry) for each entry]) // // Sorted by *name*, as §3 says, so the digest does not depend on the order -// the filesystem happened to hand the entries back in. Every piece mixed in -// is a fixed-width digest (plus the single exec byte), per hash.odin's Merkle -// rule - a name and a symlink target both hash as the Utf8 leaves they are, -// and a file's content keeps the untagged hash(content_bytes) it has as a -// value, so a file's digest doesn't change with whether you reached it -// directly or as a directory entry. +// the filesystem happened to hand the entries back in. Every piece mixed in is +// a fixed-width digest, per hash.odin's Merkle rule - a name and a symlink +// target both hash as the Utf8 leaves they are, and a file's content keeps the +// untagged hash(content_bytes) it has as a value, so a file's digest doesn't +// change with whether you reached it directly or as a directory entry. // // Three things to have in front of you before changing anything here: // @@ -26,12 +25,18 @@ import "core:slice" // directory Files - a filesystem walk. There is no way around it: §3 // defines the identity over the entries, and a directory File value holds // an open descriptor, not a snapshot of what was under it. -// * **The exec bit is only readable on Linux.** WASI's filestat carries no -// permission bits and Windows has no POSIX execute bit, so both report -// every entry as non-executable (fs.odin's Fs_Entry). A tree containing -// an executable therefore hashes differently there than on Linux. §3 says -// so explicitly; it is the resolution of the question that used to be the -// stated reason directory hashing was unbuilt. +// * **No permission bit is hashed** (§3, resolved 2026-08-31). §3 used to +// include an executable flag, and this is where it would have gone. Only +// Linux can report one - WASI's filestat has no permission bits and +// Windows has no POSIX execute bit - so hashing it made one tree two +// values depending on where it was checked out. git, which reduces to the +// same single bit, sidesteps that by *remembering* the bit in its index +// rather than re-deriving it; a directory File has no index to consult, so +// the choice here was between a digest that disagrees across targets and +// one that ignores the bit. It ignores the bit. Fs_Entry still reports it, +// and cache_store.odin still preserves it when copying a tree - that is +// about not degrading what a build produced, and is deliberately not part +// of what a directory *is*. // * **Symlinks are never followed.** Not for classification (Fs_Entry is // no-follow), not for the digest (the target string is hashed as-is), and // not for the recursion (a link to a directory is a symlink entry, not a @@ -78,15 +83,17 @@ dir_entry_digest :: proc(dir_fd: Fs_Fd, entry: Fs_Entry) -> (Value_Digest, Hash_ content, read_err := fs_read_all(fd) if read_err != .None do return {}, .Directory_Read defer delete(content) - exec: [1]u8 = {entry.is_executable ? 1 : 0} - return mix_digests(TAG_DIR_ENTRY_FILE, name_d, sha256_of(content), exec[:]), .None + // Name and content, and nothing else - see the note on permission bits + // above. entry.is_executable is deliberately not read here. + return mix_digests(TAG_DIR_ENTRY_FILE, name_d, sha256_of(content), nil), .None } } // tag || a || b || trailing - the one composite shape this file and // hash_function.odin are both built from, so neither open-codes the buffer -// arithmetic. Fixed-width inputs only, per hash.odin's Merkle rule; the -// trailing bytes are always a fixed count decided by the tag. +// arithmetic. Fixed-width inputs only, per hash.odin's Merkle rule; where a +// tag uses the trailing bytes at all, it is always a fixed count decided by +// that tag. mix_digests :: proc(tag: u8, a: Value_Digest, b: Value_Digest, trailing: []u8) -> Value_Digest { // Shadowed into locals because a `proc` parameter isn't addressable in Odin, // so it can't be sliced - the same reason digest_less (hash.odin) is written diff --git a/src/hash_test.odin b/src/hash_test.odin index 95fd5f1..9efb700 100644 --- a/src/hash_test.odin +++ b/src/hash_test.odin @@ -2,6 +2,7 @@ #+build linux, windows package hashedbuild +import "core:log" import "core:os" import "core:strings" import "core:testing" @@ -211,6 +212,44 @@ test_directory_hash_is_over_its_entries :: proc(t: ^testing.T) { testing.expect(t, !eval_bool(t, both)) } +// §3 hashes no permission bit, so setting one changes nothing about what a +// directory *is*. This is the assertion behind that decision, and it is +// Linux-only because Linux is the only target that can set such a bit at all: +// elsewhere fs_set_executable_at is a documented no-op, and the test would +// pass without having established anything. +@(test) +test_directory_hash_ignores_the_executable_bit :: proc(t: ^testing.T) { + when ODIN_OS != .Linux { + log.info("skipping: only Linux can set an executable bit for the hash to ignore") + } else { + tree := make_tree(t, "exec_bit") + defer delete(tree) + defer remove_tree(tree) + + src := strings.concatenate({`sha256 loadfile "`, tree, `"`}) + defer delete(src) + before := eval_str(t, src) + + dir_fd, open_err := fs_open_dir_path(tree) + testing.expect(t, open_err == .None, "could not open the scratch tree") + if open_err != .None do return + defer fs_close(dir_fd) + testing.expect(t, fs_set_executable_at(dir_fd, "a.txt") == .None) + + // That the bit actually landed is half the test: without it the + // comparison below would hold for the wrong reason. + entries, list_err := fs_list_dir_at(dir_fd, context.temp_allocator) + testing.expect(t, list_err == .None) + saw_executable := false + for entry in entries { + if entry.name == "a.txt" && entry.is_executable do saw_executable = true + } + testing.expect(t, saw_executable, "fs_set_executable_at did not set the bit") + + testing.expect_value(t, eval_str(t, src), before) + } +} + // A file's digest doesn't depend on whether you reached it as a value or as a // directory entry - §3 pins both to hash(content_bytes), untagged, which is // what makes `sha256 ` agree with sha256sum. From 3b33131f920f736c680724b052e174b3b7070757 Mon Sep 17 00:00:00 2001 From: Jan Strakowski Date: Mon, 31 Aug 2026 14:21:24 +0200 Subject: [PATCH 3/6] Demonstrate function and directory hashing, not just document them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `sha256 ` and `sha256 ` both work as of the `cached` work, and LANGUAGE.md described them in prose — but nothing in examples/ exercised either, and the Hashing section's snippet block still only showed a string, a file and a Table. A feature nobody outside this repo can run is not finished. examples/hashing.hb gains three entries, each asserting a property rather than a digest so they stay meaningful: - whitespace and comments are not part of a function's code; - the values it captures *are* — the property `cached` rests on, since without it one entry would serve every argument; - a directory is its own kind of value, not its contents run together. LANGUAGE.md's snippet block gains the two new forms, and the Function bullet gains the pair of one-liners above. Every snippet in it was run before it was written down. Co-Authored-By: Claude Opus 5 --- LANGUAGE.md | 12 +++++++++++- examples/hashing.hb | 26 ++++++++++++++++++++++++-- src/examples_test.odin | 2 +- 3 files changed, 36 insertions(+), 4 deletions(-) diff --git a/LANGUAGE.md b/LANGUAGE.md index d346f60..7340637 100644 --- a/LANGUAGE.md +++ b/LANGUAGE.md @@ -317,6 +317,8 @@ digest of the value it evaluates to, base64-encoded as `Utf8`: sha256 "hello" // => "Ar9oHTBiuRDqs+ZdbYD2daaU7RcvIDTJNB3UICNP92A=" sha256 loadfile "pkg.tar.gz" // exactly what sha256sum reports for that file sha256 { .a = 1, .b = 2 } +sha256 loadfile "src" // a whole directory, hashed over its entries +sha256 func (1 + 2) // a function, hashed as code plus what it captures ``` This is not a checksum utility bolted on the side — it is the value identity @@ -354,7 +356,15 @@ one when `cached` was built, since `cached` needs them: value.) - A **`Function`** hashes as its code (the shape of the expression, so reformatting it or writing a comment inside changes nothing), its captured - `ctx`, and the values of the names it uses. + `ctx`, and the values of the names it uses: + + ```hashedbuild + (sha256 func (1 + 2)) == (sha256 func ( 1 /* two */ + 2 )) // true + (let x 1; sha256 func (x + 1)) == (let x 2; sha256 func (x + 1)) // false + ``` + + That second line is the one `cached` depends on — without it, an expression + would share one cache entry across every value it closes over. - **`ctx.cache`** hashes as a constant. It is write-only and has no identity to distinguish one from another, and making it a constant is what keeps a cache directory valid after it is moved or copied — see below. diff --git a/examples/hashing.hb b/examples/hashing.hb index 8094343..3d7fbb6 100644 --- a/examples/hashing.hb +++ b/examples/hashing.hb @@ -11,15 +11,37 @@ // two such tables compare equal. // // That identity is why the two `loadfile`s below are one value: a `File` is -// its content, not the path it was reached by. Evaluates to +// its content, not the path it was reached by. +// +// The last three entries are the kinds that have no obvious "content" to hash +// and get one anyway, which is what `cached` (§15) is built on: +// +// - A **function** hashes as its code plus everything it captures. The code +// is hashed structurally, so whitespace and comments are not part of it - +// but the values of the names it uses are, which is exactly what stops +// `cached` handing one argument's answer to another. +// - A **directory** hashes over its entries: each name, each file's content, +// each subdirectory's own hash, each symlink's target string unresolved. +// No permission bits, on any target (§3) - so a tree hashes the same +// wherever it was checked out. +// +// Evaluates to // { text: "Ar9oHTBiuRDqs+ZdbYD2daaU7RcvIDTJNB3UICNP92A=", // file: "ZT6vBQgoXEojRYd890EDlZWhUF/uGfXa+C9BNGBykI0=", // key_order_is_irrelevant: true, same_content_same_file: true, -// integer_is_not_float: false }. +// integer_is_not_float: false, layout_is_not_part_of_a_function: true, +// captured_values_are: false, directory_is_not_a_file: false }. { .text = sha256 "hello", .file = sha256 loadfile "optiona.txt", .key_order_is_irrelevant = (sha256 { .a = 1, .b = 2 }) == (sha256 { .b = 2, .a = 1 }), .same_content_same_file = (loadfile "optiona.txt") == (loadfile "optiona.txt"), .integer_is_not_float = (sha256 5) == (sha256 5.0), + + // Same function, written two ways. + .layout_is_not_part_of_a_function = (sha256 func (1 + 2)) == (sha256 func ( 1 /* two */ + 2 )), + // Same code, two different captured values - so two different functions. + .captured_values_are = (let x 1; sha256 func (x + 1)) == (let x 2; sha256 func (x + 1)), + // A directory is its own kind of value, not its contents run together. + .directory_is_not_a_file = (sha256 loadfile ".") == (sha256 loadfile "optiona.txt"), } diff --git a/src/examples_test.odin b/src/examples_test.odin index 906813f..d5952cf 100644 --- a/src/examples_test.odin +++ b/src/examples_test.odin @@ -41,7 +41,7 @@ EXAMPLE_CASES := []Example_Case{ {"functions.hb", "121"}, {"functions-and-holes.hb", "{section: 11, explicit: 49, nested: 507, stored: 42, asserted: 9}"}, {"guard-chain.hb", "5"}, - {"hashing.hb", `{text: "Ar9oHTBiuRDqs+ZdbYD2daaU7RcvIDTJNB3UICNP92A=", file: "ZT6vBQgoXEojRYd890EDlZWhUF/uGfXa+C9BNGBykI0=", key_order_is_irrelevant: true, same_content_same_file: true, integer_is_not_float: false}`}, + {"hashing.hb", `{text: "Ar9oHTBiuRDqs+ZdbYD2daaU7RcvIDTJNB3UICNP92A=", file: "ZT6vBQgoXEojRYd890EDlZWhUF/uGfXa+C9BNGBykI0=", key_order_is_irrelevant: true, same_content_same_file: true, integer_is_not_float: false, layout_is_not_part_of_a_function: true, captured_values_are: false, directory_is_not_a_file: false}`}, {"numeric-literals.hb", "{hex: 42, octal: 42, binary: 42, grouped: 1000000, exponent: 1500, bases_agree: true}"}, {"nothing-and-empty.hb", "{unit: nothing, zero_table: {}, present_case: 42, empty_case: -1, same: false}"}, {"optional.hb", "42"}, From daa43d186e52cd54ef51f5b2d1110dc31eb8f5ec Mon Sep 17 00:00:00 2001 From: Jan Strakowski Date: Mon, 31 Aug 2026 14:39:36 +0200 Subject: [PATCH 4/6] =?UTF-8?q?Drop=20the=20executable=20bit=20from=20?= =?UTF-8?q?=C2=A73's=20directory=20hash?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverses the resolution #17 recorded earlier the same day. A directory entry now hashes as name and content, with no permission input on any target. #17 kept a single executable flag — the same reduction git makes — read where a target can see one and false elsewhere, and cited `core.filemode` as taking the same position. Looking at what git actually *does* rather than what it records reverses that. Git never re-derives the bit on a target that cannot report one: `core.fileMode=false` on Windows means it carries the mode from the index, so a committed `100755` round-trips through the repository without ever existing in the working tree. The same commit is executable on Linux and not on Windows. So hashing the bit made one tree two values depending on where it was checked out — and not hypothetically. This repository has four `100755` files, so `sha256 loadfile "scripts"` would have answered two ways. Remembering the bit instead, as git does, is not available to a `File`: it is a handle onto a live directory with no index beside it. Between a digest that disagrees across targets and one that ignores a bit two of the three cannot see, §3 now takes the second. The bit is still read and still restored when `cached` copies a tree, so caching a build output doesn't silently strip it — fidelity in the store, not identity in the language, and the comments say so. hash_linux_test.odin's assertion inverts: it now sets the bit, confirms it actually landed (otherwise the check would hold for the wrong reason), and asserts the digest is unchanged. Co-Authored-By: Claude Opus 5 --- LANGUAGE.md | 25 +++++++++------------ SPEC.md | 6 ++--- examples/hashing-directories.hb | 16 ++++++------- src/examples_test.odin | 9 ++++---- src/fs.odin | 19 +++++++++------- src/fs_linux.odin | 7 +++--- src/hash.odin | 30 ++++++++++++++++--------- src/hash_linux_test.odin | 40 +++++++++++++++++++++------------ 8 files changed, 86 insertions(+), 66 deletions(-) diff --git a/LANGUAGE.md b/LANGUAGE.md index e91954a..7d8b960 100644 --- a/LANGUAGE.md +++ b/LANGUAGE.md @@ -414,10 +414,9 @@ same value when their bytes match, however they were reached. A **directory** `File` hashes over its entries (§3), sorted by name so the digest is the tree's rather than the order the filesystem listed it in. Each -entry contributes its name, plus its content hash and executable bit for a -regular file, its own directory hash for a sub-directory, or its target string -for a symlink — which is never followed, so a link to a directory is a link, -not a directory. +entry contributes its name, plus its content hash for a regular file, its own +directory hash for a sub-directory, or its target string for a symlink — which +is never followed, so a link to a directory is a link, not a directory. ```hashedbuild sha256 loadfile "examples" // the digest of a whole tree @@ -433,13 +432,13 @@ Two things about it are worth knowing before you rely on it: revoked `io` can't pull a tree's contents through a handle it was handed. It happens once: a `File` is an immutable handle, so the digest is fixed from then on, and seeing a change means loading the directory again. -- **The executable bit is Linux-only, and a tree containing one hashes - differently elsewhere.** WASI's `filestat` has no permission bits and Windows - has no POSIX executable bit, so on those targets every file hashes as - non-executable. That is the language's answer rather than a gap — a Windows - checkout genuinely has no executable bits, and reporting one would be - inventing it, the same position git takes with `core.filemode`. If you need a - digest that agrees across all three, keep executables out of the tree. +- **No permission bits go into it**, so a tree hashes the same wherever it was + checked out. Only Linux can report an executable bit, and hashing one would + have made the same source tree two different values depending on the machine + — git makes that concrete, since `core.fileMode=false` on Windows carries a + committed `100755` through the repository without the bit ever existing in + the working tree. (`cached` still restores the bit when it copies a tree; that + is the store being faithful, not the bit being part of the value.) An entry that is neither a file, a directory, nor a symlink — a socket, a device node — fails rather than being skipped, since a digest that ignored part @@ -615,9 +614,7 @@ Parsed, specified, and rejected by the evaluator with "not implemented": **Hashing is complete**: `sha256` answers for every value, including the three that used to be listed here — a directory `File`, a `Function`, and a cyclic value — plus `ctx.cache`. See the Hashing section above for what each of them -encodes, and for the one place the answer is target-specific: a directory -containing an executable hashes differently on Windows and WASI than on Linux, -because neither of those has an executable bit to report. +encodes. A given value has the same digest on every target. Also absent: `true`/`false` literals, loops of any kind (recursion is the only repetition there is — see above), a `Bytes`-returning counterpart to diff --git a/SPEC.md b/SPEC.md index fb8f335..d835afa 100644 --- a/SPEC.md +++ b/SPEC.md @@ -53,11 +53,9 @@ A condition (of `then`, `and`, `or`, `is`) can itself be, or contain, an async e ``` A symlink entry hashes the link itself (its target path string) rather than resolving through it — consistent with symlink handling being a property of the containing directory, not a standalone `File` value in its own right. An entry that is none of the three — a fifo, a socket, a device node — has no encoding here, and hashing a directory containing one fails rather than skipping it: a digest that ignored part of a tree would call two different trees the same value. - **The executable bit where it cannot be observed (resolved 2026-08-31).** `is_executable` above is the entry's owner-execute bit *where the host reports one*, and **false wherever it does not**. WASI's `filestat` carries no permission bits at all and Windows has no POSIX executable bit, so on those two targets every regular file hashes as non-executable. The consequence is deliberate and worth stating plainly: a tree containing an executable has a different `dir_hash` on Linux than in a browser or on Windows. The alternative considered was dropping the bit from the encoding entirely, which would make one digest hold everywhere at the cost of a distinction a build language wants — a checked-out `configure` that lost its bit is not the same tree. Reporting the bit as *set* where it cannot be seen was never on the table; a Windows checkout genuinely has no executable bits, so claiming one would be inventing data. This is the same position git takes with `core.filemode`. - **When the entries are read (resolved 2026-08-31).** A directory `File`'s children live on the filesystem rather than in the value, so unlike every other digest this one performs I/O. It happens **at the first demand** — the first `sha256` or comparison that needs the digest — and is subject to `ctx.permissions.io` (§9) at that moment, exactly as `loadfile` is: a context that has revoked `io` cannot read a tree through a handle it was passed. The result is then part of the value and is never re-read, which is what `File` being an immutable handle (above) requires: a value whose digest changed under a program because someone touched the tree would not be one. Observing a change means loading the directory again, which produces a new value. - **No permission bits (resolved 2026-08-31).** An earlier draft of this section hashed an executable flag alongside a file entry's content — "executable flag only, not full POSIX mode". It is gone, and no permission bit of any kind is part of a value's identity. + **No permission bits (resolved 2026-08-31, reversing an earlier resolution of the same day).** This section first kept an executable flag alongside a file entry's content — "executable flag only, not full POSIX mode" — and the directory hash was built that way, with the bit read where a target could see one and false elsewhere. It is gone. No permission bit of any kind is part of a value's identity. The reason is that only Linux can report one. WASI's `filestat` carries no permission bits at all, and Windows has no POSIX execute bit; neither can be made to answer, and neither can be given a stand-in (an `.exe` extension is not the same question, and what a POSIX emulation layer guesses from a shebang is a guess). Hashing the bit therefore made **one tree two values depending on where it was checked out** — not hypothetically: git records exactly this one bit (`100644` vs `100755`) and, on Windows, sets `core.fileMode=false` so the bit round-trips through the repository without ever existing in the working tree. The same commit checks out executable on Linux and not on Windows. This repository has four such files, so its own `scripts/` directory would have hashed two ways. @@ -65,7 +63,7 @@ A condition (of `then`, `and`, `or`, `is`) can itself be, or contain, an async e Implementations may still *preserve* the bit when copying a tree — §15's cache does, so that caching a build output does not silently strip it. That is fidelity in the store, not identity in the language. - This was the open question that kept the directory hash unimplemented; it is settled, and the hash is built. + What tipped it was looking at how git behaves rather than at what it records. The first resolution cited `core.filemode` as agreeing; it does the opposite, because git never re-derives the bit on a target that cannot report one — it carries the one from the index, which is a move this language has no equivalent of. **Display (resolved 2026-08-27).** A `File` displays **the path it was reached by**, made absolute and cleaned of `.`/`..` segments — recorded when the value is constructed, not resolved through the filesystem afterwards. Two consequences worth stating: a symlinked route displays as the route taken rather than the target it resolves to, and a file renamed after it was loaded still displays the path it was loaded from. This is what makes the rule implementable off Linux at all — WASI (and the portable `*at()` family) offer no way to turn an open descriptor back into a path. diff --git a/examples/hashing-directories.hb b/examples/hashing-directories.hb index a093a79..e8324fb 100644 --- a/examples/hashing-directories.hb +++ b/examples/hashing-directories.hb @@ -4,15 +4,15 @@ // // §3 computes it over the directory's entries, each hashed with its name and // sorted by name so the answer is the tree's rather than readdir's: a regular -// file contributes its content hash and whether it is executable, a -// sub-directory contributes its own directory hash, and a symlink contributes -// its target *string*, never followed. Nothing about where the directory sits -// enters into it, which is why two handles on the same tree are one value. +// file contributes its content hash, a sub-directory contributes its own +// directory hash, and a symlink contributes its target *string*, never +// followed. No permission bits enter into it, and nothing about where the +// directory sits does either - which is why two handles on the same tree are +// one value, and why a tree hashes the same on every target. // -// The digests themselves are deliberately not written down here: a tree -// containing an executable hashes differently on Windows and in the browser -// than on Linux, because neither of those targets has an executable bit to -// report (see LANGUAGE.md). What is the same everywhere are the properties +// The digests themselves are still not written down here: this directory's +// contents change as examples are added, so a literal would be a value that +// went stale the next time someone wrote one. What holds are the properties // below. // // Reading a directory is I/O, so the first `sha256` of one needs diff --git a/src/examples_test.odin b/src/examples_test.odin index c681167..4d27a4e 100644 --- a/src/examples_test.odin +++ b/src/examples_test.odin @@ -44,11 +44,10 @@ EXAMPLE_CASES := []Example_Case{ {"guard-chain.hb", "5"}, {"hashing.hb", `{text: "Ar9oHTBiuRDqs+ZdbYD2daaU7RcvIDTJNB3UICNP92A=", file: "ZT6vBQgoXEojRYd890EDlZWhUF/uGfXa+C9BNGBykI0=", key_order_is_irrelevant: true, same_content_same_file: true, integer_is_not_float: false}`}, // The three hashing examples assert properties rather than digests, on - // purpose. A directory containing an executable hashes differently on - // Windows and WASI than on Linux (§3, and LANGUAGE.md on why), and a - // closure's digest includes its body's own source text - so a literal here - // would be a value that is either target-specific or broken by reformatting - // the example it came from. + // purpose. A directory's digest changes as examples are added to the tree, + // and a closure's includes its body's own source text - so a literal here + // would be a value that went stale the next time someone added an example or + // reformatted the one it came from. {"hashing-cyclic.hb", "{a_cycle_hashes: true, equal_cycles_hash_alike: true, shape_still_matters: true, either_end_agrees: true}"}, {"hashing-directories.hb", "{a_tree_is_not_its_file: true, one_tree_is_one_value: true, reading_twice_agrees: true}"}, {"hashing-functions.hb", "{same_body_same_hash: true, different_body_differs: true, captures_count: true, neighbours_do_not: true, builtins_hash_by_what_they_are: true, a_builtin_carries_its_argument: true}"}, diff --git a/src/fs.odin b/src/fs.odin index d8d4dcb..4ab1f4c 100644 --- a/src/fs.odin +++ b/src/fs.odin @@ -69,9 +69,10 @@ Fs_Entry :: struct { } // One entry of a directory in the detail SPEC.md §3's directory hash needs: -// which of the three shapes it is, and - for a regular file - whether it is -// executable. Distinct from Fs_Entry above, which answers the editor's much -// smaller question (a name, and whether to descend into it). +// which of the three shapes it is - plus, for a regular file, whether it is +// executable, which §3 does not hash but §15's cache preserves. Distinct from +// Fs_Entry above, which answers the editor's much smaller question (a name, +// and whether to descend into it). // // `kind` is decided **without following symlinks**: §3 hashes a link entry as // its target string rather than resolving through it, so a link *to* a @@ -86,11 +87,13 @@ Fs_Node_Kind :: enum { Fs_Dir_Entry :: struct { name: string, kind: Fs_Node_Kind, - // .Regular only. **False wherever the target cannot report the bit**, rather - // than a third "unknown" state: WASI's filestat carries no permission bits - // at all and Windows has no POSIX exec bit, so on those two targets this is - // always false. That is the language's answer, not a gap - see hash.odin's - // directory section and LANGUAGE.md. + // .Regular only, and **not part of any hash**: §3 carries no permission bit + // (see hash.odin's directory section), precisely because this is the one + // field a target can be unable to answer - WASI's filestat has no permission + // bits at all and Windows has no POSIX exec bit, so on those two it is always + // false. It exists for cache_store.odin, which puts the bit back when it + // copies a tree, so that caching a build output does not strip it. Anything + // asking what a directory *is* should ignore this field. is_executable: bool, } diff --git a/src/fs_linux.odin b/src/fs_linux.odin index 7a4d403..2e8ce9f 100644 --- a/src/fs_linux.odin +++ b/src/fs_linux.odin @@ -204,9 +204,10 @@ fs_list_entries_at :: proc(parent: Fs_Fd, allocator := context.allocator) -> ([] append(&entries, Fs_Dir_Entry { name = strings.clone(name, allocator), kind = kind, - // §3 hashes "the executable flag only - not full POSIX mode", and the - // owner bit is the one that means "this is a program". Group and other - // are part of who may run it, which is not part of what it is. + // Not for §3, which hashes no permission bit - this is for the cache, + // which restores the bit when it copies a tree (cache_store.odin). The + // owner bit is the one that means "this is a program"; group and other + // are about who may run it. is_executable = kind == .Regular && .IXUSR in stat.mode, }) } diff --git a/src/hash.odin b/src/hash.odin index 6858b98..cd86318 100644 --- a/src/hash.odin +++ b/src/hash.odin @@ -319,7 +319,7 @@ value_digest_walk :: proc(v: Value, w: ^Hash_Walk) -> (Value_Digest, Hash_Fail) // §3 spells the directory hash out: // // dir_entry_hash(name, entry) = -// hash(name, "file", content_hash, is_executable) +// hash(name, "file", content_hash) // hash(name, "dir", child_dir_hash) // hash(name, "symlink", target_path_string) // dir_hash = hash(sorted [dir_entry_hash(name, entry) for each entry]) @@ -332,13 +332,22 @@ value_digest_walk :: proc(v: Value, w: ^Hash_Walk) -> (Value_Digest, Hash_Fail) // in principle collide with a regular file whose content happened to be its // entry digests laid end to end. // -// **The executable bit is the file's owner-execute bit where the target can -// see one, and false everywhere else.** WASI's filestat has no permission bits -// and Windows has no POSIX exec bit (fs.odin), so a tree containing an -// executable hashes differently there than on Linux. That is the language's -// answer rather than a gap: a Windows checkout genuinely has no exec bits, so -// reporting one would be inventing it - the same position git takes with -// core.filemode. LANGUAGE.md says so where a user will meet it. +// **No permission bit is hashed** (§3, resolved 2026-08-31). An earlier form of +// §3 kept the owner-execute bit here, false on the targets that cannot see one. +// The trouble is that this makes a tree's identity depend on where it was +// checked out, and git shows why that is not hypothetical: it records exactly +// this one bit (100644 vs 100755) and sets core.fileMode=false on Windows, so +// the bit round-trips through a repository without ever existing in the working +// tree. The same commit is executable on Linux and not on Windows - this +// repository's own scripts/ would have hashed two ways. +// +// git escapes that by *remembering* the bit rather than re-deriving it, which +// is not available here: a File is a handle onto a live directory, with no +// index beside it to consult. Between a digest that disagrees across targets +// and one that ignores a bit two of the three cannot see, §3 takes the second. +// Fs_Dir_Entry still reports it and cache_store.odin still restores it when +// copying a tree, so caching a build output does not silently strip it - that +// is fidelity in the store, not identity in the language. // // A symlink is hashed as its target *string*, never resolved (§3), which is // also what keeps this recursion finite: nothing here follows a link, so the @@ -396,11 +405,12 @@ dir_entry_digest :: proc(dir: Fs_Fd, entry: Fs_Dir_Entry) -> (Value_Digest, Hash } defer delete(content) - payload: [2 * DIGEST_SIZE + 1]u8 + // Name and content, and nothing else - see the note on permission bits + // above. entry.is_executable is deliberately not read here. + payload: [2 * DIGEST_SIZE]u8 ch := sha256_of(content) copy(payload[0:], name[:]) copy(payload[DIGEST_SIZE:], ch[:]) - payload[2 * DIGEST_SIZE] = 1 if entry.is_executable else 0 return sha256_tagged(TAG_DIR_ENTRY_FILE, payload[:]), HASH_OK case .Directory: diff --git a/src/hash_linux_test.odin b/src/hash_linux_test.odin index c6a1a0a..b9be229 100644 --- a/src/hash_linux_test.odin +++ b/src/hash_linux_test.odin @@ -1,8 +1,8 @@ -// The executable bit is the one part of SPEC.md §3's directory hash that only -// one target can see, so it is the one part tested on only one target. WASI's +// SPEC.md §3 hashes no permission bit, and Linux is the only target that can +// prove it: it is the only one that has an executable bit to set. WASI's // filestat carries no permission bits and Windows has no POSIX exec bit -// (fs.odin), and hashing there treats every file as non-executable - which is -// the language's answer, not a gap, and is what LANGUAGE.md documents. +// (fs.odin), so the same assertion elsewhere would hold without establishing +// anything - which is why this file is `#+build linux` rather than a `when`. #+build linux package hashedbuild @@ -27,7 +27,7 @@ eval_digest :: proc(t: ^testing.T, path: string) -> string { } @(test) -test_the_executable_bit_is_part_of_a_directory_hash :: proc(t: ^testing.T) { +test_the_executable_bit_is_not_part_of_a_directory_hash :: proc(t: ^testing.T) { root := strings.concatenate({repo_root(), "/.hash_linux_test_exec"}) defer delete(root) file := strings.concatenate({root, "/build.sh"}) @@ -40,10 +40,10 @@ test_the_executable_bit_is_part_of_a_directory_hash :: proc(t: ^testing.T) { testing.expect(t, os.make_directory(root) == nil, "could not create the scratch tree") _ = os.write_entire_file(file, transmute([]u8)string("#!/bin/sh\necho hi\n")) - // Two hashes of the same bytes under the same name, differing only in - // whether the file is a program. §3 asks for exactly that distinction, and - // it is a real one in a build: a checked-out `configure` that lost its bit - // is not the same tree. + // Two hashes of the same bytes under the same name, differing only in whether + // the file is a program. §3 does not distinguish them: only Linux can see the + // difference at all, so hashing it would make one source tree two values + // depending on where it was checked out. cname := strings.clone_to_cstring(file, context.temp_allocator) testing.expect(t, linux.chmod(cname, {.IRUSR, .IWUSR}) == .NONE) plain := eval_digest(t, root) @@ -51,11 +51,23 @@ test_the_executable_bit_is_part_of_a_directory_hash :: proc(t: ^testing.T) { testing.expect(t, linux.chmod(cname, {.IRUSR, .IWUSR, .IXUSR}) == .NONE) executable := eval_digest(t, root) - testing.expect(t, plain != executable, "the exec bit is part of what a directory is") + // That the bit actually landed is half the test - without it the comparison + // would hold for the wrong reason. + dir_fd, oerr := fs_open_dir_path(root) + testing.expect(t, oerr == .None, "could not open the scratch tree") + if oerr == .None { + defer fs_close(dir_fd) + entries, lerr := fs_list_entries_at(dir_fd, context.temp_allocator) + testing.expect(t, lerr == .None) + saw := false + for entry in entries do if entry.name == "build.sh" && entry.is_executable do saw = true + testing.expect(t, saw, "chmod did not set a bit for the digest to ignore") + } - // ...and it is only the owner bit that counts. §3 says "the executable flag - // only - not full POSIX mode", so who else may run it is not part of what - // the file is. + testing.expect_value(t, plain, executable) + + // No other mode bit counts either - the rule is that permissions are not part + // of what a tree is, not that one particular bit was singled out. testing.expect(t, linux.chmod(cname, {.IRUSR, .IWUSR, .IXUSR, .IRGRP, .IROTH}) == .NONE) - testing.expect_value(t, eval_digest(t, root), executable) + testing.expect_value(t, eval_digest(t, root), plain) } From 89f8561ef429ee1abe57257c8f420f1d070ca8d5 Mon Sep 17 00:00:00 2001 From: Jan Strakowski Date: Mon, 31 Aug 2026 14:41:31 +0200 Subject: [PATCH 5/6] Show a cached cycle in the example, not just in the docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Caching a value that reaches itself is a user-visible capability and LANGUAGE.md describes it, but nothing in examples/ ran one. cached.hb now caches a self-referential Table and asserts the digest survives the round trip - which is the whole test, since §6 compares cyclic values by bisimulation, so a back-edge that came back as an unfolding of the wrong depth would not compare equal to what was stored. Co-Authored-By: Claude Opus 5 --- examples/cached.hb | 13 ++++++++++++- src/examples_test.odin | 2 +- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/examples/cached.hb b/examples/cached.hb index 68918fd..9bd5414 100644 --- a/examples/cached.hb +++ b/examples/cached.hb @@ -20,9 +20,13 @@ // HashedBuild text in `sha256-.hb/value.hb`, with any `File` it holds // stored beside it and referred to by name. // +// A value that reaches itself (§10) is written with a label on each repeated +// Table and a back-reference after it - `node "1" { ..., .self = ref "1" }` - +// which is what gives a cycle a finite written form. +// // Evaluates to // { answer: 42, asking_again_agrees: true, per_argument: { small: 2, large: 11 }, -// file_survives_the_round_trip: true }. +// file_survives_the_round_trip: true, a_cycle_survives_too: true }. let answer cached (6 * 7); @@ -38,9 +42,16 @@ let bump func (cached (#arg + 1)); let file_survives_the_round_trip ((sha256 cached (loadfile "optiona.txt")) == (sha256 loadfile "optiona.txt")); +// So does a value that reaches itself. The comparison is the whole test: §6 +// compares cyclic values by bisimulation, so a back-edge that came back as an +// unfolding of the wrong depth would not be equal to what was stored. +let rec ring { .name = "ring", .self = ring }; +let a_cycle_survives_too ((sha256 cached ring) == (sha256 ring)); + { .answer = answer, .asking_again_agrees = asking_again_agrees, .per_argument = { .small = bump 1, .large = bump 10 }, .file_survives_the_round_trip = file_survives_the_round_trip, + .a_cycle_survives_too = a_cycle_survives_too, } diff --git a/src/examples_test.odin b/src/examples_test.odin index 4d27a4e..19f9428 100644 --- a/src/examples_test.odin +++ b/src/examples_test.odin @@ -33,7 +33,7 @@ EXAMPLE_CASES := []Example_Case{ {"async-basics.hb", `"This is the payload for option A.\nThis is the payload for option B.\n"`}, {"async-branching.hb", `"medium"`}, {"async-table.hb", `{a: 2, b: 6, c: "This is the payload for option A.\n"}`}, - {"cached.hb", "{answer: 42, asking_again_agrees: true, per_argument: {small: 2, large: 11}, file_survives_the_round_trip: true}"}, + {"cached.hb", "{answer: 42, asking_again_agrees: true, per_argument: {small: 2, large: 11}, file_survives_the_round_trip: true, a_cycle_survives_too: true}"}, {"check-and-invariants.hb", "100"}, {"comparison-and-logic.hb", "{ordered: true, both: true, either: true, mixed: false}"}, {"context-permissions.hb", "{ambient: {io: nothing}, io_denied: {}, replaced: {}, still_ambient: {io: nothing}}"}, From 23bee63f9fce80ec52d2160ed63f5bc5b9b04d9a Mon Sep 17 00:00:00 2001 From: Jan Strakowski Date: Mon, 31 Aug 2026 14:44:00 +0200 Subject: [PATCH 6/6] Make `cached` work on WASI: rename rights, and the playground's shim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two failures CI caught and this machine cannot, both from the four fs write operations the cache needs. **wasi-threads**: `cached.hb` died with "could not publish cache entry (Access)" under iwasm. preview1 checks a rename against two separate rights — PATH_RENAME_SOURCE on the source descriptor and PATH_RENAME_TARGET on the destination's — and DIR_RIGHTS asked for neither, so every commit-by-rename came back ENOTCAPABLE, which fs.odin folds into a bare .Access. **playground**: the browser refused to instantiate the module at all — `LinkError: "path_rename": function import requires a callable`. docs/wasi.js implements preview1 for the playground, and nothing had needed to remove or rename anything before, so three imports simply weren't there. Added path_rename, path_unlink_file and path_remove_directory, plus the FileSystem `remove`/`rename` they sit on, plus the arity entries a spawned thread marshals them through. `rename` refuses an existing destination, which is what makes it the cache's commit rather than an overwrite. ENOTEMPTY (55) joins the ERRNO table, since an rmdir of a non-empty directory now has a way to say so. Co-Authored-By: Claude Opus 5 --- docs/wasi.js | 57 +++++++++++++++++++++++++++++++++++++++++++++++- src/fs_wasi.odin | 6 +++++ 2 files changed, 62 insertions(+), 1 deletion(-) diff --git a/docs/wasi.js b/docs/wasi.js index 609180c..317ec88 100644 --- a/docs/wasi.js +++ b/docs/wasi.js @@ -8,7 +8,7 @@ const ERRNO = { SUCCESS: 0, ACCESS: 2, BADF: 8, EXIST: 20, INVAL: 28, IO: 29, - ISDIR: 31, NOENT: 44, NOTDIR: 54, NOTCAPABLE: 76, + ISDIR: 31, NOENT: 44, NOTDIR: 54, NOTEMPTY: 55, NOTCAPABLE: 76, }; const FILETYPE = { UNKNOWN: 0, DIRECTORY: 3, REGULAR_FILE: 4, SYMBOLIC_LINK: 7 }; @@ -103,6 +103,36 @@ export class FileSystem { return true; } + // The three below exist for SPEC.md §15's cache, which builds an entry under + // a temporary name and renames it into place. Nothing else in the playground + // removes or renames anything. + + remove(path, wantDir) { + const { parent, name } = this.parentOf(path); + if (!parent || !name) return "NOENT"; + const node = parent.entries.get(name); + if (!node) return "NOENT"; + const isDir = node.type === "dir"; + if (isDir !== wantDir) return isDir ? "ISDIR" : "NOTDIR"; + if (isDir && node.entries.size > 0) return "NOTEMPTY"; + parent.entries.delete(name); + return null; + } + + // Deliberately refuses an existing destination, which is what makes it the + // cache's commit: a name already taken means another writer got there first. + rename(from, to) { + const src = this.parentOf(from); + const dst = this.parentOf(to); + if (!src.parent || !src.name || !dst.parent || !dst.name) return "NOENT"; + const node = src.parent.entries.get(src.name); + if (!node) return "NOENT"; + if (dst.parent.entries.has(dst.name)) return "EXIST"; + src.parent.entries.delete(src.name); + dst.parent.entries.set(dst.name, node); + return null; + } + list() { const snapshot = this.snapshot(); return Object.keys(snapshot).sort().map((path) => ({ path, ...snapshot[path] })); @@ -436,6 +466,28 @@ export class WASI { return self.fs.symlink(path, target) ? ERRNO.SUCCESS : ERRNO.NOENT; }, + path_unlink_file(dirfd, pathPtr, pathLen) { + const path = self.resolve(dirfd, self.readString(pathPtr, pathLen)); + if (path === null) return ERRNO.ACCESS; + const err = self.fs.remove(path, false); + return err ? (ERRNO[err] ?? ERRNO.IO) : ERRNO.SUCCESS; + }, + + path_remove_directory(dirfd, pathPtr, pathLen) { + const path = self.resolve(dirfd, self.readString(pathPtr, pathLen)); + if (path === null) return ERRNO.ACCESS; + const err = self.fs.remove(path, true); + return err ? (ERRNO[err] ?? ERRNO.IO) : ERRNO.SUCCESS; + }, + + path_rename(dirfd, fromPtr, fromLen, toFd, toPtr, toLen) { + const from = self.resolve(dirfd, self.readString(fromPtr, fromLen)); + const to = self.resolve(toFd, self.readString(toPtr, toLen)); + if (from === null || to === null) return ERRNO.ACCESS; + const err = self.fs.rename(from, to); + return err ? (ERRNO[err] ?? ERRNO.IO) : ERRNO.SUCCESS; + }, + path_readlink(dirfd, pathPtr, pathLen, bufPtr, bufLen, usedPtr) { const path = self.resolve(dirfd, self.readString(pathPtr, pathLen)); if (path === null) return ERRNO.ACCESS; @@ -591,7 +643,10 @@ export const REMOTE_CALLS = [ ["path_filestat_get", "iiiii"], ["path_open", "iiiiiIIii"], ["path_readlink", "iiiiii"], + ["path_remove_directory", "iii"], + ["path_rename", "iiiiii"], ["path_symlink", "iiiii"], + ["path_unlink_file", "iii"], ["random_get", "ii"], ]; diff --git a/src/fs_wasi.odin b/src/fs_wasi.odin index 115177c..97ff0ba 100644 --- a/src/fs_wasi.odin +++ b/src/fs_wasi.odin @@ -40,6 +40,12 @@ DIR_RIGHTS :: wasi.rights_t{ .FD_READDIR, .FD_FILESTAT_GET, .PATH_OPEN, .PATH_CREATE_FILE, .PATH_CREATE_DIRECTORY, .PATH_FILESTAT_GET, .PATH_READLINK, .PATH_SYMLINK, .PATH_UNLINK_FILE, .PATH_REMOVE_DIRECTORY, + // Both halves of a rename, for §15's cache: an entry is built under a + // temporary name and renamed into place, and preview1 checks the source + // descriptor for one right and the destination's for the other. Asking for + // only one gets ENOTCAPABLE, which surfaces as a bare "Access" - which is + // exactly what the WASI smoke test caught. + .PATH_RENAME_SOURCE, .PATH_RENAME_TARGET, } @(private = "file")