diff --git a/GETTING_STARTED.md b/GETTING_STARTED.md index 7518586..6942f39 100644 --- a/GETTING_STARTED.md +++ b/GETTING_STARTED.md @@ -52,7 +52,7 @@ If a freshly built binary refuses to start with *"An Application Control policy - **`./hb`** (no arguments) - a line-based REPL. Type an expression, then an empty line to evaluate it; `:q` to quit. - **`./hb -a path/to/program.hb`** - print the full AST before evaluating. Works with `-e` too. - **`./hb -i`** - the live terminal editor (needs a real terminal, not a pipe). -- **`./hb --cache-dir ...`** - override where `ctx.cache` writes to (defaults to your XDG cache dir; `%LOCALAPPDATA%\hashedbuild` on Windows). +- **`./hb --cache-dir ...`** - override where `ctx.cache` writes to, and where `cached` keeps its entries (defaults to your XDG cache dir; `%LOCALAPPDATA%\hashedbuild` on Windows). Handy for a throwaway cache: point it somewhere temporary and `cached` starts from nothing. - **`./hb --help`**, **`./hb --version`** - usage and version. ## Try each part of the video diff --git a/LANGUAGE.md b/LANGUAGE.md index 7b4fe3c..7d8b960 100644 --- a/LANGUAGE.md +++ b/LANGUAGE.md @@ -382,6 +382,8 @@ digest of the value it evaluates to, base64-encoded as `Utf8`: sha256 "hello" // => "Ar9oHTBiuRDqs+ZdbYD2daaU7RcvIDTJNB3UICNP92A=" sha256 loadfile "pkg.tar.gz" // exactly what sha256sum reports for that file sha256 { .a = 1, .b = 2 } +sha256 loadfile "src" // a whole directory, hashed over its entries +sha256 func (1 + 2) // a function, hashed as code plus what it captures ``` This is not a checksum utility bolted on the side — it is the value identity @@ -412,10 +414,9 @@ same value when their bytes match, however they were reached. A **directory** `File` hashes over its entries (§3), sorted by name so the digest is the tree's rather than the order the filesystem listed it in. Each -entry contributes its name, plus its content hash and executable bit for a -regular file, its own directory hash for a sub-directory, or its target string -for a symlink — which is never followed, so a link to a directory is a link, -not a directory. +entry contributes its name, plus its content hash for a regular file, its own +directory hash for a sub-directory, or its target string for a symlink — which +is never followed, so a link to a directory is a link, not a directory. ```hashedbuild sha256 loadfile "examples" // the digest of a whole tree @@ -431,13 +432,13 @@ Two things about it are worth knowing before you rely on it: revoked `io` can't pull a tree's contents through a handle it was handed. It happens once: a `File` is an immutable handle, so the digest is fixed from then on, and seeing a change means loading the directory again. -- **The executable bit is Linux-only, and a tree containing one hashes - differently elsewhere.** WASI's `filestat` has no permission bits and Windows - has no POSIX executable bit, so on those targets every file hashes as - non-executable. That is the language's answer rather than a gap — a Windows - checkout genuinely has no executable bits, and reporting one would be - inventing it, the same position git takes with `core.filemode`. If you need a - digest that agrees across all three, keep executables out of the tree. +- **No permission bits go into it**, so a tree hashes the same wherever it was + checked out. Only Linux can report an executable bit, and hashing one would + have made the same source tree two different values depending on the machine + — git makes that concrete, since `core.fileMode=false` on Windows carries a + committed `100755` through the repository without the bit ever existing in + the working tree. (`cached` still restores the bit when it copies a tree; that + is the store being faithful, not the bit being part of the value.) An entry that is neither a file, a directory, nor a symlink — a socket, a device node — fails rather than being skipped, since a digest that ignored part @@ -489,6 +490,70 @@ one value. → `examples/hashing-cyclic.hb` (§6, §10, §15) +## Caching + +`cached` takes one trailing expression, like `sha256`, and evaluates it once: +every later run reads the answer back instead of computing it. + +```hashedbuild +cached (6 * 7) // => 42, and again on every later run +cached (loadfile "pkg.tar.gz") // the bytes as they were the first time +``` + +**The key is the expression treated as a function**, hashed with the one hash +system above (§15). Three things go into it, and it is worth knowing which: + +- **the code**, structurally — so reformatting an expression or writing a + comment inside it does not throw its entry away; +- **the `ctx` it runs under** (§9), whole; +- **the values it reads** — the names it uses, and anything it reaches through + `#arg`/`#self`. So `let bump func (cached (#arg + 1))` gets one entry per + argument, not one entry. + +What is deliberately *not* in the key is anything the expression goes and reads +at run time. That is the point of a cache, and its one sharp edge: cache +`loadfile "pkg.tar.gz"` and you keep getting the bytes from the first run, +however the file changes afterwards. + +**Where entries live.** In `ctx.cache`'s directory — `--cache-dir `, else +`$XDG_CACHE_HOME/hashedbuild`, else the per-user default — one entry per key, +named `sha256-`: + +``` +/sha256- a File value, stored as itself +/sha256-.hb/ anything else +/sha256-.hb/value.hb the value, as HashedBuild text +/sha256-.hb/sha256- each File inside it, by content hash +``` + +A `File` value stays a file and a directory value stays a directory, so what a +build produced is still something you can open, `diff` or copy out. Anything +else is written as text you can read; the `File`s it holds cannot go in text, +so they are stored beside it and referred to by name. A value that reaches +itself is written with a label on each repeated Table and a back-reference +after it, which is what lets a cycle have a finite written form: + +``` +node "1" { .name = "alice", .friend = node "2" { .name = "bob", .friend = ref "1" } } +``` Entries are built under a +temporary name and renamed into place, so an interrupted run leaves a `.tmp` +rather than a half-written entry, and two runs racing on one key settle it +without locking anything. + +**What it refuses.** `cached` needs `ctx.permissions.io`, since it reads and +writes files, and it needs a `ctx` that still carries `.cache`. A value holding +a `Function`, `ctx.cache`, or an un-awaited `async` handle cannot be written +down and read back as itself, so caching one fails rather than storing +something that would come back different. All of these are fatal failures like +any other (§8). + +`async` is positional, and the two placements mean different things: +`async cached ` makes the cache lookup itself asynchronous, while +`cached async ` runs the expression on a thread with the caching wrapper +around it synchronous. + +→ `examples/cached.hb` (§15) + ## Context and permissions `ctx` is the ambient context. The filesystem builtins check @@ -508,7 +573,10 @@ is the entire point. `ctx.cache` is a write-only, content-addressed store: `createfile { .dir = ctx.cache, .content = … }` writes under the content's own hash, deduplicating -across runs, and returns the `File` it wrote. +across runs, and returns the `File` it wrote. Its directory is also where +`cached` keeps its entries (see "Caching"); the two are told apart by their +names, `sha256_` for a blob written this way and `sha256-` for a +cache entry. → `examples/context-permissions.hb`, `examples/option-picker.hb` (§9, §16) @@ -541,14 +609,12 @@ uncatchable, but the work already in flight finishes first. ## What isn't built yet Parsed, specified, and rejected by the evaluator with "not implemented": -`import` and `cached`. +`import`. **Hashing is complete**: `sha256` answers for every value, including the three that used to be listed here — a directory `File`, a `Function`, and a cyclic value — plus `ctx.cache`. See the Hashing section above for what each of them -encodes, and for the one place the answer is target-specific: a directory -containing an executable hashes differently on Windows and WASI than on Linux, -because neither of those has an executable bit to report. +encodes. A given value has the same digest on every target. Also absent: `true`/`false` literals, loops of any kind (recursion is the only repetition there is — see above), a `Bytes`-returning counterpart to diff --git a/SPEC.md b/SPEC.md index 1e7ad20..d835afa 100644 --- a/SPEC.md +++ b/SPEC.md @@ -42,26 +42,34 @@ A condition (of `then`, `and`, `or`, `is`) can itself be, or contain, an async e **Hashing / equality / ordering.** A `File`'s identity is **pure content, independent of path.** Where it was read from is how the value was obtained, not part of the value itself — two `File`s built from different paths are equal whenever their content matches. Ordering follows the same generic total-order mechanism as every other value (§6), keyed off this hash. -- A **regular file**'s hash is just `hash(content_bytes)`. Its own permission bits (e.g. executable) are *not* part of a bare `File` value's identity — permissions only become relevant as metadata about how the file sits inside a directory (next point). +- A **regular file**'s hash is just `hash(content_bytes)`. Its own permission bits are *not* part of its identity — and, as of 2026-08-31, not part of a directory entry's either (next point). - A **directory**'s hash is computed over its entries, sorted by name for determinism (independent of filesystem readdir order): ``` dir_entry_hash(name, entry) = - hash(name, "file", content_hash, is_executable) // executable flag only — not full POSIX mode - hash(name, "dir", child_dir_hash) // directories carry no exec bit - hash(name, "symlink", target_path_string) // target is NOT followed/resolved + hash(name, "file", content_hash) // no permission bits, of any kind + hash(name, "dir", child_dir_hash) + hash(name, "symlink", target_path_string) // target is NOT followed/resolved dir_hash = hash(sorted [dir_entry_hash(name, entry) for each entry]) ``` A symlink entry hashes the link itself (its target path string) rather than resolving through it — consistent with symlink handling being a property of the containing directory, not a standalone `File` value in its own right. An entry that is none of the three — a fifo, a socket, a device node — has no encoding here, and hashing a directory containing one fails rather than skipping it: a digest that ignored part of a tree would call two different trees the same value. - **The executable bit where it cannot be observed (resolved 2026-08-31).** `is_executable` above is the entry's owner-execute bit *where the host reports one*, and **false wherever it does not**. WASI's `filestat` carries no permission bits at all and Windows has no POSIX executable bit, so on those two targets every regular file hashes as non-executable. The consequence is deliberate and worth stating plainly: a tree containing an executable has a different `dir_hash` on Linux than in a browser or on Windows. The alternative considered was dropping the bit from the encoding entirely, which would make one digest hold everywhere at the cost of a distinction a build language wants — a checked-out `configure` that lost its bit is not the same tree. Reporting the bit as *set* where it cannot be seen was never on the table; a Windows checkout genuinely has no executable bits, so claiming one would be inventing data. This is the same position git takes with `core.filemode`. - **When the entries are read (resolved 2026-08-31).** A directory `File`'s children live on the filesystem rather than in the value, so unlike every other digest this one performs I/O. It happens **at the first demand** — the first `sha256` or comparison that needs the digest — and is subject to `ctx.permissions.io` (§9) at that moment, exactly as `loadfile` is: a context that has revoked `io` cannot read a tree through a handle it was passed. The result is then part of the value and is never re-read, which is what `File` being an immutable handle (above) requires: a value whose digest changed under a program because someone touched the tree would not be one. Observing a change means loading the directory again, which produces a new value. + **No permission bits (resolved 2026-08-31, reversing an earlier resolution of the same day).** This section first kept an executable flag alongside a file entry's content — "executable flag only, not full POSIX mode" — and the directory hash was built that way, with the bit read where a target could see one and false elsewhere. It is gone. No permission bit of any kind is part of a value's identity. + + The reason is that only Linux can report one. WASI's `filestat` carries no permission bits at all, and Windows has no POSIX execute bit; neither can be made to answer, and neither can be given a stand-in (an `.exe` extension is not the same question, and what a POSIX emulation layer guesses from a shebang is a guess). Hashing the bit therefore made **one tree two values depending on where it was checked out** — not hypothetically: git records exactly this one bit (`100644` vs `100755`) and, on Windows, sets `core.fileMode=false` so the bit round-trips through the repository without ever existing in the working tree. The same commit checks out executable on Linux and not on Windows. This repository has four such files, so its own `scripts/` directory would have hashed two ways. + + The alternative would have been to *remember* the bit rather than re-derive it, which is what git does. That is not available here: a `File` is a handle onto a live directory, with no index alongside it to consult. Given a choice between a digest that disagrees across targets and one that ignores a bit two of the three cannot see, this takes the second. Two trees differing only in an executable bit are one value, and the platform a build runs on stops changing what its inputs hash to. + + Implementations may still *preserve* the bit when copying a tree — §15's cache does, so that caching a build output does not silently strip it. That is fidelity in the store, not identity in the language. + + What tipped it was looking at how git behaves rather than at what it records. The first resolution cited `core.filemode` as agreeing; it does the opposite, because git never re-derives the bit on a target that cannot report one — it carries the one from the index, which is a move this language has no equivalent of. + **Display (resolved 2026-08-27).** A `File` displays **the path it was reached by**, made absolute and cleaned of `.`/`..` segments — recorded when the value is constructed, not resolved through the filesystem afterwards. Two consequences worth stating: a symlinked route displays as the route taken rather than the target it resolves to, and a file renamed after it was loaded still displays the path it was loaded from. This is what makes the rule implementable off Linux at all — WASI (and the portable `*at()` family) offer no way to turn an open descriptor back into a path. Printing/displaying a `File` — in the REPL, the live editor's result pane, or anywhere else a value gets shown to a human — shows its actual filesystem path. This holds for **every** `File` value, regardless of how it was obtained (`loadfile`, `createfile`, `symlink`'s containing directory, etc.), not just the ones `createfile` writes into `ctx.cache` (§16) — that case is simply the one where a path is otherwise unreachable, so it's the one worth calling out explicitly there. The only exception is `ctx.cache` itself: a distinct, "magic" pseudo-directory type (§16) that is *not* a `File` and has no path of its own to show. Path stays display-only either way — there is still no builtin that lets HashedBuild source read a `File`'s path back out as a `Utf8` value; this is purely about what a human sees when a value is printed, not a new capability for programs. -> TODO: full POSIX mode bits (owner/group/other rwx) are deliberately excluded as noise — confirm this holds up in practice, or whether some other bit besides "executable" ever needs to round-trip through a build. +> TODO: no permission bit is part of a value's identity at all as of 2026-08-31 (above), which resolves the old form of this question — "is the executable bit enough, or does some other mode bit need to round-trip" — by removing the one bit that was in. What is left open is whether a build ever genuinely needs a permission to be part of *identity* rather than merely preserved by the store; if one does, it needs an answer for the two targets that cannot see one, which is what sank the executable bit. **Numeric literals** (proposed 2026-08-26, loosely modeled on C but simplified — no type-width suffixes needed, since there's exactly one `Integer` width and one `Float` width): @@ -260,7 +268,7 @@ Added 2026-08-26, alongside `ctx`/`withctx` above and the filesystem builtins ( - **`ctx.permissions`** is a `Table` used as a *set*: it conventionally holds only `nothing` as every value, and a permission is granted iff its key is **present** (not iff its value is "truthy," since `Nothing` has no such notion) — e.g. `ctx.permissions.io` being present at all, regardless of its (always-`nothing`) value, means I/O is allowed. This works because, unlike Lua, a HashedBuild `Table` (§5/§6) genuinely distinguishes "key absent" from "key present with value `nothing`" — the exact distinction a set-of-flags idiom needs, and Lua's conflation of nil-valued and absent keys can't express. - **The root context** — active at the very start of a program, before any `withctx` — starts with `{ .permissions = { .io = nothing }, .cache = }`: I/O is allowed by default. `withctx` is how you *narrow* permissions around a sub-computation (e.g. before calling `import`ed code, §13), not how you grant them from nothing. Since `withctx` replaces the context wholesale (§7), a program that narrows permissions via a hand-built `Table` rather than `ctx concat {...}` loses `.cache` too unless it explicitly carries it over. - **Builtins read `ctx` live, not captured — the one deliberate exception to the closure-capture rule above.** The capture rule protects a function *from* a caller trying to grant it more authority than it was made with; a builtin like `loadfile` (§16) needs the opposite property — it must see whatever `ctx` is *actually* active at its call site, so that wrapping a call in `... withctx (ctx concat { .permissions = empty })` genuinely denies it from the outside. If builtins captured `ctx` at (interpreter-startup) creation time instead, `withctx` could never restrict them at all. -- **`ctx.cache`** (added 2026-08-27) is its own type, distinct from `File` — see §16 for the full write-up. It's "accepted as a directory" (usable as `createfile`'s `.dir`) without actually being one: it can't be read from, traversed, or passed to `loadfile`/`symlink`/`readlink`, only written to via `createfile`. +- **`ctx.cache`** (added 2026-08-27) is its own type, distinct from `File` — see §16 for the full write-up. It's "accepted as a directory" (usable as `createfile`'s `.dir`) without actually being one: it can't be read from, traversed, or passed to `loadfile`/`symlink`/`readlink`, only written to via `createfile`. **It does hash** (resolved 2026-08-31), as a domain-separated constant over an empty payload: it is write-only, unnamed and unlistable, so there is nothing about one cache that distinguishes it from another. This became necessary rather than optional once §15's cache key was defined to include the whole `ctx`, which contains it. Hashing it as its *directory path* was rejected for a concrete reason: the path would then be baked into every key, so moving or copying a cache directory would invalidate everything in it. ## 10. Name scopes / bindings @@ -347,7 +355,33 @@ Two things follow, and are the point rather than a limitation. Two closures hash **`cached`'s mechanism (resolved 2026-08-26):** the cached expression is treated *as a function* and hashed as one — the cache key is the hash (per the one system above) of that function representation, not a hash of its resolved output value. `cached` is not inherently async by itself; async-ness is controlled explicitly by *where* `async` (§2) is placed: `async cached ` makes the cache lookup/store itself asynchronous, while `cached async ` instead makes the underlying expression's own evaluation asynchronous, with the caching wrapper around it synchronous. -> TODO: Where does the cache actually live (on-disk location, process-local vs. shared/distributed) — not addressed by this round's resolution of the cache-*key* mechanism. +**What a function hashes as (resolved 2026-08-31).** §6 said every value is hashable and this section relied on it, but the encoding of a closure was never given. It is three things mixed together: + +- **the code** — the expression's syntax tree, encoded structurally: node kinds, plus the source text of leaves only. Reformatting an expression, or writing a comment inside it, therefore does not change the key. How a literal is *spelled* does (`1_000` and `1000` are different text), which splits an entry rather than merging two, and is left that way deliberately. +- **the captured `ctx`** (§9), whole. +- **the values of the free names the code uses**, each paired with its name. Collection is deliberately conservative: a name the expression shadows locally is still included if a binding of that name exists outside. Over-collecting only splits one entry into two; under-collecting would return a wrong value. + +A native builtin hashes as the name it is bound to plus whatever it captured — a proc address is not stable between runs. A function that reaches itself (`let rec`) hashes its second occurrence as a back-reference, counted from the innermost enclosing one, so recursion terminates and two identically-written recursive functions still agree. + +**`#arg`/`#self` are covered separately (resolved 2026-08-31).** They are dynamic lookups into their own stacks (§9) and are captured by no closure, so the function hash above cannot see them — which would make `let f func (cached (#arg + 1))` one entry for every argument, answering `f 10` with `f 1`'s result. `cached` therefore mixes in the stack entries the expression can reach, on top of the function hash. How far it can reach is bounded statically by the largest `N` written anywhere in it (a bare `#arg` or an omitted operand counting as 1); that bound holds through functions the expression itself calls, since entering one pushes a frame. An expression mentioning no implicit name is unaffected. + +**Where the cache lives (resolved 2026-08-31, replacing this section's TODO).** On disk, in `ctx.cache`'s directory (§16 — `--cache-dir`, else `$XDG_CACHE_HOME/hashedbuild`, else the per-user fallback). Process-local in the sense that nothing coordinates between machines; shared in the sense that the directory is an ordinary one, and two processes on it are safe. One entry per key, in one of two shapes: + +``` +/sha256- the value is a File — stored as itself +/sha256-.hb/ anything else +/sha256-.hb/value.hb the value, written as HashedBuild text +/sha256-.hb/sha256- each File inside that value +``` + +- **A `File` value is stored as a file, a directory value as a directory.** Not wrapped, not encoded: what a build produced stays something a person can open, `diff`, or copy out, which is most of the point of a content-addressed store. A directory is copied faithfully enough to hash as the original did (§3) — names, contents, and symlink targets stored without being followed — plus the executable bit where the target has one, which §3 does *not* hash but which a store has no reason to throw away. +- **Anything else is written as text**, in a subset of HashedBuild's own syntax: literals, tables, and two names the language has no literal for (`true`/`false`, and `bytes "…"`). Since text cannot hold a file, each `File` inside the value is written out beside it, named by its own content hash (§3), and referred to from the text as `file "…"` or `dir "…"` — systematically, however deeply nested. Reading it back is a separate reader that accepts literals and nothing else, not `import`: a hand-edited entry is a parse failure rather than code that runs. +- **A value that reaches itself** (§10) is written with a label: a Table occurring more than once is `node "N" { … }` at its first occurrence and `ref "N"` after that. A definition always precedes its references, since the label is assigned in the same walk that emits it, so nothing has to be patched up on the way back in — the reader creates each Table before reading its entries, exactly as §10's evaluation order does. Labels are given to a merely *shared* Table too; that is not needed for correctness, since §6 compares structurally, but it keeps a shared value from being written out exponentially. A restored cycle is bisimulation-equal to the stored one, which is what §6 requires of it. +- **`` is the cache key**, base64url without padding, since a lookup has nothing else to go on. The `-` separator distinguishes these from the `sha256_` blobs `createfile { .dir = ctx.cache }` writes into the same directory (§16). +- **Entries are committed by rename.** Each is built under a temporary name and renamed into place, so an interrupted run leaves a stray temporary rather than a truncated entry that a later run would read as a hit. The rename is also how two runs racing on one key settle it — the loser removes its temporary and reads the winner's entry, which holds the same value, since the key is the same. Nothing is overwritten and nothing is locked. +- **A store reads its own entry back** rather than returning the value it just computed, so the first run and every later one return the same value — `File`s included, displaying the cache's paths (§3) rather than wherever that particular run built them. + +**Gating and failure.** `cached` requires `ctx.permissions.io` (§9), like §16's builtins and for the same reason: it reads and writes files. It also requires a `ctx` that still carries `.cache`, which a hand-built one need not. A value holding a `Function`, `ctx.cache`, or an un-awaited `async` handle has no written form and cannot be cached — a cyclic value *can* be, per the previous point. A present-but-unreadable entry is a failure, deliberately rather than a miss — silently recomputing over a corrupt cache would hide the corruption for as long as the cache lived. All of these are fatal (§8). ## 16. Filesystem builtins @@ -372,7 +406,7 @@ The four filesystem operations below are gated by [`ctx.permissions.io`](#9-impl Added 2026-08-27. A write-only, content-addressed blob store, its own type (distinct from `File`) even though `createfile`'s `.dir` accepts it exactly like a directory handle — reading, traversal, and `loadfile`/`symlink`/`readlink` all refuse it, since there's no meaningful name to look anything up by (see below). Gated by `io` like the other filesystem operations. - **Location.** Resolved once per program run, in order: the CLI's `--cache-dir ` if given; otherwise `$XDG_CACHE_HOME/hashedbuild`; otherwise `$HOME/.cache/hashedbuild`. The directory (and any missing ancestor) is created lazily, on the first actual write — a program that never touches `ctx.cache` never creates it. -- **Naming.** Every entry is stored under `sha256_` — the name *is* the content's own hash, computed by the write itself. This is why "names of the children don't matter": whatever the caller might otherwise think to call an entry is irrelevant, since the store assigns the name, not the caller. +- **Naming.** Every entry is stored under `sha256_` — the name *is* the content's own hash, computed by the write itself. This is why "names of the children don't matter": whatever the caller might otherwise think to call an entry is irrelevant, since the store assigns the name, not the caller. The underscore is load-bearing as of 2026-08-31: §15's `cached` keeps its entries in the same directory, named `sha256-` with a hyphen, and the separator is what tells a blob written here from a cache entry written there. - **Writing.** `createfile { .dir = ctx.cache, .content = }` — note there's no `.path`; it wouldn't mean anything, since the name isn't caller-chosen. Writing content whose hash already has an entry on disk (from this run or an earlier one) **dedupes**: the existing entry is reused as-is, silently, rather than failing like an ordinary exclusive `createfile` or writing a redundant duplicate — this is what makes it a cache across runs, not just a one-shot content dump. - **The returned `File` is real, and displays like any other.** `createfile`'s result here is an ordinary `File` (§3, not the `ctx.cache` type itself) — per §3's general display rule, printing it (e.g. in the REPL or the live editor's result pane) shows its actual absolute path on disk, same as any non-magic `File`. But there's still no builtin that lets HashedBuild source read that path back out as a `Utf8` value. The program can hold the handle and pass it around, but it never learns *where* its data physically landed; only a human inspecting the program's output does. -- **Not searchable.** There's no `loadfile`/`symlink`/`readlink` counterpart for the cache - `.dir = ctx.cache` is only ever accepted by `createfile`. A program can't ask "is this content already cached?" directly; it can only write and let the store's own dedup decide. +- **Not searchable.** There's no `loadfile`/`symlink`/`readlink` counterpart for the cache - `.dir = ctx.cache` is only ever accepted by `createfile`. A program can't ask "is this content already cached?" directly; it can only write and let the store's own dedup decide. §15's `cached` does read entries back out of the same directory, but not through `ctx.cache`: it is its own syntax with its own keys, and shares nothing with this type beyond the directory the two write into. diff --git a/docs/wasi.js b/docs/wasi.js index 609180c..317ec88 100644 --- a/docs/wasi.js +++ b/docs/wasi.js @@ -8,7 +8,7 @@ const ERRNO = { SUCCESS: 0, ACCESS: 2, BADF: 8, EXIST: 20, INVAL: 28, IO: 29, - ISDIR: 31, NOENT: 44, NOTDIR: 54, NOTCAPABLE: 76, + ISDIR: 31, NOENT: 44, NOTDIR: 54, NOTEMPTY: 55, NOTCAPABLE: 76, }; const FILETYPE = { UNKNOWN: 0, DIRECTORY: 3, REGULAR_FILE: 4, SYMBOLIC_LINK: 7 }; @@ -103,6 +103,36 @@ export class FileSystem { return true; } + // The three below exist for SPEC.md §15's cache, which builds an entry under + // a temporary name and renames it into place. Nothing else in the playground + // removes or renames anything. + + remove(path, wantDir) { + const { parent, name } = this.parentOf(path); + if (!parent || !name) return "NOENT"; + const node = parent.entries.get(name); + if (!node) return "NOENT"; + const isDir = node.type === "dir"; + if (isDir !== wantDir) return isDir ? "ISDIR" : "NOTDIR"; + if (isDir && node.entries.size > 0) return "NOTEMPTY"; + parent.entries.delete(name); + return null; + } + + // Deliberately refuses an existing destination, which is what makes it the + // cache's commit: a name already taken means another writer got there first. + rename(from, to) { + const src = this.parentOf(from); + const dst = this.parentOf(to); + if (!src.parent || !src.name || !dst.parent || !dst.name) return "NOENT"; + const node = src.parent.entries.get(src.name); + if (!node) return "NOENT"; + if (dst.parent.entries.has(dst.name)) return "EXIST"; + src.parent.entries.delete(src.name); + dst.parent.entries.set(dst.name, node); + return null; + } + list() { const snapshot = this.snapshot(); return Object.keys(snapshot).sort().map((path) => ({ path, ...snapshot[path] })); @@ -436,6 +466,28 @@ export class WASI { return self.fs.symlink(path, target) ? ERRNO.SUCCESS : ERRNO.NOENT; }, + path_unlink_file(dirfd, pathPtr, pathLen) { + const path = self.resolve(dirfd, self.readString(pathPtr, pathLen)); + if (path === null) return ERRNO.ACCESS; + const err = self.fs.remove(path, false); + return err ? (ERRNO[err] ?? ERRNO.IO) : ERRNO.SUCCESS; + }, + + path_remove_directory(dirfd, pathPtr, pathLen) { + const path = self.resolve(dirfd, self.readString(pathPtr, pathLen)); + if (path === null) return ERRNO.ACCESS; + const err = self.fs.remove(path, true); + return err ? (ERRNO[err] ?? ERRNO.IO) : ERRNO.SUCCESS; + }, + + path_rename(dirfd, fromPtr, fromLen, toFd, toPtr, toLen) { + const from = self.resolve(dirfd, self.readString(fromPtr, fromLen)); + const to = self.resolve(toFd, self.readString(toPtr, toLen)); + if (from === null || to === null) return ERRNO.ACCESS; + const err = self.fs.rename(from, to); + return err ? (ERRNO[err] ?? ERRNO.IO) : ERRNO.SUCCESS; + }, + path_readlink(dirfd, pathPtr, pathLen, bufPtr, bufLen, usedPtr) { const path = self.resolve(dirfd, self.readString(pathPtr, pathLen)); if (path === null) return ERRNO.ACCESS; @@ -591,7 +643,10 @@ export const REMOTE_CALLS = [ ["path_filestat_get", "iiiii"], ["path_open", "iiiiiIIii"], ["path_readlink", "iiiiii"], + ["path_remove_directory", "iii"], + ["path_rename", "iiiiii"], ["path_symlink", "iiiii"], + ["path_unlink_file", "iii"], ["random_get", "ii"], ]; diff --git a/examples/README.md b/examples/README.md index 981eb28..ea744f0 100644 --- a/examples/README.md +++ b/examples/README.md @@ -59,6 +59,7 @@ prose around each of these. | `hashing-directories.hb` | A directory's hash: its entries, and the one digest that reads | | `hashing-functions.hb` | A closure's hash: its body's shape and the values it captures | | `hashing-cyclic.hb` | Hashing a value that reaches itself, canonically | +| `cached.hb` | `cached` — evaluate once, read the answer back on every later run | | `option-picker.hb` | A real program: read, branch, write into `ctx.cache` | | `context-permissions.hb` | `ctx`, `chctx chperm`, `withctx` — capability narrowing | | `async-basics.hb` | Two reads in flight at once, awaited implicitly | diff --git a/examples/cached.hb b/examples/cached.hb new file mode 100644 index 0000000..9bd5414 --- /dev/null +++ b/examples/cached.hb @@ -0,0 +1,57 @@ +// `cached` (SPEC.md §15): evaluate an expression once, and read the answer +// back on every later run. Like `sha256`, `func` and `async`, it is a bare +// keyword prefix taking one trailing expression - no parentheses of its own. +// +// The cache key is the expression *treated as a function*, hashed as one - so +// two things are part of it besides the code: the `ctx` it runs under, and the +// values of the names it uses. `bump 1` and `bump 10` below are therefore two +// entries and not one, which is what keeps the second call from being answered +// with the first call's result. +// +// What is *not* part of the key is anything the expression goes and reads at +// run time. That is the point of a cache and also its one sharp edge: cache +// `loadfile "pkg.tar.gz"` and you get the bytes as they were the first time, +// however the file changes afterwards. +// +// Entries live in `ctx.cache`'s directory (`--cache-dir`, else the per-user +// default), one per key, named `sha256-`. A `File` value is stored as an +// ordinary file and a directory value as an ordinary directory, so what a +// build produced is still something you can open; anything else is written as +// HashedBuild text in `sha256-.hb/value.hb`, with any `File` it holds +// stored beside it and referred to by name. +// +// A value that reaches itself (§10) is written with a label on each repeated +// Table and a back-reference after it - `node "1" { ..., .self = ref "1" }` - +// which is what gives a cycle a finite written form. +// +// Evaluates to +// { answer: 42, asking_again_agrees: true, per_argument: { small: 2, large: 11 }, +// file_survives_the_round_trip: true, a_cycle_survives_too: true }. + +let answer cached (6 * 7); + +// The second ask evaluates nothing: it reads back what the first one wrote. +let asking_again_agrees ((cached (6 * 7)) == 42); + +// One entry per captured value, not one per expression. +let bump func (cached (#arg + 1)); + +// A cached `File` comes back as a `File`, and as the *same* value - a `File` +// is its content (§3), so this holds even though the copy in the cache is at +// a different path than the one it was read from. +let file_survives_the_round_trip + ((sha256 cached (loadfile "optiona.txt")) == (sha256 loadfile "optiona.txt")); + +// So does a value that reaches itself. The comparison is the whole test: §6 +// compares cyclic values by bisimulation, so a back-edge that came back as an +// unfolding of the wrong depth would not be equal to what was stored. +let rec ring { .name = "ring", .self = ring }; +let a_cycle_survives_too ((sha256 cached ring) == (sha256 ring)); + +{ + .answer = answer, + .asking_again_agrees = asking_again_agrees, + .per_argument = { .small = bump 1, .large = bump 10 }, + .file_survives_the_round_trip = file_survives_the_round_trip, + .a_cycle_survives_too = a_cycle_survives_too, +} diff --git a/examples/hashing-directories.hb b/examples/hashing-directories.hb index a093a79..e8324fb 100644 --- a/examples/hashing-directories.hb +++ b/examples/hashing-directories.hb @@ -4,15 +4,15 @@ // // §3 computes it over the directory's entries, each hashed with its name and // sorted by name so the answer is the tree's rather than readdir's: a regular -// file contributes its content hash and whether it is executable, a -// sub-directory contributes its own directory hash, and a symlink contributes -// its target *string*, never followed. Nothing about where the directory sits -// enters into it, which is why two handles on the same tree are one value. +// file contributes its content hash, a sub-directory contributes its own +// directory hash, and a symlink contributes its target *string*, never +// followed. No permission bits enter into it, and nothing about where the +// directory sits does either - which is why two handles on the same tree are +// one value, and why a tree hashes the same on every target. // -// The digests themselves are deliberately not written down here: a tree -// containing an executable hashes differently on Windows and in the browser -// than on Linux, because neither of those targets has an executable bit to -// report (see LANGUAGE.md). What is the same everywhere are the properties +// The digests themselves are still not written down here: this directory's +// contents change as examples are added, so a literal would be a value that +// went stale the next time someone wrote one. What holds are the properties // below. // // Reading a directory is I/O, so the first `sha256` of one needs diff --git a/src/builtins_fs.odin b/src/builtins_fs.odin index 4d4cfa6..bfcd617 100644 --- a/src/builtins_fs.odin +++ b/src/builtins_fs.odin @@ -93,9 +93,10 @@ make_global_env :: proc() -> ^Env { return env } -// Package-visible rather than file-private because hash.odin asks it too: a +// Package-visible rather than file-private because two other places ask it: a // directory File's digest is read off the disk the first time anything needs -// it, and that read is an I/O operation like any other here (SPEC.md §3/§9). +// it (hash.odin), and §15's `cached` reads and writes cache entries +// (eval.odin). Both are I/O operations like any other here (SPEC.md §3/§9). ctx_allows_io :: proc(interp: ^Interpreter) -> bool { t, is_table := interp.current_ctx.(^Table_Value) if !is_table do return false @@ -524,8 +525,16 @@ cache_entry_name :: proc(content: []u8) -> string { // Opens (creating if necessary) the cache's backing directory, the first // time it's actually needed - not at program start, so a program that never -// touches the cache never creates it. -@(private = "file") +// touches the cache never creates it. §15's `cached` shares the directory and +// so calls this too (cache_store.odin), with the same laziness. +// +// Unsynchronised, and safe to leave that way: two `async` branches reaching +// here at once both open the directory and one of the two descriptors is +// dropped on the floor, which costs a descriptor and nothing else - both are +// valid, and both name the same directory. A mutex would be the wrong shape +// anyway, since the store below is already written to be safe against *other +// processes* on the same directory (see cache_store.odin on committing by +// rename), which is the harder case and covers this one. ensure_cache_dir_open :: proc(cache: ^Cache_Value) -> Fs_Error { if cache.opened do return .None fs_make_dirs(cache.dir_path) diff --git a/src/cache_format.odin b/src/cache_format.odin new file mode 100644 index 0000000..86fd953 --- /dev/null +++ b/src/cache_format.odin @@ -0,0 +1,557 @@ +package hashedbuild + +import "core:fmt" +import "core:math" +import "core:strconv" +import "core:strings" + +// The text a `cached` entry's `value.hb` holds, and the reader that turns it +// back into a Value. This is the "subset of HB" half of §15's on-disk layout - +// see cache_store.odin for the layout itself. +// +// **It is written to be read.** A cache directory is something a person opens +// to see what a build produced, so an entry says what it is in the language +// the program was written in: +// +// { .name = "libz", .version = 3, .built = true, .src = file "sha256-Ab3_", .out = dir "sha256-9xQ1" } +// +// **A separate reader, not `import`.** Everything above is HashedBuild syntax +// as it stands - `true`, `bytes`, `file` and `dir` are ordinary identifiers, +// and `file "sha256-Ab3_"` is an ordinary call - but none of those names are +// *bound* to anything, so HashedBuild's own evaluator could not read this back +// even though its parser can read the shape. Reading it here instead is also +// what keeps a cache entry from being able to run anything: this parser +// accepts literals and nothing else, so a tampered-with entry is a parse +// failure rather than code that executes. +// +// **Values that reach themselves** get two more names. `let rec` can build a +// cyclic Table (§10), and a cycle has no finite written form without a way to +// point backwards, so a Table that is reached more than once is written once +// with a label and referred to after that: +// +// node "1" { .name = "alice", .friend = node "2" { .name = "bob", .friend = ref "1" } } +// +// A definition always precedes every reference to it, because the label is +// assigned at the node's first occurrence in the same walk that emits it - so +// the reader never has to patch anything up afterwards. It does have to create +// each Table *before* filling it in, which is exactly how the evaluator builds +// one (§10: the Table is bound to the name before any entry runs). +// +// Labels are also given to a Table that is merely *shared* rather than cyclic. +// Sharing is not observable in the value model - §6 compares structurally - so +// this is not required for correctness; it is what stops a deeply shared value +// from being written out exponentially. +// +// The one spelling that goes beyond HashedBuild's own grammar is `\xNN` inside +// a string, for bytes that have no other escape (HashedBuild's lexer knows +// `\n`, `\t`, `\r`, `\"` and `\\`, and stops a string at a newline). Utf8 +// values can hold such bytes, so the format needs a way to write them. +// +// Round-tripping is the whole contract: a value written here and read back has +// to hash identically (hash.odin), or `cached` would return something that is +// not the value it stored. Floats are the only case where that takes care - +// see write_float. + +// ---- writing ---------------------------------------------------------------- + +// `file_names` maps each File in the value to the entry name it was stored +// under, which cache_store.odin fills in as it writes them out. A File that +// isn't in the map cannot be written, and says so. +cache_format_write :: proc(v: Value, file_names: map[^File_Value]string) -> (text: string, ok: bool, why: string) { + w := Write_Ctx{file_names = file_names} + w.labels = make(map[^Table_Value]int) + w.emitted = make(map[^Table_Value]bool) + defer delete(w.labels) + defer delete(w.emitted) + + // Pass one decides which Tables need a label; pass two writes. Two passes + // rather than one because a label has to be on a node's *first* occurrence, + // and whether a node recurs is only known after the whole value is walked. + seen := make(map[^Table_Value]bool) + defer delete(seen) + plan_labels(v, &seen, &w.labels) + + b: strings.Builder + strings.builder_init(&b) + if why = write_value(&b, v, &w); why != "" { + strings.builder_destroy(&b) + return "", false, why + } + return strings.to_string(b), true, "" +} + +@(private = "file") +Write_Ctx :: struct { + file_names: map[^File_Value]string, + labels: map[^Table_Value]int, // Tables reached more than once, and which label + emitted: map[^Table_Value]bool, // ...and whether the definition has been written yet +} + +// Marks every Table reached more than once. Terminates on a cycle because a +// node already in `seen` is recorded and not descended into again. +@(private = "file") +plan_labels :: proc(v: Value, seen: ^map[^Table_Value]bool, labels: ^map[^Table_Value]int) { + resolved, ok := resolve_forward(v) + if !ok do return + + t, is_table := resolved.(^Table_Value) + if !is_table do return + + if seen[t] { + if _, already := labels[t]; !already do labels[t] = len(labels) + 1 + return + } + seen[t] = true + for entry in t.entries { + plan_labels(entry.key, seen, labels) + plan_labels(entry.value, seen, labels) + } +} + +// Returns "" on success, or a sentence naming what it could not write. The +// reason travels back rather than a bare false because "this value cannot be +// cached" is not actionable on its own - which part of it, and why, is. +@(private = "file") +write_value :: proc(b: ^strings.Builder, v: Value, w: ^Write_Ctx) -> string { + // A resolved back-edge is written as whatever it points at; an unresolved one + // is a Table still being built, which cannot escape into a finished value. + resolved, forward_ok := resolve_forward(v) + if !forward_ok do return "a value still being constructed cannot be cached" + + switch av in resolved { + case Nothing_Value: + strings.write_string(b, "nothing") + + case bool: + strings.write_string(b, av ? "true" : "false") + + case i64: + strings.write_i64(b, av, 10) + + case f64: + write_float(b, av) + + case string: + write_quoted(b, transmute([]u8)av) + + case []u8: + strings.write_string(b, "bytes ") + write_quoted(b, av) + + case ^Table_Value: + if label, labelled := w.labels[av]; labelled { + if w.emitted[av] { + strings.write_string(b, "ref ") + write_quoted(b, transmute([]u8)fmt.tprintf("%d", label)) + return "" + } + w.emitted[av] = true + strings.write_string(b, "node ") + write_quoted(b, transmute([]u8)fmt.tprintf("%d", label)) + strings.write_string(b, " ") + } + strings.write_string(b, "{") + for entry, i in av.entries { + if i > 0 do strings.write_string(b, ",") + strings.write_string(b, " ") + // `.name = v` is the readable form, and exact: in HashedBuild it means + // the literal Utf8 key "name". Anything else needs the general + // `[key] = v` form, since a key can be any value at all. + if key, is_str := entry.key.(string); is_str && is_identifier_shaped(key) { + strings.write_string(b, ".") + strings.write_string(b, key) + } else { + strings.write_string(b, "[") + if why := write_value(b, entry.key, w); why != "" do return why + strings.write_string(b, "]") + } + strings.write_string(b, " = ") + if why := write_value(b, entry.value, w); why != "" do return why + } + strings.write_string(b, len(av.entries) > 0 ? " }" : "}") + + case ^File_Value: + name, found := w.file_names[av] + if !found { + // The caller writes every File out before asking for the text, so this + // is a bug here rather than anything the program did. + return fmt.tprintf("a %s in the value was not written to the cache first", + av.kind == .Directory ? "directory" : "file") + } + strings.write_string(b, av.kind == .Directory ? "dir " : "file ") + write_quoted(b, transmute([]u8)name) + + // The three that have no written form. A closure's meaning is its + // environment, ctx.cache has no content of its own, and an un-awaited handle + // is a running thread - none of them is a thing that can be written down and + // read back as itself. + case ^Function_Value: + return "a Function cannot be cached - a closure's meaning is its environment" + case ^Cache_Value: + return "ctx.cache cannot be cached - it is write-only and has no content" + case ^Async_Handle: + return "an un-awaited async handle cannot be cached" + + case ^Forward_Ref_Value: + // resolve_forward above returns the ref itself only when it is unresolved, + // and that case already returned. Unreachable, and spelled out rather than + // left to a #partial switch so a new Value case fails to compile here. + return "a value still being constructed cannot be cached" + } + return "" +} + +// Shortest decimal that reads back as the same f64, checked rather than +// assumed: if the short form doesn't round-trip, 17 significant digits always +// does. A float that came back a hair different would hash differently +// (hash.odin encodes the IEEE bits), so `cached` would hand back a value that +// isn't the one it stored. +// +// The result is also shaped as a HashedBuild Float literal, which is stricter +// than strconv's output: a digit is required on both sides of the `.` (§3), so +// a mantissa without one gets ".0" appended, and an exponent's redundant "+" +// goes. `inf`/`-inf`/`nan` are spelled as the bare names HashedBuild has no +// literal for - the reader below knows them; nothing else does. +@(private = "file") +write_float :: proc(b: ^strings.Builder, f: f64) { + switch { + case f != f: + strings.write_string(b, "nan") + return + case math.is_inf(f, 1): + strings.write_string(b, "inf") + return + case math.is_inf(f, -1): + strings.write_string(b, "-inf") + return + } + + buf: [40]u8 + text := strconv.write_float(buf[:], f, 'g', -1, 64) + if parsed, ok := strconv.parse_f64(text); !ok || parsed != f { + buf2: [40]u8 + text = strconv.write_float(buf2[:], f, 'g', 17, 64) + write_float_literal(b, text) + return + } + write_float_literal(b, text) +} + +@(private = "file") +write_float_literal :: proc(b: ^strings.Builder, text: string) { + // strconv writes a leading '+' on a positive number here; HashedBuild's + // grammar has no unary '+', so it goes. + t := text + if len(t) > 0 && t[0] == '+' do t = t[1:] + + mantissa := t + exponent := "" + if e := strings.index_any(t, "eE"); e >= 0 { + mantissa = t[:e] + exponent = t[e + 1:] + if len(exponent) > 0 && exponent[0] == '+' do exponent = exponent[1:] + } + + strings.write_string(b, mantissa) + if !strings.contains(mantissa, ".") do strings.write_string(b, ".0") + if exponent != "" { + strings.write_string(b, "e") + strings.write_string(b, exponent) + } +} + +@(private = "file") +write_quoted :: proc(b: ^strings.Builder, data: []u8) { + strings.write_byte(b, '"') + for c in data { + switch c { + case '"': strings.write_string(b, "\\\"") + case '\\': strings.write_string(b, "\\\\") + case '\n': strings.write_string(b, "\\n") + case '\r': strings.write_string(b, "\\r") + case '\t': strings.write_string(b, "\\t") + case: + // Anything else printable goes through as itself, so a path or a name + // reads as a path or a name. The rest - control bytes, and NUL, which + // would end the string as far as any C-shaped reader is concerned - gets + // the one escape this format has that HashedBuild's does not. + if c < 0x20 || c == 0x7f { + hex := HEX + strings.write_string(b, "\\x") + strings.write_byte(b, hex[c >> 4]) + strings.write_byte(b, hex[c & 0xf]) + } else { + strings.write_byte(b, c) + } + } + } + strings.write_byte(b, '"') +} + +@(private = "file") +HEX :: "0123456789abcdef" + +// Whether a Utf8 key can be written as `.name`. Deliberately ASCII-only and +// conservative - a key that doesn't qualify still round-trips perfectly +// through the `[key] = value` form, so the only cost of saying no is a +// slightly noisier entry. +@(private = "file") +is_identifier_shaped :: proc(s: string) -> bool { + if len(s) == 0 do return false + for i in 0 ..< len(s) { + c := s[i] + is_alpha := (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '_' + is_digit := c >= '0' && c <= '9' + if !is_alpha && !(is_digit && i > 0) do return false + } + return true +} + +// ---- reading ---------------------------------------------------------------- + +// A `file "..."` / `dir "..."` reference, resolved by the caller: the parser +// knows the entry name, cache_store.odin knows the directory it sits in. +Cache_File_Ref :: proc(entry_name: string, is_dir: bool, userdata: rawptr) -> (Value, bool) + +@(private = "file") +Reader :: struct { + src: string, + pos: int, + resolve: Cache_File_Ref, + userdata: rawptr, + nodes: map[string]^Table_Value, // labels defined so far, for `ref` +} + +// Parses one value and requires the text to end there. Every failure is a +// plain false: an unreadable entry is a corrupt or hand-edited cache, and the +// caller's answer to that is to treat it as a miss or as a fatal read failure, +// not to try to make sense of half of it. +cache_format_read :: proc(src: string, resolve: Cache_File_Ref, userdata: rawptr) -> (Value, bool) { + r := Reader{src = src, resolve = resolve, userdata = userdata} + r.nodes = make(map[string]^Table_Value) + defer delete(r.nodes) + v, ok := read_value(&r) + if !ok do return nil, false + skip_space(&r) + if r.pos != len(r.src) do return nil, false + return v, true +} + +@(private = "file") +skip_space :: proc(r: ^Reader) { + for r.pos < len(r.src) { + switch r.src[r.pos] { + case ' ', '\t', '\r', '\n': r.pos += 1 + case: return + } + } +} + +@(private = "file") +accept :: proc(r: ^Reader, lit: string) -> bool { + skip_space(r) + if strings.has_prefix(r.src[r.pos:], lit) { + r.pos += len(lit) + return true + } + return false +} + +// A bare word, for the handful of keyword-shaped values (`nothing`, `true`, +// `bytes`, `dir`, ...). Matched as a whole word so that `nothingness` is not +// read as `nothing` followed by junk. +@(private = "file") +accept_word :: proc(r: ^Reader, word: string) -> bool { + skip_space(r) + rest := r.src[r.pos:] + if !strings.has_prefix(rest, word) do return false + if len(rest) > len(word) { + c := rest[len(word)] + if is_identifier_shaped(string([]u8{c})) do return false + } + r.pos += len(word) + return true +} + +@(private = "file") +read_value :: proc(r: ^Reader) -> (Value, bool) { + skip_space(r) + if r.pos >= len(r.src) do return nil, false + + switch { + case accept_word(r, "nothing"): return Nothing_Value{}, true + case accept_word(r, "true"): return true, true + case accept_word(r, "false"): return false, true + case accept_word(r, "nan"): return math.nan_f64(), true + case accept_word(r, "inf"): return math.inf_f64(1), true + + case accept_word(r, "bytes"): + s, ok := read_string(r) + if !ok do return nil, false + return transmute([]u8)s, true + + case accept_word(r, "file"), accept_word(r, "dir"): + // accept_word already consumed it; which one it was is recoverable from + // the byte just before the current position. + is_dir := r.src[r.pos - 1] == 'r' + name, ok := read_string(r) + if !ok || r.resolve == nil do return nil, false + return r.resolve(name, is_dir, r.userdata) + + case r.src[r.pos] == '"': + s, ok := read_string(r) + if !ok do return nil, false + return s, true + + case accept_word(r, "node"): + // The Table is created and registered *before* its entries are read, which + // is what lets an entry refer back to it - the same order §10 gives a + // `let rec` Table in the evaluator. + label, label_ok := read_string(r) + if !label_ok do return nil, false + if _, taken := r.nodes[label]; taken do return nil, false // a label defined twice + t := new(Table_Value) + r.nodes[label] = t + if !read_table_into(r, t) do return nil, false + return t, true + + case accept_word(r, "ref"): + label, label_ok := read_string(r) + if !label_ok do return nil, false + t, defined := r.nodes[label] + if !defined do return nil, false // a reference with no definition before it + return t, true + + case r.src[r.pos] == '{': + t := new(Table_Value) + if !read_table_into(r, t) do return nil, false + return t, true + } + return read_number(r) +} + +// Fills an already-created Table, so a `node` definition can register it before +// its own entries are read. +@(private = "file") +read_table_into :: proc(r: ^Reader, t: ^Table_Value) -> bool { + if !accept(r, "{") do return false + if accept(r, "}") do return true + + for { + key: Value + switch { + case accept(r, "."): + start := r.pos + for r.pos < len(r.src) && is_identifier_shaped(r.src[r.pos:r.pos + 1]) do r.pos += 1 + if r.pos == start do return false + key = r.src[start:r.pos] + case accept(r, "["): + k, ok := read_value(r) + if !ok || !accept(r, "]") do return false + key = k + case: + return false + } + + if !accept(r, "=") do return false + value, ok := read_value(r) + if !ok do return false + append(&t.entries, Table_Entry_Value{key = key, value = value}) + + if accept(r, ",") do continue + if accept(r, "}") do return true + return false + } +} + +@(private = "file") +read_number :: proc(r: ^Reader) -> (Value, bool) { + skip_space(r) + start := r.pos + if r.pos < len(r.src) && r.src[r.pos] == '-' { + r.pos += 1 + if accept_word(r, "inf") do return math.inf_f64(-1), true + } + is_float := false + for r.pos < len(r.src) { + c := r.src[r.pos] + switch { + case c >= '0' && c <= '9': + r.pos += 1 + case c == '.': + is_float = true + r.pos += 1 + case c == 'e' || c == 'E': + is_float = true + r.pos += 1 + if r.pos < len(r.src) && (r.src[r.pos] == '-' || r.src[r.pos] == '+') do r.pos += 1 + case: + return finish_number(r.src[start:r.pos], is_float) + } + } + return finish_number(r.src[start:r.pos], is_float) +} + +@(private = "file") +finish_number :: proc(text: string, is_float: bool) -> (Value, bool) { + if text == "" || text == "-" do return nil, false + if is_float { + f, ok := strconv.parse_f64(text) + if !ok do return nil, false + return f, true + } + i, ok := strconv.parse_i64_of_base(text, 10) + if !ok do return nil, false + return i, true +} + +@(private = "file") +read_string :: proc(r: ^Reader) -> (string, bool) { + skip_space(r) + if r.pos >= len(r.src) || r.src[r.pos] != '"' do return "", false + r.pos += 1 + + b: strings.Builder + strings.builder_init(&b) + for r.pos < len(r.src) { + c := r.src[r.pos] + r.pos += 1 + switch c { + case '"': + return strings.to_string(b), true + case '\\': + if r.pos >= len(r.src) do break + e := r.src[r.pos] + r.pos += 1 + switch e { + case 'n': strings.write_byte(&b, '\n') + case 't': strings.write_byte(&b, '\t') + case 'r': strings.write_byte(&b, '\r') + case '"': strings.write_byte(&b, '"') + case '\\': strings.write_byte(&b, '\\') + case 'x': + if r.pos + 1 >= len(r.src) do return "", false + hi, hi_ok := hex_digit(r.src[r.pos]) + lo, lo_ok := hex_digit(r.src[r.pos + 1]) + if !hi_ok || !lo_ok do return "", false + r.pos += 2 + strings.write_byte(&b, hi << 4 | lo) + case: + return "", false // an escape this format doesn't define: corrupt entry + } + case: + strings.write_byte(&b, c) + } + } + return "", false // unterminated +} + +@(private = "file") +hex_digit :: proc(c: u8) -> (u8, bool) { + switch { + case c >= '0' && c <= '9': return c - '0', true + case c >= 'a' && c <= 'f': return c - 'a' + 10, true + case c >= 'A' && c <= 'F': return c - 'A' + 10, true + } + return 0, false +} diff --git a/src/cache_store.odin b/src/cache_store.odin new file mode 100644 index 0000000..b5201b6 --- /dev/null +++ b/src/cache_store.odin @@ -0,0 +1,434 @@ +package hashedbuild + +import "core:encoding/base64" +import "core:fmt" +import "core:slice" +import "core:strings" + +// SPEC.md §15's `cached`, on disk. The key mechanism lives in +// hash_function.odin (a cached expression hashes as the closure it is); the +// text format lives in cache_format.odin; this file is the layout and the two +// operations over it, lookup and store. +// +// ---- the layout ------------------------------------------------------------- +// +// One entry per key, in the same directory ctx.cache writes its blobs to (§16 +// - `--cache-dir`, else $XDG_CACHE_HOME/hashedbuild, else the per-user +// fallback). Two shapes, decided by what the cached value *is*: +// +// /sha256- the value is a File - stored as itself +// /sha256-.hb/ anything else - a directory holding +// /sha256-.hb/value.hb the value, written as HashedBuild text +// /sha256-.hb/sha256- one entry per File inside that value +// +// A File value is kept as a file, and a directory value as a directory, so +// that what a build produced is still a thing you can open, `cat`, `diff` or +// copy out - the point of a content-addressed store is lost if everything in +// it is an opaque blob. Anything that is not a File has no such natural form, +// so it becomes text; and since text cannot hold a file, any File *inside* it +// is written out beside it and referenced by name, systematically, however +// deep it is nested. +// +// `` is the cache key - the hash of the expression as a closure - because +// that is the only thing a lookup has to go on. `` on a nested entry is +// that File's own content hash (§3), which is what makes two entries that +// contain the same file share one copy of it inside their own directories. +// Both are base64url without padding, so they are legal filenames on every +// target. The separator is `-`, where ctx.cache's own blobs use `_` +// (builtins_fs.odin): the two kinds of entry live in one directory, and the +// separator is what tells them apart at a glance. +// +// ---- committing ------------------------------------------------------------- +// +// An entry is built under a temporary name and renamed into place, so a run +// that is interrupted mid-write leaves a `.tmp` behind rather than a truncated +// entry that the next run would read as a hit. The rename is also how the race +// between two runs computing the same key is settled: the loser's rename +// fails, it removes its own temporary, and both go on to read the winner's +// entry. Nothing is ever overwritten, and nothing has to be locked. +// +// ---- what a hit returns ----------------------------------------------------- +// +// A store reads the entry back rather than returning the value it just +// computed. It costs a re-read, and it buys the property that matters: the +// first run and the second return the *same* value, Files included - pointing +// at the cache, displaying the cache's paths - instead of a value whose Files +// happen to point wherever this particular run built them. + +// The prefix and alphabet an entry name is built with. Never change either: +// both are baked into the name of every entry already on disk. +@(private = "file") +ENTRY_PREFIX :: "sha256-" +@(private = "file") +VALUE_FILE :: "value.hb" +@(private = "file") +HB_SUFFIX :: ".hb" + +cache_entry_name_for :: proc(d: Value_Digest) -> string { + d := d + encoded, _ := base64.encode(d[:], base64.ENC_URL_TABLE, context.temp_allocator) + return strings.concatenate({ENTRY_PREFIX, strings.trim_right(encoded, "=")}) +} + +// ---- lookup ----------------------------------------------------------------- + +// Whether `key_name` has an entry, and if so the value it holds. A missing +// entry is (nil, false, "") - an ordinary miss, not a failure. A *present* +// entry that cannot be read is a failure with a message, deliberately rather +// than a miss: silently recomputing over a corrupt cache would hide the +// corruption for as long as the cache lived. +cache_lookup :: proc(cache: ^Cache_Value, key_name: string) -> (val: Value, found: bool, err_msg: string) { + if errno := ensure_cache_dir_open(cache); errno != .None { + return nil, false, fmt.tprintf("could not open cache directory %s (%v)", cache.dir_path, errno) + } + + // The File shape first: a bare `sha256-`, stored as itself. + if is_dir, stat_err := fs_stat_is_dir_at(cache.dir_fd, key_name, true); stat_err == .None { + v, ok, msg := load_file_entry(cache, key_name, is_dir) + return v, ok, msg + } + + text_name := strings.concatenate({key_name, HB_SUFFIX}, context.temp_allocator) + if is_dir, stat_err := fs_stat_is_dir_at(cache.dir_fd, text_name, true); stat_err != .None || !is_dir { + return nil, false, "" // a miss + } + return load_text_entry(cache, text_name) +} + +@(private = "file") +load_file_entry :: proc(cache: ^Cache_Value, name: string, is_dir: bool) -> (Value, bool, string) { + display := strings.concatenate({cache.dir_path, "/", name}) + fv, msg := open_as_file_value(cache.dir_fd, name, is_dir, display) + if msg != "" do return nil, false, msg + return fv, true, "" +} + +@(private = "file") +load_text_entry :: proc(cache: ^Cache_Value, dir_name: string) -> (Value, bool, string) { + entry_fd, open_err := fs_open_dir_at(cache.dir_fd, dir_name, true) + if open_err != .None { + return nil, false, fmt.tprintf("could not open cache entry %s (%v)", dir_name, open_err) + } + defer fs_close(entry_fd) + + text_fd, text_err := fs_open_read_at(entry_fd, VALUE_FILE, true) + if text_err != .None { + return nil, false, fmt.tprintf("cache entry %s has no %s (%v)", dir_name, VALUE_FILE, text_err) + } + text, read_err := fs_read_all(text_fd) + fs_close(text_fd) + if read_err != .None { + return nil, false, fmt.tprintf("could not read %s/%s (%v)", dir_name, VALUE_FILE, read_err) + } + + ctx := Load_Ctx{ + entry_fd = entry_fd, + display_base = strings.concatenate({cache.dir_path, "/", dir_name}, context.temp_allocator), + } + v, ok := cache_format_read(string(text), resolve_entry_file, &ctx) + if !ok { + return nil, false, fmt.tprintf("cache entry %s/%s is not readable as a value", dir_name, VALUE_FILE) + } + if ctx.failure != "" do return nil, false, ctx.failure + return v, true, "" +} + +@(private = "file") +Load_Ctx :: struct { + entry_fd: Fs_Fd, + display_base: string, + failure: string, // set by the resolver, which can only report a bool +} + +@(private = "file") +resolve_entry_file :: proc(entry_name: string, is_dir: bool, userdata: rawptr) -> (Value, bool) { + ctx := (^Load_Ctx)(userdata) + // The name comes out of the entry's own text, so it is contained the same + // way §16's `.dir` sub-paths are: a separator in it would mean the entry was + // hand-edited, and there is nothing legitimate it could name. + if entry_name == "" || index_path_sep(entry_name) >= 0 || strings.contains(entry_name, "..") { + ctx.failure = fmt.tprintf("cache entry names %q, which is not a name this store writes", entry_name) + return nil, false + } + display := strings.concatenate({ctx.display_base, "/", entry_name}) + fv, msg := open_as_file_value(ctx.entry_fd, entry_name, is_dir, display) + if msg != "" { + ctx.failure = msg + return nil, false + } + return fv, true +} + +// One File value for a name inside the store, of the kind the text said it +// was. A mismatch is reported rather than followed: it means the entry and +// what is on disk have drifted apart. +@(private = "file") +open_as_file_value :: proc(dir_fd: Fs_Fd, name: string, is_dir: bool, display: string) -> (^File_Value, string) { + actually_dir, stat_err := fs_stat_is_dir_at(dir_fd, name, true) + if stat_err != .None do return nil, fmt.tprintf("could not read cache entry %s (%v)", name, stat_err) + if actually_dir != is_dir { + return nil, fmt.tprintf("cache entry %s is not the kind of File the entry says it is", name) + } + + fv := new(File_Value) + fv.display_path = display + if is_dir { + fd, err := fs_open_dir_at(dir_fd, name, true) + if err != .None do return nil, fmt.tprintf("could not open cache entry %s (%v)", name, err) + fv.kind = .Directory + fv.dir_fd = fd + return fv, "" + } + + fd, err := fs_open_read_at(dir_fd, name, true) + if err != .None do return nil, fmt.tprintf("could not open cache entry %s (%v)", name, err) + defer fs_close(fd) + content, read_err := fs_read_all(fd) + if read_err != .None do return nil, fmt.tprintf("could not read cache entry %s (%v)", name, read_err) + fv.kind = .Regular + fv.content = content + return fv, "" +} + +// ---- store ------------------------------------------------------------------ + +// Writes `v` under `key_name` and returns what a lookup of that key now +// yields. Losing the race to another run is not a failure: the entry that +// won holds the same value, since the key is the same. +// `interp` is what hashing a nested File needs (§3's directory digest reads +// the tree, and that read is gated on `io` like any other). +cache_store :: proc(cache: ^Cache_Value, key_name: string, v: Value, interp: ^Interpreter) -> (Value, bool, string) { + if errno := ensure_cache_dir_open(cache); errno != .None { + return nil, false, fmt.tprintf("could not open cache directory %s (%v)", cache.dir_path, errno) + } + + final_name := key_name + if _, is_file := v.(^File_Value); !is_file { + final_name = strings.concatenate({key_name, HB_SUFFIX}) + } + + temp_name, temp_ok := make_temp_dir(cache.dir_fd, final_name) + if !temp_ok { + return nil, false, "could not create a temporary directory in the cache" + } + temp_fd, temp_err := fs_open_dir_at(cache.dir_fd, temp_name, true) + if temp_err != .None { + remove_tree_at(cache.dir_fd, temp_name) + return nil, false, fmt.tprintf("could not open the temporary cache entry (%v)", temp_err) + } + + // A File value is written *as* the entry, so it is built one level down and + // that inner name is what gets renamed into place; everything else fills the + // temporary directory itself. + published := temp_name + if fv, is_file := v.(^File_Value); is_file { + if msg := write_file_value(temp_fd, "entry", fv); msg != "" { + fs_close(temp_fd) + remove_tree_at(cache.dir_fd, temp_name) + return nil, false, msg + } + published = strings.concatenate({temp_name, "/entry"}, context.temp_allocator) + } else if msg := write_text_entry(temp_fd, v, interp); msg != "" { + fs_close(temp_fd) + remove_tree_at(cache.dir_fd, temp_name) + return nil, false, msg + } + fs_close(temp_fd) + + rename_err := fs_rename_at(cache.dir_fd, published, final_name) + // A File entry was built one level down, so the wrapper directory is still + // there to clear away after a successful rename; a text entry *is* the + // temporary, so there is only something to remove when the rename failed. + if rename_err != .None || published != temp_name { + remove_tree_at(cache.dir_fd, temp_name) + } + + if rename_err != .None { + // Either another run published this key first - the ordinary case, and not + // a failure, since the key determines the value - or the store is broken. + // The lookup below is what tells the two apart. + if stored, found, msg := cache_lookup(cache, key_name); msg == "" && found { + return stored, true, "" + } + return nil, false, fmt.tprintf("could not publish cache entry %s (%v)", final_name, rename_err) + } + + stored, found, msg := cache_lookup(cache, key_name) + if msg != "" do return nil, false, msg + if !found do return nil, false, fmt.tprintf("cache entry %s vanished immediately after being written", final_name) + return stored, true, "" +} + +// `.tmpN`, first N that doesn't already exist. Uniqueness comes from +// the exclusive mkdir itself rather than from a random name, which needs no +// source of randomness and no process id - neither of which every target here +// has. The cap only has to exceed the number of runs racing on one key at +// once; anything near it means something else is wrong. +@(private = "file") +make_temp_dir :: proc(dir_fd: Fs_Fd, final_name: string) -> (string, bool) { + for i in 0 ..< 64 { + name := fmt.tprintf("%s.tmp%d", final_name, i) + if fs_mkdir_at(dir_fd, name) == .None do return strings.clone(name), true + } + return "", false +} + +@(private = "file") +write_text_entry :: proc(entry_fd: Fs_Fd, v: Value, interp: ^Interpreter) -> string { + // Every File in the value, written out first, so the text can refer to each + // by the name it landed under. + files := make([dynamic]^File_Value, 0, 4, context.temp_allocator) + seen := make(map[^Table_Value]bool) + defer delete(seen) + collect_files(v, &files, &seen) + + // Not the temp allocator, unlike almost everything else here: this map has + // to survive every write below, and writing a File hashes it - which for a + // directory value walks a whole tree, allocating temporary buffers the whole + // way down. A map living in that same arena does not reliably come out the + // other side. + names := make(map[^File_Value]string, len(files)) + defer delete(names) + for fv in files { + if _, already := names[fv]; already do continue + d, herr := value_digest(fv, interp) + if herr.kind != .None do return fmt.tprintf("cannot cache this value: %s", hash_error_message(herr)) + name := cache_entry_name_for(d) + if msg := write_file_value(entry_fd, name, fv); msg != "" do return msg + names[fv] = name + } + + text, ok, why := cache_format_write(v, names) + if !ok do return fmt.tprintf("cannot cache this value: %s", why) + return write_bytes(entry_fd, VALUE_FILE, transmute([]u8)text, false) +} + +// Every File reachable in `v`, in a deterministic order. Tables are the only +// thing that can hold one; a Function's environment is not part of a value's +// content, and nothing else nests. +// +// `seen` is not an optimisation: `let rec` can build a Table that reaches +// itself (§10), and without it this walks a cycle forever. +@(private = "file") +collect_files :: proc(v: Value, out: ^[dynamic]^File_Value, seen: ^map[^Table_Value]bool) { + resolved, ok := resolve_forward(v) + if !ok do return // still under construction; write_value reports it + + #partial switch av in resolved { + case ^File_Value: + append(out, av) + case ^Table_Value: + if seen[av] do return + seen[av] = true + for entry in av.entries { + collect_files(entry.key, out, seen) + collect_files(entry.value, out, seen) + } + } +} + +// ---- writing a File out ----------------------------------------------------- + +@(private = "file") +write_file_value :: proc(dir_fd: Fs_Fd, name: string, fv: ^File_Value) -> string { + if fv.kind == .Regular do return write_bytes(dir_fd, name, fv.content, false) + + if err := fs_mkdir_at(dir_fd, name); err != .None { + return fmt.tprintf("could not create %s in the cache (%v)", name, err) + } + dst, open_err := fs_open_dir_at(dir_fd, name, true) + if open_err != .None do return fmt.tprintf("could not open %s in the cache (%v)", name, open_err) + defer fs_close(dst) + return copy_tree(fv.dir_fd, dst) +} + +// A recursive copy of everything SPEC.md §3's directory hash reads - names, +// file contents, and symlink targets stored without being followed - so a +// restored directory hashes as the one it was copied from, which is what makes +// a hit and a miss return the same value. +// +// Plus one thing §3 does *not* read: the executable bit, where the target has +// one. It is not part of a directory's identity (§3 hashes no permission bit), +// so copying it changes no digest; it is copied because caching a build output +// and getting back something you can no longer run would be a poor trade for a +// build system. Nothing else about a directory is copied. +@(private = "file") +copy_tree :: proc(src_fd: Fs_Fd, dst_fd: Fs_Fd) -> string { + entries, list_err := fs_list_entries_at(src_fd, context.temp_allocator) + if list_err != .None do return fmt.tprintf("could not read a directory being cached (%v)", list_err) + slice.sort_by(entries, proc(a, b: Fs_Dir_Entry) -> bool { return a.name < b.name }) + + for entry in entries { + switch entry.kind { + case .Symlink: + target, err := fs_readlink_at(src_fd, entry.name) + if err != .None do return fmt.tprintf("could not read the symlink %s (%v)", entry.name, err) + if serr := fs_symlink_at(dst_fd, entry.name, target); serr != .None { + return fmt.tprintf("could not recreate the symlink %s in the cache (%v)", entry.name, serr) + } + + case .Directory: + if err := fs_mkdir_at(dst_fd, entry.name); err != .None { + return fmt.tprintf("could not create %s in the cache (%v)", entry.name, err) + } + child_src, src_err := fs_open_dir_at(src_fd, entry.name, true) + if src_err != .None do return fmt.tprintf("could not open %s (%v)", entry.name, src_err) + defer fs_close(child_src) + child_dst, dst_err := fs_open_dir_at(dst_fd, entry.name, true) + if dst_err != .None do return fmt.tprintf("could not open %s in the cache (%v)", entry.name, dst_err) + defer fs_close(child_dst) + if msg := copy_tree(child_src, child_dst); msg != "" do return msg + + case .Regular: + fd, open_err := fs_open_read_at(src_fd, entry.name, true) + if open_err != .None do return fmt.tprintf("could not open %s (%v)", entry.name, open_err) + content, read_err := fs_read_all(fd) + fs_close(fd) + if read_err != .None do return fmt.tprintf("could not read %s (%v)", entry.name, read_err) + defer delete(content) + if msg := write_bytes(dst_fd, entry.name, content, entry.is_executable); msg != "" do return msg + + case .Other: + // §3 encodes three shapes and this is none of them, so the directory has + // no digest either (hash.odin says the same). Refusing here rather than + // skipping keeps the copy and the hash agreeing about what a tree is. + return fmt.tprintf("%s is not a file, directory or symlink, and cannot be cached", entry.name) + } + } + return "" +} + +@(private = "file") +write_bytes :: proc(dir_fd: Fs_Fd, name: string, data: []u8, executable: bool) -> string { + fd, err := fs_create_exclusive_at(dir_fd, name) + if err != .None do return fmt.tprintf("could not create %s in the cache (%v)", name, err) + werr := fs_write_all(fd, data) + fs_close(fd) + if werr != .None do return fmt.tprintf("could not write %s in the cache (%v)", name, werr) + if executable do fs_set_executable_at(dir_fd, name) + return "" +} + +// ---- removing a temporary --------------------------------------------------- + +// Best-effort: this only ever runs on a temporary the caller just built, and +// the caller is already on its way to reporting something else (or to using +// the entry another run published). Leaving a `.tmpN` behind is untidy, not +// wrong - the next run picks a different N. +@(private = "file") +remove_tree_at :: proc(parent: Fs_Fd, name: string) { + if fs_unlink_at(parent, name) == .None do return + if fd, err := fs_open_dir_at(parent, name, true); err == .None { + if entries, lerr := fs_list_entries_at(fd, context.temp_allocator); lerr == .None { + for entry in entries { + if entry.kind == .Directory { + remove_tree_at(fd, entry.name) + } else { + fs_unlink_at(fd, entry.name) + } + } + } + fs_close(fd) + } + fs_rmdir_at(parent, name) +} diff --git a/src/cache_test.odin b/src/cache_test.odin new file mode 100644 index 0000000..954c14d --- /dev/null +++ b/src/cache_test.odin @@ -0,0 +1,523 @@ +// Tests run natively, never in a WASI build - see eval_test.odin. +#+build linux, windows +package hashedbuild + +import "core:fmt" +import "core:os" +import "core:strings" +import "core:testing" + +// SPEC.md §15's `cached`: the key (hash_function.odin), the on-disk layout +// (cache_store.odin) and the text format (cache_format.odin), exercised +// through the language itself wherever that is possible. +// +// Every test here gets its own cache directory. `odin test` runs tests +// concurrently, and these actually write - two tests sharing a directory +// would race on the entries in it. + +@(private = "file") +cache_scratch :: proc(name: string) -> string { + return strings.concatenate({repo_root(), "/.cache_test_", name}) +} + +@(private = "file") +remove_cache_scratch :: proc(path: string) { + remove_recursively(path) +} + +// os.remove refuses a non-empty directory, and a cache entry holding a +// directory value nests arbitrarily deep, so cleanup has to recurse. +// +// The is_dir check is not an optimisation: os.open succeeds on a regular file +// too, and reading a directory listing out of that handle is not something +// every target survives. +@(private = "file") +remove_recursively :: proc(path: string) { + if is_dir, err := fs_stat_is_dir_at(fs_cwd_dir(), path, true); err == .None && is_dir { + entries, _ := fs_list_dir(path, context.temp_allocator) + for entry in entries do remove_recursively(fmt.tprintf("%s/%s", path, entry.name)) + } + os.remove(path) +} + +// Evaluates with the real global environment and a root context whose +// ctx.cache points at `cache_dir` - the same setup a real run has, since +// `cached` is only meaningful against a real store. +@(private = "file") +eval_cached_src :: proc(src: string, cache_dir: string) -> (val: Value, ok: bool, err: string) { + ast := parse(source_t{name = "test", n_bytes = u64(len(src)), data = raw_data(src)}, ast_t{}) + interp := Interpreter{ast = &ast, src = src, current_ctx = make_root_context(cache_dir)} + val, ok = eval_program(&interp, ast.root, make_global_env()) + return val, ok, interp.error_message +} + +@(private = "file") +expect_int :: proc(t: ^testing.T, src: string, cache_dir: string, want: i64) { + val, ok, err := eval_cached_src(src, cache_dir) + testing.expect(t, ok, err) + got, is_int := val.(i64) + testing.expect(t, is_int, "expected an Integer result") + testing.expect_value(t, got, want) +} + +@(private = "file") +expect_failure :: proc(t: ^testing.T, src: string, cache_dir: string) -> string { + _, ok, err := eval_cached_src(src, cache_dir) + testing.expect(t, !ok, "expected this to fail") + return err +} + +// The names directly inside a directory, sorted - what a test asserts the +// layout with. +@(private = "file") +entry_names :: proc(path: string) -> []string { + entries, err := fs_list_dir(path, context.temp_allocator) + if err != .None do return nil + names := make([dynamic]string, 0, len(entries), context.temp_allocator) + for entry in entries do append(&names, entry.name) + return names[:] +} + +// ---- the basic contract ----------------------------------------------------- + +@(test) +test_cached_returns_the_value_and_writes_one_entry :: proc(t: ^testing.T) { + dir := cache_scratch("basic") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + expect_int(t, `cached (1 + 2)`, dir, 3) + expect_int(t, `cached (1 + 2)`, dir, 3) + + names := entry_names(dir) + testing.expect_value(t, len(names), 1) + if len(names) != 1 do return + testing.expect(t, strings.has_prefix(names[0], "sha256-"), "an entry is named by its key") + testing.expect(t, strings.has_suffix(names[0], ".hb"), "a non-File value is stored as text") +} + +// The one test that distinguishes a cache from a very elaborate way of +// evaluating twice: the file the expression reads is changed underneath it, +// and the second call still answers with what the first one stored. Nothing +// in the key mentions the file's contents, so the entry stays valid. +@(test) +test_cached_hit_survives_the_source_changing :: proc(t: ^testing.T) { + dir := cache_scratch("hit") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + data := cache_scratch("hit_data") + defer delete(data) + defer remove_cache_scratch(data) + os.make_directory(data) + path := fmt.tprintf("%s/f.txt", data) + _ = os.write_entire_file(path, transmute([]u8)string("first")) + + src := fmt.aprintf(`filetext cached (loadfile "%s")`, path) + defer delete(src) + + val, ok, err := eval_cached_src(src, dir) + testing.expect(t, ok, err) + testing.expect_value(t, val.(string), "first") + + os.remove(path) + _ = os.write_entire_file(path, transmute([]u8)string("SECOND")) + + // Uncached, the change is visible... + fresh := fmt.aprintf(`filetext (loadfile "%s")`, path) + defer delete(fresh) + val2, ok2, err2 := eval_cached_src(fresh, dir) + testing.expect(t, ok2, err2) + testing.expect_value(t, val2.(string), "SECOND") + + // ...and through `cached` it is not: the stored answer is returned. + val3, ok3, err3 := eval_cached_src(src, dir) + testing.expect(t, ok3, err3) + testing.expect_value(t, val3.(string), "first") +} + +// The half of §15's key that has to be right for a hit to be correct: two +// expressions with identical code but different captured values are different +// entries. Get this wrong and the second call returns the first one's answer. +@(test) +test_cached_key_covers_the_values_the_expression_uses :: proc(t: ^testing.T) { + dir := cache_scratch("key") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + expect_int(t, `let x 1; cached (x + 10)`, dir, 11) + expect_int(t, `let x 2; cached (x + 10)`, dir, 12) + expect_int(t, `let x 1; cached (x + 10)`, dir, 11) + + testing.expect_value(t, len(entry_names(dir)), 2) +} + +// A closure captures its environment; `#arg`/`#self` (§9) are dynamic lookups +// that it does not capture, so the key has to reach them separately. This is +// the case that returned a *wrong answer* before it did: identical code, a +// different argument, one entry. +@(test) +test_cached_key_covers_the_implicit_names_it_reads :: proc(t: ^testing.T) { + dir := cache_scratch("implicit") + defer delete(dir) + defer remove_cache_scratch(dir) + + expect_int(t, `let f func (cached (#arg + 1)); f 1`, dir, 2) + expect_int(t, `let f func (cached (#arg + 1)); f 10`, dir, 11) + expect_int(t, `let f func (cached (#arg + 1)); f 1`, dir, 2) + + // The same through `|>`, which pushes onto the same stack... + expect_int(t, `1 |> cached (#arg + 100)`, dir, 101) + expect_int(t, `5 |> cached (#arg + 100)`, dir, 105) + + // ...and at a level further out, where the reach has to be counted rather + // than assumed to be one. + expect_int(t, `let f func ((func (cached (#arg2 * 2))) 0); f 3`, dir, 6) + expect_int(t, `let f func ((func (cached (#arg2 * 2))) 0); f 4`, dir, 8) + expect_int(t, `let f func ((func (cached (#arg2 * 2))) 0); f 3`, dir, 6) +} + +// An expression that reads no implicit name is not affected by any of the +// above: its key is the closure digest alone, and stays stable across calls +// made at different depths. +@(test) +test_cached_key_is_unchanged_by_depth_when_nothing_is_read :: proc(t: ^testing.T) { + dir := cache_scratch("depth") + defer delete(dir) + defer remove_cache_scratch(dir) + + expect_int(t, `cached (6 * 7)`, dir, 42) + expect_int(t, `let f func (cached (6 * 7)); f 1`, dir, 42) + expect_int(t, `let f func (cached (6 * 7)); f 99`, dir, 42) + + testing.expect_value(t, len(entry_names(dir)), 1) +} + +// ---- the three layouts ------------------------------------------------------ + +// "if it is straight a file, keep it that way": a File value is the entry, not +// something wrapped in one, so what a build produced stays a file you can open. +@(test) +test_cached_file_value_is_stored_as_a_file :: proc(t: ^testing.T) { + dir := cache_scratch("file") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + src := fmt.aprintf(`cached (loadfile "%s/README.md")`, repo_root()) + defer delete(src) + val, ok, err := eval_cached_src(src, dir) + testing.expect(t, ok, err) + + fv, is_file := val.(^File_Value) + testing.expect(t, is_file, "a cached File is still a File") + if !is_file do return + testing.expect_value(t, fv.kind, File_Kind.Regular) + testing.expect(t, strings.has_prefix(fv.display_path, dir), "a hit displays where it lives (§3)") + + names := entry_names(dir) + testing.expect_value(t, len(names), 1) + if len(names) != 1 do return + testing.expect(t, !strings.has_suffix(names[0], ".hb"), "a File entry carries no .hb suffix") +} + +// The same for a directory, which is the case that needs the whole tree copied +// - and copied faithfully enough that the restored value hashes as the +// original did (§3). That equality is the test: it covers names, contents and +// nesting in one assertion. +@(test) +test_cached_directory_value_round_trips :: proc(t: ^testing.T) { + dir := cache_scratch("tree") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + // A tree built here rather than one of the repo's own: nothing else writes + // into it (async-branching drops marker files into examples/ as it runs, and + // tests run concurrently), and the two hashes here have to see one tree. + tree := cache_scratch("tree_src") + defer delete(tree) + defer remove_cache_scratch(tree) + os.make_directory(tree) + _ = os.write_entire_file(fmt.tprintf("%s/a.txt", tree), transmute([]u8)string("alpha")) + os.make_directory(fmt.tprintf("%s/sub", tree)) + _ = os.write_entire_file(fmt.tprintf("%s/sub/b.txt", tree), transmute([]u8)string("beta")) + + src := fmt.aprintf( + `(sha256 cached (loadfile "%s")) == (sha256 loadfile "%s")`, tree, tree, + ) + defer delete(src) + val, ok, err := eval_cached_src(src, dir) + testing.expect(t, ok, err) + testing.expect(t, val.(bool), "a cached directory is the same value it was") + + names := entry_names(dir) + testing.expect_value(t, len(names), 1) + if len(names) != 1 do return + is_dir, _ := fs_stat_is_dir_at(fs_cwd_dir(), fmt.tprintf("%s/%s", dir, names[0]), true) + testing.expect(t, is_dir, "a directory value is stored as a directory") +} + +// Anything else becomes `value.hb`, with each File it holds written out beside +// it and named by that File's own content hash - the "systematically" half of +// the layout. A composite with one file in it therefore has exactly two names +// inside its entry. +@(test) +test_cached_composite_holds_its_files_beside_the_text :: proc(t: ^testing.T) { + dir := cache_scratch("composite") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + // strings.concatenate, not fmt.aprintf: the value being cached is a Table + // literal, and fmt reads its `{` as the start of a format verb. + src := strings.concatenate({`cached { .doc = loadfile "`, repo_root(), `/README.md", .n = 7 }`}) + defer delete(src) + val, ok, err := eval_cached_src(src, dir) + testing.expect(t, ok, err) + + table, is_table := val.(^Table_Value) + testing.expect(t, is_table, "a cached Table is still a Table") + if !is_table do return + doc, found := table_find(table, "doc") + testing.expect(t, found) + _, doc_is_file := doc.(^File_Value) + testing.expect(t, doc_is_file, "the File inside came back as a File") + + outer := entry_names(dir) + testing.expect_value(t, len(outer), 1) + if len(outer) != 1 do return + inner := entry_names(fmt.tprintf("%s/%s", dir, outer[0])) + testing.expect_value(t, len(inner), 2) + + has_text, has_file := false, false + for name in inner { + if name == "value.hb" do has_text = true + if strings.has_prefix(name, "sha256-") do has_file = true + } + testing.expect(t, has_text, "the entry holds its value.hb") + testing.expect(t, has_file, "and the File it refers to, named by content") +} + +// §10's `let rec` can build a Table that reaches itself, so the format has to +// write one down and read it back. The assertion that matters is the digest: +// the restored value has to *be* the value that was stored, and §6 compares +// cyclic values by bisimulation, so an unfolding to the wrong depth or a lost +// back-edge would show up here. +@(test) +test_cached_round_trips_a_cyclic_value :: proc(t: ^testing.T) { + dir := cache_scratch("cyclic") + defer delete(dir) + defer remove_cache_scratch(dir) + + self := `let rec t { .name = "alice", .self = t }; ` + same := strings.concatenate({self, `(sha256 cached t) == (sha256 t)`}) + defer delete(same) + val, ok, err := eval_cached_src(same, dir) + testing.expect(t, ok, err) + testing.expect(t, val.(bool), "a cached cyclic value is the value it was") + + // ...and the back-edge is a real one, not an unfolding that ran out. + deep := strings.concatenate({self, `(cached t).self.self.self.name`}) + defer delete(deep) + val2, ok2, err2 := eval_cached_src(deep, dir) + testing.expect(t, ok2, err2) + testing.expect_value(t, val2.(string), "alice") +} + +// Two Tables that reach each other, which is the shape §10 actually produces - +// a `let rec` whose entries mention their siblings. The cycle here runs through +// two nodes rather than one, so a writer that only handled self-reference would +// pass the test above and fail this one. +@(test) +test_cached_round_trips_a_mutual_cycle :: proc(t: ^testing.T) { + dir := cache_scratch("mutual") + defer delete(dir) + defer remove_cache_scratch(dir) + + people := `let rec people { .alice = { .name = "alice", .friend = people.bob }, ` + + `.bob = { .name = "bob", .friend = people.alice } }; ` + src := strings.concatenate({people, `(sha256 cached people) == (sha256 people)`}) + defer delete(src) + val, ok, err := eval_cached_src(src, dir) + testing.expect(t, ok, err) + testing.expect(t, val.(bool)) + + hop := strings.concatenate({people, `(cached people).alice.friend.friend.name`}) + defer delete(hop) + val2, ok2, err2 := eval_cached_src(hop, dir) + testing.expect(t, ok2, err2) + testing.expect_value(t, val2.(string), "alice") +} + +// ---- what `cached` refuses -------------------------------------------------- + +@(test) +test_cached_needs_io_and_a_cache :: proc(t: ^testing.T) { + dir := cache_scratch("refuse") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + denied := expect_failure(t, `(cached 1) withctx { .permissions = empty, .cache = ctx.cache }`, dir) + testing.expect(t, strings.contains(denied, "io permission"), denied) + + // §9 lets a program build a context by hand; one that doesn't carry .cache + // over has no store to use, and says so rather than inventing one. + no_cache := expect_failure(t, `(cached 1) withctx { .permissions = ctx.permissions }`, dir) + testing.expect(t, strings.contains(no_cache, "no .cache"), no_cache) + + // Nothing was created: the directory is made lazily, on a real write. + _, exists := os.stat(dir, context.temp_allocator) + testing.expect(t, exists != nil, "a refused `cached` creates no cache directory") +} + +// A closure's meaning is its environment, so there is nothing to write down +// and read back. The same goes for ctx.cache. Both fail rather than storing +// something that would come back as a different value. +@(test) +test_cached_refuses_a_value_it_cannot_write :: proc(t: ^testing.T) { + dir := cache_scratch("unwritable") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + for src in ([]string{`cached (func 1)`, `cached ctx.cache`, `cached { .f = func 1 }`}) { + msg := expect_failure(t, src, dir) + testing.expect(t, strings.contains(msg, "cannot cache"), msg) + } +} + +// A present-but-unreadable entry is a failure, not a miss. Recomputing over a +// corrupt cache would work, and would hide the corruption for as long as the +// cache lived. +@(test) +test_cached_reports_a_corrupt_entry :: proc(t: ^testing.T) { + dir := cache_scratch("corrupt") + defer delete(dir) // LIFO: the path has to outlive the cleanup + defer remove_cache_scratch(dir) + + expect_int(t, `cached (1 + 2)`, dir, 3) + names := entry_names(dir) + testing.expect_value(t, len(names), 1) + if len(names) != 1 do return + + value_path := fmt.tprintf("%s/%s/value.hb", dir, names[0]) + os.remove(value_path) + _ = os.write_entire_file(value_path, transmute([]u8)string("{ this is not a value")) + + msg := expect_failure(t, `cached (1 + 2)`, dir) + testing.expect(t, strings.contains(msg, "not readable as a value"), msg) +} + +// ---- the text format -------------------------------------------------------- + +// cache_format.odin's round trip, over every kind of value that has a written +// form - and, for the numbers, over the awkward ones. A value that reads back +// even slightly different would hash differently, and `cached` would be +// handing out something other than what it stored. +@(test) +test_cache_format_round_trips_every_writable_value :: proc(t: ^testing.T) { + nested := new(Table_Value) + append(&nested.entries, Table_Entry_Value{key = i64(1), value = "one"}) + append(&nested.entries, Table_Entry_Value{key = "needs quotes", value = true}) + + table := new(Table_Value) + append(&table.entries, Table_Entry_Value{key = "nothing", value = Nothing_Value{}}) + append(&table.entries, Table_Entry_Value{key = "yes", value = true}) + append(&table.entries, Table_Entry_Value{key = "no", value = false}) + append(&table.entries, Table_Entry_Value{key = "neg", value = i64(-9223372036854775808)}) + append(&table.entries, Table_Entry_Value{key = "big", value = i64(9223372036854775807)}) + append(&table.entries, Table_Entry_Value{key = "tenth", value = 0.1}) + append(&table.entries, Table_Entry_Value{key = "tiny", value = 5.0e-324}) + append(&table.entries, Table_Entry_Value{key = "third", value = 1.0 / 3.0}) + append(&table.entries, Table_Entry_Value{key = "whole", value = 2.0}) + append(&table.entries, Table_Entry_Value{key = "escapes", value = "quote\" back\\ tab\t nl\n nul\x00 hi\x7f"}) + append(&table.entries, Table_Entry_Value{key = "unicode", value = "é中文 \U0001f600"}) + append(&table.entries, Table_Entry_Value{key = "raw", value = []u8{0, 1, 2, 255}}) + append(&table.entries, Table_Entry_Value{key = "inner", value = nested}) + append(&table.entries, Table_Entry_Value{key = "empty", value = new(Table_Value)}) + + names: map[^File_Value]string + text, wrote, why := cache_format_write(table, names) + testing.expect(t, wrote, why) + + back, read := cache_format_read(text, nil, nil) + testing.expect(t, read, text) + + // Hash equality rather than field-by-field comparison: it is the property + // `cached` actually depends on, and it covers the whole structure at once. + testing.expect(t, values_hash_equal(table, back), text) +} + +// The format's own round trip for a cycle, below the language: a Table holding +// itself, written and read back. Asserted through the digest, which §6 computes +// by bisimulation for a cyclic value - so this checks the graph, not the text. +@(test) +test_cache_format_round_trips_a_cycle :: proc(t: ^testing.T) { + cyclic := new(Table_Value) + append(&cyclic.entries, Table_Entry_Value{key = "name", value = "alice"}) + append(&cyclic.entries, Table_Entry_Value{key = "self", value = cyclic}) + + names: map[^File_Value]string + text, wrote, why := cache_format_write(cyclic, names) + testing.expect(t, wrote, why) + testing.expect(t, strings.contains(text, `node "1"`), text) + testing.expect(t, strings.contains(text, `ref "1"`), text) + + back, read := cache_format_read(text, nil, nil) + testing.expect(t, read, text) + testing.expect(t, values_hash_equal(cyclic, back), text) +} + +// A Table reached twice without any cycle is written once and referred to, +// which is why a deeply shared value doesn't expand exponentially on the way +// out. Sharing isn't observable in the value model, so this is about size - +// but the value still has to come back equal. +@(test) +test_cache_format_keeps_sharing :: proc(t: ^testing.T) { + shared := new(Table_Value) + append(&shared.entries, Table_Entry_Value{key = "n", value = i64(1)}) + + outer := new(Table_Value) + append(&outer.entries, Table_Entry_Value{key = "a", value = shared}) + append(&outer.entries, Table_Entry_Value{key = "b", value = shared}) + + names: map[^File_Value]string + text, wrote, why := cache_format_write(outer, names) + testing.expect(t, wrote, why) + testing.expect(t, strings.contains(text, `ref "1"`), text) + + back, read := cache_format_read(text, nil, nil) + testing.expect(t, read, text) + testing.expect(t, values_hash_equal(outer, back), text) +} + +// A `ref` with no definition before it, and a label defined twice: both are +// entries that no writer here produces, so both are corruption. +@(test) +test_cache_format_rejects_a_broken_reference :: proc(t: ^testing.T) { + for src in ([]string{ + `{ .a = ref "1" }`, + `{ .a = node "1" { .x = 1 }, .b = node "1" { .y = 2 } }`, + `ref "nope"`, + `node "1" 5`, + }) { + _, ok := cache_format_read(src, nil, nil) + testing.expect(t, !ok, src) + } +} + +// The reader accepts literals and nothing else, so an entry someone edited +// into a program is a parse failure rather than something that runs. +@(test) +test_cache_format_rejects_anything_that_is_not_a_value :: proc(t: ^testing.T) { + for src in ([]string{ + `loadfile "/etc/passwd"`, + `1 + 2`, + `{ .a = 1 } concat { .b = 2 }`, + `func 1`, + `{ .a = }`, + `"unterminated`, + `nothing extra`, + `file "../../escape"`, + }) { + _, ok := cache_format_read(src, nil, nil) + testing.expect(t, !ok, src) + } +} diff --git a/src/eval.odin b/src/eval.odin index 4299bc2..92d8c0b 100644 --- a/src/eval.odin +++ b/src/eval.odin @@ -6,7 +6,7 @@ import "core:strings" // Tree-walking evaluator for the pure-expression core of SPEC.md. Deliberately // scoped out for this pass (each needs real design/OS decisions this project -// hasn't made yet): `cached`, `import`, `#context`, and the static-vs-runtime +// hasn't made yet): `import`, `#context`, and the static-vs-runtime // distinction for `check`/`static_check` (both just run as runtime checks // here). // @@ -430,7 +430,10 @@ eval :: proc(interp: ^Interpreter, node: Node_Idx, env: ^Env) -> (ret_val: Value case .Sha256_Expr: return eval_sha256(interp, node, env) - case .Cached_Expr, .Import_Expr: + case .Cached_Expr: + return eval_cached(interp, node, env) + + case .Import_Expr: return fail(interp, fmt.tprintf("%v is not implemented by this evaluator yet", n.kind)) } return fail(interp, fmt.tprintf("evaluation not implemented for %v", n.kind)) @@ -1128,6 +1131,113 @@ eval_sha256 :: proc(interp: ^Interpreter, node: Node_Idx, env: ^Env) -> (Value, return encoded, true } +// §15: evaluates an expression once and remembers the answer, or hands back +// the answer an earlier run already stored. The cache lives on disk, in the +// directory `ctx.cache` writes to - see cache_store.odin for the layout, +// hash_function.odin for what a closure hashes as, and hash_implicit.odin +// for the part of the key a closure's digest cannot carry. +// +// **The key is computed before the expression runs**, which is the whole point: +// a hit must not have to evaluate anything. §15 pins what it is - the cached +// expression treated as a function, hashed as one - so that is exactly what is +// built here, a closure over the operand in the current environment and +// `ctx`. Its digest covers the code, the captured `ctx`, and the values of the +// names the code uses, so `cached (x + 1)` is a different entry for each `x`. +// +// **Gated by `ctx.permissions.io`** (§9), like §16's builtins and for the same +// reason: this reads and writes files. A denied `io` is a failure rather than a +// quiet fall-through to evaluating uncached, because the two differ in what +// gets written to disk, and silently doing the other one is not something a +// program should have to guess at. +// +// **Async is positional**, as §15 says. `cached async ` is this +// procedure wrapped around an operand that evaluates on another thread: the +// await below is the "caching wrapper around it synchronous" half. `async +// cached ` is the other placement, and needs nothing here - it is an +// Async_Expr whose body happens to be this one. +@(private = "file") +eval_cached :: proc(interp: ^Interpreter, node: Node_Idx, env: ^Env) -> (Value, bool) { + n := interp.ast.nodes[node] + operand := interp.ast.extra_children[n.children_start] + + if !ctx_allows_io(interp) do return fail(interp, "cached: io permission not granted in the current context") + + cache, has_cache := cache_of_ctx(interp.current_ctx) + if !has_cache { + return fail(interp, "cached: the current context has no .cache (SPEC.md §9 - a hand-built ctx has to carry it over)") + } + + key, herr := value_digest(new_function(interp, operand, env), interp) + if herr.kind != .None do return fail(interp, fmt.tprintf("cached: %s", hash_error_message(herr))) + key, herr = mix_implicit_reach(interp, operand, key) + if herr.kind != .None do return fail(interp, fmt.tprintf("cached: %s", hash_error_message(herr))) + key_name := cache_entry_name_for(key) + + if stored, found, msg := cache_lookup(cache, key_name); msg != "" { + return fail(interp, fmt.tprintf("cached: %s", msg)) + } else if found { + return stored, true + } + + val, ok := eval_slot(interp, operand, env) + if !ok do return nil, false + val, ok = await_value(interp, val) + if !ok do return nil, false + + stored, stored_ok, msg := cache_store(cache, key_name, val, interp) + if !stored_ok do return fail(interp, fmt.tprintf("cached: %s", msg)) + return stored, true +} + +// A closure captures its environment and its `ctx`, but `#arg`/`#self` (§9) are +// dynamic lookups into the interpreter's own stacks and are captured by +// neither - so the closure digest cannot see them, and the key has to. Without +// this, `let f func (cached (#arg + 1)); f 1` and `f 10` would be one entry. +// +// Only the levels the expression can actually reach are mixed in, and only when +// it reaches any at all: an expression that mentions no implicit name gets the +// closure digest untouched. See hash_implicit.odin for why a static bound on +// the reach is sound. +@(private = "file") +mix_implicit_reach :: proc(interp: ^Interpreter, operand: Node_Idx, key: Value_Digest) -> (Value_Digest, Hash_Fail) { + max_arg, max_self := 0, 0 + implicit_reach(interp, operand, &max_arg, &max_self) + if max_arg == 0 && max_self == 0 do return key, HASH_OK + + return implicit_reach_digest( + key, + top_of(interp.arg_stack[:], max_arg), + top_of(interp.self_stack[:], max_self), + interp, + ) +} + +// The top `count` entries, innermost first, padded with `nothing` where the +// stack is shorter than the reach - so the shape of what gets mixed in depends +// only on the expression, never on how deep the program happens to be. +@(private = "file") +top_of :: proc(stack: []Value, count: int) -> []Value { + out := make([]Value, count, context.temp_allocator) + for i in 0 ..< count { + idx := len(stack) - 1 - i + out[i] = idx >= 0 ? stack[idx] : Nothing_Value{} + } + return out +} + +// `ctx.cache`, if this context has one. §9 lets a program build a context by +// hand, and one built without carrying `.cache` over simply hasn't got a cache +// to use - so this is a question, not an assertion. +@(private = "file") +cache_of_ctx :: proc(ctx: Value) -> (^Cache_Value, bool) { + t, is_table := ctx.(^Table_Value) + if !is_table do return nil, false + val, found := table_find(t, "cache") + if !found do return nil, false + cache, is_cache := val.(^Cache_Value) + return cache, is_cache +} + // ---- check / static_check / error (§11) ---------------------------------------- @(private = "file") diff --git a/src/examples_test.odin b/src/examples_test.odin index b2ceaf4..19f9428 100644 --- a/src/examples_test.odin +++ b/src/examples_test.odin @@ -33,6 +33,7 @@ EXAMPLE_CASES := []Example_Case{ {"async-basics.hb", `"This is the payload for option A.\nThis is the payload for option B.\n"`}, {"async-branching.hb", `"medium"`}, {"async-table.hb", `{a: 2, b: 6, c: "This is the payload for option A.\n"}`}, + {"cached.hb", "{answer: 42, asking_again_agrees: true, per_argument: {small: 2, large: 11}, file_survives_the_round_trip: true, a_cycle_survives_too: true}"}, {"check-and-invariants.hb", "100"}, {"comparison-and-logic.hb", "{ordered: true, both: true, either: true, mixed: false}"}, {"context-permissions.hb", "{ambient: {io: nothing}, io_denied: {}, replaced: {}, still_ambient: {io: nothing}}"}, @@ -43,11 +44,10 @@ EXAMPLE_CASES := []Example_Case{ {"guard-chain.hb", "5"}, {"hashing.hb", `{text: "Ar9oHTBiuRDqs+ZdbYD2daaU7RcvIDTJNB3UICNP92A=", file: "ZT6vBQgoXEojRYd890EDlZWhUF/uGfXa+C9BNGBykI0=", key_order_is_irrelevant: true, same_content_same_file: true, integer_is_not_float: false}`}, // The three hashing examples assert properties rather than digests, on - // purpose. A directory containing an executable hashes differently on - // Windows and WASI than on Linux (§3, and LANGUAGE.md on why), and a - // closure's digest includes its body's own source text - so a literal here - // would be a value that is either target-specific or broken by reformatting - // the example it came from. + // purpose. A directory's digest changes as examples are added to the tree, + // and a closure's includes its body's own source text - so a literal here + // would be a value that went stale the next time someone added an example or + // reformatted the one it came from. {"hashing-cyclic.hb", "{a_cycle_hashes: true, equal_cycles_hash_alike: true, shape_still_matters: true, either_end_agrees: true}"}, {"hashing-directories.hb", "{a_tree_is_not_its_file: true, one_tree_is_one_value: true, reading_twice_agrees: true}"}, {"hashing-functions.hb", "{same_body_same_hash: true, different_body_differs: true, captures_count: true, neighbours_do_not: true, builtins_hash_by_what_they_are: true, a_builtin_carries_its_argument: true}"}, @@ -196,9 +196,19 @@ read_dir_names :: proc(dir: string) -> []string { return names } +// Recursive, because a `cached` entry (§15) is a directory: a value that is +// not a File is stored as `sha256-.hb/value.hb`, and a directory value as +// the tree itself. os.remove won't take a non-empty directory, so a one-level +// sweep would leave the scratch cache behind for the next run to trip over. +// +// The is_dir check is not an optimisation: os.open succeeds on a regular file +// too, and reading a directory listing out of that handle is not something +// every target survives. @(private = "file") remove_dir_and_entries :: proc(dir: string) { - for name in read_dir_names(dir) do os.remove(fmt.tprintf("%s/%s", dir, name)) + if is_dir, err := fs_stat_is_dir_at(fs_cwd_dir(), dir, true); err == .None && is_dir { + for name in read_dir_names(dir) do remove_dir_and_entries(fmt.tprintf("%s/%s", dir, name)) + } os.remove(dir) } diff --git a/src/fs.odin b/src/fs.odin index b4a9412..4ab1f4c 100644 --- a/src/fs.odin +++ b/src/fs.odin @@ -50,6 +50,16 @@ FS_INVALID_FD :: Fs_Fd(-1) // fs_open_dir_path open a directory by path - ctx.cache only (§9) // fs_make_dirs mkdir -p by path - ctx.cache only (§9) // fs_list_dir names in a directory - the editor's file pickers +// +// The five below were added for §15's `cached` (see cache_store.odin), which is +// the first thing here that has to write a whole directory back out again. +// Reading one is `fs_list_entries_at`, further down. +// +// fs_mkdir_at create one directory, failing if it exists +// fs_rename_at rename within one directory - the atomic commit +// fs_unlink_at remove one non-directory name +// fs_rmdir_at remove one empty directory +// fs_set_executable_at set the owner-execute bit, where the target has one // One entry of a directory listing. Deliberately minimal: the editor wants // names, and whether to descend. @@ -59,9 +69,10 @@ Fs_Entry :: struct { } // One entry of a directory in the detail SPEC.md §3's directory hash needs: -// which of the three shapes it is, and - for a regular file - whether it is -// executable. Distinct from Fs_Entry above, which answers the editor's much -// smaller question (a name, and whether to descend into it). +// which of the three shapes it is - plus, for a regular file, whether it is +// executable, which §3 does not hash but §15's cache preserves. Distinct from +// Fs_Entry above, which answers the editor's much smaller question (a name, +// and whether to descend into it). // // `kind` is decided **without following symlinks**: §3 hashes a link entry as // its target string rather than resolving through it, so a link *to* a @@ -76,11 +87,13 @@ Fs_Node_Kind :: enum { Fs_Dir_Entry :: struct { name: string, kind: Fs_Node_Kind, - // .Regular only. **False wherever the target cannot report the bit**, rather - // than a third "unknown" state: WASI's filestat carries no permission bits - // at all and Windows has no POSIX exec bit, so on those two targets this is - // always false. That is the language's answer, not a gap - see hash.odin's - // directory section and LANGUAGE.md. + // .Regular only, and **not part of any hash**: §3 carries no permission bit + // (see hash.odin's directory section), precisely because this is the one + // field a target can be unable to answer - WASI's filestat has no permission + // bits at all and Windows has no POSIX exec bit, so on those two it is always + // false. It exists for cache_store.odin, which puts the bit back when it + // copies a tree, so that caching a build output does not strip it. Anything + // asking what a directory *is* should ignore this field. is_executable: bool, } diff --git a/src/fs_linux.odin b/src/fs_linux.odin index 009f0c5..2e8ce9f 100644 --- a/src/fs_linux.odin +++ b/src/fs_linux.odin @@ -1,5 +1,6 @@ package hashedbuild +import "core:fmt" import "core:os" import "core:strings" import "core:sys/linux" @@ -203,9 +204,10 @@ fs_list_entries_at :: proc(parent: Fs_Fd, allocator := context.allocator) -> ([] append(&entries, Fs_Dir_Entry { name = strings.clone(name, allocator), kind = kind, - // §3 hashes "the executable flag only - not full POSIX mode", and the - // owner bit is the one that means "this is a program". Group and other - // are part of who may run it, which is not part of what it is. + // Not for §3, which hashes no permission bit - this is for the cache, + // which restores the bit when it copies a tree (cache_store.odin). The + // owner bit is the one that means "this is a program"; group and other + // are about who may run it. is_executable = kind == .Regular && .IXUSR in stat.mode, }) } @@ -226,3 +228,50 @@ fs_list_dir :: proc(path: string, allocator := context.allocator) -> ([]Fs_Entry } return entries[:], .None } + +// ---- directory-as-a-value operations (§3's directory hash, §15's cached) ----- + +@(private = "file") S_IFMT :: u32(0o170000) +@(private = "file") S_IFDIR :: u32(0o040000) +@(private = "file") S_IFREG :: u32(0o100000) +@(private = "file") S_IFLNK :: u32(0o120000) + + +fs_mkdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + cname := strings.clone_to_cstring(name, context.temp_allocator) + ret := linux.syscall(linux.SYS_mkdirat, linux.Fd(parent), cast(rawptr)cname, u32(0o755)) + if ret < 0 do return fs_errno_to_error(linux.Errno(-ret)) + return .None +} + +fs_rename_at :: proc(parent: Fs_Fd, old_name: string, new_name: string) -> Fs_Error { + cold := strings.clone_to_cstring(old_name, context.temp_allocator) + cnew := strings.clone_to_cstring(new_name, context.temp_allocator) + ret := linux.syscall( + linux.SYS_renameat, linux.Fd(parent), cast(rawptr)cold, linux.Fd(parent), cast(rawptr)cnew, + ) + if ret < 0 do return fs_errno_to_error(linux.Errno(-ret)) + return .None +} + +fs_unlink_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + cname := strings.clone_to_cstring(name, context.temp_allocator) + return fs_errno_to_error(linux.unlinkat(linux.Fd(parent), cname, nil)) +} + +fs_rmdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + cname := strings.clone_to_cstring(name, context.temp_allocator) + return fs_errno_to_error(linux.unlinkat(linux.Fd(parent), cname, {.REMOVEDIR})) +} + +// The executable bit, on the one target that has one. Only ever used to put +// back a bit that was read off a file being copied into the cache, so that +// caching a build output does not quietly strip it - never to grant execute to +// something that did not already have it. Nothing about a value's identity +// depends on it (§3 hashes no permission bit); this is fidelity, not semantics. +fs_set_executable_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + cname := strings.clone_to_cstring(name, context.temp_allocator) + ret := linux.syscall(linux.SYS_fchmodat, linux.Fd(parent), cast(rawptr)cname, u32(0o755), 0) + if ret < 0 do return fs_errno_to_error(linux.Errno(-ret)) + return .None +} diff --git a/src/fs_wasi.odin b/src/fs_wasi.odin index 9e65b65..97ff0ba 100644 --- a/src/fs_wasi.odin +++ b/src/fs_wasi.odin @@ -40,6 +40,12 @@ DIR_RIGHTS :: wasi.rights_t{ .FD_READDIR, .FD_FILESTAT_GET, .PATH_OPEN, .PATH_CREATE_FILE, .PATH_CREATE_DIRECTORY, .PATH_FILESTAT_GET, .PATH_READLINK, .PATH_SYMLINK, .PATH_UNLINK_FILE, .PATH_REMOVE_DIRECTORY, + // Both halves of a rename, for §15's cache: an entry is built under a + // temporary name and renamed into place, and preview1 checks the source + // descriptor for one right and the destination's for the other. Asking for + // only one gets ENOTCAPABLE, which surfaces as a bare "Access" - which is + // exactly what the WASI smoke test caught. + .PATH_RENAME_SOURCE, .PATH_RENAME_TARGET, } @(private = "file") @@ -351,3 +357,40 @@ fs_list_dir :: proc(path: string, allocator := context.allocator) -> ([]Fs_Entry } return entries[:], .None } + +// ---- directory-as-a-value operations (§3's directory hash, §15's cached) --------- + + +fs_mkdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, rel, ok := rebase_absolute(parent, name) + if !ok || dir == FS_INVALID_FD do return .Access + return to_fs_error(wasi.path_create_directory(wasi.fd_t(dir), rel)) +} + +fs_rename_at :: proc(parent: Fs_Fd, old_name: string, new_name: string) -> Fs_Error { + old_dir, old_rel, old_ok := rebase_absolute(parent, old_name) + new_dir, new_rel, new_ok := rebase_absolute(parent, new_name) + if !old_ok || !new_ok || old_dir == FS_INVALID_FD || new_dir == FS_INVALID_FD do return .Access + return to_fs_error(wasi.path_rename(wasi.fd_t(old_dir), old_rel, wasi.fd_t(new_dir), new_rel)) +} + +fs_unlink_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, rel, ok := rebase_absolute(parent, name) + if !ok || dir == FS_INVALID_FD do return .Access + return to_fs_error(wasi.path_unlink_file(wasi.fd_t(dir), rel)) +} + +fs_rmdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, rel, ok := rebase_absolute(parent, name) + if !ok || dir == FS_INVALID_FD do return .Access + return to_fs_error(wasi.path_remove_directory(wasi.fd_t(dir), rel)) +} + +// preview1 has no chmod of any kind, and nothing to set - see Fs_Entry. The +// caller only ever asks for a bit it just read back as set, and this target +// never reads one as set, so succeeding without doing anything is exactly +// right rather than a swallowed failure. It cannot affect a digest either +// way: §3 hashes no permission bit. +fs_set_executable_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + return .None +} diff --git a/src/fs_windows.odin b/src/fs_windows.odin index 2a92cc0..1858cfb 100644 --- a/src/fs_windows.odin +++ b/src/fs_windows.odin @@ -672,3 +672,56 @@ name_length :: proc(buf: []u16) -> int { for c, i in buf do if c == 0 do return i return len(buf) } + +// ---- directory-as-a-value operations (§3's directory hash, §15's cached) ----- +// +// All six reach a child the same way every other operation on this target +// does: look the descriptor's directory path out of the slot table, join the +// name onto it, and call Win32 with the result. See this file's header for +// what that costs relative to the *at() family. + + +fs_mkdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, ok := dir_path_of(parent) + if !ok do return .Not_Directory + if !windows.CreateDirectoryW(to_win_path(join_child(dir, name)), nil) do return last_error() + return .None +} + +// The atomic commit a cache entry is published by. MoveFileExW without +// MOVEFILE_REPLACE_EXISTING fails when the destination is taken, which is +// what the caller wants: a name already there means another run won the race, +// and its entry is the one to use. +fs_rename_at :: proc(parent: Fs_Fd, old_name: string, new_name: string) -> Fs_Error { + dir, ok := dir_path_of(parent) + if !ok do return .Not_Directory + moved := windows.MoveFileExW( + to_win_path(join_child(dir, old_name)), + to_win_path(join_child(dir, new_name)), + 0, + ) + if !moved do return last_error() + return .None +} + +fs_unlink_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, ok := dir_path_of(parent) + if !ok do return .Not_Directory + if !windows.DeleteFileW(to_win_path(join_child(dir, name))) do return last_error() + return .None +} + +fs_rmdir_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + dir, ok := dir_path_of(parent) + if !ok do return .Not_Directory + if !windows.RemoveDirectoryW(to_win_path(join_child(dir, name))) do return last_error() + return .None +} + +// Nothing to set - Windows has no POSIX execute bit, and this target never +// reports one as set either, so the caller only ever reaches here for a bit +// that was already false. See fs.odin's Fs_Entry and fs_wasi.odin's copy of +// this note. +fs_set_executable_at :: proc(parent: Fs_Fd, name: string) -> Fs_Error { + return .None +} diff --git a/src/hash.odin b/src/hash.odin index f09b6a0..cd86318 100644 --- a/src/hash.odin +++ b/src/hash.odin @@ -67,6 +67,10 @@ TAG_NATIVE :: 0x0e TAG_AST :: 0x0f TAG_CYCLIC :: 0x10 TAG_CYCLIC_NODE :: 0x11 +// Not a value's digest at all: §15's cache key, which is a closure's digest +// plus what the expression can read out of the dynamic `#arg`/`#self` stacks. +// It lives here so the tag space stays in one place. See hash_implicit.odin. +TAG_IMPLICIT_REACH :: 0x12 // Byte-lexicographic order on digests. Written as an explicit loop rather than // slice.cmp because a `proc` parameter isn't addressable, so it can't be sliced. @@ -315,7 +319,7 @@ value_digest_walk :: proc(v: Value, w: ^Hash_Walk) -> (Value_Digest, Hash_Fail) // §3 spells the directory hash out: // // dir_entry_hash(name, entry) = -// hash(name, "file", content_hash, is_executable) +// hash(name, "file", content_hash) // hash(name, "dir", child_dir_hash) // hash(name, "symlink", target_path_string) // dir_hash = hash(sorted [dir_entry_hash(name, entry) for each entry]) @@ -328,13 +332,22 @@ value_digest_walk :: proc(v: Value, w: ^Hash_Walk) -> (Value_Digest, Hash_Fail) // in principle collide with a regular file whose content happened to be its // entry digests laid end to end. // -// **The executable bit is the file's owner-execute bit where the target can -// see one, and false everywhere else.** WASI's filestat has no permission bits -// and Windows has no POSIX exec bit (fs.odin), so a tree containing an -// executable hashes differently there than on Linux. That is the language's -// answer rather than a gap: a Windows checkout genuinely has no exec bits, so -// reporting one would be inventing it - the same position git takes with -// core.filemode. LANGUAGE.md says so where a user will meet it. +// **No permission bit is hashed** (§3, resolved 2026-08-31). An earlier form of +// §3 kept the owner-execute bit here, false on the targets that cannot see one. +// The trouble is that this makes a tree's identity depend on where it was +// checked out, and git shows why that is not hypothetical: it records exactly +// this one bit (100644 vs 100755) and sets core.fileMode=false on Windows, so +// the bit round-trips through a repository without ever existing in the working +// tree. The same commit is executable on Linux and not on Windows - this +// repository's own scripts/ would have hashed two ways. +// +// git escapes that by *remembering* the bit rather than re-deriving it, which +// is not available here: a File is a handle onto a live directory, with no +// index beside it to consult. Between a digest that disagrees across targets +// and one that ignores a bit two of the three cannot see, §3 takes the second. +// Fs_Dir_Entry still reports it and cache_store.odin still restores it when +// copying a tree, so caching a build output does not silently strip it - that +// is fidelity in the store, not identity in the language. // // A symlink is hashed as its target *string*, never resolved (§3), which is // also what keeps this recursion finite: nothing here follows a link, so the @@ -392,11 +405,12 @@ dir_entry_digest :: proc(dir: Fs_Fd, entry: Fs_Dir_Entry) -> (Value_Digest, Hash } defer delete(content) - payload: [2 * DIGEST_SIZE + 1]u8 + // Name and content, and nothing else - see the note on permission bits + // above. entry.is_executable is deliberately not read here. + payload: [2 * DIGEST_SIZE]u8 ch := sha256_of(content) copy(payload[0:], name[:]) copy(payload[DIGEST_SIZE:], ch[:]) - payload[2 * DIGEST_SIZE] = 1 if entry.is_executable else 0 return sha256_tagged(TAG_DIR_ENTRY_FILE, payload[:]), HASH_OK case .Directory: diff --git a/src/hash_implicit.odin b/src/hash_implicit.odin new file mode 100644 index 0000000..0d56e35 --- /dev/null +++ b/src/hash_implicit.odin @@ -0,0 +1,95 @@ +package hashedbuild + +import "core:strconv" +import "core:strings" + +// The part of a cached expression's identity that a *closure* digest cannot +// carry, and the reason §15's cache key is not simply `sha256 `. +// +// hash_function.odin encodes a closure as its body's shape plus the values it +// captures, which is exactly right for a Function value. But `#arg`/`#argN`, +// `#self`/`#selfN` and a bare Hole are not captures: they are dynamic lookups +// into the interpreter's own stacks (§9), which is precisely what lets them +// reach through a hard boundary the way a lexical name cannot. Nothing about +// them is in the closure, so nothing about them is in its digest. +// +// Left there, `let f func (cached (#arg + 1))` would have one entry for every +// argument, and `f 10` would answer 2. `cached` therefore mixes the reachable +// stack entries in on top of the closure digest (see eval.odin's eval_cached). +// +// **How far an expression can reach is decided statically**, by the largest N +// written anywhere in it - a bare `#arg`, `#self` or Hole counting as 1. That +// bound holds even for a `#argN` written inside a function the expression +// itself calls: entering that function pushes a frame, so an N there reaches +// N-1 of the frames that were already on the stack when `cached` ran. Whatever +// the expression pushes for itself follows from its code and its captured +// environment, and the closure digest already covers both. +// +// An expression that mentions none of them is left alone entirely - its key is +// the closure digest unchanged, so this costs nothing in the ordinary case. + +// `base`, plus the stack entries the expression can reach, innermost first. +// Tagged separately from everything in hash.odin because it is not a value's +// digest: it is a cache key, and only `cached` ever computes one. +implicit_reach_digest :: proc( + base: Value_Digest, args: []Value, selves: []Value, interp: ^Interpreter, +) -> (Value_Digest, Hash_Fail) { + s: Digest_Stream + digest_stream_begin(&s, TAG_IMPLICIT_REACH) + digest_stream_digest(&s, base) + + for group in ([][]Value{args, selves}) { + count: [8]u8 + n := u64(len(group)) + for i in 0 ..< 8 do count[i] = u8((n >> (8 * uint(i))) & 0xff) + digest_stream_bytes(&s, count[:]) + for v in group { + d, f := value_digest(v, interp) + if f.kind != .None do return {}, f + digest_stream_digest(&s, d) + } + } + return digest_stream_end(&s), HASH_OK +} + +// How deep into each stack the expression can reach. Zero for both means it +// reads neither. +implicit_reach :: proc(interp: ^Interpreter, idx: Node_Idx, max_arg: ^int, max_self: ^int) { + n := interp.ast.nodes[idx] + + #partial switch n.kind { + case .Hole: + // An evaluated Hole is `#arg` by another spelling (eval.odin). + if max_arg^ < 1 do max_arg^ = 1 + case .Implicit_Name: + note_implicit(node_text(interp, idx), max_arg, max_self) + } + + for i in 0 ..< int(n.children_count) { + implicit_reach(interp, interp.ast.extra_children[int(n.children_start) + i], max_arg, max_self) + } +} + +// "#arg" / "#arg3" / "#self" / "#self2". Anything else - `#context`, or a +// malformed name - is left alone: it has no stack to reach into, and it fails +// when the expression runs. +@(private = "file") +note_implicit :: proc(text: string, max_arg: ^int, max_self: ^int) { + rest := text[1:] if len(text) > 0 else text + + target: ^int + prefix: string + switch { + case strings.has_prefix(rest, "arg"): target, prefix = max_arg, "arg" + case strings.has_prefix(rest, "self"): target, prefix = max_self, "self" + case: return + } + + level := 1 + if digits := rest[len(prefix):]; len(digits) > 0 { + v, ok := strconv.parse_int(digits) + if !ok do return + level = v + } + if level > target^ do target^ = level +} diff --git a/src/hash_linux_test.odin b/src/hash_linux_test.odin index c6a1a0a..b9be229 100644 --- a/src/hash_linux_test.odin +++ b/src/hash_linux_test.odin @@ -1,8 +1,8 @@ -// The executable bit is the one part of SPEC.md §3's directory hash that only -// one target can see, so it is the one part tested on only one target. WASI's +// SPEC.md §3 hashes no permission bit, and Linux is the only target that can +// prove it: it is the only one that has an executable bit to set. WASI's // filestat carries no permission bits and Windows has no POSIX exec bit -// (fs.odin), and hashing there treats every file as non-executable - which is -// the language's answer, not a gap, and is what LANGUAGE.md documents. +// (fs.odin), so the same assertion elsewhere would hold without establishing +// anything - which is why this file is `#+build linux` rather than a `when`. #+build linux package hashedbuild @@ -27,7 +27,7 @@ eval_digest :: proc(t: ^testing.T, path: string) -> string { } @(test) -test_the_executable_bit_is_part_of_a_directory_hash :: proc(t: ^testing.T) { +test_the_executable_bit_is_not_part_of_a_directory_hash :: proc(t: ^testing.T) { root := strings.concatenate({repo_root(), "/.hash_linux_test_exec"}) defer delete(root) file := strings.concatenate({root, "/build.sh"}) @@ -40,10 +40,10 @@ test_the_executable_bit_is_part_of_a_directory_hash :: proc(t: ^testing.T) { testing.expect(t, os.make_directory(root) == nil, "could not create the scratch tree") _ = os.write_entire_file(file, transmute([]u8)string("#!/bin/sh\necho hi\n")) - // Two hashes of the same bytes under the same name, differing only in - // whether the file is a program. §3 asks for exactly that distinction, and - // it is a real one in a build: a checked-out `configure` that lost its bit - // is not the same tree. + // Two hashes of the same bytes under the same name, differing only in whether + // the file is a program. §3 does not distinguish them: only Linux can see the + // difference at all, so hashing it would make one source tree two values + // depending on where it was checked out. cname := strings.clone_to_cstring(file, context.temp_allocator) testing.expect(t, linux.chmod(cname, {.IRUSR, .IWUSR}) == .NONE) plain := eval_digest(t, root) @@ -51,11 +51,23 @@ test_the_executable_bit_is_part_of_a_directory_hash :: proc(t: ^testing.T) { testing.expect(t, linux.chmod(cname, {.IRUSR, .IWUSR, .IXUSR}) == .NONE) executable := eval_digest(t, root) - testing.expect(t, plain != executable, "the exec bit is part of what a directory is") + // That the bit actually landed is half the test - without it the comparison + // would hold for the wrong reason. + dir_fd, oerr := fs_open_dir_path(root) + testing.expect(t, oerr == .None, "could not open the scratch tree") + if oerr == .None { + defer fs_close(dir_fd) + entries, lerr := fs_list_entries_at(dir_fd, context.temp_allocator) + testing.expect(t, lerr == .None) + saw := false + for entry in entries do if entry.name == "build.sh" && entry.is_executable do saw = true + testing.expect(t, saw, "chmod did not set a bit for the digest to ignore") + } - // ...and it is only the owner bit that counts. §3 says "the executable flag - // only - not full POSIX mode", so who else may run it is not part of what - // the file is. + testing.expect_value(t, plain, executable) + + // No other mode bit counts either - the rule is that permissions are not part + // of what a tree is, not that one particular bit was singled out. testing.expect(t, linux.chmod(cname, {.IRUSR, .IWUSR, .IXUSR, .IRGRP, .IROTH}) == .NONE) - testing.expect_value(t, eval_digest(t, root), executable) + testing.expect_value(t, eval_digest(t, root), plain) } diff --git a/src/main.odin b/src/main.odin index c833370..89b297b 100644 --- a/src/main.odin +++ b/src/main.odin @@ -17,7 +17,8 @@ Options: -a, --ast Print the parsed AST before evaluating -e, --eval Evaluate like one REPL submission and exit --cache-dir - Override ctx.cache's location (SPEC.md §16) + Override where ctx.cache and cached keep + their entries (SPEC.md §15/§16) -h, --help Print this help and exit --version Print the version and exit`