From 6d625272e655e95fffbb48ec70ea73cd7c197855 Mon Sep 17 00:00:00 2001 From: Jan Strakowski Date: Mon, 31 Aug 2026 09:56:43 +0200 Subject: [PATCH 1/2] Finish hashing: directories, closures, cycles, and ctx.cache MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `sha256` now answers for every value. The three kinds LANGUAGE.md listed as unbuilt each needed a language decision first, and each was taken: **A directory File** hashes over its entries per SPEC.md §3 - name plus content hash and exec bit for a regular file, the child's own directory hash for a sub-directory, the target string for a symlink, sorted by name. The open question was the exec bit, which WASI and Windows cannot report: unobservable now means *not executable*, so a tree containing an executable hashes differently there than on Linux. §3 stays as written, and a Windows checkout genuinely has no exec bits - the same position git takes with core.filemode. An entry that is none of the three shapes fails rather than being skipped. This is the one digest that does I/O, so it happens at the first demand and needs ctx.permissions.io at that moment, then is memoised - a File is an immutable handle (§3), so its digest cannot change under a program. The evaluator warms both operands before a comparison (hash_materialize), since values_equal is reached from table_find and can do no I/O there. **A Function** is its body's shape plus the values it captures. Free-variable analysis is what makes that useful: a closure's digest survives an unrelated binding changing nearby, which is what §15's `cached` will need. The encoding is of the program, not its meaning - renaming a local is a different closure. Builtins hash as the operation they are, and carry what they were partially applied to. **A cyclic value** goes through SCC decomposition: everything outside a cycle is folded as usual, and nodes inside one get a canonical form of the cycle - bisimulation classes, numbered by a deterministic walk from the node itself. That has to agree with values_equal, which already compares cycles by bisimulation, or equal values would have two content addresses. **ctx.cache** hashes as a bare tag. The only thing that would tell two apart is the directory it is rooted at, which is exactly the path §9 keeps out of the language. One new fs operation, `fs_list_entries_at`, on all three backends: a descriptor's entries with their kind and exec bit. sha256_tagged now streams rather than concatenating, so hashing a large value no longer copies it. SPEC.md records the four resolutions; LANGUAGE.md's "what isn't built yet" loses the hashing paragraph and gains three sections with runnable snippets; three new examples, asserted by the suite. The exec bit is tested on Linux only, because it is the only target that has one. Co-Authored-By: Claude Opus 5 --- LANGUAGE.md | 109 +++++-- SPEC.md | 14 +- examples/README.md | 3 + examples/hashing-cyclic.hb | 39 +++ examples/hashing-directories.hb | 42 +++ examples/hashing-functions.hb | 41 +++ src/builtins_fs.odin | 26 +- src/eval.odin | 18 +- src/examples_test.odin | 9 + src/fs.odin | 33 +++ src/fs_linux.odin | 58 ++++ src/fs_wasi.odin | 45 +++ src/fs_windows.odin | 40 +++ src/hash.odin | 469 ++++++++++++++++++++++++------ src/hash_cyclic.odin | 495 ++++++++++++++++++++++++++++++++ src/hash_function.odin | 261 +++++++++++++++++ src/hash_linux_test.odin | 61 ++++ src/hash_test.odin | 299 ++++++++++++++++++- src/rec_build_test.odin | 132 ++++++++- src/value.odin | 37 ++- 20 files changed, 2091 insertions(+), 140 deletions(-) create mode 100644 examples/hashing-cyclic.hb create mode 100644 examples/hashing-directories.hb create mode 100644 examples/hashing-functions.hb create mode 100644 src/hash_cyclic.odin create mode 100644 src/hash_function.odin create mode 100644 src/hash_linux_test.odin diff --git a/LANGUAGE.md b/LANGUAGE.md index 5f6aa99..7b4fe3c 100644 --- a/LANGUAGE.md +++ b/LANGUAGE.md @@ -406,14 +406,89 @@ same value when their bytes match, however they were reached. (loadfile "a.txt") == (loadfile "copy-of-a.txt") // true, if the bytes match ``` -Three kinds of value have no digest yet, and say so rather than inventing one: -a **directory** `File` (§3 hashes one over its entries including each file's -executable bit, which only the Linux target can report — see below), a -`Function`, and -`ctx.cache`. Hashing one is a fatal failure like any other (§8). - → `examples/hashing.hb` (§3, §6, §15) +### Directories + +A **directory** `File` hashes over its entries (§3), sorted by name so the +digest is the tree's rather than the order the filesystem listed it in. Each +entry contributes its name, plus its content hash and executable bit for a +regular file, its own directory hash for a sub-directory, or its target string +for a symlink — which is never followed, so a link to a directory is a link, +not a directory. + +```hashedbuild +sha256 loadfile "examples" // the digest of a whole tree +(loadfile "examples") == (loadfile "examples") // true — one tree, two handles +``` + +Two things about it are worth knowing before you rely on it: + +- **It reads the disk, and that read needs `io`.** Everything else hashes what + the value already holds; a directory's children are on the filesystem. The + read happens at the first `sha256` or comparison that needs the digest and is + checked against `ctx.permissions.io` at that moment, so a context that + revoked `io` can't pull a tree's contents through a handle it was handed. It + happens once: a `File` is an immutable handle, so the digest is fixed from + then on, and seeing a change means loading the directory again. +- **The executable bit is Linux-only, and a tree containing one hashes + differently elsewhere.** WASI's `filestat` has no permission bits and Windows + has no POSIX executable bit, so on those targets every file hashes as + non-executable. That is the language's answer rather than a gap — a Windows + checkout genuinely has no executable bits, and reporting one would be + inventing it, the same position git takes with `core.filemode`. If you need a + digest that agrees across all three, keep executables out of the tree. + +An entry that is neither a file, a directory, nor a symlink — a socket, a +device node — fails rather than being skipped, since a digest that ignored part +of a tree would call two different trees the same value. + +→ `examples/hashing-directories.hb` (§3, §9) + +### Functions + +A **closure** hashes as the shape of its body plus the values it captures +(§15) — so two functions hash alike exactly when they would compute the same +thing, and what else happened to be in scope where they were written doesn't +enter into it: + +```hashedbuild +(sha256 (let x 1; let unrelated "zz"; func (#arg + x))) + == (sha256 (let x 1; func (#arg + x))) // true +(sha256 (let x 1; func (#arg + x))) + == (sha256 (let x 2; func (#arg + x))) // false — a different capture +``` + +The digest is of the program, though, not of what the program means: renaming a +local or respelling a literal gives a different function. A builtin has no body +to take a shape from, so it hashes as the operation it is — and a partially +applied one carries what it was applied to, which is why two `chperm` results +differ exactly when they grant different things. + +→ `examples/hashing-functions.hb` (§15, §16) + +### Values that reach themselves + +A cyclic `let rec` value (see above) hashes too, but not by the same route: an +ordinary digest folds up from the leaves, and a cycle has none. Such a value is +instead reduced to a canonical form of the cycle and hashed from that, with two +properties that are the whole reason for the exercise — the digest doesn't +depend on which node you started from, and it doesn't depend on how the cycle +was written: + +```hashedbuild +let rec g { .n = 1, .next = g }; +let rec h { .a = { .n = 1, .next = h.b }, .b = { .n = 1, .next = h.a } }; +(g == h.a) and ((sha256 g) == (sha256 h.a)) // true — both halves +``` + +That pairing is the requirement, not a coincidence. `g` and `h.a` are *equal* +under §6 because unrolling either gives the same infinite tree, so a digest +that told them apart would give a content-addressed language two addresses for +one value. + +→ `examples/hashing-cyclic.hb` (§6, §10, §15) + ## Context and permissions `ctx` is the ambient context. The filesystem builtins check @@ -468,22 +543,12 @@ uncatchable, but the work already in flight finishes first. Parsed, specified, and rejected by the evaluator with "not implemented": `import` and `cached`. -Partly built: **hashing**. `sha256` works for every value except a directory -`File`, a `Function`, and `ctx.cache`. The directory case is the interesting -one — `SPEC.md` §3 defines a directory's hash over its entries including each -file's executable bit, and two of the three targets cannot report one: WASI's -`filestat` has no permission bits at all, and Windows has no POSIX exec bit. -So there is no way to compute the specified digest everywhere the interpreter -runs. Building it means first deciding what a directory hashes as somewhere -that cannot see an exec bit. `Function` is unbuilt because §15 needs it for `cached` but never -says how a closure is encoded. A **cyclic value** (above) is the third case, and -unbuilt for a related reason: the digest is a Merkle fold, a composite's hash -built from its children's, and a cycle has no bottom to start from. `SPEC.md` -§6 describes what the answer looks like — components hashed canonically, so the -digest does not depend on where the walk entered the cycle — but §3 pins what a -digest encodes, so it is a spec decision first. `sha256` of one fails cleanly -meanwhile. Equality over cyclic values *is* built, and does not depend on any of -this. +**Hashing is complete**: `sha256` answers for every value, including the three +that used to be listed here — a directory `File`, a `Function`, and a cyclic +value — plus `ctx.cache`. See the Hashing section above for what each of them +encodes, and for the one place the answer is target-specific: a directory +containing an executable hashes differently on Windows and WASI than on Linux, +because neither of those has an executable bit to report. Also absent: `true`/`false` literals, loops of any kind (recursion is the only repetition there is — see above), a `Bytes`-returning counterpart to diff --git a/SPEC.md b/SPEC.md index 2c5272b..1e7ad20 100644 --- a/SPEC.md +++ b/SPEC.md @@ -51,7 +51,11 @@ A condition (of `then`, `and`, `or`, `is`) can itself be, or contain, an async e hash(name, "symlink", target_path_string) // target is NOT followed/resolved dir_hash = hash(sorted [dir_entry_hash(name, entry) for each entry]) ``` - A symlink entry hashes the link itself (its target path string) rather than resolving through it — consistent with symlink handling being a property of the containing directory, not a standalone `File` value in its own right. + A symlink entry hashes the link itself (its target path string) rather than resolving through it — consistent with symlink handling being a property of the containing directory, not a standalone `File` value in its own right. An entry that is none of the three — a fifo, a socket, a device node — has no encoding here, and hashing a directory containing one fails rather than skipping it: a digest that ignored part of a tree would call two different trees the same value. + + **The executable bit where it cannot be observed (resolved 2026-08-31).** `is_executable` above is the entry's owner-execute bit *where the host reports one*, and **false wherever it does not**. WASI's `filestat` carries no permission bits at all and Windows has no POSIX executable bit, so on those two targets every regular file hashes as non-executable. The consequence is deliberate and worth stating plainly: a tree containing an executable has a different `dir_hash` on Linux than in a browser or on Windows. The alternative considered was dropping the bit from the encoding entirely, which would make one digest hold everywhere at the cost of a distinction a build language wants — a checked-out `configure` that lost its bit is not the same tree. Reporting the bit as *set* where it cannot be seen was never on the table; a Windows checkout genuinely has no executable bits, so claiming one would be inventing data. This is the same position git takes with `core.filemode`. + + **When the entries are read (resolved 2026-08-31).** A directory `File`'s children live on the filesystem rather than in the value, so unlike every other digest this one performs I/O. It happens **at the first demand** — the first `sha256` or comparison that needs the digest — and is subject to `ctx.permissions.io` (§9) at that moment, exactly as `loadfile` is: a context that has revoked `io` cannot read a tree through a handle it was passed. The result is then part of the value and is never re-read, which is what `File` being an immutable handle (above) requires: a value whose digest changed under a program because someone touched the tree would not be one. Observing a change means loading the directory again, which produces a new value. **Display (resolved 2026-08-27).** A `File` displays **the path it was reached by**, made absolute and cleaned of `.`/`..` segments — recorded when the value is constructed, not resolved through the filesystem afterwards. Two consequences worth stating: a symlinked route displays as the route taken rather than the target it resolves to, and a file renamed after it was loaded still displays the path it was loaded from. This is what makes the rule implementable off Linux at all — WASI (and the portable `*at()` family) offer no way to turn an open descriptor back into a path. @@ -333,6 +337,14 @@ Two builtins operationalizing §6's "every value is hashable" claim. Like `impor **Resolved 2026-08-26, amended 2026-08-28**: the hash is one consistent underlying system — one canonical byte encoding, one hash mechanism, shared with §6's general value-hash (used for ordering/equality everywhere, e.g. `Table`'s key-sorted hash and `File`'s directory hash, §3/§5). The encoding is a Merkle construction: a composite value hashes its children to fixed-width digests and mixes *those*, never inlining a child's own encoding. That is forced by §3, which pins a regular `File`'s hash to `hash(content_bytes)` with no tag or length of its own — so that `sha256 ` is the digest `sha256sum` reports for the same bytes — and an untagged, variable-length encoding cannot be inlined unambiguously. Leaves are otherwise domain-separated by a tag byte, so that values of different types with the same payload (`Integer` 5 and `Float` 5.0, which §6 does not equate) do not collide. +**A `Function`'s encoding (resolved 2026-08-31).** `cached` below hashes an expression "as a function", which needs a closure to have an encoding; this is it. A closure is **the shape of its body, mixed with the values it captures** — the body's syntax tree node by node, each leaf including its own spelling, and then for every free name of that body the digest of the value that name stood for where the closure was made, mixed in under the name. A builtin (§16) has no body, so it encodes as its operation's name together with anything it was partially applied to. A closure that reads `ctx` (§9) mixes in the context it captured; one that does not, does not — ambient authority is part of what a closure is only where the closure can see it. + +Two things follow, and are the point rather than a limitation. Two closures hash alike exactly when they would compute the same thing, so a cache key survives an unrelated binding changing nearby — which is what makes `cached` hit at all. And the encoding is of the *program*, not of what the program means: renaming a local, or spelling a literal differently, is a different closure. Alpha-equivalence is not promised. + +**A cyclic value's encoding (resolved 2026-08-31).** §10's `let rec` can build a `Table` that reaches itself, and §6 says every value is hashable. A Merkle fold has no bottom to start from on a cycle, so a value that reaches itself is encoded differently: the graph is split into strongly connected components, everything outside a cycle is folded as usual, and each node **inside** one is encoded by a canonical form of the cycle reachable from it — the nodes reduced by bisimulation, then numbered in the order a deterministic walk from that node first meets them. Two consequences are the whole requirement: the digest does not depend on which node the walk entered by, and it does not depend on how the cycle was written. A one-node cycle and a two-node cycle that unroll to the same infinite tree are **equal** under §6, so they must — and do — hash alike. Anything weaker would give a content-addressed language two addresses for one value. + +**`ctx.cache`'s encoding (resolved 2026-08-31).** §9's cache is a value, so §6 makes it hashable; it encodes as a bare tag and nothing else, so all of them hash alike. It has no content to hash, and the one thing that would tell two apart — the directory it is rooted at — is precisely the path §9 spends its last paragraph keeping out of reach of programs. Hashing it to that path would hand the path back through a side door. + **`cached`'s mechanism (resolved 2026-08-26):** the cached expression is treated *as a function* and hashed as one — the cache key is the hash (per the one system above) of that function representation, not a hash of its resolved output value. `cached` is not inherently async by itself; async-ness is controlled explicitly by *where* `async` (§2) is placed: `async cached ` makes the cache lookup/store itself asynchronous, while `cached async ` instead makes the underlying expression's own evaluation asynchronous, with the caching wrapper around it synchronous. > TODO: Where does the cache actually live (on-disk location, process-local vs. shared/distributed) — not addressed by this round's resolution of the cache-*key* mechanism. diff --git a/examples/README.md b/examples/README.md index 6f14431..981eb28 100644 --- a/examples/README.md +++ b/examples/README.md @@ -56,6 +56,9 @@ prose around each of these. | `files-sandboxed.hb` | Directory handles, contained sub-paths, `filetext`, path display | | `files-symlink.hb` | `readlink` — and why symlinks aren't values of their own | | `hashing.hb` | `sha256`, and the content identity that makes two Files one value | +| `hashing-directories.hb` | A directory's hash: its entries, and the one digest that reads | +| `hashing-functions.hb` | A closure's hash: its body's shape and the values it captures | +| `hashing-cyclic.hb` | Hashing a value that reaches itself, canonically | | `option-picker.hb` | A real program: read, branch, write into `ctx.cache` | | `context-permissions.hb` | `ctx`, `chctx chperm`, `withctx` — capability narrowing | | `async-basics.hb` | Two reads in flight at once, awaited implicitly | diff --git a/examples/hashing-cyclic.hb b/examples/hashing-cyclic.hb new file mode 100644 index 0000000..0e502de --- /dev/null +++ b/examples/hashing-cyclic.hb @@ -0,0 +1,39 @@ +// Hashing a value that reaches itself (SPEC.md §6/§10). `let rec` can build a +// Table that contains itself (see examples/cyclic-data.hb), and every value is +// hashable - including that one. +// +// It cannot be hashed the way everything else is. An ordinary digest is a fold +// from the leaves upward, and a cycle has no leaves to start from. So a cyclic +// value's digest is instead a canonical form of the cycle itself: the shape is +// reduced to what genuinely differs, and the digest is read off that. The +// point of "canonical" is the two properties below - it does not matter which +// node you started from, and it does not matter how the cycle was written. +// +// That second one is the same rule equality already follows. `g` below is a +// one-node cycle and `h.a` is one node of a two-node cycle, and they are +// *equal*, because unrolling either gives the same infinite tree. A digest +// that disagreed with that would be a language where two equal values have two +// different content addresses. +// +// There are no boolean literals, so "false" is spelled `1 > 2`. +// +// Evaluates to { a_cycle_hashes: true, equal_cycles_hash_alike: true, +// shape_still_matters: true, either_end_agrees: true }. +let rec g { .n = 1, .next = g }; +let rec h { .a = { .n = 1, .next = h.b }, .b = { .n = 1, .next = h.a } }; +let rec differs { .n = 2, .next = differs }; +{ + // It terminates and produces a digest, which is the first thing to want. + .a_cycle_hashes = (sha256 g) == (sha256 g), + + // A 1-cycle and a 2-cycle that unroll the same way are one value, and hash + // as one value. + .equal_cycles_hash_alike = ((g == h.a) and ((sha256 g) == (sha256 h.a))), + + // Canonical is not constant: cycles that unroll differently still differ. + .shape_still_matters = ((sha256 g) == (sha256 differs)) == (1 > 2), + + // The same node reached two ways is one digest - the walk's entry point is + // not part of the answer. + .either_end_agrees = (sha256 g) == (sha256 g.next.next.next), +} diff --git a/examples/hashing-directories.hb b/examples/hashing-directories.hb new file mode 100644 index 0000000..a093a79 --- /dev/null +++ b/examples/hashing-directories.hb @@ -0,0 +1,42 @@ +// A directory `File`'s hash (SPEC.md §3). A directory is a value like any +// other, so `sha256` answers for one - but its children are on the disk rather +// than in the value, which makes this the one digest that reads. +// +// §3 computes it over the directory's entries, each hashed with its name and +// sorted by name so the answer is the tree's rather than readdir's: a regular +// file contributes its content hash and whether it is executable, a +// sub-directory contributes its own directory hash, and a symlink contributes +// its target *string*, never followed. Nothing about where the directory sits +// enters into it, which is why two handles on the same tree are one value. +// +// The digests themselves are deliberately not written down here: a tree +// containing an executable hashes differently on Windows and in the browser +// than on Linux, because neither of those targets has an executable bit to +// report (see LANGUAGE.md). What is the same everywhere are the properties +// below. +// +// Reading a directory is I/O, so the first `sha256` of one needs +// `ctx.permissions.io` like `loadfile` does - and it is only the first, since +// a `File` is an immutable handle (§3) and the digest is fixed once read. +// +// There are no boolean literals, so "false" is spelled `1 > 2` here, the same +// way examples/comparison-and-logic.hb spells it. +// +// Evaluates to { a_tree_is_not_its_file: true, one_tree_is_one_value: true, +// reading_twice_agrees: true }. +let here loadfile "."; +{ + // A directory holding a file is not that file. Both digests are built from + // the same bytes on disk, and §3's tagging is what keeps them apart. + .a_tree_is_not_its_file = ((sha256 here) == (sha256 loadfile "optiona.txt")) == (1 > 2), + + // Two separate handles, one tree. §3 makes a File's identity its content, + // so these are the same value even though they are different handles - and + // comparing them is what reads the second one. + .one_tree_is_one_value = here == (loadfile "."), + + // Two independent walks of the same directory agree. That is §3's "sorted + // by name for determinism" doing its job: the digest is the tree's, not the + // order the filesystem happened to hand the entries back in. + .reading_twice_agrees = (sha256 here) == (sha256 loadfile "."), +} diff --git a/examples/hashing-functions.hb b/examples/hashing-functions.hb new file mode 100644 index 0000000..ffee0c0 --- /dev/null +++ b/examples/hashing-functions.hb @@ -0,0 +1,41 @@ +// A `Function`'s hash (SPEC.md §15). A closure is a value, so `sha256` +// answers for one - and what it answers is the two things a closure actually +// is: the shape of its body, and the values it captures. +// +// So two functions hash alike exactly when they would compute the same thing. +// The same expression reading the same values is one function; a different +// body, or the same body reading a different value, is a different one. What +// does *not* enter into it is everything else that happened to be in scope +// where the closure was written - which is the property §15's `cached` needs, +// since a cache key that moved whenever an unrelated neighbour changed would +// miss every time. +// +// A builtin (§16) has no body to take a shape from, so it hashes as the +// operation it is. A partially applied one carries what it was built from, +// which is why the two `chperm` results below are told apart by the +// permission they grant rather than by being two objects. +// +// There are no boolean literals, so "false" is spelled `1 > 2`. +// +// Evaluates to { same_body_same_hash: true, different_body_differs: true, +// captures_count: true, neighbours_do_not: true, +// builtins_hash_by_what_they_are: true, +// a_builtin_carries_its_argument: true }. +{ + .same_body_same_hash = (sha256 func (#arg + 1)) == (sha256 func (#arg + 1)), + .different_body_differs = ((sha256 func (#arg + 1)) == (sha256 func (#arg + 2))) == (1 > 2), + + // Same body, different captured value: a different function. + .captures_count = + ((sha256 (let x 1; func (#arg + x))) == (sha256 (let x 2; func (#arg + x)))) == (1 > 2), + + // Same body, same captured value, different surroundings: one function. + .neighbours_do_not = + (sha256 (let x 1; let unrelated "zz"; func (#arg + x))) + == (sha256 (let x 1; func (#arg + x))), + + .builtins_hash_by_what_they_are = ((sha256 loadfile) == (sha256 createfile)) == (1 > 2), + .a_builtin_carries_its_argument = + ((sha256 chperm { .name = "io", .enabled = 1 < 2 }) + == (sha256 chperm { .name = "io", .enabled = 1 > 2 })) == (1 > 2), +} diff --git a/src/builtins_fs.odin b/src/builtins_fs.odin index 0be6367..4d4cfa6 100644 --- a/src/builtins_fs.odin +++ b/src/builtins_fs.odin @@ -66,9 +66,15 @@ resolve_cache_dir :: proc(override: string) -> string { return strings.concatenate({os.get_env_alloc("HOME", context.temp_allocator), "/.cache/hashedbuild"}) } +// `name` is what the builtin hashes as (hash_function.odin): a native has no +// body to take a shape from, so its identity is the operation it *is*, and +// the name is how that gets written down. It is the binding's own spelling +// below, and must stay stable for the same reason a tag byte must - see the +// note on renumbering in hash.odin. @(private = "file") -new_native_function :: proc(fn: Native_Fn, closure: Value = nil) -> Value { +new_native_function :: proc(name: string, fn: Native_Fn, closure: Value = nil) -> Value { f := new(Function_Value) + f.name = name f.native = fn f.native_closure = closure return f @@ -78,16 +84,18 @@ new_native_function :: proc(fn: Native_Fn, closure: Value = nil) -> Value { // the filesystem builtins, pre-bound by name (§16). make_global_env :: proc() -> ^Env { env := env_make_child(nil) - env_bind(env, "loadfile", new_native_function(builtin_loadfile)) - env_bind(env, "createfile", new_native_function(builtin_createfile)) - env_bind(env, "symlink", new_native_function(builtin_symlink)) - env_bind(env, "readlink", new_native_function(builtin_readlink)) - env_bind(env, "chperm", new_native_function(builtin_chperm)) - env_bind(env, "filetext", new_native_function(builtin_filetext)) + env_bind(env, "loadfile", new_native_function("loadfile", builtin_loadfile)) + env_bind(env, "createfile", new_native_function("createfile", builtin_createfile)) + env_bind(env, "symlink", new_native_function("symlink", builtin_symlink)) + env_bind(env, "readlink", new_native_function("readlink", builtin_readlink)) + env_bind(env, "chperm", new_native_function("chperm", builtin_chperm)) + env_bind(env, "filetext", new_native_function("filetext", builtin_filetext)) return env } -@(private = "file") +// Package-visible rather than file-private because hash.odin asks it too: a +// directory File's digest is read off the disk the first time anything needs +// it, and that read is an I/O operation like any other here (SPEC.md §3/§9). ctx_allows_io :: proc(interp: ^Interpreter) -> bool { t, is_table := interp.current_ctx.(^Table_Value) if !is_table do return false @@ -627,7 +635,7 @@ builtin_chperm :: proc(interp: ^Interpreter, _: Value, arg: Value) -> (Value, bo closure := new(Table_Value) append(&closure.entries, Table_Entry_Value{key = "name", value = name_str}) append(&closure.entries, Table_Entry_Value{key = "enabled", value = enabled_bool}) - return new_native_function(apply_chperm, closure), true + return new_native_function("chperm.apply", apply_chperm, closure), true } // The actual oldctx -> newctx function chperm returns: a copy of `old_ctx` diff --git a/src/eval.odin b/src/eval.odin index c5e5f9d..4299bc2 100644 --- a/src/eval.odin +++ b/src/eval.odin @@ -610,6 +610,17 @@ eval_binary :: proc(interp: ^Interpreter, node: Node_Idx, env: ^Env) -> (Value, if !lok do return nil, false r, rok = concrete_value(interp, r) if !rok do return nil, false + // Comparing a directory File means comparing what is inside it (§3), so + // the comparison is a read - and a read is where `io` is checked (§9). + // values_equal itself is reached from table_find on the hot path of every + // field access and can do no I/O, so the operands are warmed here, while + // there is still a context to ask. See hash.odin's hash_materialize. + if f := hash_materialize(interp, l); f.kind != .None { + return fail(interp, hash_error_message(f)) + } + if f := hash_materialize(interp, r); f.kind != .None { + return fail(interp, hash_error_message(f)) + } if op == .Op_EqEq do return values_equal(l, r), true return compare_ordered(interp, op, l, r) @@ -1107,8 +1118,11 @@ eval_sha256 :: proc(interp: ^Interpreter, node: Node_Idx, env: ^Env) -> (Value, val, ok = concrete_value(interp, val) if !ok do return nil, false - encoded, herr := value_digest_base64(val) - if herr != .None { + // The interpreter goes along: a directory File's digest is read off the disk + // the first time anything asks (§3), and that read needs this context's `io` + // permission (§9). It is also what a Function's shape is read from. + encoded, herr := value_digest_base64(val, interp) + if herr.kind != .None { return fail(interp, fmt.tprintf("sha256: %s", hash_error_message(herr))) } return encoded, true diff --git a/src/examples_test.odin b/src/examples_test.odin index 89c18c9..b2ceaf4 100644 --- a/src/examples_test.odin +++ b/src/examples_test.odin @@ -42,6 +42,15 @@ EXAMPLE_CASES := []Example_Case{ {"functions-and-holes.hb", "{section: 11, explicit: 49, nested: 507, stored: 42, asserted: 9}"}, {"guard-chain.hb", "5"}, {"hashing.hb", `{text: "Ar9oHTBiuRDqs+ZdbYD2daaU7RcvIDTJNB3UICNP92A=", file: "ZT6vBQgoXEojRYd890EDlZWhUF/uGfXa+C9BNGBykI0=", key_order_is_irrelevant: true, same_content_same_file: true, integer_is_not_float: false}`}, + // The three hashing examples assert properties rather than digests, on + // purpose. A directory containing an executable hashes differently on + // Windows and WASI than on Linux (§3, and LANGUAGE.md on why), and a + // closure's digest includes its body's own source text - so a literal here + // would be a value that is either target-specific or broken by reformatting + // the example it came from. + {"hashing-cyclic.hb", "{a_cycle_hashes: true, equal_cycles_hash_alike: true, shape_still_matters: true, either_end_agrees: true}"}, + {"hashing-directories.hb", "{a_tree_is_not_its_file: true, one_tree_is_one_value: true, reading_twice_agrees: true}"}, + {"hashing-functions.hb", "{same_body_same_hash: true, different_body_differs: true, captures_count: true, neighbours_do_not: true, builtins_hash_by_what_they_are: true, a_builtin_carries_its_argument: true}"}, {"numeric-literals.hb", "{hex: 42, octal: 42, binary: 42, grouped: 1000000, exponent: 1500, bases_agree: true}"}, {"nothing-and-empty.hb", "{unit: nothing, zero_table: {}, present_case: 42, empty_case: -1, same: false}"}, {"optional.hb", "42"}, diff --git a/src/fs.odin b/src/fs.odin index c8644d0..b4a9412 100644 --- a/src/fs.odin +++ b/src/fs.odin @@ -58,6 +58,39 @@ Fs_Entry :: struct { is_dir: bool, } +// One entry of a directory in the detail SPEC.md §3's directory hash needs: +// which of the three shapes it is, and - for a regular file - whether it is +// executable. Distinct from Fs_Entry above, which answers the editor's much +// smaller question (a name, and whether to descend into it). +// +// `kind` is decided **without following symlinks**: §3 hashes a link entry as +// its target string rather than resolving through it, so a link *to* a +// directory is .Symlink here, never .Directory. +Fs_Node_Kind :: enum { + Regular, + Directory, + Symlink, + Other, // a fifo, socket, or device node - §3 describes no hash for one +} + +Fs_Dir_Entry :: struct { + name: string, + kind: Fs_Node_Kind, + // .Regular only. **False wherever the target cannot report the bit**, rather + // than a third "unknown" state: WASI's filestat carries no permission bits + // at all and Windows has no POSIX exec bit, so on those two targets this is + // always false. That is the language's answer, not a gap - see hash.odin's + // directory section and LANGUAGE.md. + is_executable: bool, +} + +// fs_list_entries_at the above, for every name in an open directory +// +// Named here rather than in the list above because it is the one operation +// added for hashing, and a directory's digest is its only caller. It takes a +// descriptor, not a path, because §16's containment is descriptor-relative and +// the walk must not be able to step outside the handle it was handed. + // What went wrong, in terms both targets can express. Deliberately coarse: // these become the parenthesised detail in a §16 failure message, where the // distinctions that matter are "wasn't there", "already there", "not allowed" diff --git a/src/fs_linux.odin b/src/fs_linux.odin index bb904a3..009f0c5 100644 --- a/src/fs_linux.odin +++ b/src/fs_linux.odin @@ -155,6 +155,64 @@ fs_make_dirs :: proc(path: string) -> Fs_Error { return .None } +// ---- the directory hash's listing (SPEC.md §3) ------------------------------- + +// getdents64 against a descriptor, plus one fstatat per name for the kind and +// the mode. The d_type getdents already reports is deliberately *not* trusted +// on its own: it is documented as possibly .UNKNOWN (some filesystems fill it +// in, some don't), and the exec bit needs the stat regardless - so one call +// answers both questions rather than two answering one each. +// +// The listing is done through a *fresh* descriptor rather than `parent` +// itself. getdents advances the descriptor's own offset, and `parent` is a +// long-lived handle a program keeps using as a `.dir` (§16) - reading its +// entries must not be something the program can observe afterwards. Opening +// "." relative to it costs one syscall and keeps the caller's handle exactly +// as it was found. +fs_list_entries_at :: proc(parent: Fs_Fd, allocator := context.allocator) -> ([]Fs_Dir_Entry, Fs_Error) { + listing, open_err := open_at(parent, ".", {.DIRECTORY}) + if open_err != .None do return nil, open_err + defer fs_close(listing) + + entries := make([dynamic]Fs_Dir_Entry, 0, 16, allocator) + buf := make([]u8, 4096, context.temp_allocator) + for { + written, err := linux.getdents(linux.Fd(listing), buf) + if err != .NONE do return entries[:], fs_errno_to_error(err) + if written == 0 do break + + offset := 0 + for dirent in linux.dirent_iterate_buf(buf[:written], &offset) { + name := linux.dirent_name(dirent) + if name == "." || name == ".." do continue + + cname := strings.clone_to_cstring(name, context.temp_allocator) + stat: linux.Stat + // AT_SYMLINK_NOFOLLOW: a link is hashed as its target string (§3), so + // the entry's own type is what matters, never what it points at. + if serr := linux.fstatat(linux.Fd(listing), cname, &stat, {.SYMLINK_NOFOLLOW}); serr != .NONE { + return entries[:], fs_errno_to_error(serr) + } + + kind := Fs_Node_Kind.Other + switch { + case linux.S_ISREG(stat.mode): kind = .Regular + case linux.S_ISDIR(stat.mode): kind = .Directory + case linux.S_ISLNK(stat.mode): kind = .Symlink + } + append(&entries, Fs_Dir_Entry { + name = strings.clone(name, allocator), + kind = kind, + // §3 hashes "the executable flag only - not full POSIX mode", and the + // owner bit is the one that means "this is a program". Group and other + // are part of who may run it, which is not part of what it is. + is_executable = kind == .Regular && .IXUSR in stat.mode, + }) + } + } + return entries[:], .None +} + // Listing goes through core:os here, which is a perfectly good directory // reader on a platform that has a working directory. (WASI does not, which is // why fs_wasi.odin implements this against the preopen table instead.) diff --git a/src/fs_wasi.odin b/src/fs_wasi.odin index 976e7f6..9e65b65 100644 --- a/src/fs_wasi.odin +++ b/src/fs_wasi.odin @@ -271,6 +271,51 @@ fs_make_dirs :: proc(path: string) -> Fs_Error { // preview1's fd_readdir: entries come back packed as a 24-byte header (next // cookie, inode, name length, filetype) followed by the raw name, and the // caller keeps asking until a pass returns less than it asked for. +// ---- the directory hash's listing (SPEC.md §3) ------------------------------- + +// preview1's fd_readdir is cookie-driven rather than offset-driven, so unlike +// the Linux side this can read `parent` itself: nothing about the descriptor +// changes, and a caller still holding it as a `.dir` handle (§16) sees no +// difference. +// +// `is_executable` is always false here, and that is the language's answer +// rather than a missing feature: a preview1 `filestat` has no permission bits +// of any kind to report. See hash.odin's directory section. +fs_list_entries_at :: proc(parent: Fs_Fd, allocator := context.allocator) -> ([]Fs_Dir_Entry, Fs_Error) { + entries := make([dynamic]Fs_Dir_Entry, 0, 16, allocator) + buf := make([]u8, 4096, context.temp_allocator) + cookie := wasi.dircookie_t(0) + + for { + used, read_err := wasi.fd_readdir(wasi.fd_t(parent), buf, cookie) + if read_err != .SUCCESS do return entries[:], to_fs_error(read_err) + if used == 0 do break + + offset := 0 + for offset + size_of(wasi.dirent_t) <= int(used) { + dirent := (^wasi.dirent_t)(raw_data(buf[offset:]))^ + name_start := offset + size_of(wasi.dirent_t) + name_end := name_start + int(dirent.d_namlen) + if name_end > int(used) do break // a name split across reads: ask again from d_next + + name := string(buf[name_start:name_end]) + if name != "." && name != ".." { + kind := Fs_Node_Kind.Other + #partial switch dirent.d_type { + case .REGULAR_FILE: kind = .Regular + case .DIRECTORY: kind = .Directory + case .SYMBOLIC_LINK: kind = .Symlink + } + append(&entries, Fs_Dir_Entry{name = strings.clone(name, allocator), kind = kind}) + } + cookie = dirent.d_next + offset = name_end + } + if int(used) < len(buf) do break + } + return entries[:], .None +} + fs_list_dir :: proc(path: string, allocator := context.allocator) -> ([]Fs_Entry, Fs_Error) { dir, err := fs_open_dir_path(path) if err != .None do return nil, err diff --git a/src/fs_windows.odin b/src/fs_windows.odin index d698435..2a92cc0 100644 --- a/src/fs_windows.odin +++ b/src/fs_windows.odin @@ -605,6 +605,46 @@ fs_make_dirs :: proc(path: string) -> Fs_Error { // The editor's file pickers. FindFirstFileW wants a wildcard rather than a // directory, and always reports "." and ".." first, which no caller wants. +// ---- the directory hash's listing (SPEC.md §3) ------------------------------- + +// FindFirstFileW against the path this descriptor stands for, which is how +// every other operation on this target reaches a child (see the header). The +// attributes the enumeration already carries answer the kind, so unlike the +// Linux side there is no per-entry stat: a reparse point is a link, and +// FILE_ATTRIBUTE_DIRECTORY decides the rest. +// +// `is_executable` is always false here, and that is the language's answer +// rather than a missing feature: Windows has no POSIX exec bit, and what it +// has instead - an extension the shell knows how to run - is a property of the +// name, not of the file. See hash.odin's directory section. +fs_list_entries_at :: proc(parent: Fs_Fd, allocator := context.allocator) -> ([]Fs_Dir_Entry, Fs_Error) { + dir, ok := dir_path_of(parent) + if !ok do return nil, .Not_Directory + + data: windows.WIN32_FIND_DATAW + h := windows.FindFirstFileW(to_win_path(join_child(dir, "*")), &data) + if h == windows.INVALID_HANDLE_VALUE do return nil, last_error() + defer windows.FindClose(h) + + entries := make([dynamic]Fs_Dir_Entry, 0, 16, allocator) + for { + name, err := windows.utf16_to_utf8(data.cFileName[:name_length(data.cFileName[:])], context.temp_allocator) + if err == nil && name != "." && name != ".." { + attrs := data.dwFileAttributes + kind := Fs_Node_Kind.Regular + switch { + // Checked first: a junction or a symlink *to* a directory carries both + // bits, and §3 hashes a link as its target string without resolving it. + case (attrs & windows.FILE_ATTRIBUTE_REPARSE_POINT) != 0: kind = .Symlink + case (attrs & windows.FILE_ATTRIBUTE_DIRECTORY) != 0: kind = .Directory + } + append(&entries, Fs_Dir_Entry{name = strings.clone(name, allocator), kind = kind}) + } + if !windows.FindNextFileW(h, &data) do break + } + return entries[:], .None +} + fs_list_dir :: proc(path: string, allocator := context.allocator) -> ([]Fs_Entry, Fs_Error) { abs := absolute_dir_path(path) defer delete(abs) diff --git a/src/hash.odin b/src/hash.odin index 7da15f4..f09b6a0 100644 --- a/src/hash.odin +++ b/src/hash.odin @@ -2,6 +2,7 @@ package hashedbuild import "core:crypto/hash" import "core:encoding/base64" +import "core:fmt" import "core:slice" // SPEC.md §6's "every value is hashable", and the §15 `sha256` builtin built @@ -23,6 +24,20 @@ import "core:slice" // Leaves are domain-separated by a tag byte so that values of different types // with the same payload stay distinct - Integer 5 and Float 5.0 are not equal // (value.odin), so they must not hash alike either. +// +// The three kinds a Merkle fold does not reach on its own live in this file +// and its two companions: +// +// * a **directory File**, below - its children are on the disk rather than +// in the value, so it is the one digest that does I/O; +// * a **Function** (hash_function.odin) - a closure is its body's shape plus +// the values it captures, so hashing one is a static analysis and a fold; +// * a **cyclic value** (hash_cyclic.odin) - a fold has no bottom to start +// from, so those nodes are hashed by a canonical form instead. +// +// Everything else is the fold, and the fold is the fast path: value_digest +// tries it first, allocates nothing for an acyclic value that holds no +// directory, and only reaches for the graph machinery when it meets a cycle. DIGEST_SIZE :: 32 @@ -31,24 +46,30 @@ Value_Digest :: [DIGEST_SIZE]u8 // Tag bytes. Never renumber these: a digest that changes meaning silently // invalidates every cache entry and every recorded hash a user has written // down. Appending a new tag for a new type is fine. -@(private = "file") +// +// 0x06 is unused - it was the regular File's, before §3 pinned that digest to +// the untagged content hash. It stays unused rather than being recycled, for +// the same reason the rest are never renumbered. TAG_NOTHING :: 0x00 -@(private = "file") TAG_BOOLEAN :: 0x01 -@(private = "file") TAG_INTEGER :: 0x02 -@(private = "file") TAG_FLOAT :: 0x03 -@(private = "file") TAG_UTF8 :: 0x04 -@(private = "file") TAG_BYTES :: 0x05 -@(private = "file") TAG_TABLE :: 0x07 +TAG_DIRECTORY :: 0x08 +TAG_DIR_ENTRY_FILE :: 0x09 +TAG_DIR_ENTRY_DIR :: 0x0a +TAG_DIR_ENTRY_LINK :: 0x0b +TAG_CACHE :: 0x0c +TAG_FUNCTION :: 0x0d +TAG_NATIVE :: 0x0e +TAG_AST :: 0x0f +TAG_CYCLIC :: 0x10 +TAG_CYCLIC_NODE :: 0x11 // Byte-lexicographic order on digests. Written as an explicit loop rather than // slice.cmp because a `proc` parameter isn't addressable, so it can't be sliced. -@(private = "file") digest_less :: proc(a, b: Value_Digest) -> bool { for i in 0 ..< DIGEST_SIZE { if a[i] != b[i] do return a[i] < b[i] @@ -56,81 +77,152 @@ digest_less :: proc(a, b: Value_Digest) -> bool { return false } -@(private = "file") sha256_of :: proc(data: []u8) -> Value_Digest { digest: Value_Digest hash.hash_bytes_to_buffer(.SHA256, data, digest[:]) return digest } -@(private = "file") +// The tag byte and the payload, fed to one digest without joining them first. +// Streamed rather than concatenated because the payload is sometimes the whole +// value - a Bytes leaf, a directory's entry digests - and a hash that copied +// what it hashes would double the cost of every large value in a build. sha256_tagged :: proc(tag: u8, payload: []u8) -> Value_Digest { - buf := make([]u8, 1 + len(payload), context.temp_allocator) - buf[0] = tag - copy(buf[1:], payload) - return sha256_of(buf) + s: Digest_Stream + digest_stream_begin(&s, tag) + digest_stream_bytes(&s, payload) + return digest_stream_end(&s) +} + +// A digest built a piece at a time, for a composite whose payload is a +// sequence of children rather than one buffer. Same bytes as assembling the +// payload and hashing it in one go - SHA-256 is a streaming construction, so +// this is the identity, not an approximation. +Digest_Stream :: struct { + ctx: hash.Context, +} + +digest_stream_begin :: proc(s: ^Digest_Stream, tag: u8) { + hash.init(&s.ctx, .SHA256) + tag_byte := [1]u8{tag} + hash.update(&s.ctx, tag_byte[:]) +} + +digest_stream_bytes :: proc(s: ^Digest_Stream, data: []u8) { + hash.update(&s.ctx, data) +} + +digest_stream_digest :: proc(s: ^Digest_Stream, d: Value_Digest) { + child := d // a parameter isn't addressable, and hashing one needs a slice + hash.update(&s.ctx, child[:]) +} + +digest_stream_end :: proc(s: ^Digest_Stream) -> Value_Digest { + d: Value_Digest + hash.final(&s.ctx, d[:]) + return d } -// Why a value has no digest, so the caller can say which value and why rather -// than emitting one "not hashable" for every case. +// A digest of a Utf8 payload, which several composites need for a name: an +// entry's filename (§3), a captured variable's spelling (hash_function.odin). +// It is the same digest the Utf8 *value* has, deliberately - there is one +// encoding per type, not one per use site. +sha256_text :: proc(s: string) -> Value_Digest { + return sha256_tagged(TAG_UTF8, transmute([]u8)s) +} + +// ---- failures ---------------------------------------------------------------- + +// Why a value has no digest. Every one of these is either a refusal the OS +// made or a shape §3 describes no encoding for - the kinds that used to sit +// here for "not built yet" are gone, because they are built. Hash_Error :: enum { None, - Directory_File, // §3's directory hash needs an exec bit only Linux can report - Function, // §15 needs it for `cached`, but never specifies the encoding - Cache, // §9's ctx.cache is write-only and has no identity to hash - Async, // an un-awaited handle - callers await before hashing - Cyclic, // §10's cyclic Table - a Merkle fold has no bottom to start from + Io_Denied, // a directory's first read, in a context without `io` (§9) + Directory_Read, // the OS refused somewhere in the tree + Unhashable_Entry, // a fifo/socket/device in a tree - §3 encodes three shapes + Directory_Unread, // a cold directory digest, asked for where no read is possible + No_Program, // a closure's shape is its AST's, and there is no AST here + Async, // an un-awaited handle - callers await before hashing + Cyclic, // internal: the fold met a cycle. Never escapes value_digest. +} + +// The failure, plus the entry or reason the kind alone doesn't carry. `detail` +// is temp-allocated, so it lives exactly as long as the failure is being +// reported - which is the same turn, in every caller. +Hash_Fail :: struct { + kind: Hash_Error, + detail: string, +} + +HASH_OK :: Hash_Fail{} + +@(private = "file") +fail_kind :: proc(kind: Hash_Error) -> Hash_Fail { + return Hash_Fail{kind = kind} +} + +@(private = "file") +fail_detail :: proc(kind: Hash_Error, detail: string) -> Hash_Fail { + return Hash_Fail{kind = kind, detail = detail} } -hash_error_message :: proc(e: Hash_Error) -> string { - switch e { +hash_error_message :: proc(f: Hash_Fail) -> string { + switch f.kind { case .None: return "" - case .Directory_File: - return "a directory File has no hash yet (see LANGUAGE.md on what isn't built yet)" - case .Function: - return "a Function has no hash yet (see LANGUAGE.md on what isn't built yet)" - case .Cache: - return "ctx.cache has no hash - it is write-only and has no identity (SPEC.md §9)" + case .Io_Denied: + return "reading a directory File's entries needs the io permission, and the current context does not grant it" + case .Directory_Read: + return f.detail + case .Unhashable_Entry: + return f.detail + case .Directory_Unread: + return "a directory File's entries have not been read yet, and cannot be read from here" + case .No_Program: + return "a Function's hash is its body's shape, which needs the program it was written in" case .Async: return "an un-awaited async handle has no hash" case .Cyclic: - return "a cyclic value has no hash yet (see LANGUAGE.md on what isn't built yet)" + return "a cyclic value could not be hashed" } return "" } +// ---- the walk ---------------------------------------------------------------- + +// State threaded through the fold. `interp` is what makes a directory's first +// read possible - and permitted; it is nil wherever hashing is asked for +// outside the evaluator (values_hash_equal, reached from table_find), and +// there a cold directory is a failure rather than a silent read. +Hash_Walk :: struct { + interp: ^Interpreter, + open: [dynamic]rawptr, // the composite nodes currently being folded +} + // The digest of a value, per the encoding described at the top of this file. -// Fails (rather than inventing a digest) for the kinds §3/§15 leave open - -// see Hash_Error. -value_digest :: proc(v: Value) -> (Value_Digest, Hash_Error) { - path := make([dynamic]rawptr, 0, 8, context.temp_allocator) - return value_digest_walk(v, &path) -} - -// The encoding above is a Merkle fold: a composite's digest is built from its -// children's. A cyclic Table (§10) has no bottom to start that fold from, and -// no amount of recursion reaches one - so it is refused rather than hung on. -// Hashing one *is* definable (decompose into strongly connected components, -// Merkle-fold the acyclic part, and give each component a digest canonical -// under bisimulation so it does not depend on which node you entered by) but -// §3 pins the encoding of a digest, so choosing one is a spec decision, not an -// implementation detail. Until that is made, `sha256` of a cyclic value fails -// the same way a directory File or a Function does. // -// `path` is the chain of Tables currently open above this one, so the test is -// for a genuine back-edge - a Table that appears twice in different branches, -// which is ordinary sharing rather than a cycle, still hashes fine. -@(private = "file") -value_digest_walk :: proc(v: Value, path: ^[dynamic]rawptr) -> (Value_Digest, Hash_Error) { +// Two passes, and the second one almost never runs: the fold is tried first, +// and only a value that turns out to contain a cycle falls through to the +// graph algorithm in hash_cyclic.odin. That ordering is what keeps an ordinary +// Table's digest - the overwhelmingly common case - a plain recursive hash +// with no graph analysis behind it. +value_digest :: proc(v: Value, interp: ^Interpreter = nil) -> (Value_Digest, Hash_Fail) { + w := Hash_Walk{interp = interp, open = make([dynamic]rawptr, 0, 8, context.temp_allocator)} + d, f := value_digest_walk(v, &w) + if f.kind != .Cyclic do return d, f + return value_digest_cyclic(v, interp) +} + +value_digest_walk :: proc(v: Value, w: ^Hash_Walk) -> (Value_Digest, Hash_Fail) { resolved, rok := resolve_forward(v) - if !rok do return {}, .Cyclic + if !rok do return {}, fail_kind(.Cyclic) switch av in resolved { case Nothing_Value: - return sha256_tagged(TAG_NOTHING, nil), .None + return sha256_tagged(TAG_NOTHING, nil), HASH_OK case bool: - return sha256_tagged(TAG_BOOLEAN, {1 if av else 0}), .None + return sha256_tagged(TAG_BOOLEAN, {1 if av else 0}), HASH_OK case i64: // Little-endian two's complement, fixed 8 bytes - Integer is exactly one @@ -138,7 +230,7 @@ value_digest_walk :: proc(v: Value, path: ^[dynamic]rawptr) -> (Value_Digest, Ha buf: [8]u8 u := transmute(u64)av for i in 0 ..< 8 do buf[i] = u8((u >> (8 * uint(i))) & 0xff) - return sha256_tagged(TAG_INTEGER, buf[:]), .None + return sha256_tagged(TAG_INTEGER, buf[:]), HASH_OK case f64: // IEEE-754 bits, little-endian. Two normalisations keep this consistent @@ -152,19 +244,19 @@ value_digest_walk :: proc(v: Value, path: ^[dynamic]rawptr) -> (Value_Digest, Ha if f != f do u = 0x7ff8_0000_0000_0000 // any NaN -> one canonical quiet NaN buf: [8]u8 for i in 0 ..< 8 do buf[i] = u8((u >> (8 * uint(i))) & 0xff) - return sha256_tagged(TAG_FLOAT, buf[:]), .None + return sha256_tagged(TAG_FLOAT, buf[:]), HASH_OK case string: - return sha256_tagged(TAG_UTF8, transmute([]u8)av), .None + return sha256_tagged(TAG_UTF8, transmute([]u8)av), HASH_OK case []u8: - return sha256_tagged(TAG_BYTES, av), .None + return sha256_tagged(TAG_BYTES, av), HASH_OK case ^File_Value: // §3: "a regular file's hash is just hash(content_bytes)" - deliberately // untagged, so it matches what sha256sum reports for the same bytes. - if av.kind == .Directory do return {}, .Directory_File - return sha256_of(av.content), .None + if av.kind == .Directory do return file_directory_digest(av, w.interp) + return sha256_of(av.content), HASH_OK case ^Table_Value: // §5/§6: a Table's hash is key-sorted, so it does not depend on the order @@ -172,63 +264,268 @@ value_digest_walk :: proc(v: Value, path: ^[dynamic]rawptr) -> (Value_Digest, Ha // already implements (it matches entries by key, ignoring position). // Sorting by key digest gives a deterministic order without needing §6's // cross-type value ordering, which isn't built. - for open in path^ { - if open == rawptr(av) do return {}, .Cyclic + for open in w.open { + if open == rawptr(av) do return {}, fail_kind(.Cyclic) } - append(path, rawptr(av)) - defer pop(path) + append(&w.open, rawptr(av)) + defer pop(&w.open) pairs := make([][2]Value_Digest, len(av.entries), context.temp_allocator) for entry, i in av.entries { - kd, kerr := value_digest_walk(entry.key, path) - if kerr != .None do return {}, kerr - vd, verr := value_digest_walk(entry.value, path) - if verr != .None do return {}, verr + kd, kerr := value_digest_walk(entry.key, w) + if kerr.kind != .None do return {}, kerr + vd, verr := value_digest_walk(entry.value, w) + if verr.kind != .None do return {}, verr pairs[i] = {kd, vd} } slice.sort_by(pairs, proc(a, b: [2]Value_Digest) -> bool { return digest_less(a[0], b[0]) }) - buf := make([]u8, len(pairs) * 2 * DIGEST_SIZE, context.temp_allocator) - for i in 0 ..< len(pairs) { - copy(buf[i * 2 * DIGEST_SIZE:], pairs[i][0][:]) - copy(buf[(i * 2 + 1) * DIGEST_SIZE:], pairs[i][1][:]) + s: Digest_Stream + digest_stream_begin(&s, TAG_TABLE) + for pair in pairs { + digest_stream_digest(&s, pair[0]) + digest_stream_digest(&s, pair[1]) } - return sha256_tagged(TAG_TABLE, buf), .None + return digest_stream_end(&s), HASH_OK case ^Function_Value: - return {}, .Function + return function_digest(av, w) case ^Cache_Value: - return {}, .Cache + // §9's ctx.cache. A bare tag and nothing else: there is exactly one per + // context, it has no content a program can observe, and the one thing that + // *would* tell two of them apart - the directory it is rooted at - is the + // path §9 spends its last paragraph keeping out of the language. Hashing + // it to the path would hand that back through a side door. + return sha256_tagged(TAG_CACHE, nil), HASH_OK case ^Async_Handle: - return {}, .Async + return {}, fail_kind(.Async) case ^Forward_Ref_Value: // Unreachable: resolve_forward above returned either a non-forward value // or .Cyclic. Present so the switch stays exhaustive. - return {}, .Cyclic + return {}, fail_kind(.Cyclic) } - return sha256_tagged(TAG_NOTHING, nil), .None + return sha256_tagged(TAG_NOTHING, nil), HASH_OK } +// ---- directories (SPEC.md §3) ------------------------------------------------- + +// §3 spells the directory hash out: +// +// dir_entry_hash(name, entry) = +// hash(name, "file", content_hash, is_executable) +// hash(name, "dir", child_dir_hash) +// hash(name, "symlink", target_path_string) +// dir_hash = hash(sorted [dir_entry_hash(name, entry) for each entry]) +// +// with the three spelled-out tags becoming the three TAG_DIR_ENTRY_* bytes, +// and "sorted" meaning by name, byte-wise - so the digest is the tree's, not +// the order readdir happened to hand it back in. The directory's own digest is +// tagged, which §3 leaves open: only a *regular* file's digest is pinned +// untagged (so it matches sha256sum), and without a tag here a directory could +// in principle collide with a regular file whose content happened to be its +// entry digests laid end to end. +// +// **The executable bit is the file's owner-execute bit where the target can +// see one, and false everywhere else.** WASI's filestat has no permission bits +// and Windows has no POSIX exec bit (fs.odin), so a tree containing an +// executable hashes differently there than on Linux. That is the language's +// answer rather than a gap: a Windows checkout genuinely has no exec bits, so +// reporting one would be inventing it - the same position git takes with +// core.filemode. LANGUAGE.md says so where a user will meet it. +// +// A symlink is hashed as its target *string*, never resolved (§3), which is +// also what keeps this recursion finite: nothing here follows a link, so the +// walk is bounded by the tree's real depth and a link loop is just a string. +@(private = "file") +file_directory_digest :: proc(fv: ^File_Value, interp: ^Interpreter) -> (Value_Digest, Hash_Fail) { + if fv.dir_digest_known do return fv.dir_digest, HASH_OK + // No evaluator to ask, so no permission to check and no read to make. This + // is values_hash_equal's path (value.odin) - see the note there on why the + // evaluator warms the digest before a comparison rather than doing it here. + if interp == nil do return {}, fail_kind(.Directory_Unread) + if !ctx_allows_io(interp) do return {}, fail_kind(.Io_Denied) + + d, f := directory_digest_at(fv.dir_fd) + if f.kind != .None do return {}, f + fv.dir_digest = d + fv.dir_digest_known = true + return d, HASH_OK +} + +@(private = "file") +directory_digest_at :: proc(dir: Fs_Fd) -> (Value_Digest, Hash_Fail) { + entries, err := fs_list_entries_at(dir, context.temp_allocator) + if err != .None { + return {}, fail_detail(.Directory_Read, fmt.tprintf("could not read a directory's entries (%v)", err)) + } + slice.sort_by(entries, proc(a, b: Fs_Dir_Entry) -> bool { return a.name < b.name }) + + s: Digest_Stream + digest_stream_begin(&s, TAG_DIRECTORY) + for entry in entries { + ed, f := dir_entry_digest(dir, entry) + if f.kind != .None do return {}, f + digest_stream_digest(&s, ed) + } + return digest_stream_end(&s), HASH_OK +} + +@(private = "file") +dir_entry_digest :: proc(dir: Fs_Fd, entry: Fs_Dir_Entry) -> (Value_Digest, Hash_Fail) { + name := sha256_text(entry.name) + + switch entry.kind { + case .Regular: + // no_follow throughout: the listing already decided this is a regular + // file rather than a link, and opening it must not be able to disagree. + fd, oerr := fs_open_read_at(dir, entry.name, true) + if oerr != .None { + return {}, fail_detail(.Directory_Read, fmt.tprintf("could not open %s while hashing a directory (%v)", entry.name, oerr)) + } + content, rerr := fs_read_all(fd) + fs_close(fd) + if rerr != .None { + return {}, fail_detail(.Directory_Read, fmt.tprintf("could not read %s while hashing a directory (%v)", entry.name, rerr)) + } + defer delete(content) + + payload: [2 * DIGEST_SIZE + 1]u8 + ch := sha256_of(content) + copy(payload[0:], name[:]) + copy(payload[DIGEST_SIZE:], ch[:]) + payload[2 * DIGEST_SIZE] = 1 if entry.is_executable else 0 + return sha256_tagged(TAG_DIR_ENTRY_FILE, payload[:]), HASH_OK + + case .Directory: + child, oerr := fs_open_dir_at(dir, entry.name, true) + if oerr != .None { + return {}, fail_detail(.Directory_Read, fmt.tprintf("could not open %s while hashing a directory (%v)", entry.name, oerr)) + } + cd, f := directory_digest_at(child) + fs_close(child) + if f.kind != .None do return {}, f + + payload: [2 * DIGEST_SIZE]u8 + copy(payload[0:], name[:]) + copy(payload[DIGEST_SIZE:], cd[:]) + return sha256_tagged(TAG_DIR_ENTRY_DIR, payload[:]), HASH_OK + + case .Symlink: + target, rerr := fs_readlink_at(dir, entry.name) + if rerr != .None { + return {}, fail_detail(.Directory_Read, fmt.tprintf("could not read the link %s while hashing a directory (%v)", entry.name, rerr)) + } + defer delete(target) + + payload: [2 * DIGEST_SIZE]u8 + td := sha256_text(target) + copy(payload[0:], name[:]) + copy(payload[DIGEST_SIZE:], td[:]) + return sha256_tagged(TAG_DIR_ENTRY_LINK, payload[:]), HASH_OK + + case .Other: + // §3 encodes three shapes, and this is none of them. Refused rather than + // skipped: a digest that quietly ignored part of a tree would claim two + // different trees were the same value. + return {}, fail_detail( + .Unhashable_Entry, + fmt.tprintf("%s is neither a file, a directory, nor a symlink, and SPEC.md §3 gives no hash for one", entry.name), + ) + } + return {}, fail_kind(.Directory_Read) +} + +// ---- what the evaluator calls ------------------------------------------------ + // §15: the digest, base64-encoded as Utf8. Standard alphabet with padding - // the form README.md's worked example writes a package hash in, and what // `sha256sum ... | xxd -r -p | base64` prints. (ctx.cache's entry names use // the URL alphabet instead, because those become filenames; see builtins_fs.odin.) -value_digest_base64 :: proc(v: Value, allocator := context.allocator) -> (string, Hash_Error) { - d, err := value_digest(v) - if err != .None do return "", err - return base64.encode(d[:], base64.ENC_TABLE, allocator), .None +value_digest_base64 :: proc(v: Value, interp: ^Interpreter = nil, allocator := context.allocator) -> (string, Hash_Fail) { + d, f := value_digest(v, interp) + if f.kind != .None do return "", f + encoded, _ := base64.encode(d[:], base64.ENC_TABLE, allocator) + return encoded, HASH_OK } // Whether two values hash alike. Used for §3's File identity, where equality // *is* the content hash - two Files with the same bytes are the same value // however they were reached. +// +// No interpreter, on purpose: this is reached from table_find, which is the +// hot path of every field access and has no business opening files. A +// directory File whose digest has not been read yet therefore compares as +// "not equal to anything but itself" here, and the evaluator warms it first - +// see hash_materialize below. values_hash_equal :: proc(a: Value, b: Value) -> bool { - da, aerr := value_digest(a) - if aerr != .None do return false - db, berr := value_digest(b) - if berr != .None do return false + da, af := value_digest(a) + if af.kind != .None do return false + db, bf := value_digest(b) + if bf.kind != .None do return false return da == db } + +// Read the entries of every directory File reachable from `v`, so that a +// later comparison - which cannot do I/O - finds the digests already there. +// +// This is what makes "the first read is the read" observable at the right +// moment. Comparing a directory File means comparing its contents (§3), so +// the comparison *is* a read, and a read needs `io` (§9); but the comparison +// itself happens deep inside values_equal with no context to ask. So the +// evaluator warms the operands here, where it still has one, and fails +// honestly if the permission is missing rather than quietly answering "not +// equal" - which is what an un-warmed comparison would otherwise say. +// +// Cheap for everything else: a value holding no File and no Table returns at +// the first switch, and one that has already been hashed re-walks a value it +// has, without touching the disk. +hash_materialize :: proc(interp: ^Interpreter, v: Value) -> Hash_Fail { + // Every `==` in every program comes through here, and almost none of them + // involve a File. Nothing but the three kinds that can hold one is worth + // allocating a visited set for. + #partial switch av in v { + case ^File_Value: + if av.kind != .Directory || av.dir_digest_known do return HASH_OK + case ^Table_Value, ^Function_Value, ^Forward_Ref_Value: + case: + return HASH_OK + } + seen := make([dynamic]rawptr, 0, 8, context.temp_allocator) + return materialize_walk(interp, v, &seen) +} + +@(private = "file") +materialize_walk :: proc(interp: ^Interpreter, v: Value, seen: ^[dynamic]rawptr) -> Hash_Fail { + resolved, rok := resolve_forward(v) + if !rok do return HASH_OK // mid-construction; nothing a program can compare yet + + #partial switch av in resolved { + case ^File_Value: + if av.kind != .Directory || av.dir_digest_known do return HASH_OK + _, f := file_directory_digest(av, interp) + return f + + case ^Table_Value: + for s in seen^ do if s == rawptr(av) do return HASH_OK + append(seen, rawptr(av)) + for entry in av.entries { + if f := materialize_walk(interp, entry.key, seen); f.kind != .None do return f + if f := materialize_walk(interp, entry.value, seen); f.kind != .None do return f + } + + case ^Function_Value: + // A closure holds its captures, and one of them can be a directory. The + // captured values are what function_digest mixes in, so they are exactly + // what has to be warm before a comparison reaches for a digest. + for s in seen^ do if s == rawptr(av) do return HASH_OK + append(seen, rawptr(av)) + if av.native != nil do return materialize_walk(interp, av.native_closure, seen) + for captured in function_captures(interp, av) { + if f := materialize_walk(interp, captured.value, seen); f.kind != .None do return f + } + } + return HASH_OK +} diff --git a/src/hash_cyclic.odin b/src/hash_cyclic.odin new file mode 100644 index 0000000..b169502 --- /dev/null +++ b/src/hash_cyclic.odin @@ -0,0 +1,495 @@ +package hashedbuild + +import "core:slice" + +// Hashing a value that reaches itself (SPEC.md §10's cyclic `let rec`). +// +// hash.odin's encoding is a Merkle fold: a composite's digest is built from +// its children's. A cycle has no bottom to start that fold from, so this file +// supplies the other half of the answer §6 asks for - "some arbitrary total +// order... such that any value can be compared with any other" - by giving +// each node in a cycle a digest that depends only on *what it is*, never on +// where a walk entered it. +// +// The requirement is sharper than it sounds. `values_equal` (value.odin) +// already compares cyclic values by bisimulation: two separately built cycles +// of the same shape are equal, and a 2-cycle whose halves are identical is +// equal to a 1-cycle, because unrolling them gives the same infinite tree. A +// digest that disagreed with that would break the one property a content- +// addressed language cannot do without - equal values hashing alike. So the +// digest here is not merely deterministic; it is canonical **under +// bisimulation**, which is exactly the equality that already exists. +// +// Three steps: +// +// 1. **Build the graph.** Tables and Functions are the nodes - the only two +// kinds that hold other values - and everything else is a leaf whose +// digest the ordinary fold gives. Each node's out-edges carry a *slot* - +// a Table's key digest, a closure's captured name - and are in the order +// hash.odin's fold would put them in, which is what lets the two paths +// encode the same node the same way. +// +// 2. **Fold what can be folded.** Tarjan's algorithm splits the graph into +// strongly connected components and hands them back successors-first, so +// every component is reached only once everything it points at has a +// digest. A component of one node with no self-edge is not a cycle at all, +// and gets the plain Merkle encoding - byte for byte the one hash.odin +// computes, which is what keeps `sha256 t` the same answer whether or not +// some unrelated corner of the value turned out to be cyclic. +// +// 3. **Canonicalise the rest.** Within a genuine component, partition +// refinement finds the bisimulation classes, and each node's digest is +// then a canonical encoding of the quotient graph reachable from its own +// class - classes numbered in the order a deterministic walk from that +// class first meets them. Two bisimilar nodes reach isomorphic quotients +// in the same order, so they encode identically; two that are not +// bisimilar differ in the first round of refinement that told them apart. + +// An out-edge. `slot` is what names this child within its parent - a key's +// digest for a Table entry, a name's for a capture - and is part of the +// encoding, not just a sort key. `target` indexes into the node list, or is +// NOT_A_NODE for a child that is an ordinary value. +@(private = "file") +NOT_A_NODE :: -1 + +@(private = "file") +Graph_Edge :: struct { + slot: Value_Digest, + has_slot: bool, // false for a builtin's carried closure, which has no name + target: int, + leaf: Value_Digest, // valid when target == NOT_A_NODE +} + +@(private = "file") +Graph_Node :: struct { + ptr: rawptr, + tag: u8, + // What the node is before its children are looked at: nothing for a Table + // (its identity is entirely its entries), the body's shape for a closure, + // the operation's name for a builtin. + prefix: []u8, + edges: []Graph_Edge, + digest: Value_Digest, + done: bool, + + // Tarjan's bookkeeping. + index: int, + lowlink: int, + on_stack: bool, + visited: bool, +} + +@(private = "file") +Graph :: struct { + interp: ^Interpreter, + nodes: [dynamic]Graph_Node, + by_ptr: map[rawptr]int, + fail: Hash_Fail, +} + +// The entry point hash.odin falls through to when its fold meets a cycle. +value_digest_cyclic :: proc(v: Value, interp: ^Interpreter) -> (Value_Digest, Hash_Fail) { + g := Graph { + interp = interp, + nodes = make([dynamic]Graph_Node, 0, 16, context.temp_allocator), + by_ptr = make(map[rawptr]int, 16, context.temp_allocator), + } + + root, is_node := graph_add(&g, v) + if g.fail.kind != .None do return {}, g.fail + // Not a node, so it holds nothing and cannot have been what was cyclic. + // Reachable only if a caller asks about a value the fold already answered. + if !is_node { + w := Hash_Walk{interp = interp, open = make([dynamic]rawptr, 0, 4, context.temp_allocator)} + return value_digest_walk(v, &w) + } + + if f := solve(&g); f.kind != .None do return {}, f + return g.nodes[root].digest, HASH_OK +} + +// ---- building the graph -------------------------------------------------------- + +// Adds `v` and everything below it, returning its node index. `is_node` is +// false for a value that is not a Table or a Function, which is a leaf: it +// holds no other values, so the ordinary fold answers it outright. +@(private = "file") +graph_add :: proc(g: ^Graph, v: Value) -> (index: int, is_node: bool) { + resolved, rok := resolve_forward(v) + if !rok { + g.fail = Hash_Fail{kind = .Cyclic} + return NOT_A_NODE, false + } + + #partial switch av in resolved { + case ^Table_Value: + return graph_add_table(g, av), true + case ^Function_Value: + return graph_add_function(g, av), true + } + return NOT_A_NODE, false +} + +// The digest of a value that is not a graph node. Its own fold cannot reach a +// node - only Tables and Functions hold values - so this cannot recurse back +// into the cycle. +@(private = "file") +graph_leaf_digest :: proc(g: ^Graph, v: Value) -> Value_Digest { + w := Hash_Walk{interp = g.interp, open = make([dynamic]rawptr, 0, 4, context.temp_allocator)} + d, f := value_digest_walk(v, &w) + if f.kind != .None && g.fail.kind == .None do g.fail = f + return d +} + +// A child, as an edge: a node reference if it is one, its digest if it is not. +@(private = "file") +graph_edge_to :: proc(g: ^Graph, slot: Value_Digest, has_slot: bool, child: Value) -> Graph_Edge { + if index, is_node := graph_add(g, child); is_node { + return Graph_Edge{slot = slot, has_slot = has_slot, target = index} + } + return Graph_Edge{slot = slot, has_slot = has_slot, target = NOT_A_NODE, leaf = graph_leaf_digest(g, child)} +} + +@(private = "file") +graph_add_table :: proc(g: ^Graph, t: ^Table_Value) -> int { + if existing, found := g.by_ptr[rawptr(t)]; found do return existing + + index := len(g.nodes) + g.by_ptr[rawptr(t)] = index + append(&g.nodes, Graph_Node{ptr = rawptr(t), tag = TAG_TABLE, index = NOT_A_NODE}) + + edges := make([dynamic]Graph_Edge, 0, len(t.entries), context.temp_allocator) + for entry in t.entries { + // A key goes through the ordinary fold rather than becoming a node of its + // own, because a key has to have a digest *now*: it is what puts this + // node's children in an order, and an order is what the whole encoding + // below rests on. A cyclic key would therefore have nowhere to go - but it + // is also not a shape a program can build, since making one means + // inspecting an unresolved forward reference and that fails while the + // `let rec` is still running (rec_build.odin). If one ever arrived, the + // fold reports it here instead of being quietly mis-ordered. + append(&edges, graph_edge_to(g, graph_leaf_digest(g, entry.key), true, entry.value)) + } + sort_edges(edges[:]) + g.nodes[index].edges = edges[:] + return index +} + +@(private = "file") +graph_add_function :: proc(g: ^Graph, fv: ^Function_Value) -> int { + if existing, found := g.by_ptr[rawptr(fv)]; found do return existing + + index := len(g.nodes) + g.by_ptr[rawptr(fv)] = index + append(&g.nodes, Graph_Node{ptr = rawptr(fv), index = NOT_A_NODE}) + + if fv.native != nil { + name := sha256_text(fv.name) + prefix := make([]u8, DIGEST_SIZE, context.temp_allocator) + copy(prefix, name[:]) + edges := make([]Graph_Edge, 1, context.temp_allocator) + edges[0] = graph_edge_to(g, {}, false, fv.native_closure) + g.nodes[index].tag = TAG_NATIVE + g.nodes[index].prefix = prefix + g.nodes[index].edges = edges + return index + } + + if g.interp == nil { + if g.fail.kind == .None do g.fail = Hash_Fail{kind = .No_Program} + g.nodes[index].tag = TAG_FUNCTION + return index + } + + shape := function_shape_digest(g.interp, fv) + prefix := make([]u8, DIGEST_SIZE, context.temp_allocator) + copy(prefix, shape[:]) + + // Deliberately NOT sort_edges: function_captures already returns them sorted + // by name, and that order is part of the encoding hash_function.odin writes. + // Re-sorting by the name's *digest* would put them in a different order and + // give the same closure two digests depending on which path reached it. + captures := function_captures(g.interp, fv) + edges := make([dynamic]Graph_Edge, 0, len(captures), context.temp_allocator) + for capture in captures { + append(&edges, graph_edge_to(g, sha256_text(capture.name), true, capture.value)) + } + g.nodes[index].tag = TAG_FUNCTION + g.nodes[index].prefix = prefix + g.nodes[index].edges = edges[:] + return index +} + +@(private = "file") +sort_edges :: proc(edges: []Graph_Edge) { + slice.sort_by(edges, proc(a, b: Graph_Edge) -> bool { return digest_less(a.slot, b.slot) }) +} + +// ---- the Merkle half ----------------------------------------------------------- + +// Byte for byte what hash.odin's fold produces for the same node. This is not +// a convenience: a value hashed here and the same value hashed there have to +// agree, or a Table's digest would depend on whether something else in the +// value it was reached through happened to be cyclic. +@(private = "file") +merkle_digest :: proc(g: ^Graph, node: ^Graph_Node) -> Value_Digest { + buf := make([dynamic]u8, 0, len(node.prefix) + len(node.edges) * 2 * DIGEST_SIZE, context.temp_allocator) + append(&buf, ..node.prefix) + for edge in node.edges { + slot := edge.slot + if edge.has_slot do append(&buf, ..slot[:]) + child := edge.leaf + if edge.target != NOT_A_NODE do child = g.nodes[edge.target].digest + append(&buf, ..child[:]) + } + return sha256_tagged(node.tag, buf[:]) +} + +// ---- components ---------------------------------------------------------------- + +@(private = "file") +Tarjan :: struct { + next: int, + stack: [dynamic]int, +} + +// Tarjan's algorithm, iterative rather than recursive because the graph is a +// user's data structure and its depth is theirs to choose - the evaluator has +// a depth budget (§12) but a value that is already built does not. +@(private = "file") +solve :: proc(g: ^Graph) -> Hash_Fail { + if g.fail.kind != .None do return g.fail + + t := Tarjan{stack = make([dynamic]int, 0, len(g.nodes), context.temp_allocator)} + for i in 0 ..< len(g.nodes) { + if !g.nodes[i].visited do strongconnect(g, &t, i) + } + return g.fail +} + +@(private = "file") +Frame :: struct { + node: int, + edge: int, +} + +@(private = "file") +strongconnect :: proc(g: ^Graph, t: ^Tarjan, start: int) { + frames := make([dynamic]Frame, 0, 16, context.temp_allocator) + open_node(g, t, start) + append(&frames, Frame{node = start}) + + for len(frames) > 0 { + frame := &frames[len(frames) - 1] + node := &g.nodes[frame.node] + + if frame.edge < len(node.edges) { + edge := node.edges[frame.edge] + frame.edge += 1 + if edge.target == NOT_A_NODE do continue + + child := &g.nodes[edge.target] + if !child.visited { + open_node(g, t, edge.target) + append(&frames, Frame{node = edge.target}) + } else if child.on_stack { + node.lowlink = min(node.lowlink, child.index) + } + continue + } + + // Every edge walked: this node's component is settled if it is the root. + if node.lowlink == node.index do close_component(g, t, frame.node) + finished := frame.node + pop(&frames) + if len(frames) > 0 { + parent := &g.nodes[frames[len(frames) - 1].node] + parent.lowlink = min(parent.lowlink, g.nodes[finished].lowlink) + } + } +} + +@(private = "file") +open_node :: proc(g: ^Graph, t: ^Tarjan, i: int) { + node := &g.nodes[i] + node.visited = true + node.index = t.next + node.lowlink = t.next + t.next += 1 + node.on_stack = true + append(&t.stack, i) +} + +// Pops one component and gives every node in it a digest. Successors are +// already done - that is what makes the Merkle case below possible at all. +@(private = "file") +close_component :: proc(g: ^Graph, t: ^Tarjan, root: int) { + members := make([dynamic]int, 0, 4, context.temp_allocator) + for { + i := pop(&t.stack) + g.nodes[i].on_stack = false + append(&members, i) + if i == root do break + } + + if len(members) == 1 && !has_self_edge(g, members[0]) { + node := &g.nodes[members[0]] + node.digest = merkle_digest(g, node) + node.done = true + return + } + canonicalise(g, members[:]) +} + +@(private = "file") +has_self_edge :: proc(g: ^Graph, i: int) -> bool { + for edge in g.nodes[i].edges do if edge.target == i do return true + return false +} + +// ---- canonical digests for a component ----------------------------------------- + +// Partition refinement, then one canonical encoding per bisimulation class. +// +// Refinement starts with every node in the component in one class and splits +// on each round: a node's signature is what it is, plus - for each child in +// slot order - either that child's finished digest (it is outside the +// component) or the class its child currently sits in. Two nodes stay together +// only while nothing has told them apart, which is precisely the greatest +// fixed point bisimulation is defined as. n rounds suffice for n nodes: every +// round that changes anything splits at least one class, and there are at most +// n classes. +@(private = "file") +canonicalise :: proc(g: ^Graph, members: []int) { + class := make(map[int]int, len(members), context.temp_allocator) + for m in members do class[m] = 0 + class_count := 1 + + for _ in 0 ..< len(members) { + sigs := make([]Value_Digest, len(members), context.temp_allocator) + for m, i in members do sigs[i] = refine_signature(g, m, class) + + ordered := make([]Value_Digest, len(members), context.temp_allocator) + copy(ordered, sigs) + slice.sort_by(ordered, proc(a, b: Value_Digest) -> bool { return digest_less(a, b) }) + + // Distinct signatures, in byte order: that ordering is a property of the + // signatures themselves, so the numbering does not depend on the order + // the component's members were popped in. + next_id := 0 + ids := make(map[Value_Digest]int, len(members), context.temp_allocator) + for sig, i in ordered { + if i > 0 && ordered[i - 1] == sig do continue + ids[sig] = next_id + next_id += 1 + } + for m, i in members do class[m] = ids[sigs[i]] + + if next_id == class_count do break // nothing split; the partition is stable + class_count = next_id + } + + // One encoding per class, shared by every node in it - which is what makes + // two bisimilar nodes hash alike even when they are different objects. + encoded := make(map[int]Value_Digest, class_count, context.temp_allocator) + for m in members { + c := class[m] + if _, done := encoded[c]; !done do encoded[c] = class_digest(g, members, class, c) + g.nodes[m].digest = encoded[c] + g.nodes[m].done = true + } +} + +@(private = "file") +refine_signature :: proc(g: ^Graph, i: int, class: map[int]int) -> Value_Digest { + node := &g.nodes[i] + buf := make([dynamic]u8, 0, 64, context.temp_allocator) + append(&buf, node.tag) + append(&buf, ..node.prefix) + for edge in node.edges { + slot := edge.slot + if edge.has_slot do append(&buf, ..slot[:]) + if edge.target == NOT_A_NODE || edge.target not_in class { + // Outside the component, so already final. + append(&buf, 0x00) + child := edge.leaf + if edge.target != NOT_A_NODE do child = g.nodes[edge.target].digest + append(&buf, ..child[:]) + } else { + append(&buf, 0x01) + append(&buf, ..u32_bytes(u32(class[edge.target]))) + } + } + return sha256_of(buf[:]) +} + +// The canonical encoding of the quotient graph reachable from class `c`. +// +// A deterministic walk numbers the classes it meets - `c` is 0, and each new +// class takes the next number in the order the walk first reaches it - and +// then every class is written out in that numbering, with intra-component +// children written as numbers rather than digests. Because the walk visits +// children in slot order and bisimilar nodes have identical slots, two +// bisimilar classes produce the same numbering and therefore the same bytes. +@(private = "file") +class_digest :: proc(g: ^Graph, members: []int, class: map[int]int, c: int) -> Value_Digest { + representative := make(map[int]int, len(members), context.temp_allocator) + for m in members { + if _, seen := representative[class[m]]; !seen do representative[class[m]] = m + } + + order := make(map[int]int, len(members), context.temp_allocator) + visit_order := make([dynamic]int, 0, len(members), context.temp_allocator) + + // Breadth-first from `c`, taking each node's children in slot order and + // numbering a class the first time it is reached. Breadth-first rather than + // depth-first for no deeper reason than that it is the one whose numbering + // is obvious from the code: `visit_order` is the queue and the numbering at + // once, so "the order the walk first meets them" is literally what it holds. + order[c] = 0 + append(&visit_order, c) + for at := 0; at < len(visit_order); at += 1 { + node := &g.nodes[representative[visit_order[at]]] + for edge in node.edges { + if edge.target == NOT_A_NODE || edge.target not_in class do continue + child := class[edge.target] + if _, seen := order[child]; seen do continue + order[child] = len(visit_order) + append(&visit_order, child) + } + } + + buf := make([dynamic]u8, 0, 256, context.temp_allocator) + for visited in visit_order { + node := &g.nodes[representative[visited]] + append(&buf, TAG_CYCLIC_NODE) + append(&buf, node.tag) + prefix := sha256_of(node.prefix) + append(&buf, ..prefix[:]) + append(&buf, ..u32_bytes(u32(len(node.edges)))) + for edge in node.edges { + slot := edge.slot + append(&buf, 0x01 if edge.has_slot else 0x00) + append(&buf, ..slot[:]) + if edge.target == NOT_A_NODE || edge.target not_in class { + append(&buf, 0x00) + child := edge.leaf + if edge.target != NOT_A_NODE do child = g.nodes[edge.target].digest + append(&buf, ..child[:]) + } else { + append(&buf, 0x01) + ref: Value_Digest + copy(ref[:], u32_bytes(u32(order[class[edge.target]]))) + append(&buf, ..ref[:]) + } + } + } + return sha256_tagged(TAG_CYCLIC, buf[:]) +} + +@(private = "file") +u32_bytes :: proc(v: u32) -> []u8 { + buf := make([]u8, 4, context.temp_allocator) + for i in 0 ..< 4 do buf[i] = u8((v >> (8 * uint(i))) & 0xff) + return buf +} diff --git a/src/hash_function.odin b/src/hash_function.odin new file mode 100644 index 0000000..48d1e4c --- /dev/null +++ b/src/hash_function.odin @@ -0,0 +1,261 @@ +package hashedbuild + +import "core:slice" + +// A Function's digest (SPEC.md §15), which the rest of hash.odin's Merkle fold +// treats as just another composite. +// +// §15 says `cached` hashes the cached expression "as a function", and asks for +// the cache key to be the hash of that function representation - but it never +// says what a closure's representation *is*, which is the decision this file +// makes. A closure is two things and nothing else: +// +// * **the shape of its body** - the AST, node kind by node kind, with each +// leaf's own spelling folded in; and +// * **the values it captures** - the free names of that body, looked up in +// the environment the closure was made in, each hashed as an ordinary +// value and mixed in under its name. +// +// So two closures hash alike exactly when they would compute the same thing: +// the same expression, reading the same values. Where they were written, and +// what else happened to be in scope there, does not enter into it - which is +// the property `cached` actually needs, since a cache keyed on irrelevant +// surroundings misses every time the surroundings change. +// +// Three things follow, and are worth being plain about: +// +// * **Bound names count.** `func (let x 1; x)` and `func (let y 1; y)` are +// the same function and hash differently, because the shape includes every +// leaf's spelling. Alpha-equivalence would need the binders renumbered, +// which is a bigger analysis than this buys back - a closure's digest +// surviving a rename is not something §15 asks for. +// * **Literals hash as written.** `"a"` and `"\x61"` are the same Utf8 value +// and hash differently as *shapes*, for the same reason. The digest is of +// the program, not of what the program would evaluate to. +// * **The free-name set is an over-approximation.** free_names below collects +// every identifier in a reference position, including ones an inner `let` +// goes on to bind. A name bound inside the body simply is not in the +// closure's environment, so it contributes nothing; one that is *also* a +// name outside contributes a value the body never reads. That costs +// precision - two closures that differ only in such a shadow hash apart - +// and never correctness, which is the direction that matters: every name +// the body can actually read is in the set. +// +// A builtin (§16) has no body to take a shape from, so it hashes as its name +// plus whatever it carries in `native_closure` - `chperm { .name = "io" }` +// returns a function, and two of those are the same function when they were +// built from the same permission. + +// One captured name and the value it stood for. Exposed because two other +// places walk a closure's children: hash.odin warms directory digests before a +// comparison, and hash_cyclic.odin treats a closure as a graph node whose +// out-edges are exactly these. +Function_Capture :: struct { + name: string, + value: Value, +} + +function_digest :: proc(fv: ^Function_Value, w: ^Hash_Walk) -> (Value_Digest, Hash_Fail) { + if fv.native != nil { + closure_digest, f := value_digest_walk(fv.native_closure, w) + if f.kind != .None do return {}, f + payload: [2 * DIGEST_SIZE]u8 + nd := sha256_text(fv.name) + copy(payload[0:], nd[:]) + copy(payload[DIGEST_SIZE:], closure_digest[:]) + return sha256_tagged(TAG_NATIVE, payload[:]), HASH_OK + } + + if w.interp == nil do return {}, Hash_Fail{kind = .No_Program} + + // A closure that captures itself - `let rec f func (f #arg)`, the ordinary + // way to write a recursive function - is a cycle like any other, and goes + // to hash_cyclic.odin by the same route a cyclic Table does. + for open in w.open { + if open == rawptr(fv) do return {}, Hash_Fail{kind = .Cyclic} + } + append(&w.open, rawptr(fv)) + defer pop(&w.open) + + shape := function_shape_digest(w.interp, fv) + captures := function_captures(w.interp, fv) + + buf := make([]u8, DIGEST_SIZE + len(captures) * 2 * DIGEST_SIZE, context.temp_allocator) + copy(buf[0:], shape[:]) + for capture, i in captures { + vd, f := value_digest_walk(capture.value, w) + if f.kind != .None do return {}, f + nd := sha256_text(capture.name) + at := DIGEST_SIZE + i * 2 * DIGEST_SIZE + copy(buf[at:], nd[:]) + copy(buf[at + DIGEST_SIZE:], vd[:]) + } + return sha256_tagged(TAG_FUNCTION, buf), HASH_OK +} + +// The body's shape, plus the captured `ctx` when - and only when - the body +// can see it. A closure carries the context it was made in (§9), and two +// closures with different authority genuinely are different functions; but +// folding `ctx` in unconditionally would make every closure's digest depend on +// the whole permission table, so a body that never writes `ctx` does not pay +// for it. +function_shape_digest :: proc(interp: ^Interpreter, fv: ^Function_Value) -> Value_Digest { + body := node_shape_digest(interp, fv.body) + _, uses_ctx := free_names(interp, fv.body) + if !uses_ctx do return body + + ctx_walk := Hash_Walk{interp = interp, open = make([dynamic]rawptr, 0, 4, context.temp_allocator)} + ctx_digest, f := value_digest_walk(fv.ctx, &ctx_walk) + // A context that cannot be hashed - it holds a directory nobody has read, say + // - folds in as a bare tag rather than failing the whole function: `ctx` is + // ambient authority, not an argument, and refusing to hash a closure because + // of what its caller could do is not a distinction §15 wants. + if f.kind != .None do ctx_digest = sha256_tagged(TAG_CACHE, nil) + + payload: [2 * DIGEST_SIZE]u8 + copy(payload[0:], body[:]) + copy(payload[DIGEST_SIZE:], ctx_digest[:]) + return sha256_tagged(TAG_FUNCTION, payload[:]) +} + +// The free names of the closure's body, paired with what they stood for where +// it was written. Sorted by name, so the digest does not depend on the order +// the walk happened to meet them in. A name the environment does not hold is +// dropped: it is bound inside the body (see the over-approximation note above), +// or it is an undefined name the program will fail on anyway. +// +// **That order is part of the encoding**, not an internal detail: a closure +// caught in a cycle is encoded by hash_cyclic.odin instead, and it mirrors this +// order exactly rather than imposing its own. The two must agree, or the same +// closure would have two digests depending on how it was reached. +function_captures :: proc(interp: ^Interpreter, fv: ^Function_Value) -> []Function_Capture { + if fv.native != nil do return nil + + names, _ := free_names(interp, fv.body) + captures := make([dynamic]Function_Capture, 0, len(names), context.temp_allocator) + for name in names { + if v, found := env_lookup(fv.env, name); found { + append(&captures, Function_Capture{name = name, value = v}) + } + } + slice.sort_by(captures[:], proc(a, b: Function_Capture) -> bool { return a.name < b.name }) + return captures[:] +} + +// ---- the body's shape --------------------------------------------------------- + +// One node as a fixed-width record - kind, the flags that change meaning, +// child count, then either the leaf's own text or the children's digests. The +// count is what makes concatenating the children unambiguous without any +// separator, the same argument the Merkle encoding rests on in hash.odin. +@(private = "file") +node_shape_digest :: proc(interp: ^Interpreter, idx: Node_Idx) -> Value_Digest { + n := interp.ast.nodes[idx] + + // Only the two flags that mean something semantically. Has_Error and + // Is_Missing describe a program that isn't going to run. + flags: u8 + if .Computed_Key in n.flags do flags |= 1 + if .Is_Rec in n.flags do flags |= 2 + + header: [5]u8 + header[0] = u8(u16(n.kind) & 0xff) + header[1] = u8(u16(n.kind) >> 8) + header[2] = flags + header[3] = u8(n.children_count & 0xff) + header[4] = u8(n.children_count >> 8) + + if n.children_count == 0 { + text := sha256_text(node_text(interp, idx)) + payload: [5 + DIGEST_SIZE]u8 + copy(payload[0:], header[:]) + copy(payload[5:], text[:]) + return sha256_tagged(TAG_AST, payload[:]) + } + + buf := make([]u8, 5 + int(n.children_count) * DIGEST_SIZE, context.temp_allocator) + copy(buf[0:], header[:]) + for i in 0 ..< int(n.children_count) { + child := node_shape_digest(interp, interp.ast.extra_children[int(n.children_start) + i]) + copy(buf[5 + i * DIGEST_SIZE:], child[:]) + } + return sha256_tagged(TAG_AST, buf) +} + +// A leaf's own spelling. Operators and punctuation have a kind and nothing +// else worth hashing, but taking the span uniformly costs nothing and means a +// new leaf kind that *does* carry text needs no change here. +@(private = "file") +node_text :: proc(interp: ^Interpreter, idx: Node_Idx) -> string { + n := interp.ast.nodes[idx] + if int(n.span.end) > len(interp.src) || n.span.start > n.span.end do return "" + return interp.src[n.span.start:n.span.end] +} + +// ---- free names ---------------------------------------------------------------- + +// Every identifier in the subtree that is a *reference* to a name, plus +// whether the subtree mentions `ctx`. The work is in telling a reference from +// the several places an Identifier leaf means something else entirely - a +// field's spelling, a binder, a pattern's selector - none of which reads +// anything from the enclosing scope. +@(private = "file") +free_names :: proc(interp: ^Interpreter, idx: Node_Idx) -> (names: []string, uses_ctx: bool) { + found := make([dynamic]string, 0, 8, context.temp_allocator) + ctx_seen := false + collect_names(interp, idx, &found, &ctx_seen) + + slice.sort(found[:]) + unique := make([dynamic]string, 0, len(found), context.temp_allocator) + for name, i in found { + if i > 0 && found[i - 1] == name do continue + append(&unique, name) + } + return unique[:], ctx_seen +} + +@(private = "file") +collect_names :: proc(interp: ^Interpreter, idx: Node_Idx, out: ^[dynamic]string, uses_ctx: ^bool) { + n := interp.ast.nodes[idx] + + #partial switch n.kind { + case .Identifier: + append(out, node_text(interp, idx)) + return + case .Ctx_Expr: + uses_ctx^ = true + return + } + + for i in 0 ..< int(n.children_count) { + if child_is_a_spelling(interp, n, i) do continue + collect_names(interp, interp.ast.extra_children[int(n.children_start) + i], out, uses_ctx) + } +} + +// Whether child `i` of `n` is an Identifier used as a literal spelling rather +// than as a variable. Each of these is a place the parser reuses the +// Identifier leaf for a name that is written down rather than looked up, and +// descending into one would invent a capture out of a field name. +@(private = "file") +child_is_a_spelling :: proc(interp: ^Interpreter, n: Node, i: int) -> bool { + #partial switch n.kind { + case .Binary_Expr: + // [left, op_leaf, right]. `a.b` and `a !.b` name a field on the left + // operand; the right leaf is that field's spelling, not a variable. + if i != 2 do return false + op := interp.ast.nodes[interp.ast.extra_children[int(n.children_start) + 1]].kind + return op == .Op_Dot || op == .Op_CheckDot + case .Table_Entry: + // [key, value]. A `.name = v` key is the literal text; a `[expr] = v` key + // is an expression, and Computed_Key is the only thing telling them apart. + return i == 0 && .Computed_Key not_in n.flags + case .Let_Bind: + return i == 1 // [bound_expr, name_leaf, body] + case .Pattern_Bind: + return i == 1 // [pattern, name_leaf] + case .Table_Pattern_Field: + return i == 0 // [name_leaf] + } + return false +} diff --git a/src/hash_linux_test.odin b/src/hash_linux_test.odin new file mode 100644 index 0000000..c6a1a0a --- /dev/null +++ b/src/hash_linux_test.odin @@ -0,0 +1,61 @@ +// The executable bit is the one part of SPEC.md §3's directory hash that only +// one target can see, so it is the one part tested on only one target. WASI's +// filestat carries no permission bits and Windows has no POSIX exec bit +// (fs.odin), and hashing there treats every file as non-executable - which is +// the language's answer, not a gap, and is what LANGUAGE.md documents. +#+build linux +package hashedbuild + +import "core:os" +import "core:strings" +import "core:sys/linux" +import "core:testing" + +@(private = "file") +eval_digest :: proc(t: ^testing.T, path: string) -> string { + src := strings.concatenate({`sha256 loadfile "`, path, `"`}) + defer delete(src) + ast := parse(source_t{name = "test", n_bytes = u64(len(src)), data = raw_data(src)}, ast_t{}) + cache := strings.concatenate({repo_root(), "/.hash_linux_test_cache"}) + defer delete(cache) + interp := Interpreter{ast = &ast, src = src, current_ctx = make_root_context(cache)} + val, ok := eval_program(&interp, ast.root, make_global_env()) + testing.expect(t, ok, interp.error_message) + digest, is_str := val.(string) + testing.expect(t, is_str, "expected a Utf8 digest") + return digest +} + +@(test) +test_the_executable_bit_is_part_of_a_directory_hash :: proc(t: ^testing.T) { + root := strings.concatenate({repo_root(), "/.hash_linux_test_exec"}) + defer delete(root) + file := strings.concatenate({root, "/build.sh"}) + defer delete(file) + defer os.remove(root) + defer os.remove(file) + + os.remove(file) + os.remove(root) + testing.expect(t, os.make_directory(root) == nil, "could not create the scratch tree") + _ = os.write_entire_file(file, transmute([]u8)string("#!/bin/sh\necho hi\n")) + + // Two hashes of the same bytes under the same name, differing only in + // whether the file is a program. §3 asks for exactly that distinction, and + // it is a real one in a build: a checked-out `configure` that lost its bit + // is not the same tree. + cname := strings.clone_to_cstring(file, context.temp_allocator) + testing.expect(t, linux.chmod(cname, {.IRUSR, .IWUSR}) == .NONE) + plain := eval_digest(t, root) + + testing.expect(t, linux.chmod(cname, {.IRUSR, .IWUSR, .IXUSR}) == .NONE) + executable := eval_digest(t, root) + + testing.expect(t, plain != executable, "the exec bit is part of what a directory is") + + // ...and it is only the owner bit that counts. §3 says "the executable flag + // only - not full POSIX mode", so who else may run it is not part of what + // the file is. + testing.expect(t, linux.chmod(cname, {.IRUSR, .IWUSR, .IXUSR, .IRGRP, .IROTH}) == .NONE) + testing.expect_value(t, eval_digest(t, root), executable) +} diff --git a/src/hash_test.odin b/src/hash_test.odin index 3aabb16..7d9b4a9 100644 --- a/src/hash_test.odin +++ b/src/hash_test.odin @@ -2,6 +2,8 @@ #+build linux, windows package hashedbuild +import "core:log" +import "core:os" import "core:strings" import "core:testing" @@ -127,15 +129,298 @@ test_sha256_awaits_an_async_operand :: proc(t: ^testing.T) { testing.expect(t, eval_bool(t, `(sha256 async ("a" concat "b")) == (sha256 "ab")`)) } -// The kinds §3/§15 leave open fail by name rather than inventing a digest. +// ---- directories (§3) ---------------------------------------------------------- + +// A scratch tree, built entry by entry so a test can say exactly what it holds +// and then change one thing about it. +@(private = "file") +Tree :: struct { + root: string, +} + +@(private = "file") +make_tree :: proc(t: ^testing.T, name: string) -> Tree { + root := strings.concatenate({repo_root(), "/.hash_test_", name}) + clear_tree_at(root) // a leftover from an interrupted run + err := os.make_directory(root) + testing.expect(t, err == nil, "could not create the scratch tree") + return Tree{root = root} +} + +@(private = "file") +tree_write :: proc(tree: Tree, rel: string, content: string) { + path := strings.concatenate({tree.root, "/", rel}, context.temp_allocator) + _ = os.write_entire_file(path, transmute([]u8)content) +} + +@(private = "file") +tree_subdir :: proc(tree: Tree, rel: string) { + _ = os.make_directory(strings.concatenate({tree.root, "/", rel}, context.temp_allocator)) +} + +// One level of nesting deep, which is all these trees ever have. +@(private = "file") +clear_tree_at :: proc(root: string) { + if handle, err := os.open(root); err == nil { + entries, _ := os.read_dir(handle, -1, context.temp_allocator) + for entry in entries { + child := strings.concatenate({root, "/", entry.name}, context.temp_allocator) + // Only a directory gets opened and listed. Handing a file's handle to + // os.read_dir is not a no-op on Windows - it walks a structure that + // isn't there. + if entry.type == .Directory { + if inner, ierr := os.open(child); ierr == nil { + grandchildren, _ := os.read_dir(inner, -1, context.temp_allocator) + for g in grandchildren do os.remove(strings.concatenate({child, "/", g.name}, context.temp_allocator)) + os.close(inner) + } + } + os.remove(child) + } + os.close(handle) + } + os.remove(root) +} + +@(private = "file") +remove_tree :: proc(tree: Tree) { + clear_tree_at(tree.root) + delete(tree.root) +} + +@(private = "file") +tree_digest :: proc(t: ^testing.T, tree: Tree) -> string { + src := strings.concatenate({`sha256 loadfile "`, tree.root, `"`}, context.temp_allocator) + return eval_str(t, src) +} + +// §3 defines a directory's hash over its entries. The point of the whole +// exercise is that the digest is the *tree's*, not the path's: two directories +// holding the same thing are the same value, and one byte anywhere inside is a +// different one. @(test) -test_unhashable_values_fail_with_a_reason :: proc(t: ^testing.T) { - testing.expect(t, strings.contains(eval_failure(t, `sha256 func 1`), "Function has no hash")) - testing.expect(t, strings.contains(eval_failure(t, `sha256 ctx.cache`), "ctx.cache has no hash")) +test_directory_hash_is_its_contents :: proc(t: ^testing.T) { + a := make_tree(t, "dir_a") + defer remove_tree(a) + b := make_tree(t, "dir_b") + defer remove_tree(b) + + for tree in ([]Tree{a, b}) { + tree_write(tree, "one.txt", "hello") + tree_subdir(tree, "nested") + tree_write(tree, "nested/two.txt", "world") + } + testing.expect_value(t, tree_digest(t, a), tree_digest(t, b)) + + // One byte, one level down. + before := tree_digest(t, a) + c := make_tree(t, "dir_c") + defer remove_tree(c) + tree_write(c, "one.txt", "hello") + tree_subdir(c, "nested") + tree_write(c, "nested/two.txt", "worlds") + testing.expect(t, before != tree_digest(t, c), "a changed byte is a changed tree") +} + +@(test) +test_directory_hash_covers_names_not_just_content :: proc(t: ^testing.T) { + // §3 hashes each entry with its name, so the same bytes under a different + // name is a different directory. Without the name in the entry digest these + // two would collide. + a := make_tree(t, "name_a") + defer remove_tree(a) + tree_write(a, "alpha.txt", "same") + + b := make_tree(t, "name_b") + defer remove_tree(b) + tree_write(b, "beta.txt", "same") + + testing.expect(t, tree_digest(t, a) != tree_digest(t, b)) +} + +@(test) +test_a_directory_is_not_its_only_file :: proc(t: ^testing.T) { + // A regular File hashes as its bare content (§3, untagged); a directory + // holding just that file must not land on the same digest. + tree := make_tree(t, "dir_vs_file") + defer remove_tree(tree) + tree_write(tree, "only.txt", "content") + + file_src := strings.concatenate({`sha256 loadfile "`, tree.root, `/only.txt"`}, context.temp_allocator) + testing.expect(t, tree_digest(t, tree) != eval_str(t, file_src)) +} + +@(test) +test_two_directory_handles_on_one_tree_are_equal :: proc(t: ^testing.T) { + // §3: a File's identity is content, not path. Two separate handles are two + // objects, so this only holds because the comparison hashes them - which + // means the evaluator warmed both digests first (hash.odin). + tree := make_tree(t, "dir_equality") + defer remove_tree(tree) + tree_write(tree, "x.txt", "same") + + src := strings.concatenate({ + `let a loadfile "`, tree.root, `"; let b loadfile "`, tree.root, `"; a == b`, + }, context.temp_allocator) + testing.expect(t, eval_bool(t, src), "two handles on one tree are one value") +} - dir := strings.concatenate({`sha256 loadfile "`, repo_root(), `/examples"`}) - defer delete(dir) - testing.expect(t, strings.contains(eval_failure(t, dir), "directory File has no hash")) +@(test) +test_different_trees_are_not_equal :: proc(t: ^testing.T) { + a := make_tree(t, "neq_a") + defer remove_tree(a) + tree_write(a, "x.txt", "one") + b := make_tree(t, "neq_b") + defer remove_tree(b) + tree_write(b, "x.txt", "two") + + src := strings.concatenate({ + `let a loadfile "`, a.root, `"; let b loadfile "`, b.root, `"; a == b`, + }, context.temp_allocator) + testing.expect(t, !eval_bool(t, src)) +} + +// Reading a tree is I/O, and I/O is what §9's permission governs. The handle +// is obtained while io is granted; the *hash* is asked for after it has been +// revoked, which is the moment the read would happen. +@(test) +test_hashing_a_directory_needs_io :: proc(t: ^testing.T) { + tree := make_tree(t, "dir_perm") + defer remove_tree(tree) + tree_write(tree, "x.txt", "content") + + src := strings.concatenate({ + `let d loadfile "`, tree.root, `"; (sha256 d) chctx chperm { .name = "io", .enabled = 1 > 2 }`, + }, context.temp_allocator) + testing.expect(t, strings.contains(eval_failure(t, src), "needs the io permission")) +} + +@(test) +test_a_directory_digest_is_read_once :: proc(t: ^testing.T) { + // §3 calls a File an immutable handle, so its digest is fixed at the first + // read: revoking io afterwards cannot make the same value unhashable, and + // changing the tree afterwards cannot make it a different value. + tree := make_tree(t, "dir_memo") + defer remove_tree(tree) + tree_write(tree, "x.txt", "before") + + src := strings.concatenate({ + `let d loadfile "`, tree.root, `";`, + ` (sha256 d) == ((sha256 d) chctx chperm { .name = "io", .enabled = 1 > 2 })`, + }, context.temp_allocator) + testing.expect(t, eval_bool(t, src), "the second ask is answered from the first read") +} + +// §3 hashes a symlink entry as its target *string*, without resolving it - so +// two links pointing at different names are two different trees even when +// neither target exists, and a link is never confused with what it points at. +// +// Skipped where a symlink cannot be created: on Windows that needs Developer +// Mode or an elevated shell, the same privilege examples/files-symlink.hb +// wants (see examples_test.odin). +@(test) +test_symlink_entries_hash_as_their_target :: proc(t: ^testing.T) { + tree := make_tree(t, "dir_symlink") + defer remove_tree(tree) + + dir_fd, oerr := fs_open_dir_path(tree.root) + testing.expect(t, oerr == .None, "could not open the scratch tree") + if oerr != .None do return + defer fs_close(dir_fd) + + if err := fs_symlink_at(dir_fd, "link", "somewhere.txt"); err != .None { + log.infof("skipping: this environment cannot create a symlink (%v) - on Windows that needs Developer Mode", err) + return + } + with_first := tree_digest(t, tree) + + other := make_tree(t, "dir_symlink_other") + defer remove_tree(other) + other_fd, oerr2 := fs_open_dir_path(other.root) + testing.expect(t, oerr2 == .None) + if oerr2 != .None do return + defer fs_close(other_fd) + testing.expect(t, fs_symlink_at(other_fd, "link", "elsewhere.txt") == .None) + + testing.expect(t, with_first != tree_digest(t, other), "the target string is part of the entry") +} + +// ---- ctx.cache (§9) ------------------------------------------------------------ + +@(test) +test_ctx_cache_hashes_as_a_bare_tag :: proc(t: ^testing.T) { + // §6 says every value is hashable, and ctx.cache is a value. It has no + // content to hash and its one distinguishing feature - the directory it is + // rooted at - is the path §9 keeps out of the language, so it is a tag and + // nothing more. What that has to be is stable and unlike anything else. + testing.expect(t, eval_bool(t, `(sha256 ctx.cache) == (sha256 ctx.cache)`)) + testing.expect(t, !eval_bool(t, `(sha256 ctx.cache) == (sha256 nothing)`)) +} + +// ---- functions (§15) ----------------------------------------------------------- + +@(test) +test_functions_hash_by_body_and_captures :: proc(t: ^testing.T) { + // The same expression reading the same values is the same function... + testing.expect(t, eval_bool(t, `(sha256 func (#arg + 1)) == (sha256 func (#arg + 1))`)) + testing.expect(t, eval_bool(t, `(sha256 (let x 1; func (#arg + x))) == (sha256 (let x 1; func (#arg + x)))`)) + // ...and a different body, or a different captured value, is not. + testing.expect(t, !eval_bool(t, `(sha256 func (#arg + 1)) == (sha256 func (#arg + 2))`)) + testing.expect(t, !eval_bool(t, `(sha256 (let x 1; func (#arg + x))) == (sha256 (let x 2; func (#arg + x)))`)) +} + +@(test) +test_function_hash_ignores_the_rest_of_the_scope :: proc(t: ^testing.T) { + // The property `cached` needs: a closure's digest is what it reads, not what + // happened to be in scope where it was written. Without free-variable + // analysis this would fail, and every cache lookup would miss whenever an + // unrelated binding nearby changed. + testing.expect(t, eval_bool(t, ` + (sha256 (let x 1; let unrelated "zz"; func (#arg + x))) + == (sha256 (let x 1; func (#arg + x)))`)) +} + +@(test) +test_builtins_hash_by_name :: proc(t: ^testing.T) { + // A builtin has no body to take a shape from (§16), so its identity is the + // operation it is. + testing.expect(t, eval_bool(t, `(sha256 loadfile) == (sha256 loadfile)`)) + testing.expect(t, !eval_bool(t, `(sha256 loadfile) == (sha256 createfile)`)) + // A partially applied one carries what it was built from, so two `chperm` + // results are the same function exactly when they grant the same thing. + testing.expect(t, eval_bool(t, ` + (sha256 chperm { .name = "io", .enabled = 1 < 2 }) + == (sha256 chperm { .name = "io", .enabled = 1 < 2 })`)) + testing.expect(t, !eval_bool(t, ` + (sha256 chperm { .name = "io", .enabled = 1 < 2 }) + == (sha256 chperm { .name = "io", .enabled = 1 > 2 })`)) +} + +@(test) +test_a_recursive_function_hashes :: proc(t: ^testing.T) { + // `let rec` over a closure captures the scope the closure itself is bound + // in, so the function reaches itself: a cycle, and it goes the same way a + // cyclic Table does (hash_cyclic.odin). What matters is that it terminates + // and stays consistent. + fact := `let rec fact (let n; (n == 0) then 1 else n * (fact (n - 1))); sha256 fact` + digest := eval_str(t, fact) + testing.expect(t, len(digest) == 44) + testing.expect(t, eval_bool(t, strings.concatenate({ + `(`, fact, `) == "`, digest, `"`, + }, context.temp_allocator)), "the same recursive function hashes the same way twice") +} + +// The two kinds §3/§15 still describe no digest for. Both are shapes a program +// cannot hold: an un-awaited handle is awaited by every operator that meets +// one (§2), and a `.Other` directory entry is a device node in a build tree. +@(test) +test_unhashable_values_fail_with_a_reason :: proc(t: ^testing.T) { + // Everything the old version of this test listed - a Function, ctx.cache, a + // directory File - now hashes; the tests above are what replaced it. What is + // left is worth one assertion: `serialize` was removed, and the surface says + // so by these being ordinary names rather than reserved words. + testing.expect(t, len(eval_str(t, `sha256 func 1`)) == 44) + testing.expect(t, len(eval_str(t, `sha256 ctx.cache`)) == 44) } // `serialize`/`serialize_file` were removed from the language, so they are diff --git a/src/rec_build_test.odin b/src/rec_build_test.odin index 301a521..7de07de 100644 --- a/src/rec_build_test.odin +++ b/src/rec_build_test.odin @@ -19,6 +19,20 @@ run :: proc(src: string) -> (val: Value, ok: bool, err: string) { return val, ok, interp.error_message } +// Like `run`, but the AST and the interpreter are heap-allocated and kept: a +// Function's digest is the shape of its body, which is a subtree of the +// program it was written in, so hashing one after the run needs both alive. +@(private = "file") +run_keeping_the_program :: proc(src: string) -> (val: Value, interp: ^Interpreter, ok: bool) { + ast := new(ast_t) + ast^ = parse(source_t{name = "test", n_bytes = u64(len(src)), data = raw_data(src)}, ast_t{}) + interp = new(Interpreter) + interp.ast = ast + interp.src = src + val, ok = eval(interp, ast.root, env_make_child(nil)) + return +} + @(private = "file") expect_prints :: proc(t: ^testing.T, src: string, want: string) { val, ok, err := run(src) @@ -246,15 +260,119 @@ test_printing_leaves_acyclic_sharing_alone :: proc(t: ^testing.T) { // ---- hashing ----------------------------------------------------------------- @(test) -test_hashing_a_cyclic_value_is_refused :: proc(t: ^testing.T) { - // §3 pins what a digest encodes, so a cyclic one is a spec decision rather - // than an implementation detail - until it is made, this must fail cleanly - // rather than recurse forever. See hash.odin. +test_hashing_a_cyclic_value_terminates :: proc(t: ^testing.T) { + // The Merkle fold has no bottom to start from here, so this goes through + // hash_cyclic.odin instead. What it must not do is recurse forever. val, ok, err := run("let rec p { .n = 1, .self = p }; p") testing.expect(t, ok, err) if !ok do return _, herr := value_digest(val) - testing.expect_value(t, herr, Hash_Error.Cyclic) + testing.expect_value(t, herr.kind, Hash_Error.None) +} + +@(test) +test_cyclic_digests_agree_with_cyclic_equality :: proc(t: ^testing.T) { + // The property the whole of hash_cyclic.odin exists for: values_equal + // compares cycles by bisimulation, so the digest has to be canonical under + // bisimulation too, or a content-addressed language would have equal values + // that hash apart. These are the same three pairs test_cyclic_equality_is_ + // bisimulation asserts equality for. + same := []([2]string) { + // Two separately built cycles of the same shape. + {"let rec p { .n = 1, .self = p }; p", "let rec q { .n = 1, .self = q }; q"}, + // A 1-cycle and a 2-cycle that unroll to the same infinite tree. + { + "let rec p { .n = 1, .self = p }; p", + "let rec p { .n = 1, .self = { .n = 1, .self = p } }; p", + }, + } + for pair in same { + a, aok, aerr := run(pair[0]) + testing.expect(t, aok, aerr) + b, bok, berr := run(pair[1]) + testing.expect(t, bok, berr) + if !aok || !bok do continue + + testing.expect(t, values_equal(a, b), "the two are equal, so this pair is the interesting one") + da, ea := value_digest(a) + db, eb := value_digest(b) + testing.expect_value(t, ea.kind, Hash_Error.None) + testing.expect_value(t, eb.kind, Hash_Error.None) + testing.expect(t, da == db, "equal cyclic values must hash alike") + } +} + +@(test) +test_the_two_hash_paths_agree_on_acyclic_values :: proc(t: ^testing.T) { + // §6 asks for *one* encoding, and there are two implementations of it: the + // Merkle fold (hash.odin) and the graph algorithm (hash_cyclic.odin), which + // takes over the moment a value contains a cycle anywhere. The parts of that + // value which are *not* cyclic must come out the same either way - otherwise + // a Table's digest would depend on whether some unrelated corner of the + // value it was reached through happened to loop. + // + // Asked of the two procedures directly, because from inside the language the + // fold always wins for an acyclic value and the disagreement would be + // invisible until something cyclic wrapped it. + sources := []string { + `{ .a = 1, .b = { .c = "x", .d = 2.5 } }`, + `{ 1, 2, 3 }`, + // A closure, whose captures the two paths order independently - the one + // place they have actually drifted. + `let alpha 1; let zeta 2; let beta 3; { .f = func (#arg + alpha + zeta + beta) }`, + `empty`, + } + for src in sources { + // The interpreter has to outlive the evaluation here: a closure's digest + // is its body's shape, and the body is a subtree of that program's AST. + val, interp, ok := run_keeping_the_program(src) + testing.expect(t, ok, interp.error_message) + if !ok do continue + + folded, ferr := value_digest(val, interp) + graphed, gerr := value_digest_cyclic(val, interp) + testing.expect_value(t, ferr.kind, Hash_Error.None) + testing.expect_value(t, gerr.kind, Hash_Error.None) + testing.expect(t, folded == graphed, "the fold and the graph must encode the same value the same way") + } +} + +@(test) +test_cyclic_digests_separate_unequal_cycles :: proc(t: ^testing.T) { + // Canonical must not mean constant: two cycles that unroll differently have + // to hash apart, or every cyclic value would share one digest. + a, aok, aerr := run("let rec p { .n = 1, .self = p }; p") + testing.expect(t, aok, aerr) + b, bok, berr := run("let rec p { .n = 2, .self = p }; p") + testing.expect(t, bok, berr) + if !aok || !bok do return + da, _ := value_digest(a) + db, _ := value_digest(b) + testing.expect(t, da != db, "different cycles must hash differently") +} + +@(test) +test_a_cycle_hashes_the_same_from_either_end :: proc(t: ^testing.T) { + // The entry-point independence the canonical form is for: `.a` and `.b` of a + // two-node cycle are not bisimilar (their entries differ), but reaching the + // *same* node down two different paths must give one digest either way. + val, ok, err := run("let rec p { .a = { .tag = 1, .back = p }, .b = p.a }; p") + testing.expect(t, ok, err) + if !ok do return + table, is_table := val.(^Table_Value) + testing.expect(t, is_table) + if !is_table do return + + a, has_a := table_find(table, "a") + b, has_b := table_find(table, "b") + testing.expect(t, has_a && has_b) + if !has_a || !has_b do return + + da, ea := value_digest(a) + db, eb := value_digest(b) + testing.expect_value(t, ea.kind, Hash_Error.None) + testing.expect_value(t, eb.kind, Hash_Error.None) + testing.expect(t, da == db, "one node reached two ways is one digest") } @(test) @@ -268,7 +386,7 @@ test_hashing_acyclic_values_is_unaffected :: proc(t: ^testing.T) { if !aok || !bok do return da, ea := value_digest(a) db, eb := value_digest(b) - testing.expect_value(t, ea, Hash_Error.None) - testing.expect_value(t, eb, Hash_Error.None) + testing.expect_value(t, ea.kind, Hash_Error.None) + testing.expect_value(t, eb.kind, Hash_Error.None) testing.expect(t, da == db, "equal values must hash alike") } diff --git a/src/value.odin b/src/value.odin index 2ada6e9..6a5ee67 100644 --- a/src/value.odin +++ b/src/value.odin @@ -33,6 +33,10 @@ Function_Value :: struct { ctx: Value, // captured `ctx` (SPEC.md §9) at the point the closure was made - see eval.odin's call_function native: Native_Fn, // non-nil for a builtin (§16) - call_function invokes this instead of evaluating body/env native_closure: Value, // passed as `closure` to `native`, if any + // Builtins only: the operation's name, which is what one hashes as + // (hash_function.odin). Empty for a closure, whose identity is its body's + // shape and its captures instead. + name: string, } // SPEC.md §3's File: a handle to a filesystem entity, file or directory only @@ -53,6 +57,20 @@ File_Value :: struct { // - a program holds the handle without ever learning where its data lives. // Empty only if that resolution failed (see builtins_fs.odin's path_of_fd). display_path: string, + + // Directory only: §3's directory hash, read off the disk the first time + // anything asks for it and kept thereafter (hash.odin). A Regular file's + // content is already in `content`, so it needs no such field - this is the + // one kind whose digest is not a function of what the value already holds. + // + // Memoised rather than recomputed because §3 calls a File an *immutable* + // handle: a value whose digest changed under a program because someone + // touched the tree would not be one. The first read is therefore the read, + // and a later `sha256` of the same value answers the same thing forever. + // (A fresh `loadfile` of the same path is a new value, and sees the tree as + // it is then - which is how a build observes a change.) + dir_digest: Value_Digest, + dir_digest_known: bool, } // SPEC.md §9's ctx.cache: a write-only, content-addressed blob store rooted @@ -280,12 +298,19 @@ values_equal_bisim :: proc(a: Value, b: Value, bs: ^Bisim) -> bool { if !ok do return false // SPEC.md §3: a File's identity is pure content, independent of path - // two Files built from different paths are equal whenever their content - // matches. A Regular file compares by its content digest (hash.odin). - // A Directory still compares by reference: §3 hashes one over its entries - // including each file's executable bit, which only the Linux target can - // report - WASI's filestat has no permission bits at all, and Windows has - // no POSIX exec bit - so that half isn't built (see LANGUAGE.md). - if x.kind == .Directory || y.kind == .Directory do return x == y + // matches. A Regular file compares by its content digest, a Directory by + // §3's entry-wise directory digest (hash.odin). + // + // A directory's digest is read off the disk on first demand, and reading + // is an I/O operation - so it needs the `io` permission and an + // interpreter to ask, neither of which exists down here (this is reached + // from table_find, on the hot path of every field access). The evaluator + // therefore warms both operands before comparing them (eval.odin's + // hash_materialize), and what is left here is a pure question about + // memoised digests. The `x == y` shortcut is what makes a directory still + // compare equal to itself if that warming was skipped or refused. + if x.kind != y.kind do return false + if x == y do return true return values_hash_equal(x, y) case ^Cache_Value: y, ok := bv.(^Cache_Value) From 52b2c65d269168164d90b19088eb863945c831d6 Mon Sep 17 00:00:00 2001 From: Jan Strakowski Date: Mon, 31 Aug 2026 11:03:50 +0200 Subject: [PATCH 2/2] Test cyclic hashing on molecules, and pin what the model cannot see rec_build_test.odin exercises the canonical form on two-node toys. A ring of atoms is a cycle in the literal sense, so molecules push it much harder - and being real structures, they say something about the encoding rather than just covering lines. Five of them, each for a different shape of cycle: * **benzene** - six interchangeable carbons, one bisimulation class. Every carbon hashes alike, which is entry-point independence at its starkest. * **cyclohexane and the Kekule drawing** - the same six-carbon ring with different atoms and bonds. Three rings, three digests: the encoding sees what is in the cycle, not only its shape. * **toluene** - one methyl splits the ring into six positions by distance from it, so the refinement has to find six classes where benzene had one. * **the three xylenes** - same formula, methyls 1,2- / 1,3- / 1,4-. All three separate, and para finds its own two-fold rotation. * **naphthalene and azulene** - both C10H8, a ten-carbon perimeter plus one chord, differing only in where the chord lands. Two interlocking cycles in one component; naphthalene's five classes come out exactly right, and azulene's lopsided fusion correctly has no rotation to find. Two tests pin limitations rather than capabilities, because both turned up while writing the others and both are properties of the *model*, not bugs: * A perfectly symmetric ring cannot count itself. Benzene and the cyclopropenyl cation are the same repeating unit at different periods, so bisimulation calls them one value - and values_equal agrees, which is what makes the shared digest correct rather than a collision. Any substituent recovers the ring size, as the toluene test shows. * Reflection is invisible. m-xylene's two methylated carbons are equivalent to a chemist, and hash apart, because bisimulation matches Table entries by key and a reflection maps `.ring` onto an incoming bond. Listing both neighbours does not help - it just gives reflection two keys to swap. Rotation maps `.ring` onto `.ring`, which is why para works. The root cause of the second is worth recording: a Table's keys must be hashable before its entries can be ordered, so a cyclic value cannot be a key, which rules out the unordered neighbour set a molecule really wants. The digest is therefore of an oriented drawing. Pinned here so a future change to the encoding has to argue with it. Co-Authored-By: Claude Opus 5 --- src/hash_molecules_test.odin | 400 +++++++++++++++++++++++++++++++++++ 1 file changed, 400 insertions(+) create mode 100644 src/hash_molecules_test.odin diff --git a/src/hash_molecules_test.odin b/src/hash_molecules_test.odin new file mode 100644 index 0000000..dd3be13 --- /dev/null +++ b/src/hash_molecules_test.odin @@ -0,0 +1,400 @@ +// Tests run natively, never in a WASI build - see eval_test.odin. +#+build linux, windows +package hashedbuild + +import "core:strings" +import "core:testing" + +// Cyclic hashing (hash_cyclic.odin), exercised on the recursive structure it +// was actually designed for rather than on the two-node toys in +// rec_build_test.odin: molecules. +// +// A ring of atoms is a cycle in the literal sense - benzene is six carbons +// each bonded to the next until the sixth closes back onto the first - so +// `let rec` builds one directly, and hashing one is exactly the problem the +// canonical form solves. The molecules below range from a perfectly symmetric +// ring (every atom interchangeable, one bisimulation class) through +// substituted rings that break that symmetry, to a fused bicyclic where two +// cycles interlock inside a single strongly connected component. Between them +// they cover the refinement machinery far harder than a self-referential +// two-entry Table does. +// +// **The model.** An atom is a Table: its element, how many hydrogens hang off +// it, and `.ring`, the bond to the next atom around the ring. `.substituent` +// hangs a group off a ring atom, `.fused` is a cross-bond closing a second +// ring. Bonds are therefore *directed*, and that is not a stylistic choice: a +// Table's keys have to be hashable before its entries can be ordered, so a +// cyclic value cannot be a key (hash_cyclic.odin), which rules out the +// unordered set of neighbours a chemist would draw. Two consequences are +// tested below rather than glossed over - rotating the numbering is invisible +// (which is the point), but *reflecting* it is not. + +@(private = "file") +eval_molecule :: proc(t: ^testing.T, parts: ..string) -> Value { + src := strings.concatenate(parts) + defer delete(src) + ast := parse(source_t{name = "test", n_bytes = u64(len(src)), data = raw_data(src)}, ast_t{}) + interp := Interpreter{ast = &ast, src = src} + val, ok := eval(&interp, ast.root, env_make_child(nil)) + testing.expect(t, ok, interp.error_message) + return val +} + +// The digest of whatever `expr` selects, as the language itself computes it. +@(private = "file") +digest :: proc(t: ^testing.T, defs: string, expr: string) -> string { + val := eval_molecule(t, defs, "sha256 ", expr) + s, is_str := val.(string) + testing.expect(t, is_str, "expected a Utf8 digest") + return s +} + +@(private = "file") +truth :: proc(t: ^testing.T, defs: string, expr: string) -> bool { + val := eval_molecule(t, defs, expr) + b, is_bool := val.(bool) + testing.expect(t, is_bool, "expected a Boolean") + return b +} + +// ---- the molecules ------------------------------------------------------------- + +// Benzene: six aromatic CH units in a ring. Every carbon is like every other, +// so the whole ring is one bisimulation class. +@(private = "file") +BENZENE :: `let rec benzene { + .c1 = { .element = "C", .h = 1, .bond = "aromatic", .ring = benzene.c2 }, + .c2 = { .element = "C", .h = 1, .bond = "aromatic", .ring = benzene.c3 }, + .c3 = { .element = "C", .h = 1, .bond = "aromatic", .ring = benzene.c4 }, + .c4 = { .element = "C", .h = 1, .bond = "aromatic", .ring = benzene.c5 }, + .c5 = { .element = "C", .h = 1, .bond = "aromatic", .ring = benzene.c6 }, + .c6 = { .element = "C", .h = 1, .bond = "aromatic", .ring = benzene.c1 }, +};` + +// Cyclohexane: the same six-carbon ring, saturated - two hydrogens per carbon +// and single bonds throughout. +@(private = "file") +CYCLOHEXANE :: `let rec cyclohexane { + .c1 = { .element = "C", .h = 2, .bond = "single", .ring = cyclohexane.c2 }, + .c2 = { .element = "C", .h = 2, .bond = "single", .ring = cyclohexane.c3 }, + .c3 = { .element = "C", .h = 2, .bond = "single", .ring = cyclohexane.c4 }, + .c4 = { .element = "C", .h = 2, .bond = "single", .ring = cyclohexane.c5 }, + .c5 = { .element = "C", .h = 2, .bond = "single", .ring = cyclohexane.c6 }, + .c6 = { .element = "C", .h = 2, .bond = "single", .ring = cyclohexane.c1 }, +};` + +// The Kekule drawing of benzene: alternating double and single bonds rather +// than six equivalent aromatic ones. Same atoms, different bonds. +@(private = "file") +KEKULE :: `let rec kekule { + .c1 = { .element = "C", .h = 1, .bond = "double", .ring = kekule.c2 }, + .c2 = { .element = "C", .h = 1, .bond = "single", .ring = kekule.c3 }, + .c3 = { .element = "C", .h = 1, .bond = "double", .ring = kekule.c4 }, + .c4 = { .element = "C", .h = 1, .bond = "single", .ring = kekule.c5 }, + .c5 = { .element = "C", .h = 1, .bond = "double", .ring = kekule.c6 }, + .c6 = { .element = "C", .h = 1, .bond = "single", .ring = kekule.c1 }, +};` + +// Two smaller rings built from exactly the same repeating unit as the two +// above - three aromatic CH, and four carbons alternating double/single. +@(private = "file") +SMALL_RINGS :: `let rec cyclopropenyl { + .c1 = { .element = "C", .h = 1, .bond = "aromatic", .ring = cyclopropenyl.c2 }, + .c2 = { .element = "C", .h = 1, .bond = "aromatic", .ring = cyclopropenyl.c3 }, + .c3 = { .element = "C", .h = 1, .bond = "aromatic", .ring = cyclopropenyl.c1 }, +}; +let rec cyclobutadiene { + .c1 = { .element = "C", .h = 1, .bond = "double", .ring = cyclobutadiene.c2 }, + .c2 = { .element = "C", .h = 1, .bond = "single", .ring = cyclobutadiene.c3 }, + .c3 = { .element = "C", .h = 1, .bond = "double", .ring = cyclobutadiene.c4 }, + .c4 = { .element = "C", .h = 1, .bond = "single", .ring = cyclobutadiene.c1 }, +};` + +// Toluene, and the same substitution on a five-membered ring. +@(private = "file") +TOLUENE :: `let methyl { .element = "C", .h = 3 }; +let rec toluene { + .c1 = { .element = "C", .h = 0, .substituent = methyl, .ring = toluene.c2 }, + .c2 = { .element = "C", .h = 1, .ring = toluene.c3 }, + .c3 = { .element = "C", .h = 1, .ring = toluene.c4 }, + .c4 = { .element = "C", .h = 1, .ring = toluene.c5 }, + .c5 = { .element = "C", .h = 1, .ring = toluene.c6 }, + .c6 = { .element = "C", .h = 1, .ring = toluene.c1 }, +}; +let rec methylcyclopentadienyl { + .c1 = { .element = "C", .h = 0, .substituent = methyl, .ring = methylcyclopentadienyl.c2 }, + .c2 = { .element = "C", .h = 1, .ring = methylcyclopentadienyl.c3 }, + .c3 = { .element = "C", .h = 1, .ring = methylcyclopentadienyl.c4 }, + .c4 = { .element = "C", .h = 1, .ring = methylcyclopentadienyl.c5 }, + .c5 = { .element = "C", .h = 1, .ring = methylcyclopentadienyl.c1 }, +};` + +// The three xylenes: C8H10 each, two methyls 1,2- / 1,3- / 1,4- around the +// ring. Same atoms, three different molecules. +@(private = "file") +XYLENES :: `let me { .element = "C", .h = 3 }; +let rec ortho { + .c1 = { .element = "C", .h = 0, .substituent = me, .ring = ortho.c2 }, + .c2 = { .element = "C", .h = 0, .substituent = me, .ring = ortho.c3 }, + .c3 = { .element = "C", .h = 1, .ring = ortho.c4 }, + .c4 = { .element = "C", .h = 1, .ring = ortho.c5 }, + .c5 = { .element = "C", .h = 1, .ring = ortho.c6 }, + .c6 = { .element = "C", .h = 1, .ring = ortho.c1 }, +}; +let rec meta { + .c1 = { .element = "C", .h = 0, .substituent = me, .ring = meta.c2 }, + .c2 = { .element = "C", .h = 1, .ring = meta.c3 }, + .c3 = { .element = "C", .h = 0, .substituent = me, .ring = meta.c4 }, + .c4 = { .element = "C", .h = 1, .ring = meta.c5 }, + .c5 = { .element = "C", .h = 1, .ring = meta.c6 }, + .c6 = { .element = "C", .h = 1, .ring = meta.c1 }, +}; +let rec para { + .c1 = { .element = "C", .h = 0, .substituent = me, .ring = para.c2 }, + .c2 = { .element = "C", .h = 1, .ring = para.c3 }, + .c3 = { .element = "C", .h = 1, .ring = para.c4 }, + .c4 = { .element = "C", .h = 0, .substituent = me, .ring = para.c5 }, + .c5 = { .element = "C", .h = 1, .ring = para.c6 }, + .c6 = { .element = "C", .h = 1, .ring = para.c1 }, +};` + +// Naphthalene and azulene: both C10H8, both a ten-carbon perimeter closed by +// one cross-bond. Only the chord's position differs - naphthalene splits the +// perimeter into two six-rings, azulene into a five-ring fused to a seven. +// Two interlocking cycles in one strongly connected component, which is the +// hardest shape in this file. +@(private = "file") +FUSED :: `let rec naphthalene { + .c1 = { .element = "C", .h = 1, .ring = naphthalene.c2 }, + .c2 = { .element = "C", .h = 1, .ring = naphthalene.c3 }, + .c3 = { .element = "C", .h = 1, .ring = naphthalene.c4 }, + .c4 = { .element = "C", .h = 1, .ring = naphthalene.c4a }, + .c4a = { .element = "C", .h = 0, .ring = naphthalene.c5, .fused = naphthalene.c8a }, + .c5 = { .element = "C", .h = 1, .ring = naphthalene.c6 }, + .c6 = { .element = "C", .h = 1, .ring = naphthalene.c7 }, + .c7 = { .element = "C", .h = 1, .ring = naphthalene.c8 }, + .c8 = { .element = "C", .h = 1, .ring = naphthalene.c8a }, + .c8a = { .element = "C", .h = 0, .ring = naphthalene.c1, .fused = naphthalene.c4a }, +}; +let rec azulene { + .c1 = { .element = "C", .h = 1, .ring = azulene.c2 }, + .c2 = { .element = "C", .h = 1, .ring = azulene.c3 }, + .c3 = { .element = "C", .h = 1, .ring = azulene.c3a }, + .c3a = { .element = "C", .h = 0, .ring = azulene.c4, .fused = azulene.c8a }, + .c4 = { .element = "C", .h = 1, .ring = azulene.c5 }, + .c5 = { .element = "C", .h = 1, .ring = azulene.c6 }, + .c6 = { .element = "C", .h = 1, .ring = azulene.c7 }, + .c7 = { .element = "C", .h = 1, .ring = azulene.c8 }, + .c8 = { .element = "C", .h = 1, .ring = azulene.c8a }, + .c8a = { .element = "C", .h = 0, .ring = azulene.c1, .fused = azulene.c3a }, +};` + +// ---- 1. benzene: the perfectly symmetric ring ---------------------------------- + +@(test) +test_benzene_every_carbon_is_the_same_carbon :: proc(t: ^testing.T) { + // Six atoms, one bisimulation class. This is entry-point independence at its + // starkest: which carbon you name is not part of the answer, so all six + // digests coincide - and so does the digest of the carbon you arrive at + // after walking the ring any number of times. + from_c1 := digest(t, BENZENE, "benzene.c1") + for start in ([]string{"benzene.c2", "benzene.c3", "benzene.c4", "benzene.c5", "benzene.c6"}) { + testing.expect_value(t, digest(t, BENZENE, start), from_c1) + } + testing.expect_value(t, digest(t, BENZENE, "benzene.c1.ring.ring.ring"), from_c1) + testing.expect_value(t, digest(t, BENZENE, "benzene.c1.ring.ring.ring.ring.ring.ring"), from_c1) + + // The ring is genuinely traversable, not just hashable. + testing.expect(t, truth(t, BENZENE, `benzene.c1.ring.ring.ring.ring.ring.ring.element == "C"`)) +} + +// ---- 2. the same skeleton, different chemistry --------------------------------- + +@(test) +test_the_ring_skeleton_alone_is_not_the_molecule :: proc(t: ^testing.T) { + // Three six-membered carbon rings that differ only in what hangs off each + // carbon and how the bonds are drawn. If the digest saw only the cycle's + // shape these would collide; it sees the atoms too, so they do not. + aromatic := digest(t, BENZENE, "benzene.c1") + saturated := digest(t, CYCLOHEXANE, "cyclohexane.c1") + alternating := digest(t, KEKULE, "kekule.c1") + + testing.expect(t, aromatic != saturated, "benzene is not cyclohexane") + testing.expect(t, aromatic != alternating, "aromatic bonds are not alternating ones") + testing.expect(t, saturated != alternating, "cyclohexane is not the Kekule drawing") +} + +// ---- 3. toluene: one substituent breaks the symmetry --------------------------- + +@(test) +test_a_substituent_splits_the_ring_into_positions :: proc(t: ^testing.T) { + // Benzene's six carbons were one class. Hang a methyl off one of them and + // every carbon becomes distinguishable by how far it sits from the methyl, + // so the refinement has to find six classes instead of one. + positions := []string { + "toluene.c1", "toluene.c2", "toluene.c3", "toluene.c4", "toluene.c5", "toluene.c6", + } + seen := make([dynamic]string, 0, len(positions), context.temp_allocator) + for p in positions { + d := digest(t, TOLUENE, p) + for previous in seen do testing.expect(t, previous != d, "each ring position is its own class") + append(&seen, d) + } + + // And walking the whole way round still lands on the same carbon. + testing.expect_value( + t, + digest(t, TOLUENE, "toluene.c1.ring.ring.ring.ring.ring.ring"), + digest(t, TOLUENE, "toluene.c1"), + ) +} + +@(test) +test_a_substituent_makes_the_ring_size_visible :: proc(t: ^testing.T) { + // The counterpart to test_a_symmetric_ring_cannot_count_itself below. Once + // one atom stands out, distance from it distinguishes every other atom, and + // a five-ring simply has fewer distances than a six-ring - so the two hash + // apart even though their repeating unit is identical. + testing.expect( + t, + digest(t, TOLUENE, "toluene.c1") != digest(t, TOLUENE, "methylcyclopentadienyl.c1"), + "a substituted six-ring is not a substituted five-ring", + ) +} + +// ---- 4. the xylenes: positional isomers ---------------------------------------- + +@(test) +test_the_three_xylenes_are_three_values :: proc(t: ^testing.T) { + // Same formula, same atoms, methyls in three different arrangements. This is + // the "canonical is not constant" case with real stakes: a build keyed on + // these digests would confuse three different compounds if they collided. + o := digest(t, XYLENES, "ortho.c1") + m := digest(t, XYLENES, "meta.c1") + p := digest(t, XYLENES, "para.c1") + + testing.expect(t, o != m, "ortho is not meta") + testing.expect(t, o != p, "ortho is not para") + testing.expect(t, m != p, "meta is not para") +} + +@(test) +test_para_xylene_finds_its_own_symmetry :: proc(t: ^testing.T) { + // p-xylene's two methylated carbons sit directly across the ring, so + // rotating the numbering by three maps one onto the other. That is a + // rotation, which the canonical form does see - and the two hash alike. + testing.expect_value(t, digest(t, XYLENES, "para.c4"), digest(t, XYLENES, "para.c1")) + // Its unsubstituted carbons pair up the same way. + testing.expect_value(t, digest(t, XYLENES, "para.c5"), digest(t, XYLENES, "para.c2")) +} + +// ---- 5. naphthalene and azulene: two cycles in one component ------------------- + +@(test) +test_a_fused_bicyclic_hashes_and_finds_its_rotation :: proc(t: ^testing.T) { + // Ten atoms, eleven bonds, two rings sharing an edge: one strongly connected + // component that no single walk can linearise. Naphthalene has a two-fold + // rotation, so the carbons pair up across it - the bridgeheads with each + // other, and each perimeter carbon with the one five positions away. + testing.expect_value( + t, + digest(t, FUSED, "naphthalene.c8a"), + digest(t, FUSED, "naphthalene.c4a"), + ) + pairs := [][2]string { + {"naphthalene.c1", "naphthalene.c5"}, + {"naphthalene.c2", "naphthalene.c6"}, + {"naphthalene.c3", "naphthalene.c7"}, + {"naphthalene.c4", "naphthalene.c8"}, + } + for pair in pairs do testing.expect_value(t, digest(t, FUSED, pair[0]), digest(t, FUSED, pair[1])) + + // Neighbours are not interchangeable, though - the refinement stops in the + // right place rather than collapsing everything. + testing.expect( + t, + digest(t, FUSED, "naphthalene.c1") != digest(t, FUSED, "naphthalene.c2"), + "adjacent carbons are not equivalent", + ) + testing.expect( + t, + digest(t, FUSED, "naphthalene.c1") != digest(t, FUSED, "naphthalene.c4a"), + "a bridgehead is not a perimeter carbon", + ) +} + +@(test) +test_azulene_is_not_naphthalene :: proc(t: ^testing.T) { + // Both are C10H8, both a ten-carbon perimeter plus one chord. Only where the + // chord lands differs, and that is enough. + testing.expect( + t, + digest(t, FUSED, "azulene.c1") != digest(t, FUSED, "naphthalene.c1"), + "5+7 fusion is not 6+6 fusion", + ) + // Azulene's fusion is lopsided, so unlike naphthalene's its two bridgeheads + // are *not* interchangeable: one carries a five-ring on the short side. + testing.expect( + t, + digest(t, FUSED, "azulene.c3a") != digest(t, FUSED, "azulene.c8a"), + "an asymmetric fusion has no rotation to find", + ) +} + +// ---- what this model cannot see ------------------------------------------------ + +@(test) +test_a_symmetric_ring_cannot_count_itself :: proc(t: ^testing.T) { + // Benzene and the cyclopropenyl cation are built from the same repeating + // unit - an aromatic CH bonded to the next - and differ only in how many + // times it repeats. Under bisimulation that difference does not exist: + // unrolling either gives the same infinite chain, so they are the same + // value and hash alike. Kekule benzene and cyclobutadiene collide for the + // same reason at period two. + // + // This is not the hash disagreeing with equality - the second half of each + // pair below is the point. `values_equal` calls them equal too, and a digest + // that separated them would be the bug. The loss is in the model: a ring + // with nothing to distinguish any atom carries no record of its own length, + // and the containers available cannot express an unordered bond set that + // would (see this file's header). Every substituted ring above recovers it. + testing.expect_value( + t, + digest(t, strings.concatenate({BENZENE, SMALL_RINGS}, context.temp_allocator), "cyclopropenyl.c1"), + digest(t, strings.concatenate({BENZENE, SMALL_RINGS}, context.temp_allocator), "benzene.c1"), + ) + testing.expect(t, truth( + t, + strings.concatenate({BENZENE, SMALL_RINGS}, context.temp_allocator), + "benzene.c1 == cyclopropenyl.c1", + ), "the hash agrees with equality here, which is what makes it correct") + + testing.expect_value( + t, + digest(t, strings.concatenate({KEKULE, SMALL_RINGS}, context.temp_allocator), "cyclobutadiene.c1"), + digest(t, strings.concatenate({KEKULE, SMALL_RINGS}, context.temp_allocator), "kekule.c1"), + ) +} + +@(test) +test_reflection_is_invisible_but_rotation_is_not :: proc(t: ^testing.T) { + // m-xylene's two methylated carbons are equivalent to a chemist: reflect the + // ring through them and the molecule is unchanged. The digest disagrees, and + // the reason is structural rather than incidental. Bisimulation matches + // Table entries **by key**, and a reflection maps one atom's `.ring` onto + // another's *incoming* bond - a different key, or in this model no key at + // all. No amount of listing more neighbours fixes it; naming both directions + // would just give reflection two keys to swap, which matching by key can + // never do. + // + // Rotation, by contrast, maps `.ring` onto `.ring`, which is exactly why + // para above works and why every benzene carbon agrees. So: this model + // hashes an *oriented* drawing of a molecule. That is a property of the + // model, and pinned here so a future change to the encoding has to argue + // with it rather than silently alter it. + testing.expect( + t, + digest(t, XYLENES, "meta.c1") != digest(t, XYLENES, "meta.c3"), + "reflection is not visible to a walk that only goes one way round", + ) +}