Drop the external blake3 dependency; use the in-tree module

.claude/blackboard.md

@@ -3,7 +3,52 @@
 > **Read this first.** The "Polyglot Notebook" architecture below is a
 > separate/older program, not the current epoch.
 
-## 2026-07-06 (latest) — F64 GEMM completed: FMA tier + register residency + native-engine swap
+## 2026-07-29 (latest) — blake3 dependency DROPPED: call sites swapped to the in-tree module
+
+Operator: "go ahead" on the rung-3a swap, with the measured cost table in
+hand. This closes the only rung of `the-simd-ladder.md` that carried a cargo
+cycle (`blake3 → ndarray::simd`).
+
+1. **15 call sites, 8 modules** (`seal`, `merkle_tree`, `plane`, `vsa`,
+   `spo_bundle`, `crystal_encoder`, `compression_curves`, `deepnsm`) now
+   reach `crate::hpc::blake3`. Implemented as one `use super::blake3;` per
+   module rather than rewriting each `blake3::` path — the in-tree API is
+   shape-compatible (`hash`, `Hasher::{new,new_keyed,new_derive_key,update,
+   finalize,finalize_xof}`, `Hash::as_bytes → &[u8;32]`), so the diff is 10
+   added lines, not 15 edited ones. `spo_bundle`'s use is inside
+   `#[cfg(test)]`, so it takes `use crate::hpc::blake3;` there instead of
+   leaning on the subtree's blanket `unused_imports` allow.
+2. **Dependency gone.** `dep:blake3` off the `std` feature; the ~55-line
+   Cargo.toml comment block replaced with a do-not-re-add note. `blake3`
+   **and** its transitive `constant_time_eq` are both absent from
+   `Cargo.lock`. nostd matrix untouched — every call site is under
+   `pub mod hpc`, itself `#[cfg(feature = "std")]`.
+3. **What this actually bought:** the second SIMD surface. Even at
+   `default-features = false, features = ["pure"]` the crate shipped its own
+   `rust_{sse2,sse41,avx2}.rs` intrinsics beside `ndarray::simd` — the exact
+   thing the matryoshka pattern exists to prevent. The old Cargo.toml comment
+   named this and said "Tracked, not done here."
+
+**Cost, accepted with numbers in hand [MEASURED, not re-measurable]:**
+1.25–1.39× across the input sizes this crate hashes (16 B 1.34–1.39×, 256 B
+1.25–1.29×, 2 KB 1.30×); 4.7–4.9× at 64 KB, which no call site here touches.
+The A/B bench needs both implementations present and is unbuildable from this
+commit forward, by its own design note.
+
+**Loose ends.**
+- `AdaWorldAPI/BLAKE3` (v1.8.5) exists and is an **unmodified upstream
+  mirror** — no ndarray wiring, all `ffi_*`/`rust_*` files intact. Master had
+  been pinning `blake3 = { version = "1" }` from **crates.io** while that fork
+  existed, i.e. the P0 fork rule was already broken. This change resolves it
+  by deletion; wiring the fork instead would have been the other resolution
+  and would NOT have closed the second-surface problem.
+- Verified `src/hpc/blake3_test_vectors.json` is **byte-identical** (31,922 B)
+  to that fork's `test_vectors.json`, so the correctness tests run against
+  genuine upstream vectors.
+- Rung 3b (`hash_many`) still absent; that is the whole 64 KB gap. Not worth
+  building for ndarray's own call sites — nothing here hashes bulk.
+
+## 2026-07-06 — F64 GEMM completed: FMA tier + register residency + native-engine swap
 
 Operator: "complete the F64 gemm… and pr". Three moves, all on the entry
 below's foundation:

.claude/knowledge/blake3-in-tree-measured.md

@@ -1,8 +1,22 @@
 # In-tree BLAKE3 — correct, and what it costs
 
-> **Status: MEASURED, 2026-07-29.** Correctness against the official vectors;
-> throughput against the external crate. Both numbers below are reproducible
-> with the instruments in this directory.
+> **Status: MEASURED, 2026-07-29 — and SWAPPED, 2026-07-29.** Correctness
+> against the official vectors; throughput against the external crate.
+>
+> **The open decision this document existed to inform has been taken:** the
+> operator authorised the swap, all 15 call sites now reach
+> `crate::hpc::blake3`, and the external dependency is gone. The cost below
+> was accepted with the numbers in hand, not discovered afterward.
+>
+> Two consequences for anyone reading this later:
+>
+> - The **correctness** numbers remain reproducible. The vectors file
+>   (`src/hpc/blake3_test_vectors.json`) is byte-identical to upstream's
+>   `test_vectors.json` — verified against `AdaWorldAPI/BLAKE3` v1.8.5.
+> - The **throughput** numbers are now historical and cannot be re-run. The
+>   A/B bench needs both implementations present; with the crate dropped there
+>   is nothing to compare against, exactly as its own header anticipated. To
+>   re-measure, you must temporarily re-add the dependency.
 
 ## READ BY:
 - Anyone about to drop the external `blake3` dependency
@@ -21,10 +35,16 @@ cycle — the only rung of the ladder that has one (`the-simd-ladder.md`).
 Cutting it means ndarray owning BLAKE3 rather than consuming the crate.
 
 Scoping finding that made this small: **ndarray's usage is entirely
-single-input.** 14 call sites across 8 files use only `hash`,
-`Hasher::{new, new_keyed, update, finalize, finalize_xof().fill()}`,
-`Hash::as_bytes`, and `Hash` as a signature type. **No `hash_many`.** So the
-serial core suffices, and it needs no SIMD at all.
+single-input.** 15 `blake3::` references across 8 files — 14 calls plus one
+type position (`merkle_tree::truncate_hash`'s `&blake3::Hash` parameter) —
+using only `hash`, `Hasher::{new, new_keyed, update, finalize,
+finalize_xof().fill()}`, `Hash::as_bytes`, and `Hash` as a signature type.
+**No `hash_many`.** So the serial core suffices, and it needs no SIMD at all.
+
+(An earlier revision said "14 call sites", counting calls but not the type
+position, while the swap record above counts all 15 references. Raised by
+CodeRabbit on #269; both numbers were describing different things, and the
+distinction is now explicit rather than a discrepancy.)
 
 ## Correctness — proven
 
@@ -135,9 +155,16 @@ hash small nodes, `deepnsm`/`compression_curves` small. So ndarray's real
 exposure is the **1.3–1.6× band**, not the 5× one — but 1.3× on a hot encoder
 path is a real cost, not a rounding error.
 
-**Not swapped here.** The module lands and is tested; the call sites still
-use the external crate. Flipping them is a decision with a measured price
-tag, and it is the operator's.
+**Not swapped in the commit this section was written for.** The module landed
+and was tested; the call sites still used the external crate. Flipping them
+was a decision with a measured price tag, and it was the operator's.
+
+**Taken 2026-07-29.** The operator authorised the flip with this table in
+hand. All 15 call sites across the eight modules named above now reach
+`crate::hpc::blake3` via a per-module `use super::blake3;`; `blake3` and its
+transitive `constant_time_eq` are gone from `Cargo.lock`. The accepted price
+is the 1.3× small-input band — the 4.8× 64 KB row was never on ndarray's
+curve, since no call site here hashes bulk input.
 
 ## One deliberate deviation from upstream, and one restored
 

Cargo.toml

@@ -154,63 +154,48 @@ libc = { version = "0.2.82", optional = true }
 matrixmultiply = { version = "0.3.2", default-features = false, features=["cgemm"] }
 
 # =====================================================================
-# blake3 — DEPENDENCY OF `std` FEATURE (NOT `hpc-extras`). READ BEFORE
-# TOUCHING.
+# blake3 — NO LONGER A DEPENDENCY. Do not re-add it.
 # =====================================================================
 #
-# blake3 is required by the cognitive substrate modules — plane, seal,
-# merkle_tree, vsa, spo_bundle, crystal_encoder, compression_curves,
-# deepnsm — which all live under `pub mod hpc;` (itself gated on
-# `feature = "std"` in lib.rs). Those modules `use blake3;` directly
-# and unconditionally; there is no #[cfg] dance inside them.
+# BLAKE3 now lives in-tree at `src/hpc/blake3.rs`, as a **portable-only
+# transcription of the BLAKE3 reference implementation -- no SIMD, no
+# unsafe**. See that module's own header. It is the serial path every
+# conformant implementation reduces to at `simd_degree() == 1`, which
+# suffices here because ndarray's usage is entirely single-input.
 #
-# Pinning blake3 to the `std` feature (rather than `hpc-extras`) means:
-#   * ANY consumer that enables `std` automatically gets blake3, with
-#     ZERO additional feature wiring. This is the default. Cargo's
-#     transitive feature resolution does the rest.
-#   * Consumers selecting `default-features = false, features = ["std"]`
-#     (e.g. burn-ndarray, which disables hpc-extras to shed p64/fractal)
-#     STILL get blake3 — no more "missing blake3" build errors that
-#     used to require chasing down hpc-extras.
-#   * Consumers selecting `default-features = false` (no std at all,
-#     e.g. the `thumbv6m-none-eabi` CI matrix entry) do NOT pull blake3
-#     and therefore do not pull `constant_time_eq`, whose lack of
-#     `#![no_std]` declaration would otherwise fail the nostd link with
-#     `error[E0463]: can't find crate for std`.
+# The external crate was removed because even with `default-features =
+# false, features = ["pure"]` it carried its own SSE2/SSE4.1/AVX2
+# intrinsics -- a second SIMD surface beside `ndarray::simd`, which is
+# exactly what the matryoshka pattern exists to prevent. The old comment
+# block here said "Tracked, not done here"; this is it being done.
 #
-# If you find yourself wanting to make blake3 unconditional (drop the
-# `optional = true`), check first: does it still pass
-# `cargo rustc -p ndarray --target=thumbv6m-none-eabi
-#  --no-default-features --features portable-atomic-critical-section`?
-# If not, leave the `optional = true` + `std`-feature pinning in place.
+# It also breaks a cycle. `ndarray -> blake3` meant blake3 could never be
+# built ON `ndarray::simd` without cargo seeing `ndarray -> blake3 ->
+# ndarray`. Dropping the dep is what makes an ndarray-backed BLAKE3
+# possible -- NOT here (this module is portable), but for a future
+# SIMD-backed implementation and for the AdaWorldAPI/BLAKE3 fork.
 #
-# =====================================================================
-# `default-features = false` + `pure`: NO C/ASM FFI. Operator directive
-# 2026-07-28 — "I don't want any ffi with c".
-#
-# blake3's default build runs `cc::Build` over `c/blake3_{sse2,sse41,avx2,
-# avx512}_x86-64_unix.S` and links them (measured before this change: 33 `.o`
-# files plus `libblake3_avx512_assembly.a` in target/, with `blake3_*_ffi`
-# cfgs set). The `pure` feature routes build.rs to
-# `build_sse2_sse41_avx2_rust_intrinsics()` — its own comment: "No C code to
-# compile here" — which only sets `blake3_{sse2,sse41,avx2}_rust` cfgs and
-# lets the normal cargo build compile the Rust intrinsics modules.
+# Separately: blake3 carried `cc` as an unconditional build-dependency, and
+# the `pure` feature existed only to stop that `cc` from compiling the
+# bundled C/ASM (operator directive 2026-07-28, "I don't want any ffi with
+# c"). In-tree there is no C to opt out of, and `cc` is no longer pulled on
+# blake3's account. Dropping blake3 also removed `constant_time_eq`,
+# `arrayref` and `arrayvec` from the lockfile.
 #
-# What `pure` costs (build.rs:344-371): the hand-tuned x86-64 assembly is
-# replaced by Rust intrinsics, the AVX-512 path is dropped entirely, and the
-# aarch64 NEON C intrinsics are dropped. What it removes: every byte of C.
+# Cost, measured: 1.25-1.39x across the input sizes this crate actually
+# hashes (16 B word 1.34-1.39x, 256 B 1.25-1.29x, 2 KB 1.30x), rising to
+# 4.7-4.9x at 64 KB where upstream's `hash_many` has no in-tree counterpart.
+# Every call site here sits in the small-input band, not the 64 KB one. Full
+# table, methodology and the attribution of the gap:
+# `.claude/knowledge/blake3-in-tree-measured.md`.
 #
-# `default-features = false` also drops blake3's own `std` (which only gates
-# `constant_time_eq/std`); this crate's `std` feature is what pulls blake3 in
-# at all, so nothing here needs blake3's.
+# The A/B bench under `.claude/knowledge/blake3-ab-bench/` compared the two
+# implementations and is, by its own design note, unbuildable from this commit
+# onward — there is no longer a second implementation to compare against.
 #
-# NOTE: `pure` still leaves blake3's Rust SSE2/SSE4.1/AVX2 intrinsics — a
-# second SIMD surface beside `ndarray::simd`, which the matryoshka pattern
-# exists to prevent. Closing that means implementing BLAKE3's compression on
-# `ndarray::simd::U32x16` (it is a ChaCha-derived u32 ARX kernel, and that
-# lane is proven at the AVX2 instruction floor — see
-# `.claude/knowledge/td-t22-asm-investigation.md`). Tracked, not done here.
-blake3 = { version = "1", optional = true, default-features = false, features = ["pure"] }
+# Call sites live under `pub mod hpc;` (gated on `feature = "std"` in lib.rs),
+# so the nostd matrix is untouched: `blake3` and its `constant_time_eq`
+# dependency are both gone from Cargo.lock entirely.
 
 # p64 + fractal — specialized convergence / manifold math. Gated behind
 # `hpc-extras` since they pull in a dep tree burn-ndarray doesn't need.
@@ -306,7 +291,7 @@ blas = ["dep:cblas-sys", "dep:libc"]
 
 serde = ["dep:serde"]
 
-std = ["num-traits/std", "matrixmultiply/std", "dep:blake3"]
+std = ["num-traits/std", "matrixmultiply/std"]
 rayon = ["dep:rayon", "std"]
 
 # Portable-SIMD backend (NIGHTLY ONLY). Routes `crate::simd::*` types
@@ -345,13 +330,9 @@ nightly-simd = ["std"]
 runtime-dispatch = ["std"]
 
 # HPC extras: p64 palette/NARS bridge + fractal manifold.
-# (blake3 was previously listed here; it is now part of `std` directly
-# because the cognitive substrate modules under hpc/ that import blake3
-# are themselves `std`-gated. See the blake3 comment in [dependencies].)
 # These pull in a non-trivial dependency tree; downstream crates such as
 # burn-ndarray that only need the core array layer can disable this with
-# `default-features = false` (and re-enable `std` explicitly if needed —
-# blake3 will come along with `std`).
+# `default-features = false` (and re-enable `std` explicitly if needed).
 hpc-extras = ["std", "dep:p64", "dep:fractal", "fractal/std"]
 
 matrixmultiply-threading = ["matrixmultiply/threading"]

README.md

@@ -210,20 +210,26 @@ cargo test
 
 ### Transitive dependencies of the `std` feature
 
-Enabling the `std` feature (the default) pulls in **`blake3`** as a hard
-transitive dependency. The cognitive substrate modules under `hpc/` —
-`plane`, `seal`, `merkle_tree`, `vsa`, `spo_bundle`, `crystal_encoder`,
-`compression_curves`, `deepnsm` — import `blake3` directly for integrity
-hashing and XOF expansion, and there is no separate feature to enable it.
-This was previously gated behind `hpc-extras`, which caused recurring
-"missing blake3" build errors for consumers (e.g. `burn-ndarray`) that
-selected `default-features = false, features = ["std"]` to shed the
-`p64` / `fractal` dependency tree. Pinning blake3 to `std` removes that
-footgun: any `std`-enabled build automatically gets blake3.
+**None for hashing.** BLAKE3 is in-tree.
+
+The cognitive substrate modules under `hpc/` — `plane`, `seal`,
+`merkle_tree`, `vsa`, `spo_bundle`, `crystal_encoder`, `compression_curves`,
+`deepnsm` — use `hpc::blake3` for integrity hashing and XOF expansion. That
+is a portable pure-Rust transcription of the BLAKE3 reference
+implementation, shipped in this crate: no SIMD, no `unsafe`, no C, and no
+build script.
+
+Earlier revisions pulled the external **`blake3`** crate here, first gated
+behind `hpc-extras` (which caused recurring "missing blake3" build errors
+for consumers such as `burn-ndarray` selecting
+`default-features = false, features = ["std"]`), then pinned to `std`.
+**Both are gone.** `blake3` and its transitive `constant_time_eq`,
+`arrayref` and `arrayvec` no longer appear in the dependency graph at any
+feature combination, so the footgun cannot recur.
 
 Consumers building `default-features = false` (no `std`, e.g. the
-`thumbv6m-none-eabi` nostd target) skip both the `hpc` module and the
-blake3 dep, so the nostd link is unaffected.
+`thumbv6m-none-eabi` nostd target) skip the `hpc` module and the BLAKE3
+code with it, so the nostd link is unaffected.
 
 ## Ecosystem