Add persistent payload verification and improve box handling

CHANGELOG.md

@@ -6,8 +6,45 @@ All notable changes to Scrollcase are documented here. The format follows
 
 ## [Unreleased]
 
+### Added
+
+- Commit new boxes to their extracted payload through the optional signed `payloadDigest` release
+  field and the canonical `payload-digest.v1` list inside the archive. Node and Python share golden
+  byte vectors for the list format and 59 language-neutral consumer cases. `schemaVersion` remains
+  2 because the field is additive: existing v2 releases still verify normally, while an explicit
+  installed-payload check refuses a release that carries no digest rather than claiming success.
+
+  Add `attachExtractedBox` / `attach_extracted_box` so an application can install once, restart, and
+  mint a fresh process-bound `PreparedBox` without retaining or re-extracting the archive. Attached
+  receipts are marked `attached`, assert the native host, re-check execution files and on-demand
+  assets, and deliberately do not claim the payload bytes were proved.
+
+  Add the separate `verifyExtractedPayload` / `verify_extracted_payload` integrity operation and
+  `scrollcase verify --extracted <dir>`. Verification walks the authenticated list rather than the
+  directory, so later extra files are ignored; embedded assets are read, while on-demand assets keep
+  their signed per-file checks. The result detects corruption at that moment, not later mutation or
+  a live local attacker, and excludes Python bytecode caches by design.
+
+- Store already-compressed payload paths in the box archive instead of deflating them. Every path a
+  scroll declares in `assets` is stored automatically, and the new optional `uncompressedPaths`
+  names anything else the project knows to be compressed already — the tree an `assetArchives` entry
+  expanded into, a bundled corpus — matching a path itself and everything beneath it.
+
+  Weights arrive compressed, and deflating them again is loss on both sides of the trade: measured
+  on incompressible bytes, level 6 runs at 47 MB/s and produces an archive 0.03% *larger* than its
+  input, and level 1 recovers 4 MB/s because the search fails either way. Lowering the level is not
+  a fix; not compressing is. Nothing opens the file or reads its extension — the decision comes from
+  the scroll and the path alone, so a rebuild of the same commit stays byte-identical.
+
 ### Changed
 
+- Print `run`'s own status lines on stderr, and say on every run that the extraction is temporary.
+  Every other verb owns its standard output; `run` hands stdout to the box, so a caller redirecting
+  it into a file was receiving a Scrollcase status line mixed into the application's bytes, with no
+  way for the box to tell. The second line states what `run` is — one-shot, deleted on exit — rather
+  than leaving a caller to read a repeated multi-gigabyte extraction as the tool being slow. A box
+  kept across runs is `verifyAndExtractBox` plus `runExtractedBox` from the library, not this verb.
+
 - Publish the demo box as one plainly named archive per operating system —
   `hello-box-1.0.0-macos-aarch64-metal.zip` and its two siblings — that unpacks to a folder which
   already runs. A box archive has to be named for its own SHA-256 and sit beside its release

CONTRIBUTING.md

@@ -24,9 +24,13 @@ knowing them first will save you a rejected pull request.
   declared by the project in config, scroll or flags; the tool stays ignorant of who uses it.
 - **One contract, multiple implementations.** `src/contract/` and its schemas are authoritative.
   `scrollcase/consumer` and Python's `scrollcase_consumer` must prove the same observable behavior
-  against shared language-neutral conformance fixtures; neither defines a parallel format.
+  for preparation, attachment, installed-payload verification, and execution against shared
+  language-neutral conformance fixtures; neither defines a parallel format.
 - **Verification precedes execution.** No consumer path may start box code before signature,
   payload-shape, archive size/hash, safe-entry, and manifest-agreement checks succeed.
+- **Attachment and payload verification stay separate.** Attachment may mint a process-bound receipt
+  from a caller-supplied local root without reading every payload byte; installed-payload
+  verification is the explicit, potentially multi-gigabyte check against the signed entry list.
 
 ## Development
 

README.md

@@ -122,7 +122,7 @@ then performs the approved installations. If conda-forge is selected but Conda i
 | `lock [<scroll>]` | Resolve the scroll's pixi manifest into `pixi.lock`; omission opens a terminal menu |
 | `audit <scroll>` | Dependency licence inventory, derived from the lock |
 | `build [<scroll>]` | Build, self-test, archive, and sign a box; omission opens a terminal menu |
-| `verify <release.json>` | Verify signature, archive hash, and layout |
+| `verify <release.json>` | Verify a signed archive, or an extracted payload with `--extracted <dir>` |
 | `run <release.json>` | Verify, temporarily extract, and run a local box |
 
 `scrollcase help` documents every option.
@@ -132,8 +132,9 @@ workspace.
 Projects that consume the box contract directly can import the full Node surface from
 `scrollcase/contract`, or its target/document-shape helpers with no Node built-ins from
 `scrollcase/contract/browser`. Generated format types remain under `scrollcase/contract/types`.
-Local preparation and shell-free execution are available from `scrollcase/consumer`; the typed
-Python package under `python/` exposes the same semantics as `scrollcase_consumer`.
+Local preparation, process-restart attachment, opt-in extracted-payload verification, and shell-free
+execution are available from `scrollcase/consumer`; the typed Python package under `python/`
+exposes the same semantics as `scrollcase_consumer`.
 
 ## Workspace
 

docs/concepts/architecture.md

@@ -22,15 +22,16 @@ API at `scrollcase/consumer` and the Python package imported as `scrollcase_cons
 flowchart LR
   C["canonical v2 contract<br/>schemas + fixtures"] --> N["Node consumer<br/>scrollcase/consumer"]
   C --> P["Python consumer<br/>scrollcase_consumer"]
-  F["caller-supplied release, archive,<br/>trust keys, destination"] --> N
+  F["caller-supplied release, archive or root,<br/>trust keys, destination"] --> N
   F --> P
   N --> L["verified local box<br/>or child process"]
   P --> L
 ```
 
-Both consumers must agree on verification, safe extraction, execution, receipts, errors, signals,
-cleanup, and on-demand assets by passing the same language-neutral conformance cases. Generated or
-checked schema copies are projections of the canonical contract, never independent definitions.
+Both consumers must agree on verification, safe extraction, attachment across restarts,
+installed-payload checking, execution, receipts, errors, signals, cleanup, and on-demand assets by
+passing the same language-neutral conformance cases. Generated or checked schema copies are
+projections of the canonical contract, never independent definitions.
 
 The security order is fixed: validate the signed document and release shape, verify the archive
 size and hash, validate every archive entry, extract safely, compare `box.json` with the signed
@@ -41,6 +42,12 @@ All inputs are local and caller-selected. Consumer code does not choose a channe
 update an installation, promote, revoke, publish, serve, allocate a runner, or own application
 lifecycle policy.
 
+A persistent installation earns a new process-bound receipt through attachment rather than loading
+one from disk. Byte verification stays separate and opt-in: the signed release commits to the
+`payload-digest.v1` list inside the box, and the verifier checks that list instead of treating later
+extra files as corruption. The result is point-in-time integrity; operating-system permissions and
+the embedding application remain responsible for guarding the directory afterwards.
+
 ## The substrate
 
 One substrate, and only one: **pixi + conda-pack + conda-forge**.
@@ -105,8 +112,9 @@ misrepresented as consumer checks.
 **Parity after the self-test, on the same payload.** There is no point comparing accelerators in
 a box that cannot import its dependencies in the first place.
 
-**Normalise, then archive.** Timestamps are stamped to a fixed instant and files enumerated in
-one stable order, which is what makes the ZIP deterministic.
+**Commit, normalise, then archive.** After `box.json`, the builder writes `payload-digest.v1` and
+places the list's hash in the signed release. Timestamps are then stamped to a fixed instant and
+files enumerated in one stable order, which is what makes the ZIP deterministic.
 
 **Sign last, stage after.** The release commits to the archive by hash; the channel commits to
 the release document by *its* hash. The staging tree is then laid out exactly as a bucket would
@@ -199,6 +207,7 @@ src/
 ├── contract/          the box format itself — the source of truth
 │   ├── targets.mjs      target model, identity rule, per-target adapters
 │   ├── documents.mjs    signed-document envelope, namespacing
+│   ├── payload-digest.mjs canonical extracted-entry list bytes
 │   ├── schema/          eight JSON Schemas
 │   └── fixtures/        golden fixtures other implementations prove themselves against
 ├── build/             solving, packing, staging, auditing, verifying
@@ -215,8 +224,8 @@ src/
 │   ├── audit.mjs        the licence audit verb
 │   ├── project.mjs      init and doctor
 │   └── parity.mjs       the accelerator parity gate
-├── consumer/          verified local preparation and shell-free execution
-│   ├── verify-and-extract.mjs  staged extraction and the opaque prepared receipt
+├── consumer/          verified local preparation, attachment, checking and execution
+│   ├── verify-and-extract.mjs  staged extraction, attachment, payload checking, opaque receipts
 │   ├── run-extracted.mjs       interpreter invocation, assets, stdio, signals
 │   └── run-box.mjs             one-shot temporary execution and cleanup
 ├── sign/              key generation, local signing, external dispatch, verification

docs/concepts/design-decisions.md

@@ -51,6 +51,50 @@ checked generated copies of the canonical schemas; it does not hand-maintain a s
 drifts at edge cases — links, traversal, collisions, signals, or argument handling — unless both
 implementations are held to the same observable cases.
 
+## Persistent installations earn a new receipt; payload verification stays separate
+
+A prepared receipt is process-bound execution authority. Serialising it would let anyone who can
+write the receipt file manufacture an object that appears to have passed the trust chain. A process
+that starts later therefore calls `attachExtractedBox` / `attach_extracted_box`: it re-verifies the
+signed release, requires a target the current host can execute, checks the interpreter and execution
+shape, verifies on-demand assets, and binds a fresh receipt to the real directory's device and inode.
+The receipt says `attached`, not `prepared`, because no archive established the payload bytes in that
+process.
+
+Byte verification is an independent, opt-in operation. New builds write `payload-digest.v1` inside
+the payload and add optional `payloadDigest: { format, sha256 }` to the signed release. The list has
+one byte-sorted record per original file or link and is excluded from itself; the release signs its
+hash. `verifyExtractedPayload` / `verify_extracted_payload` authenticates the bounded list before
+parsing it, then visits only the paths it names. The field is additive, so `schemaVersion` stays 2
+and older v2 releases remain valid, while the specific payload-verification operation refuses one
+that carries no commitment.
+
+**Rejected:** storing the whole per-file table in the release. A conda environment routinely holds
+10,000–30,000 files, which would add megabytes to every signed document. One signed digest plus the
+list inside the payload keeps the document small without weakening which bytes it commits to.
+
+**Rejected:** a single root hash recomputed by walking the installed directory. Honest installations
+grow: Python creates caches, applications write in their working directory, and on-demand assets are
+materialised after extraction. If the directory is the input, every legitimate extra file changes
+the answer. Walking the signed list makes extras invisible by construction.
+
+**Rejected:** folding byte verification into attachment or execution, or adding a verification flag
+to attachment. Embedded weights can make the scan read tens of gigabytes, and a result at attach
+time does not guarantee the tree at a later spawn or lazy Python import. Separate operations keep
+both cost and meaning explicit: attachment answers whether a directory can mint a receipt now;
+payload verification answers whether its listed bytes match now.
+
+**Rejected:** committing file mode or modification time. Archive writing synthesises modes from the
+target and path, Windows extraction does not apply `chmod`, and no extractor restores the fixed
+build timestamp. Including either would make an honest extraction disagree with its build.
+
+The limit is stated rather than hidden. Payload verification has a check-to-use window and is not a
+defence against a live local attacker; operating-system permissions and application ownership guard
+the directory. `__pycache__` directories and `*.pyc` files are excluded by the collector and are
+therefore a permanent blind spot, not merely part of that timing window. Embedded assets are listed
+and expensive to re-read; on-demand assets are ignored extras whose separate signed descriptors are
+checked during attachment and execution.
+
 ## One substrate: pixi + conda-pack + conda-forge
 
 Scrollcase supports exactly one dependency backend.

docs/getting-started/overview.md

@@ -294,8 +294,8 @@ Scrollcase prepares the files, but does not upload them anywhere.
 
 The Node API at `scrollcase/consumer` and the Python package imported as
 `scrollcase_consumer` accept local release documents, archives, trust keys, and destinations.
-They share verification, safe extraction, execution, receipt, signal, cleanup, and on-demand
-asset semantics.
+They share verification, safe extraction, re-attachment to a persistent root, opt-in installed
+payload checking, execution, receipt, signal, cleanup, and on-demand asset semantics.
 
 For a one-shot terminal run, the CLI is a thin wrapper over the Node consumer:
 
@@ -307,6 +307,11 @@ It verifies first, runs the signed script or module without a shell, preserves t
 and removes its temporary extraction. See [Library APIs](/reference/api) and
 [CLI Commands](/reference/cli#run).
 
+An application that keeps an extracted box across restarts uses the attachment APIs rather than
+serialising a receipt. It may verify the installed bytes independently through the signed payload
+list, or with `scrollcase verify --extracted <dir>` for a manual check. See
+[Distributing Boxes](/guides/distributing-boxes#keeping-an-extracted-box-across-restarts).
+
 ## What Happens During `build`
 
 From the developer's perspective, the build is a single command:

docs/guides/distributing-boxes.md

@@ -168,6 +168,62 @@ Whatever installs your boxes should do exactly what `scrollcase verify` does, in
 Running `scrollcase verify --self-test` on the build machine covers the archive and temporary
 extraction checks, not final installation, compatibility policy, rollout, or activation.
 
+## Keeping an extracted box across restarts
+
+The archive proves the payload while it exists. A persistent installation usually discards that
+archive, and a `PreparedBox` receipt cannot be serialised across process restarts: doing so would
+turn a writable file into a forgeable execution capability. A new process earns a fresh receipt by
+re-checking the signed release and the directory's execution prerequisites:
+
+```js
+import {
+  attachExtractedBox,
+  runExtractedBox,
+  verifyExtractedPayload,
+} from 'scrollcase/consumer';
+
+// Optional and potentially expensive: proves the installed bytes at this moment.
+await verifyExtractedPayload('release.json', {
+  publicPath: 'trusted-key.json',
+  root: '/srv/boxes/my-model/1.0.0/macos-aarch64-metal',
+});
+
+// Does not re-read original payload bytes; on-demand assets are hashed separately.
+const attached = await attachExtractedBox('release.json', {
+  publicPath: 'trusted-key.json',
+  root: '/srv/boxes/my-model/1.0.0/macos-aarch64-metal',
+});
+await runExtractedBox(attached);
+```
+
+Python exposes the same sequence as `verify_extracted_payload`, `attach_extracted_box`, and
+`run_extracted_box`. Attachment deliberately does not verify every payload byte: it enumerates the
+tree and measures metadata, but the full content read is a separate decision so launch does not
+silently become a multi-gigabyte integrity scan.
+
+For a manual or maintenance check, the CLI reaches the same Node consumer operation:
+
+```sh
+scrollcase verify release.json \
+  --extracted /srv/boxes/my-model/1.0.0/macos-aarch64-metal \
+  --public-key trusted-key.json
+```
+
+`--extracted` needs no archive and cannot be combined with `--archive` or `--self-test`. It checks
+the signed `payload-digest.v1` entry list rather than walking the directory, so unrelated files that
+appeared after installation do not fail an honest box. Embedded assets are original entries and are
+read in full; on-demand assets are later extras, ignored by this digest and checked separately by
+their signed descriptors during attachment and execution.
+
+::: warning Integrity is a point-in-time result
+Payload verification detects ordinary corruption and identifies a directory against a signed
+release. It does not stop the directory changing after the check. Protect persistent installations
+with operating-system permissions and the embedding application's ownership policy.
+
+`__pycache__` directories and `*.pyc` files are excluded when the build collects payload entries,
+so the digest can never see them. Do not treat the check as proof about compiled Python caches.
+:::
+
 ## Namespaces for existing publishers
 
 If you already have boxes installed in the field under your own document kinds, keep emitting

docs/guides/managing-weights.md

@@ -77,6 +77,28 @@ When the upstream artefact is a tarball or zip, declare it as an asset and then
 - `removeAfterExtract` defaults to `true`: the compressed original is dead weight inside the
   payload once unpacked.
 
+## Compression
+
+Weights arrive already compressed, and deflating them again is pure loss. Measured on
+incompressible bytes: level 6 runs at 47 MB/s and the archive comes out **0.03% larger** than the
+input, and dropping to level 1 recovers 4 MB/s because the search fails either way. Lowering the
+level is not the fix — not compressing is.
+
+So every path you declare in `assets` is **stored** in the archive rather than deflated. You do not
+have to ask for this and there is nothing to configure.
+
+For anything else your box carries that is already compressed — the tree an `assetArchives` entry
+expanded into, a bundled corpus of JPEGs — say so:
+
+```jsonc
+"uncompressedPaths": ["model-cache/hello", "corpora/images"]
+```
+
+An entry matches that path and everything beneath it. Nothing is decided by looking at the file or
+its extension: the choice comes from the scroll alone, which is what keeps two builds of the same
+commit byte-identical. The interpreter, `site-packages` and the notices compress genuinely and
+still do.
+
 ## Files from your own repository
 
 Runtime shims, licence notices, a parity check script — anything you maintain yourself — go in