Bradley/stackbench test

[bradleyshep]

Description of Changes

This PR adds Stack Bench, a reproducible benchmark for coding agents building applications against SpacetimeDB, PostgreSQL, and MongoDB.

Benchmark definitions and grading

• Adds versioned tracks, levels, feature packs, specification packs, recipes, prompts, checks, and dependency declarations.
• Supports selecting complete recipes, individual packs, or individual checks.
• Separates what the agent is asked to build from what the finished application is evaluated against.
• Adds typed scenario actions for browser interaction, authenticated server calls, concurrent operations, direct database writes, reconnects, reloads, and service lifecycle operations.
• Records each check as passed, failed, inconclusive, or harness_failure.
• Keeps the declared score denominator fixed when evidence is missing or inconclusive.
• Adds reference, mutation, and null-control qualification:
  • reference applications for all three stacks;
  • source-bound mutation manifests;
  • exact mutation-target and collateral-failure validation;
  • blank-application controls;
  • versioned calibrations and promotion catalogs.
• Adds immutable identities for recipes, prompts, fixtures, mutations, checks, runtime images, adapters, and evidence.

Ecommerce benchmark

• Adds ecommerce L1 coverage for accounts, sessions, storefronts, carts, checkout, purchases, warehouses, reviews, authorization, accounting, concurrency, live updates, persistence, reconnects, and direct database writes.
• Adds ecommerce L2 coverage for fulfilment, shipping, warehouse transfers, cancellations, returns, historical pricing, operational views, recommendations, authorization, stock conservation, and refund accounting.
• Adds complete ecommerce reference applications for MongoDB, PostgreSQL, and SpacetimeDB.
• Promotes qualified ecommerce L1 modular 2.4.
• Keeps ecommerce L2 1.4 as the promoted L2 release.
• Adds L2 1.5 as a draft cumulative successor based on L1 2.4:
  • 76 checks;
  • 117 available points;
  • static mutation coverage for all 74 scored checks on every supported stack;
  • candidate reference fixtures, mutation manifests, and calibration;
  • live qualification still required before promotion.

Runs, repairs, and campaigns

• Adds bounded correction rounds with rollback when a repair loses previously established evidence.
• Stops cumulative execution when a lower level remains incomplete.
• Adds source checkpoints and operator-granted continuation rounds without rewriting the original result.
• Adds campaign manifests containing exact stacks, repetitions, parallelism, recipes, models, pricing, prompt treatments, and correction budgets.
• Supports concurrent attempts of the same or different stacks using isolated ports, databases, modules, containers, work directories, leases, and result directories.
• Adds durable campaign scheduling, retries, locking, interruption recovery, reconciliation, and cleanup.
• Preserves invalid and interrupted attempts instead of presenting them as comparison data.
• Adds reproducible JSON and HTML reports with scores, coverage, cost, tokens, elapsed time, correction rounds, raw evidence, screenshots, videos, traces, and prompts.

CLI, dashboard, and appliance

• Adds Stack Bench CLI commands for runs, campaigns, preflight, qualification, composition, repair continuations, recovery, release verification, and reporting.
• Adds a Dockerized local dashboard that reads and controls the same durable campaign state as the CLI.
• Adds a Linux/amd64 Docker appliance with pinned controller and build images, preflight checks, resource leases, reference qualification, release identity verification, and recovery tooling.
• Adds model-free reference adapters and campaign trials for testing orchestration without provider spend.
• Adds retained grader stdout/stderr and structured browser-initialization evidence.
• Hardens process cleanup, destructive filesystem operations, campaign locks, recovery authority, credential redaction, and failed teardown handling.
• Prevents local notes, generated reports, media, runtime state, and mutation scratch files from entering controller image builds.

SpacetimeDB TypeScript SDK

• Changes exported TableRef from an anonymous readonly type alias to a readonly interface with the same members.
• Produces shorter, named TypeScript diagnostics when a TableRef is used incorrectly.
• Changes automatic ConnectionManager reconnects to reuse the identity token previously issued by the server.
• Keeps explicit connection rebuilds bound to the token supplied by the caller.
• Adds unit tests for reconnect token reuse and TableRef diagnostics.
• Adds an opt-in live-host reconnect identity test.

SpacetimeDB CLI

• Changes spacetime dev to retain configuration changes made during direct-login publishing.
• Reuses that authenticated configuration for log streaming and subsequent rebuilds.

SpacetimeDB skills

• Expands the TypeScript, Rust, and C# server skills with examples and guidance for client visibility filters, procedures, HTTP endpoints, indexes, and uniqueness constraints.
• Updates the TypeScript client skill’s authentication and reconnect guidance.

API and ABI breaking changes

This PR changes the public Rust signature of:

spacetimedb_cli::publish::exec_from_entry

The function now accepts &mut Config instead of taking Config by value. External Rust callers must pass a mutable reference. The API-breaking label should be applied unless the previous entry point is preserved through a compatibility wrapper.

The exported TypeScript TableRef declaration changes from a type alias to an interface while retaining the same readonly member contract.

Automatic TypeScript client reconnection also changes behavior by presenting the identity token from the previous successful connection.

Expected complexity level and risk

3 / 5

Most of the implementation is isolated under tools/stack-bench. The principal risks are benchmark correctness, evidence integrity, concurrent resource isolation, interrupted-run recovery, Docker cleanup, and credential handling.

The TypeScript reconnect change and spacetimedb-cli helper signature are the only changes outside the benchmark and skill directories.

Testing

• Stack Bench host suite: 602 tests, 599 passed, 0 failed, 3 Windows-specific skips.
• Composition validation: 51 packs, 13 recipes, and 677 selected checks.
• All calibration manifests compile.
• All registered reference fixtures pass source and registry validation.
• Scenario validation completes with 0 errors.
• All 27 prompt snapshots match.
• All 4 definition goldens match.
• TypeScript reconnect and TableRef regression tests: 4/4 passed.
• cargo check -p spacetimedb-cli.
• Ecommerce L1 2.4 null control: 0/58.
• Ecommerce L1 2.4 references: 58/58 on MongoDB, PostgreSQL, and SpacetimeDB.
• Ecommerce L1 2.4 mutation qualification:
  • MongoDB: 44/44 defects caught;
  • PostgreSQL: 45/45 defects caught;
  • SpacetimeDB: 46/46 defects caught.
• Controller build-context regression coverage.
• Review whether to preserve the previous exec_from_entry Rust signature.
• Run the appliance preflight and smoke test on Linux/amd64.
• Run live Docker qualification for ecommerce L2 1.5 before promotion.
• Run the opt-in TypeScript reconnect test against a disposable live host.

[github-advanced-security]

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.