Skip to content

fix(replay): unbreak the capture/replay harness (lk resolution + env-name refresh) - #193

Open
cathysnell wants to merge 3 commits into
databricks-solutions:mainfrom
cathysnell:fix/replay-capture-kit-lk
Open

fix(replay): unbreak the capture/replay harness (lk resolution + env-name refresh)#193
cathysnell wants to merge 3 commits into
databricks-solutions:mainfrom
cathysnell:fix/replay-capture-kit-lk

Conversation

@cathysnell

Copy link
Copy Markdown
Contributor

Problem

The replay/capture launchers cannot run on current main. A fresh
capture-scenario.sh --create (and the replay/smoke launchers, which share the
same resolver) hard-fails at the first pre-project lk call:

bash: .../templates/project/common/scripts/lk: No such file or directory
capture-scenario: kit --warm failed

KIT_LK still points at templates/project/common/scripts/lk, a shim that was
removed from this repo — the scaffold lk now lives in the
@databricks-solutions/lakebase-scm-utils substrate package (see the header of
examples/replay/lk) and was made generic, so it needs LAKEBASE_KIT_PACKAGE
to know which kit to load. This is very likely why the shipped stockflow corpus
is still a v0.3.0-beta.14 recording: the capture path is broken, so it cannot be
re-recorded.

Fix

  1. lk resolution. Resolve KIT_LK via a shared kit_lk_path() helper in
    lib/pin-local-kit.sh (installed substrate path, with the legacy in-repo path
    as a fallback), and export LAKEBASE_KIT_PACKAGE from
    resolve_kit_single_source so the generic shim resolves the kit for the
    pre-project --warm / lakebase-create-project calls. A scaffolded project
    reads .lakebase/kit-package, but the launchers invoke lk before any project
    exists. Applied to capture-scenario.sh, _replay-smoke.sh, run-smoke.sh.

  2. Env-name refresh. Rename the deprecated LAKEBASE_SFTDD_{AUTO_CONTINUE,SPRINT_REQUESTS}
    to the current LAKEBASE_CONSORT_* prefix across the active launchers + docs.
    consortEnv() reads CONSORT_* first (SFTDD/TDD remain fallbacks), so this
    is behavior-preserving on the current engine; it silences the per-capture
    deprecation warnings and won't break at the scheduled v0.4.0 removal.

Tests

  • New hermetic anti-drift guard in replay-layout-guard.test.ts: the launchers
    must resolve KIT_LK via kit_lk_path (not the removed in-repo path), and
    pin-local-kit.sh must define the helper + export LAKEBASE_KIT_PACKAGE.
  • Updated the two guards that pinned the old state (npx-tax-guard.test.ts was
    asserting the removed lk path; consort-workflow-smoke.test.ts asserted the
    old env name).
  • npm run typecheck clean; the affected hermetic suites pass. Verified the fix
    end-to-end: with the patched resolver, lk resolves the pinned kit and
    lakebase-create-project runs, and a live capture-scenario.sh --create
    provisions + drives past setup.

Notes for maintainers

  • Scope is the replay harness only; no engine/kit runtime code changes.
  • Heads-up (separate from this PR): running the full npm test inside a
    linked git worktree corrupts the worktree's git — several build-lane tests
    (cycle-record, greenOpenCycle, cutExperiment, …) make real git commits
    that land on the current branch instead of an isolated fixture (branch HEAD gets
    rewritten with green: T1/T2 / refactor commits). Worth isolating those
    tests' git ops; happy to file/fix separately.

This pull request and its description were written by Isaac.

cathysnell and others added 2 commits September 4, 2026 11:59
…name refresh)

The replay/capture launchers could not run on current main.

1. KIT_LK pointed at templates/project/common/scripts/lk, a shim removed from this
   repo (moved into the @databricks-solutions/lakebase-scm-utils substrate package
   and made generic, needing LAKEBASE_KIT_PACKAGE). Every capture, replay, and
   smoke hard-failed at the first pre-project `lk` call:
     bash: .../templates/project/common/scripts/lk: No such file or directory
     capture-scenario: kit --warm failed
   Resolve KIT_LK via a shared kit_lk_path() helper in pin-local-kit.sh (installed
   substrate path, legacy in-repo path as fallback), and export LAKEBASE_KIT_PACKAGE
   from resolve_kit_single_source so the generic shim knows which kit to load for
   the pre-project --warm / lakebase-create-project calls (a scaffolded project
   reads .lakebase/kit-package; the launchers run before any project exists).
   npx-tax-guard was asserting the removed path; updated to require the helper.

2. Refresh the deprecated LAKEBASE_SFTDD_{AUTO_CONTINUE,SPRINT_REQUESTS} env names
   to the current LAKEBASE_CONSORT_* prefix across the active launchers + docs.
   consortEnv() reads CONSORT first (SFTDD/TDD remain fallbacks), so this is
   behavior-preserving; it silences the per-capture deprecation warnings and won't
   break at the v0.4.0 removal.

Adds a hermetic anti-drift guard (replay-layout-guard.test.ts) and updates the two
guards that pinned the old state. Full hermetic suite + typecheck green.

Co-authored-by: Isaac <no-reply@databricks.com>
…d on missing lk/dist

Three more capture/replay breakers surfaced by the re-record audit:

1. BREAKING , resume wedges on the feature checkout. On a RESUME (alreadyClaimed)
   HEAD is on the parent tier and the per-run .consort/.lakebase metadata
   (workflow-state.json, pipeline.json, smells.json, ...) is dirty + tracked, so
   `_replay-smoke.sh`'s plain `git checkout <feature>` ABORTS ("local changes would
   be overwritten") and the whole resume fails. Force the checkout (-f): that churn
   is disposable (the feature branch carries its own committed state) and landing on
   it is the point , the same reasoning and pattern the orchestrator's `done` phase
   already uses (orchestrator-effects.ts) and the fork-guard's metadata tolerance.

2. DEGRADED , kit_lk_path returned a dead path silently. If neither the substrate
   shim nor the legacy in-repo path exists (usually: `npm install` not run), it now
   errors + returns 1, and the three launchers bail (|| exit/return 1), instead of a
   later `bash "$KIT_LK"` dying with an opaque "No such file or directory".

3. DEGRADED , scenario-conditions reader missing was silent. When
   dist/bin/consort/scenario-conditions.cli.js is absent (unbuilt/stale dist), every
   manifest field read empty and a capture silently scaffolded the WRONG stack
   (java, no UI, wrong tiers). Warn loud (non-fatal; still degrades to --ui).

Guards added in replay-layout-guard.test.ts. typecheck + the affected hermetic
guards green.

NOTE (separate repo, not fixed here): the exact checkout that halted a live capture
resume is `consort-pipeline accept` -> mergePaired -> a plain `git checkout <into>`
in the @databricks-solutions/lakebase-scm-utils substrate (index.js checkoutBranch),
aborting on the same disposable .consort churn. That needs a substrate-side fix
(force/ignore the known-disposable run metadata, as this repo's done-phase does).

Co-authored-by: Isaac <no-reply@databricks.com>
@cathysnell
cathysnell force-pushed the fix/replay-capture-kit-lk branch from ab49a58 to 3fd0eff Compare September 4, 2026 19:55
…n, safe force-checkout, full env rename

Responds to the code review of the prior two commits:

- F1 (pin-local-kit.sh): LAKEBASE_KIT_PACKAGE is now single-sourced from
  KIT_PACKAGE_DEFAULT (the same constant local_kit_cache_link pins the cache
  symlink under), instead of honoring an inbound override the cache slot could
  not satisfy — an override would have pointed the generic lk shim at a package
  with no pinned slot.
- F2 (capture-scenario.sh): also warn when the scenario-conditions reader is
  PRESENT but yields no fields (stale/broken dist), not only when it is absent —
  the silent-default-to-wrong-stack case the guard exists to catch.
- F3 (_replay-smoke.sh): before the resume `checkout -f`, refuse when a dirty
  tracked file lives OUTSIDE .consort/.lakebase, so -f only ever discards the
  disposable run-metadata it targets and never silently nukes source/intake
  edits (mirrors cutExperiment's fork-guard tolerance).
- F4: complete the env-name migration — rename the remaining LAKEBASE_SFTDD_*
  knobs (HUMAN_PROXY, REPLAY_BUILD_DIR, REPLAY_DIR, RECORDED_INTAKE_DIR,
  GATE_ANSWER_FILE, EPHEMERAL_VERIFY, LOOP, USE_MANIFEST_STEPS, AUTO_CONTINUE)
  to LAKEBASE_CONSORT_* across the active launchers. Verified safe: no raw
  process.env reads of these names exist in the engine or the substrate package,
  so all resolution is via consortEnv() (CONSORT-first). Left the back-compat
  tests and recorded corpora untouched by design.
- F5 (replay-layout-guard.test.ts): scope the "fails loud" assertion to the
  kit_lk_path body so removing its `return 1` trips the guard even if another
  `return 1` survives elsewhere in the file.

typecheck clean; replay-layout-guard (8 tests) green.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant