fix(replay): unbreak the capture/replay harness (lk resolution + env-name refresh) - #193
Open
cathysnell wants to merge 3 commits into
Open
fix(replay): unbreak the capture/replay harness (lk resolution + env-name refresh)#193cathysnell wants to merge 3 commits into
cathysnell wants to merge 3 commits into
Conversation
…name refresh)
The replay/capture launchers could not run on current main.
1. KIT_LK pointed at templates/project/common/scripts/lk, a shim removed from this
repo (moved into the @databricks-solutions/lakebase-scm-utils substrate package
and made generic, needing LAKEBASE_KIT_PACKAGE). Every capture, replay, and
smoke hard-failed at the first pre-project `lk` call:
bash: .../templates/project/common/scripts/lk: No such file or directory
capture-scenario: kit --warm failed
Resolve KIT_LK via a shared kit_lk_path() helper in pin-local-kit.sh (installed
substrate path, legacy in-repo path as fallback), and export LAKEBASE_KIT_PACKAGE
from resolve_kit_single_source so the generic shim knows which kit to load for
the pre-project --warm / lakebase-create-project calls (a scaffolded project
reads .lakebase/kit-package; the launchers run before any project exists).
npx-tax-guard was asserting the removed path; updated to require the helper.
2. Refresh the deprecated LAKEBASE_SFTDD_{AUTO_CONTINUE,SPRINT_REQUESTS} env names
to the current LAKEBASE_CONSORT_* prefix across the active launchers + docs.
consortEnv() reads CONSORT first (SFTDD/TDD remain fallbacks), so this is
behavior-preserving; it silences the per-capture deprecation warnings and won't
break at the v0.4.0 removal.
Adds a hermetic anti-drift guard (replay-layout-guard.test.ts) and updates the two
guards that pinned the old state. Full hermetic suite + typecheck green.
Co-authored-by: Isaac <no-reply@databricks.com>
…d on missing lk/dist
Three more capture/replay breakers surfaced by the re-record audit:
1. BREAKING , resume wedges on the feature checkout. On a RESUME (alreadyClaimed)
HEAD is on the parent tier and the per-run .consort/.lakebase metadata
(workflow-state.json, pipeline.json, smells.json, ...) is dirty + tracked, so
`_replay-smoke.sh`'s plain `git checkout <feature>` ABORTS ("local changes would
be overwritten") and the whole resume fails. Force the checkout (-f): that churn
is disposable (the feature branch carries its own committed state) and landing on
it is the point , the same reasoning and pattern the orchestrator's `done` phase
already uses (orchestrator-effects.ts) and the fork-guard's metadata tolerance.
2. DEGRADED , kit_lk_path returned a dead path silently. If neither the substrate
shim nor the legacy in-repo path exists (usually: `npm install` not run), it now
errors + returns 1, and the three launchers bail (|| exit/return 1), instead of a
later `bash "$KIT_LK"` dying with an opaque "No such file or directory".
3. DEGRADED , scenario-conditions reader missing was silent. When
dist/bin/consort/scenario-conditions.cli.js is absent (unbuilt/stale dist), every
manifest field read empty and a capture silently scaffolded the WRONG stack
(java, no UI, wrong tiers). Warn loud (non-fatal; still degrades to --ui).
Guards added in replay-layout-guard.test.ts. typecheck + the affected hermetic
guards green.
NOTE (separate repo, not fixed here): the exact checkout that halted a live capture
resume is `consort-pipeline accept` -> mergePaired -> a plain `git checkout <into>`
in the @databricks-solutions/lakebase-scm-utils substrate (index.js checkoutBranch),
aborting on the same disposable .consort churn. That needs a substrate-side fix
(force/ignore the known-disposable run metadata, as this repo's done-phase does).
Co-authored-by: Isaac <no-reply@databricks.com>
cathysnell
force-pushed
the
fix/replay-capture-kit-lk
branch
from
September 4, 2026 19:55
ab49a58 to
3fd0eff
Compare
…n, safe force-checkout, full env rename Responds to the code review of the prior two commits: - F1 (pin-local-kit.sh): LAKEBASE_KIT_PACKAGE is now single-sourced from KIT_PACKAGE_DEFAULT (the same constant local_kit_cache_link pins the cache symlink under), instead of honoring an inbound override the cache slot could not satisfy — an override would have pointed the generic lk shim at a package with no pinned slot. - F2 (capture-scenario.sh): also warn when the scenario-conditions reader is PRESENT but yields no fields (stale/broken dist), not only when it is absent — the silent-default-to-wrong-stack case the guard exists to catch. - F3 (_replay-smoke.sh): before the resume `checkout -f`, refuse when a dirty tracked file lives OUTSIDE .consort/.lakebase, so -f only ever discards the disposable run-metadata it targets and never silently nukes source/intake edits (mirrors cutExperiment's fork-guard tolerance). - F4: complete the env-name migration — rename the remaining LAKEBASE_SFTDD_* knobs (HUMAN_PROXY, REPLAY_BUILD_DIR, REPLAY_DIR, RECORDED_INTAKE_DIR, GATE_ANSWER_FILE, EPHEMERAL_VERIFY, LOOP, USE_MANIFEST_STEPS, AUTO_CONTINUE) to LAKEBASE_CONSORT_* across the active launchers. Verified safe: no raw process.env reads of these names exist in the engine or the substrate package, so all resolution is via consortEnv() (CONSORT-first). Left the back-compat tests and recorded corpora untouched by design. - F5 (replay-layout-guard.test.ts): scope the "fails loud" assertion to the kit_lk_path body so removing its `return 1` trips the guard even if another `return 1` survives elsewhere in the file. typecheck clean; replay-layout-guard (8 tests) green. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The replay/capture launchers cannot run on current
main. A freshcapture-scenario.sh --create(and the replay/smoke launchers, which share thesame resolver) hard-fails at the first pre-project
lkcall:KIT_LKstill points attemplates/project/common/scripts/lk, a shim that wasremoved from this repo — the scaffold
lknow lives in the@databricks-solutions/lakebase-scm-utilssubstrate package (see the header ofexamples/replay/lk) and was made generic, so it needsLAKEBASE_KIT_PACKAGEto know which kit to load. This is very likely why the shipped
stockflowcorpusis still a v0.3.0-beta.14 recording: the capture path is broken, so it cannot be
re-recorded.
Fix
lkresolution. ResolveKIT_LKvia a sharedkit_lk_path()helper inlib/pin-local-kit.sh(installed substrate path, with the legacy in-repo pathas a fallback), and export
LAKEBASE_KIT_PACKAGEfromresolve_kit_single_sourceso the generic shim resolves the kit for thepre-project
--warm/lakebase-create-projectcalls. A scaffolded projectreads
.lakebase/kit-package, but the launchers invokelkbefore any projectexists. Applied to
capture-scenario.sh,_replay-smoke.sh,run-smoke.sh.Env-name refresh. Rename the deprecated
LAKEBASE_SFTDD_{AUTO_CONTINUE,SPRINT_REQUESTS}to the current
LAKEBASE_CONSORT_*prefix across the active launchers + docs.consortEnv()readsCONSORT_*first (SFTDD/TDDremain fallbacks), so thisis behavior-preserving on the current engine; it silences the per-capture
deprecation warnings and won't break at the scheduled v0.4.0 removal.
Tests
replay-layout-guard.test.ts: the launchersmust resolve
KIT_LKviakit_lk_path(not the removed in-repo path), andpin-local-kit.shmust define the helper + exportLAKEBASE_KIT_PACKAGE.npx-tax-guard.test.tswasasserting the removed
lkpath;consort-workflow-smoke.test.tsasserted theold env name).
npm run typecheckclean; the affected hermetic suites pass. Verified the fixend-to-end: with the patched resolver,
lkresolves the pinned kit andlakebase-create-projectruns, and a livecapture-scenario.sh --createprovisions + drives past setup.
Notes for maintainers
npm testinside alinked
git worktreecorrupts the worktree's git — several build-lane tests(
cycle-record,greenOpenCycle,cutExperiment, …) make realgit commitsthat land on the current branch instead of an isolated fixture (branch HEAD gets
rewritten with
green: T1/T2/refactorcommits). Worth isolating thosetests' git ops; happy to file/fix separately.
This pull request and its description were written by Isaac.