fix(templates): adopt agent-eval 0.135.1 - #197
Conversation
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 8741d75b
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-28T23:04:03Z
|
| State | Detail |
|---|---|
| Interrupted | webhook restarted |
No review verdict was produced for this run. Trigger a fresh review on the current PR head if the PR is still open.
tangletools · #197 · model: kimi-for-coding · updated 2026-07-28T23:05:34Z
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 8741d75b
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-28T23:06:44Z
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Concerns | 1 (1 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.0s |
| Interrogation | 81.2s (2 bridge agents) |
| Total | 81.2s |
💰 Value — sound-with-nits
Correct, complete forward bump of agent-eval 0.134.2→0.135.1 across all 22 surfaces plus a real test-runner fix; ship, with one maintenance note.
- What it does: Bumps @tangle-network/agent-eval from 0.134.2 to 0.135.1 in every place it is pinned or referenced: root + recruiter-example + both harness-family package.json files, all 6 layer/capability manifest.json packageDeps, both pnpm-workspace.yaml exclusions, the two lockfiles, doc/comment strings in AGENTS.md/README.md/campaign.ts/loop.ts, and the version-pinning regression test. It also rewrites the r
- Goals it achieves: Keep every template that composes agent-eval on the current release so composed consumer projects and the repo itself resolve one consistent version; fix a test command that Node's native runner couldn't execute on .ts files (matching the tsx convention every sibling script already uses). The system is better because no template ships a stale eval dependency and the research template's
pnpm test - Assessment: Good on its merits. A grep for 0.135.1 shows the version landed consistently in all 22 expected locations with no stragglers, and 0.134.2 no longer appears anywhere — the bump is complete, not partial. The node→tsx test-runner change is a legitimate fix (Node --test has no TS loader; tsx is already a devDependency and is what research:propose/screen/validate/sweep all use), not drive-by noise. The
- Better / existing approach: No existing tool does this job. scripts/dep-cascade-auto-bump.ts (lines 84-94) only unifies already-declared inconsistent versions and only edits package.json files via JSON parse/write — it cannot forward-bump to a version not yet present anywhere, and it models nothing about manifest packageDeps, pnpm-workspace.yaml, or doc/comment strings. scripts/cve-auto-bump.ts is single-family-single-packag
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content
🎯 Usefulness — sound
Clean version-cohort bump of agent-eval to 0.135.1 across every pin surface, plus a real fix that makes the research template's test script actually executable (node --test cannot run .ts; tsx can), with a regression test guarding it.
- Integration: Fully reachable. The families/layers are consumed by the repo's compose pipeline (its entire purpose); the research test script runs when a generated research-harness project invokes
pnpm test; the pin test runs in this repo's CI; the recruiter example workspace consumes the pinned version. Nothing is dead. - Fit with existing patterns: Matches the established version-cohort pattern exactly — there is already a CI test (
tests/agent-eval-harness-ts.test.ts:132) that asserts the exact pinned version, and this PR updates its expected value rather than introducing a new mechanism. The tsx switch aligns the test script with every other script in the same package.json (all already usetsx src/...). - Real-world viability: The tsx fix is the load-bearing real-world change: without it
pnpm testin any generated research project fails on the first.tsfile (the tests are real TypeScript importing from.js-suffixed TS modules). The version bump is mechanical but keeps generated templates on the current published release. The added regression test pins the exact script string so it cannot silently revert. - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟡 Version string is duplicated across ~22 surfaces; only 1 is regression-tested [maintenance] ``
The agent-eval version is hand-materialized in 4 package.json files, 6 manifest.json packageDeps, 2 pnpm-workspace.yaml, and several .md/.ts comment strings (grep for 0.135.1 returns 22 non-lock files). tests/agent-eval-harness-ts.test.ts:132-142 pins only registry/families/agent-eval-harness-ts/files/package.json — the other 3 package.json files and all 6 manifests are unchecked, so a future bump that misses a manifest packageDeps ships a stale eval version to composed consumer projects with no
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
✅ No Blockers —
|
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Approved — 5 non-blocking findings — 8741d75b
Full multi-shot audit completed 7/7 planned shots over 22 changed files. Global verifier still owns final merge decision.
Full immutable report for this review: trace
Summary comment for this run: full summary
tangletools · 2026-07-28T23:21:37Z · immutable trace
|
Addressed all five current review findings at
Checks on this head:
|
|
@tangletools please run the full review on current head |
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Concerns | 1 (1 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.2s |
| Interrogation | 124.6s (2 bridge agents) |
| Total | 124.8s |
💰 Value — sound-with-nits
A complete, well-scoped agent-eval 0.134.2→0.135.1 cohort bump that also fixes a stale API doc, a broken TS test runner, and replaces a one-point pin check with a proper cross-registry regression gate — ship.
- What it does: Bumps @tangle-network/agent-eval from 0.134.2 to 0.135.1 across every distinct install site: root package.json devDep (package.json:135), the recruiter example (examples/recruiter-eval-workspace/eval/package.json:13), both template families' files/package.json, all five layer manifests that declare packageDeps (auto-research, judge-pairwise, redteam, trace-multi-turn, capability/agent-eval), and a
- Goals it achieves: Keep the host tooling, the dogfood example, and every generated eval/research template on one coherent agent-eval cohort so a consumer composing family+layers never hits a mixed-version install; make future bumps drift-proof by converting a single-point assertion into a regression gate that walks the whole registry; and stop embedding version literals in comments that go stale every release. The s
- Assessment: Good change, in the grain of the codebase. This is a template registry where the same version legitimately must live in many places — each generated family/layer is an independent install root, so the pins cannot be collapsed to one source-of-truth file. Given that constraint, a comprehensive regression test is the correct mechanism, and this one is well-built: it enumerates the fixed package.json
- Better / existing approach: No materially better approach for the bump itself — separate install roots demand per-site pins, and a regression gate is the right verification layer. One adjacent gap worth noting: the repo already has a purpose-built drift detector, scripts/dep-cascade-detector.ts, but its scan roots (scripts/dep-cascade-detector.ts:25-27) cover only registry/families, registry/layers/framework, and registry/la
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content
🎯 Usefulness — sound
A complete, coherent agent-eval 0.134.2→0.135.1 bump across every pin site, plus a real fix (tsx test runner for Node-20 compat) and a stronger regression test that enforces future bumps stay cohesive.
- Integration: All pin sites are updated consistently and reached by real consumers: root package.json (devDep driving the foundry's own build/test), the recruiter example workspace, both composed family templates (agent-eval-harness-ts, agent-research-harness-ts), and all 5 layer manifests carrying packageDeps (auto-research, judge-pairwise, redteam, trace-multi-turn, capability/agent-eval). A repo-wide grep co
- Fit with existing patterns: Follows the established pattern exactly — these are the same files that held the 0.134.2 pins. The test expansion extends the existing static-shape contract test file (tests/agent-eval-harness-ts.test.ts) rather than introducing a new harness. The tsx switch aligns with CLAUDE.md's 'Scripts are TypeScript — invoked via tsx' convention and the research template already declares tsx ^4.19.0 as a dev
- Real-world viability: The tsx fix is the load-bearing robustness gain: the old
node --test ... tests/*.test.tscannot execute .ts files on Node 20 (the declared engine floor>=20) — Node only gained native .ts support via --experimental-strip-types in 22.6+. The research template's tests import .js-compiled ESM from .ts sources, so the old command was broken on the supported floor; tsx resolves it. The cohort test - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟡 dep-cascade-detector is blind to most layer categories, including agent-eval [maintenance] ``
scripts/dep-cascade-detector.ts:24-28 only scans registry/families, registry/layers/framework, and registry/layers/capability. The agent-eval layer category (auto-research, judge-pairwise, redteam, trace-multi-turn) — where 4 of the 5 pins in this PR live — is not in its roots (confirmed:
rg agent-eval scripts/dep-cascade-detector.ts→ no hits). The detector also misses agent-base, agent-channels, agent-tools, database, payments, queue, sdk, etc. This PR's new test compensates for agent-eval s
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
✅ No Blockers —
|
| glm | verifier:glm51 | aggregate | |
|---|---|---|---|
| Readiness | 71 | 92 | 71 |
| Confidence | 95 | 95 | 95 |
| Correctness | 71 | 95 | 71 |
| Security | 71 | 100 | 71 |
| Testing | 71 | 90 | 71 |
| Architecture | 71 | 95 | 71 |
Reviewer score is advisory once the run is complete and the verdict has no blockers.
Full multi-shot audit completed 7/7 planned shots over 22 changed files. Global verifier still owns final merge decision. | Clean version bump (0.134.2 -> 0.135.1) consistent across all 4 package.json files, 5 layer manifests, 3 workspace YAMLs, and 2 lock files. The runSteeringOptimization API change (evaluate-callback -> rows-based) is correctly reflected in both loop.ts implementation (line 69: PairwiseSteeringOptimizer().optimize(input.rows)) and the README example. Test coverage was substantially expanded with a cohort-consistency test that validates every pin location. All reviewer findi
🟡 LOW Orphaned historical version reference in campaign.ts comment — registry/families/agent-eval-harness-ts/files/src/eval/campaign.ts
Line 94 still says 'the bug class 0.22 was designed to eliminate' but this PR stripped the version pin from line 2 ('@tangle-network/agent-eval@0.22+' -> '@tangle-network/agent-eval'), leaving '0.22' without context in this file.
🟡 LOW README example imports SteeringOptimizationRow from upstream while local barrel re-exports it — registry/layers/agent-eval/auto-research/files/src/eval/auto-research/README.md
Line 26: import type { SteeringOptimizationRow } from '@tangle-network/agent-eval' is redundant; the local barrel (index.ts:27) already re-exports this type alongside the other imports on lines 18-25 from './eval/auto-research/index.js'. Inconsistent import style in the same example block.
🟡 LOW README example references undefined runAndScore helper with detached comment — registry/layers/agent-eval/auto-research/files/src/eval/auto-research/README.md
Line 36 uses
score: await runAndScore(bundle, scenario)but runAndScore is never declared or imported. The explanatory comment on line 42 sits after a blank line (line 41), visually attached to the next section ('2. Variable-length...') rather than
🟡 LOW README intro now understates the layer's behavior — registry/layers/agent-eval/auto-research/files/src/eval/auto-research/README.md
The closing line 'This layer ranks variants. Other layers execute, score, store, and promote them.' is imprecise: loop.ts exports runMultiShotTrajectoryOptimization, runEvolution, runPromotedImprovementLoop, and analyzeOptimization, which do optimization, promotion, and decision-packet analysis — not just ranking.
🟡 LOW Hardcoded AGENT_EVAL_VERSION couples test to manual cohort bumps — tests/agent-eval-harness-ts.test.ts
The literal '0.135.1' at line 39 is the single source of truth the test enforces across all pin locations. Every cohort bump requires editing this constant. Intentional as regression bait (file header lines 7-23 document this), but no inline comment near the constant itself reminds operators of the bump procedure.
🟡 LOW layerPins.length >= 5 assertion message omits found manifest paths — tests/agent-eval-harness-ts.test.ts
The assertion at line 180-183 fires with count only ('expected at least 5 agent-eval layer pins; got N'). If a registry refactor drops or merges a layer directory, the operator gets no visibility into which manifests remain. Currently exactly 5 manifests pin agent-eval (judge-pairwise, auto-research, trace-multi-turn, redteam, capability/agent-eval).
tangletools · 2026-07-29T00:35:16Z · trace
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Approved — 6 non-blocking findings — 7aae88d5
Full multi-shot audit completed 7/7 planned shots over 22 changed files. Global verifier still owns final merge decision. | Clean version bump (0.134.2 -> 0.135.1) consistent across all 4 package.json files, 5 layer manifests, 3 workspace YAMLs, and 2 lock files. The runSteeringOptimization API change (evaluate-callback -> rows-based) is correctly reflected in both loop.ts implementation (line 69:
Full immutable report for this review: trace
Summary comment for this run: full summary
tangletools · 2026-07-29T00:35:16Z · immutable trace
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
🟠 Value Audit — better-approach-exists
| Verdict | better-approach-exists |
| Concerns | 2 (1 medium-concern, 1 weak-concern) |
| Heuristic | 0.0s |
| Duplication | 0.0s |
| Interrogation | 137.5s (2 bridge agents) |
| Total | 137.5s |
💰 Value — better-approach-exists
Correct, coherent agent-eval 0.134.2→0.135.1 bump + upstream API adaptation + good test consolidation, but it manually re-does what the repo's own dep-cascade detector/auto-bumper was built to automate — and only has to because that tooling's scope was never widened to cover the agent-eval layer gro
- What it does: Bumps @tangle-network/agent-eval from 0.134.2 to 0.135.1 in every pin location: root package.json:135, the recruiter example (examples/recruiter-eval-workspace/eval/package.json:13), both scaffold families (registry/families/agent-eval-harness-ts/files/package.json:13, registry/families/agent-research-harness-ts/files/package.json:21), all five layer manifests' packageDeps (capability/agent-eval,
- Goals it achieves: Keep every generated install root on the current agent-eval release (templates teach the pattern, so staleness propagates). Make future bumps cheap and drift-proof: one constant moves, one assertion enforces it everywhere, and layer pins are auto-discovered rather than hand-listed. Align the auto-research scaffold with the real upstream row-based steering API so generated research projects compile
- Assessment: Sound on its merits. The functional change (row-based runSteeringOptimization) is a faithful tracking of a real upstream API change, not a cosmetic rename — SteeringOptimizationInput now carries rows: SteeringOptimizationRow[] and delegates to new PairwiseSteeringOptimizer().optimize(rows), matching the rewritten README example. The test consolidation is the right structural response to recurring
- Better / existing approach: The repo already has scripts/dep-cascade-detector.ts and scripts/dep-cascade-auto-bump.ts — a detector that finds cross-layer version drift and an auto-bumper that opens a single coordinated PR unifying every out-of-sync instance to the newest declared version (dep-cascade-auto-bump.ts:103-138 even authors the exact 'unify X → Y across N layers' commit/PR this PR hand-rolled). But collectPackageJs
- Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content
🎯 Usefulness — sound
A clean, fully-wired agent-eval 0.134.2→0.135.1 bump that hardens the version-cohort invariant test and fixes the research template's broken pnpm test command.
- Integration: Every changed pin is consumed today: the root package builds/typechecks/tests against 0.135.1, the recruiter example installs it, and the two template families + five layer manifests feed the registry compose/scaffold pipeline. The new cohort test runs in the root
node --testsuite (tests/agent-eval-harness-ts.test.ts:158). Thetsx --testfix makes the generated research project'spnpm test - Fit with existing patterns: Follows the established pattern exactly. Version pins are mirrored across the same five surfaces the repo already maintains (root package.json, recruiter example, two family files/package.json, layer manifests packageDeps, pnpm-workspace minimumReleaseAgeExclude). The consolidated cohort test replaces a single-location check with one driven by a single
AGENT_EVAL_VERSIONconstant — the grain of - Real-world viability: Holds up: the
>= 5layer-pin floor is intentionally loose (7 exist today) but the per-pin equality loop still asserts every found pin matches the cohort, so partial bumps fail regardless. The test-command regex/^tsx --test\b.*--test-concurrency=1\b.*tests\/\*\.test\.ts$/matches the actual script string. Doc cleanup (removing 0.22/0.21/0.77.0 prose from campaign.ts, loop.ts, manifest descript - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟠 Manual bump + bespoke cohort test re-do what dep-cascade detector/auto-bumper already automate [better-architecture] ``
scripts/dep-cascade-detector.ts:24-28 scans only registry/families, registry/layers/framework, and registry/layers/capability — it never visits registry/layers/agent-eval (4 of 5 agent-eval pins), and dep-cascade-auto-bump.ts:84-94 only rewrites package.json deps, never pnpm-workspace.yaml minimumReleaseAgeExclude. So the very toil this PR performs (unify one package across the registry in a single coordinated PR) is the stated purpose of dep-cascade-auto-bump.ts:1-15, but its scope gap forces a
🟡 Version-prose cleanup is inconsistent across .ts and .md [maintenance] ``
The PR stripped '@0.22+', '@^0.77.0', and '0.19 multi-shot adapter' from campaign.ts, loop.ts, loop README, manifest descriptions, and test names, but left 'agent-eval 0.21+' and 'agent-eval 0.22+' as section headings in registry/families/agent-eval-harness-ts/files/README.md:50,84 and examples/recruiter-eval-workspace/eval/README.md:50,84 (verified via git grep). Commit 7aae88d 'remove duplicated eval version prose' touched some but not these. Either keep version-arrival markers everywhere as h
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — 20692599
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-07-29T01:11:41Z
✅ No Blockers —
|
| glm | verifier:glm51 | aggregate | |
|---|---|---|---|
| Readiness | 86 | 92 | 86 |
| Confidence | 95 | 93 | 93 |
| Correctness | 86 | 92 | 86 |
| Security | 86 | 95 | 86 |
| Testing | 86 | 88 | 86 |
| Architecture | 86 | 90 | 86 |
Reviewer score is advisory once the run is complete and the verdict has no blockers.
Full multi-shot audit completed 7/7 planned shots over 22 changed files. Global verifier still owns final merge decision. | Verified all 5 install-root categories (4 package.json, 5 layer manifests, 3 pnpm-workspace.yaml, 2 lock files) are consistently bumped to 0.135.1. The new cohort-invariant test (tests/agent-eval-harness-ts.test.ts:158-211) recursively walks registry/layers for packageDeps pins and enforces parity — confirmed exactly 5 layer manifests have packageDeps (judge-rubric and scenarios do not). The README example fix (old called runSteeringOptimization with {variants,examples,ev
🟡 LOW Manifest description drops screener/validator workflow framing — registry/families/agent-research-harness-ts/manifest.json
The new description removes the explicit two-phase screener (1-rep) → validator (multi-rep bootstrap-CI) workflow framing. The bundle still ships screener.ts and validator.ts (manifest.json files[] lines 45-46), and the CLI reference 'pnpm research propose|screen|validate|sweep' still names both phases, so routing is unaffected. AGENTS.md lines 88-90 still document screener=1rep/validator=5rep with bootstrap-CI gates. Pure human-readability regression in the descrip
🟡 LOW Header comment lost the version pin with no grep fallback for drift detection — registry/layers/agent-eval/auto-research/files/src/eval/auto-research/loop.ts
loop.ts:4 and README.md:3 now say 'over @tangle-network/agent-eval' with no version. The manifest is correctly the single source of truth and the layer-pin test (tests/agent-eval-harness-ts.test.ts:176-193) enforces parity, so this is the right structural direction. The only cost is that grep '0.135.1' no longer surfaces these files on a future bump — operators must rely on the test, not grep. Not blocking; calling out so it's a conscious tradeoff rather than an accidental one.
🟡 LOW Hardcoded packageFiles/workspaceFiles lists can silently miss new install roots — tests/agent-eval-harness-ts.test.ts
The packageFiles array (lines 159-164) and workspaceFiles array (lines 195-199) are hand-listed. If a new family or workspace is added that pins @tangle-network/agent-eval, the cohort invariant silently won't cover it. The layer-manifest half of the test uses recursive findManifestFiles (line 176) and is robust; the package/workspace halves are not. Optional hardening
tangletools · 2026-07-29T01:13:41Z · trace
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
✅ Approved — 3 non-blocking findings — 20692599
Full multi-shot audit completed 7/7 planned shots over 22 changed files. Global verifier still owns final merge decision. | Verified all 5 install-root categories (4 package.json, 5 layer manifests, 3 pnpm-workspace.yaml, 2 lock files) are consistently bumped to 0.135.1. The new cohort-invariant test (tests/agent-eval-harness-ts.test.ts:158-211) recursively walks registry/layers for packageDeps pi
Full immutable report for this review: trace
Summary comment for this run: full summary
tangletools · 2026-07-29T01:13:41Z · immutable trace
tangletools
left a comment
tangletools
left a comment
There was a problem hiding this comment.
🟡 Value Audit — sound-with-nits
| Verdict | sound-with-nits |
| Concerns | 1 (1 medium-concern) |
| Heuristic | 0.0s |
| Duplication | 0.0s |
| Interrogation | 130.0s (2 bridge agents) |
| Total | 130.0s |
💰 Value — sound-with-nits
Bumps agent-eval 0.134.2→0.135.1 across every pin site, upgrades a single-point version test into comprehensive cohort coverage, fixes a broken TS test runner, and de-rots stale doc comments — coherent and worth shipping.
- What it does: Three things bundled: (1) unifies @tangle-network/agent-eval at 0.135.1 across all 8 dependency declarations (root package.json, recruiter example, both harness families, and 5 layer manifests) plus the 3 pnpm-workspace.yaml minimumReleaseAgeExclude entries; (2) replaces the old single-file version test ('package.json pins @tangle-network/agent-eval to the current cohort', which only checked regis
- Goals it achieves: Keep every generated install root on one agent-eval release so composed templates don't ship with mixed/behind versions; make future bumps a one-constant edit with CI catching any forgotten location; fix a TS test invocation that Node can't run; and stop rotting comments against version numbers that move every release.
- Assessment: Sound and in the codebase's grain. The test upgrade is the standout — this PR exists precisely because pins had drifted across many locations, and the old test only verified one of them; the new test (tests/agent-eval-harness-ts.test.ts:158-211) converts that into exhaustive coverage with a single AGENT_EVAL_VERSION constant and clear failure messages naming the offending file, which is exactly th
- Better / existing approach: Searched for existing cohort/version-pin tooling and found a purpose-built subsystem: scripts/dep-cascade-detector.ts + scripts/dep-cascade-auto-bump.ts, wired into CI at .github/workflows/cve-sweep.yml:36 and exposed as
pnpm dep:cascade(package.json:104). Its stated job is 'find npm packages used by multiple layers and flag inconsistent pinned versions' and auto-open a single unify PR. However - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 2
- Bridge warning: opencode/kimi-for-coding/k2p7: bridge stream ended without value-audit content
🎯 Usefulness — sound
Clean agent-eval 0.134.2→0.135.1 pin bump across all 13 surfaces, plus a strictly-better consolidated drift-prevention test and accurate comment refresh — nothing dead or competing.
- Integration: Fully wired and reachable. agent-eval@0.135.1 is the dependency consumed by every generated eval/research install: imported in runner.ts:43, campaign.ts:41, loop.ts:43, pareto.ts:19, judge-client.ts:8; composed by examples/recruiter-eval-workspace/eval/package.json:13; pinned in 5 layer manifests and 3 pnpm-workspace.yaml exclusions. grep confirmed zero residual 0.134.2 pins. The templates ARE the
- Fit with existing patterns: Follows the exact established pattern of prior version bumps. The test consolidation (single AGENT_EVAL_VERSION constant at tests/agent-eval-harness-ts.test.ts:69 + recursive findManifestFiles discovery) is in-grain and strictly stronger than the per-file hardcoded checks it replaced — it catches the exact multi-file drift class this PR exists to fix. The node→tsx test-runner change aligns the res
- Real-world viability: Holds up. The rewritten test uses dynamic manifest discovery with a
>= 5floor so new agent-eval layers added later are auto-covered without test edits; the tsx --test regex is correctly anchored and matches the real script; the AGENT_EVAL_VERSION constant means future bumps touch one line. The tsx switch fixes a genuine execution gap (node can't run .ts without a loader, and tsx was already dec - Model: opencode/zai-coding-plan/glm-5.2
- Bridge attempts: 1
💰 Value Audit
🟠 Existing dep-cascade subsystem has a blind spot for the agent-eval layer group [better-architecture] ``
scripts/dep-cascade-detector.ts:24-28 scans only registry/families, registry/layers/framework, registry/layers/capability. The entire registry/layers/agent-eval/* group (auto-research, judge-pairwise, redteam, trace-multi-turn) — where 4 of this PR's 8 pin edits live — is unscanned, so the CI-wired detector (.github/workflows/cve-sweep.yml:36) cannot catch version drift among eval layers. This PR therefore does manually (edit 13 files + a bespoke single-package regression test) what the cascade
What this audit checks
It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.
| Pass | What it asks |
|---|---|
| Heuristic | Vague title? Whitespace-only or cruft-bearing diff? (content signals only) |
| Duplication | Do added function/class names already exist elsewhere in the repo? |
| Value Audit | What does it do? What goal does it achieve? Is it good? Better architecture or already-exists? |
| Usefulness Audit | Does it integrate and fit? Will it hold up in real use and actually get used? |
Findings are concerns, not blocks — the human reviewer decides what to do with them.
2 participants
Summary
Checks
Security