Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/actions/prune-repository/prune.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@
# The app's condition-5 test executes the evaluation runner as a child
# process; the context root is not a workspace and must be copied explicitly.
"@apps/brunch-agent": ["libs/@hashintel/brunch-agent/evaluations"],
# Repertoire tests read sibling plugin.yaml files by relative URL.
"@hashintel/brunch-agent-repertoire": [
"libs/@hashintel/brunch-agent/packages/plugin-sdcpn/plugin.yaml",
"libs/@hashintel/brunch-agent/packages/plugin-gherkin/plugin.yaml",
],
}

TURBO_QUERY = """
Expand Down
10 changes: 10 additions & 0 deletions .github/actions/prune-repository/prune_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
TRANSPORT = "@hashintel/brunch-agent-transport-aisdk"
APP = "@apps/brunch-agent"
WEBSITE = "@apps/petrinaut-website"
REPERTOIRE = "@hashintel/brunch-agent-repertoire"


class BrunchRequestedExtras(unittest.TestCase):
Expand Down Expand Up @@ -46,6 +47,15 @@ def test_app_job_adds_the_baseline_evaluation_paths(self) -> None:
["libs/@hashintel/brunch-agent/evaluations"],
)

def test_repertoire_job_copies_sibling_plugin_yaml(self) -> None:
self.assertEqual(
extra_paths_for_requested({REPERTOIRE}),
[
"libs/@hashintel/brunch-agent/packages/plugin-sdcpn/plugin.yaml",
"libs/@hashintel/brunch-agent/packages/plugin-gherkin/plugin.yaml",
],
)

def test_sibling_or_website_job_does_not_add_brunch_extras(self) -> None:
self.assertEqual(extras_for_requested({TRANSPORT}), frozenset())
self.assertEqual(extras_for_requested({WEBSITE}), frozenset())
Expand Down
4 changes: 2 additions & 2 deletions libs/@hashintel/brunch-agent/CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ One row of a plugin's `schema` key (the `Must know` table under ADR-0006): a slo
_Avoid_: demand clause, scope expression, objective row

**Pattern**:
A discretionary, kind-indexed heuristic under a plugin's `patterns` key with a **machine-matchable trigger on node state**: the model situation that triggers it and the question that resolves it. A trigger the harness cannot match against node state (a vague quantifier, an expert who does not know) is guidance — a technique or a movement — not a pattern (ADR-0007). Surfaced by the harness when a node matches the trigger and the relevant slot is unsatisfied; the interviewer decides whether to use it. Never names a domain.
A discretionary, kind-indexed heuristic under a plugin's `patterns` key. Its machine trigger is the declared kind plus, when present, one unsatisfied demanded slot; its `when` text explains when the surfaced candidate applies and its `ask` text gives the resolving question. A trigger the harness cannot narrow to node kind and slot state (a vague quantifier, an expert who does not know) is guidance — a technique or a movement — not a pattern (ADR-0007). The interviewer decides whether to use a surfaced candidate. Never names a domain.
_Avoid_: card, kernel card (retired — ADR-0006 replaced Detects/Goal/Questions/Artifacts cards with pattern rows), technique card

**Runbook**:
Expand All @@ -133,7 +133,7 @@ One of the fixed, harness-owned headings of plugin authoring (ADR-0007): the har
_Avoid_: heading (for the authoring unit), section, property (unqualified)

**Repertoire**:
The harness's own filling of every guidance and runbook key — what it teaches every plugin about interviewing — shipped as `packages/repertoire`, rendered by bindings, never imported by a plugin. Admitted by evidence, not by plausibility; never rescoped without run evidence.
The harness's own filling of every guidance and runbook key — what it can teach any plugin about interviewing — shipped as `packages/repertoire`, rendered by bindings, never imported by a plugin. An entry may declare the value grades it applies to; the harness derives applicability from the plugin's demand rows. Admitted by evidence, not by plausibility; never rescoped without run evidence.
_Avoid_: quiver (retired name), strategy library, kernel

**Mechanism type**:
Expand Down
2 changes: 1 addition & 1 deletion libs/@hashintel/brunch-agent/docs/INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ partition is [`docs/agents/partition-worktrees.md`](agents/partition-worktrees.m
| [research/elicitation-strategy-literature](reference/research/elicitation/elicitation-strategy-literature.md) | active | gisted in FE-1360 resolution | Literature synthesis, 9 sections, verification-labeled |
| [research/re-interviewing-literature-worker-report](reference/research/elicitation/interviewing-literature-source-catalog.md) | active | noted on FE-1361 | Verbatim instruments: 34-mistake taxonomy, question typologies, LLM-interviewer results |
| [research/frontier-model-elicitor-failure-catalogue](reference/research/elicitation/frontier-model-elicitor-failure-catalogue.md) | active | FE-1407 | Test-oracle list for the harness (reclassified 2026-08-25 as test-bed material, not authority): typed frontier-model failure catalogue from the two baseline transcripts and indexed literature — mechanism, detection signature, accountable layer, bounded prevention claim, and the licensed-deferral boundary |
| [baseline evaluation evidence](evidence/evaluations/process-model-elicitation/baseline/) | active | gisted in FE-1361 resolution; FE-1431 | Baseline-control evidence: transcripts, raw snapshots, delivered models, and scored read-out for conditions 1, 2, (2026-08-25) 4 — the rendered teaching layer as prompt only — and (2026-08-25) 5 — the shipped harness in the loop, transcripts and folded store committed, read-out pending review, with a (2026-08-26) turn latency assessment (`condition-5-turn-latency.md`: 145 s/turn, 97% extraction, actions R0–R5) that STEERING carries as an immediate concern; with the executable cases and protocol under `evaluations/` it is the simulated-expert harness for the walking-skeleton run (reclassified 2026-08-25 as test-bed material) |
| [baseline evaluation evidence](evidence/evaluations/process-model-elicitation/baseline/) | active | gisted in FE-1361 resolution; FE-1431 | Baseline-control evidence: frozen conditions 1–2; cycle-one live arms from 2026-08-25 archived under `transcripts/cycle-1/`; cycle-two conditions 4–5 from 2026-08-26 in `transcripts/`; and the scored read-out freezing the plugin key catalogue while routing residual identity, quote-repair, classification, and session-control defects to harness work. Includes the first-run turn latency assessment (`condition-5-turn-latency.md`: 145 s/turn, 97% extraction, actions R0–R5) carried by STEERING; with the executable cases and protocol under `evaluations/` it is the simulated-expert harness for the walking-skeleton run (reclassified 2026-08-25 as test-bed material) |
| [ir-design](specs/intermediate-representation.md) | active | gisted in FE-1364 resolution; amended by FE-1480 | The IR design: Layer A (ratified on worked examples, FE-1397; definition sentence amended by ADR-0003) + the CPS plugin's ten-kind payload, deterministic scaffold and obligation contract (Layer B); executable code is realized downstream under ADR-0005 |
| [ir-worked-examples](evidence/proofs/design/intermediate-representation-worked-examples.md) | active | gisted in FE-1397 | Layer-A validation across Gherkin/CPS/BPMN + assurance: property verdicts, amendments, sublimation findings |
| [ir-design-plain](specs/intermediate-representation-plain.md) | active | strain findings on FE-1401; amended by FE-1480 | Plain-prose rendering of the IR design, including ADR-0005's split between deterministic scaffolding and model-assisted executable realization; notes that `plugin-sdcpn/plugin.yaml` is now the concrete rendering of Layer B |
Expand Down
2 changes: 1 addition & 1 deletion libs/@hashintel/brunch-agent/docs/control/STEERING.md
Original file line number Diff line number Diff line change
Expand Up @@ -378,7 +378,7 @@ changes the witness: latency is no longer a number in a transcript but a person
| Construct and review-and-revise share most of one runbook. | Medium-low; Lu's reading of the early plugin-schema passes; unrehearsed. | Author both job runbooks against the key schema and diff them; a cycle of the plugin loop. |
| Kind-level rows express the coatings case. | High; cycle-two [condition-5 evidence](../evidence/evaluations/process-model-elicitation/baseline/transcripts/) folds 166 captures into 51 nodes with 0 unmapped, and the 93 unsatisfied rows are declared demands, not missing vocabulary. Sufficiency for completion is still unshown. | A run with identity handling; count objective nodes. |
| Typed extraction on the verbatim floor holds up in a live run. | Medium-high; the [latency assessment](../evidence/evaluations/process-model-elicitation/baseline/condition-5-turn-latency.md) records repaired refusals and leaves cost, not correctness, as the open question. | R0 + the assessment's §6 spike: does a cheaper extraction model keep kind/node/slot agreement? |
| The shipped loop converges to completion. | Low; both condition-5 runs (cycle one, [cycle two](../evidence/evaluations/process-model-elicitation/baseline/transcripts/)) end with node identity the dominant defect and no terminal act for an expert-stopped engagement; the cycle-two readout routes both to FE-1383 as harness work, not catalogue gaps. | Give the sweep the node index (R4); rerun condition 5; count objective nodes. |
| The shipped loop converges to completion. | Low; both condition-5 runs ([cycle one](../evidence/evaluations/process-model-elicitation/baseline/transcripts/cycle-1/), [cycle two](../evidence/evaluations/process-model-elicitation/baseline/transcripts/)) end with node identity the dominant defect and no terminal act for an expert-stopped engagement; the cycle-two readout routes both to FE-1383 as harness work, not catalogue gaps. | Give the sweep the node index (R4); rerun condition 5; count objective nodes. |
| Per-turn latency is dominated by extraction, and removable from the critical path. | Medium; the [latency assessment](../evidence/evaluations/process-model-elicitation/baseline/condition-5-turn-latency.md) infers the split from output volume because per-call timing is absent. | R0, then the assessment's §6 spike on frozen turn tails. |
| The condition-5 instrument is too expensive to iterate on. | High; a full run is 12–24 turns at ~145 s each (30–60 min), `harness-run.ts` writes every artefact only at the end, `HARD_STOP_AT = 24` is hard-coded with no short-run knob, and W2's afternoon (2026-08-26, ~3 h after the credential fix) produced attribution fixes and no completed figure. The instrument's cost is now a bound on the P1 cycle and on every harness fix. | A `BRUNCH_BASELINE_HARD_STOP` env (a 3-turn run yields the purpose split in under 10 min); the §6 frozen-tail replay; and taking the first split from the Proof 0 panel run, which FE-1505 already accepts ("harness **or** panel run"). |
| The baseline and the panel measure the same interviewer. | Low; `harness-run.ts` defaults `BRUNCH_SDCPN_MODEL` to `claude-opus-5`, `sdcpn-elicitor.ts` defaults it to `claude-haiku-4-5`. The 145 s figure is opus in the baseline; the panel's production default is unmeasured and undecided. | Lu names the Proof 0 interviewer model before the run; the run records it in the proof bundle. |
Expand Down
2 changes: 1 addition & 1 deletion libs/@hashintel/brunch-agent/docs/control/STRATEGY-LOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -357,7 +357,7 @@ reference because the expert or situation pack changes.

**Evidence links:** [baseline protocol](../../evaluations/protocols/process-model-elicitation/baseline/protocol.md),
[condition-3 preregistration (amended)](../../evaluations/protocols/process-model-elicitation/baseline/condition-3-preregistration.md),
[condition-5 transcript](../evidence/evaluations/process-model-elicitation/baseline/transcripts/condition-5.md),
[first condition-5 transcript](../evidence/evaluations/process-model-elicitation/baseline/transcripts/cycle-1/condition-5.md),
[condition-4 read-out](../evidence/evaluations/process-model-elicitation/baseline/readout.md),
[turn latency assessment](../evidence/evaluations/process-model-elicitation/baseline/condition-5-turn-latency.md),
FE-1404, FE-1431, FE-1361
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Condition 5 — turn latency assessment and recommended actions

> **Provenance.** Agent-authored diagnosis, 2026-08-26, of the first condition-5 run
> (2026-08-25, [transcript](transcripts/condition-5.md), [raw record](transcripts/condition-5.raw.json),
> [folded store](transcripts/condition-5-captures.json)). Commissioned by Lu after the read-out
> (2026-08-25, [transcript](transcripts/cycle-1/condition-5.md),
> [raw record](transcripts/cycle-1/condition-5.raw.json),
> [folded store](transcripts/cycle-1/condition-5-captures.json)). Commissioned by Lu after the read-out
> showed 2.4 minutes per interviewer turn: "not going to be viable at all, for a working
> application". Inputs: the raw record's per-turn tool calls, signals, sweep results, and usage
> totals; the runner [`harness-run.ts`](../../../../../evaluations/protocols/process-model-elicitation/baseline/harness-run.ts);
Expand Down
Loading
Loading