feat(analyst): trace analysts run official DSPy RLMs; retire the Ax stack

CHANGELOG.md

@@ -38,6 +38,26 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
   Completed results retain a digest of every behavior-defining source file and are read through one strict recursive schema.
   The repository includes one pinned 32-case CodeTraceBench input, two complete 64-call GLM-5.2 Agent Eval runs, and a failure-inclusive run of pinned CodeTracer on the same trajectories.
   Exact source, input, result, resume, usage, cost, and secret-scan checks are committed with the results.
+- The Python package gains a `dspy` extra: `agent-eval-rpc[dspy]` runs official `dspy.RLM` in a sandboxed Deno/Pyodide child process with seven allowlisted trace tools and strict JSON I/O.
+
+### Changed
+
+- **Breaking:** trace analysts are recursive research programs run through an explicit analysis engine.
+  `analyzeTraces` requires an `engine` (the DSPy RLM engine is the primary implementation) and reports engine iterations; `maxTurns`, `maxSubqueries`, `onTurn`, and `AnalyzeTracesTurnSnapshot` are gone.
+  `callLlmJson` remains only as the one-shot `direct` benchmark baseline.
+- **Breaking:** `defineTraceAnalyst()` returns an inert `TraceAnalystDefinition` for registration instead of a registrable analyst, and no longer takes a `cost` declaration — analyst cost is always metered LLM usage.
+  `createTraceAnalystKind` is now `createTraceAnalyst`; `TraceAnalystKindSpec` and `CreateTraceAnalystKindOpts` are replaced by `TraceAnalystDefinition`.
+- **Breaking:** `BuildTraceAnalystSurfaceDispatchOptions.analyze` receives `instructions` instead of `actorDescription`.
+- **Breaking:** `SteeringOptimizerBackend` narrows to `'pairwise'`.
+- Citation verification is store-backed: cited trace and span ids must resolve in the trace analysis store, and encoded or foreign ids are rejected.
+- External-optimizer subprocess calls fail on HTTP 200 responses with zero input and output usage, and on output over-reservation after recording the actual charge.
+- Relative external-optimizer runner commands resolve against the caller's working directory instead of the child's temporary directory.
+
+### Removed
+
+- **Breaking:** the Ax analyst stack and the `@ax-llm/ax` dependency: `createAnalystAi`, `CreateAnalystAiConfig`, `structureFindings`, `StructureFindingsOptions`, `StructureFindingsResult`, `AxGepaSteeringOptimizer`, and `AxSteeringOptimizerConfig`.
+- **Breaking:** `createPublicBenchmarkModelRunner`; the analyst benchmark CLI defaults to the DSPy RLM runner and keeps the one-shot runner as the explicit `direct` baseline.
+- `buildTraceAnalystTools`; trace tools are built from the transport-neutral descriptors.
 
 ### Fixed
 

README.md

@@ -357,7 +357,8 @@ pnpm tsx examples/selfimprove-quickstart/index.ts
 You do not need a runnable agent to analyze data you already captured.
 Use `analyzeRuns()` for `RunRecord[]`.
 For traces, run a registry of built-in or custom analysts, measure it on labeled issues and exact span locations, then turn only reviewed findings into eval data.
-For a public quality check, convert CodeTraceBench with `traces import-codetracebench`, then run `agent-eval analyst-benchmark` against the pinned labels and a real model.
+For a public quality check, convert CodeTraceBench with `traces import-codetracebench`, then run `agent-eval analyst-benchmark` against pinned labels.
+The command compares an empty baseline with the official DSPy `RLM` trace analyst and records its trace reads, model calls, tokens, cost, runtime, and cited findings.
 
 Use `AnalystRegistry.runExact()` when the caller, rather than registry defaults, must own every execution choice.
 The ordered `analystIds` array is the execution order, and `null` explicitly disables optional budget, timeout, cancellation, cost, tag, or prior-finding channels.

benchmarks/trace-analysis/codetracebench-glm52-20260730/README.md

@@ -10,6 +10,11 @@ This directory records three real-model runs on the same 32 public CodeTraceBenc
 The result is useful but not strong.
 Agent Eval accounted for every provider call and beat this CodeTracer run on the published metric, but all three arms had low precision, weak repeatability, and frequent false positives on solved trajectories.
 
+**Implementation provenance.**
+These numbers were produced by the retired one-shot direct runner at implementation SHA-256 `4dba263b6256a30d56c7fdb2d992d3a953c0035d731f359b704db806f68f75ac` (the value recorded in every artifact here).
+The current Agent Eval analyst is a recursive DSPy RLM engine with a different implementation digest; the reproduce commands below therefore run a different analyst and will produce different numbers.
+These results describe only the retired runner, and no number here may be cited for the current engine until a fresh certified run replaces this directory.
+
 ## Inputs
 
 | Field | Value |

clients/python/README.md

@@ -217,8 +217,7 @@ Missing provider usage fails the run instead of assuming zero cost.
 
 ### DSPy
 
-DSPy programs should use DSPy's official optimizers directly.
-Install DSPy 3.2.1 and the Agent Eval metric adapter with:
+Install DSPy 3.2.1 and the Agent Eval adapters with:
 
 ```sh
 python -m pip install "agent-eval-rpc[dspy]"
@@ -249,6 +248,34 @@ Identical calls share one judge result, including concurrent calls.
 `DspyJudgeMetric` rejects DSPy's default unrestricted disk-cache pickle handling.
 Configure the official restricted cache as shown above, or call `dspy.configure_cache(enable_disk_cache=False)` before creating the metric.
 
+DSPy programs should use DSPy's official optimizers directly.
+Agent Eval also runs the official `dspy.RLM` for recursive trace analysis.
+The TypeScript API starts the Python bridge, provides authenticated trace tools, keeps provider credentials in Node, enforces model and trace-read limits, and validates cited findings.
+The Python bridge owns no trace storage or model credentials.
+
+```ts
+import { createDspyRlmTraceEngine } from '@tangle-network/agent-eval/analyst'
+import { analyzeTraces } from '@tangle-network/agent-eval/traces'
+
+const engine = createDspyRlmTraceEngine({
+  baseUrl: process.env.LLM_BASE_URL!,
+  apiKey: process.env.LLM_API_KEY!,
+  model: process.env.LLM_MODEL!,
+  pricing: {
+    inputUsdPerMillion: 3,
+    outputUsdPerMillion: 15,
+  },
+  runner: { command: '.venv/bin/python' },
+})
+
+const result = await analyzeTraces(
+  { question: 'What first caused this run to fail?' },
+  { source: 'run.otlp.jsonl', engine, toolGroup: 'singleTrace' },
+)
+```
+
+See [Trace Analysis](../../docs/trace-analysis.md) for custom definitions, limits, result fields, and the public quality benchmark.
+
 DSPy 3.2.1 pins GEPA 0.0.27.
 The general Optimize Anything bridge uses GEPA 0.1.4, so repository checks install them in separate environments:
 

clients/python/pyproject.toml

@@ -35,6 +35,7 @@ dev = [
   "ruff>=0.6",
 ]
 dspy = [
+  "deno==2.7.14",
   "dspy==3.2.1",
 ]