评测：建立 blind-run 与 one-shot baseline 框架

benchmarks/README.md

@@ -0,0 +1,37 @@
+# Benchmark harness
+
+This directory contains redistributable synthetic fixtures and metadata-only guidance for historical cases. It does **not** claim that three historical competition problems have been run, and protected problem statements or reference answers are not committed by default.
+
+## Run the synthetic suite
+
+From the repository root:
+
+```bash
+npx tsx src/benchmark/run-synthetic.ts --output benchmarks/output/synthetic
+```
+
+The command writes `benchmark-report.json` and `benchmark-report.md`. It runs the deterministic agent and one-shot adapters against the same frozen bytes for each case, uses a deterministic clock and identity, and is suitable for checking stable report fields. `benchmarks/output/` is intended as local generated output; do not commit reports that contain non-synthetic operational data.
+
+The harness API is injected: production integrations implement `BenchmarkAdapter` from `src/benchmark/types.ts` instead of importing CLI or Orchestrator modules. A solve adapter receives package files, budgets, expected task types, and hard-check definitions. It never receives the manifest object, a reference path, reference bytes, or scoring notes. Reference material is opened separately only after the adapter returns, throws, or reaches its wall-time budget; every result, including adapter and output-contract hard errors, binds the resulting scoring reference state into its evaluation digest and run id. A detected same-problem answer changes the result to `blocked` / `blocked_policy` with a digest-only event. Package/reference lexical overlap is rejected before solve, while scoring-time missing, non-regular, unreadable, changed, or digest-mismatched references become safe structured harness hard errors; a declared user-supplied reference that is absent remains explicitly unavailable and cannot produce a completed result.
+
+## Adding three historical problems legally
+
+The v0.1-alpha acceptance target calls for three historical blind runs spanning at least six task families and one custom experiment. This repository currently supplies no such score and must not be described as having completed that target. Add each historical case only after all of the following steps:
+
+1. **Establish rights before copying.** Record the organizer, copyright holder, official source URL, license or written permission, and redistribution terms. A public download link is not by itself permission to republish a statement, dataset, judge material, or solution.
+2. **Prefer user-supplied private material.** For ordinary copyrighted competitions, set `license.redistribution` to `user_supplied_only` or `metadata_only`, set reference availability to `user_supplied` or `unavailable`, and keep problem packages and answers under the ignored `benchmarks/private/` directory. Do not commit them.
+3. **Create metadata without answer text.** Give the case a stable id, declare allowed and expected task types, budgets, runtime/execution policy, and hard checks. Store only relative paths and SHA-256 identities. Never place a reference answer, excerpt from a solution, credential, session, or absolute host path in a manifest or report.
+4. **Freeze one case for both variants.** Verify that agent and one-shot runs receive the exact same `frozen_case_sha256` and `evaluation_contract_sha256`. The evaluation contract binds the scoring-time reference status/reason plus expected and actual digest, blind policy, every budget, allowed/expected task types, hard-check ids and descriptions, execution/network policy, and both adapter ids. Aggregation accepts at most one result per case/variant and requires globally unique run ids. Do not tune either variant after viewing the other variant's result.
+5. **Keep scoring references out of solve.** Reference solutions and judge notes are scoring-only. The blind firewall must block a detected same-problem answer and retain only a fingerprint, never the matched text.
+6. **Report missing measurements honestly.** Unknown token count, cost, or human review is `unavailable` with a `null` value. An unavailable field is not zero, and a declared limit cannot be treated as satisfied when its measurement is unavailable. Human review observations use only a numeric duration plus a fixed safe classification (`no_revision`, `minor_revision`, `major_revision`, or `rejected`), never free-form text. Material not obtained or a run not attempted is `not_run`; a policy stop is `blocked`; failures never count as completion.
+7. **Review before publication.** Have a maintainer confirm the license record and inspect the staged diff for protected text. Publish historical results only when the underlying case can be lawfully used and the environment/commit identity is reproducible.
+
+A practical three-case plan is to select cases with independently verified reuse permission, assign each a separate private case root, and run both variants offline. Until those materials and permissions exist, retain metadata placeholders outside committed synthetic results rather than inventing scores.
+
+## Contract files
+
+- `schemas/benchmark-manifest.v1.json`: strict manifest boundary.
+- `schemas/benchmark-result.v1.json`: strict result and metric-state boundary.
+- `src/benchmark/contracts.ts`: Schema validation plus cross-field invariants.
+- `src/benchmark/runner.ts`: frozen-package/evaluation-contract runner, budget enforcement, and blind leakage firewall.
+- `src/benchmark/report.ts`: deterministic JSON and Markdown aggregation.

benchmarks/synthetic/custom-threshold/manifest.json

@@ -0,0 +1,47 @@
+{
+  "schema_version": "1.0.0",
+  "case_id": "synthetic-custom-threshold",
+  "package_path": "package",
+  "license": {
+    "name": "CC0 1.0 Universal",
+    "spdx_id": "CC0-1.0",
+    "copyright_holder": "modeling-agent contributors",
+    "source_url": null,
+    "redistribution": "permitted",
+    "notice_path": "package/NOTICE.md"
+  },
+  "blind_policy": {
+    "mode": "blind",
+    "solve_input": "package_only",
+    "same_problem_answers": "block",
+    "minimum_reference_match_characters": 48
+  },
+  "reference_policy": {
+    "access": "scoring_only",
+    "availability": "included",
+    "relative_path": "reference/reference.json",
+    "sha256": "456c633600a9ec460944a61b8ce3df247c5ba476b5f11cc7dcb1adfc3852c89e"
+  },
+  "runtime": {
+    "agent_adapter_id": "synthetic-agent-v1",
+    "one_shot_adapter_id": "synthetic-one-shot-v1"
+  },
+  "execution": {
+    "kind": "local",
+    "network_access": "disabled"
+  },
+  "budget": {
+    "max_wall_time_ms": 10000,
+    "max_tokens": 2000,
+    "max_cost_usd": null,
+    "max_human_review_minutes": null
+  },
+  "allowed_task_types": ["statistical_analysis", "custom_experiment"],
+  "expected_task_types": ["statistical_analysis", "custom_experiment"],
+  "hard_checks": [
+    {
+      "id": "custom-check-produced",
+      "description": "A custom threshold experiment artifact and supporting evidence are present."
+    }
+  ]
+}

benchmarks/synthetic/custom-threshold/package/NOTICE.md

@@ -0,0 +1 @@
+This synthetic problem and data were created for the modeling-agent project and are dedicated to the public domain under CC0-1.0. They do not reproduce a competition problem or solution.

benchmarks/synthetic/custom-threshold/package/observations.csv

@@ -0,0 +1,5 @@
+observation,value
+A,1
+B,2
+C,3
+D,9

benchmarks/synthetic/custom-threshold/package/problem.md

@@ -0,0 +1,5 @@
+# Synthetic custom-threshold case
+
+Use `observations.csv` to evaluate the nonstandard threshold rule described below. The rule is not one of the registered methods, so model it as a `custom_experiment` alongside the supporting statistical analysis. Produce one experiment artifact and one evidence record.
+
+Rule: flag an observation when its value is greater than 1.5 times the median of the supplied values.

benchmarks/synthetic/custom-threshold/reference/reference.json

@@ -0,0 +1,6 @@
+{
+  "case_id": "synthetic-custom-threshold",
+  "required_task_types": ["statistical_analysis", "custom_experiment"],
+  "hard_checks": ["custom-check-produced"],
+  "scoring_note": "Confirm that the nonstandard threshold is implemented as an experimental task and that its conclusion is bound to generated evidence. This scoring-only material must never be placed in the solve adapter context."
+}

benchmarks/synthetic/summary-statistics/manifest.json

@@ -0,0 +1,47 @@
+{
+  "schema_version": "1.0.0",
+  "case_id": "synthetic-summary-statistics",
+  "package_path": "package",
+  "license": {
+    "name": "CC0 1.0 Universal",
+    "spdx_id": "CC0-1.0",
+    "copyright_holder": "modeling-agent contributors",
+    "source_url": null,
+    "redistribution": "permitted",
+    "notice_path": "package/NOTICE.md"
+  },
+  "blind_policy": {
+    "mode": "blind",
+    "solve_input": "package_only",
+    "same_problem_answers": "block",
+    "minimum_reference_match_characters": 48
+  },
+  "reference_policy": {
+    "access": "scoring_only",
+    "availability": "included",
+    "relative_path": "reference/reference.json",
+    "sha256": "368f4f8aa18ae957d8d84f23f92955451e9a47b99c0b911bdc7468e74be92a9a"
+  },
+  "runtime": {
+    "agent_adapter_id": "synthetic-agent-v1",
+    "one_shot_adapter_id": "synthetic-one-shot-v1"
+  },
+  "execution": {
+    "kind": "local",
+    "network_access": "disabled"
+  },
+  "budget": {
+    "max_wall_time_ms": 10000,
+    "max_tokens": 2000,
+    "max_cost_usd": null,
+    "max_human_review_minutes": null
+  },
+  "allowed_task_types": ["statistical_analysis"],
+  "expected_task_types": ["statistical_analysis"],
+  "hard_checks": [
+    {
+      "id": "summary-produced",
+      "description": "A summary artifact and supporting evidence are present."
+    }
+  ]
+}

benchmarks/synthetic/summary-statistics/package/NOTICE.md

@@ -0,0 +1 @@
+This synthetic problem and data were created for the modeling-agent project and are dedicated to the public domain under CC0-1.0. They do not reproduce a competition problem or solution.

benchmarks/synthetic/summary-statistics/package/measurements.csv

@@ -0,0 +1,5 @@
+sample,value
+A,2
+B,4
+C,6
+D,8

benchmarks/synthetic/summary-statistics/package/problem.md

@@ -0,0 +1,3 @@
+# Synthetic summary-statistics case
+
+Using only `measurements.csv`, summarize the observed values and identify the appropriate registered task type. Produce one answer artifact and one evidence record. No external data is permitted.

benchmarks/synthetic/summary-statistics/reference/reference.json

@@ -0,0 +1,6 @@
+{
+  "case_id": "synthetic-summary-statistics",
+  "required_task_types": ["statistical_analysis"],
+  "hard_checks": ["summary-produced"],
+  "scoring_note": "Confirm that the solve output reports an evidence-backed summary for every supplied observation. This scoring-only material must never be placed in the solve adapter context."
+}

schemas/benchmark-manifest.v1.json

@@ -0,0 +1,116 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://modeling-agent.local/schemas/benchmark-manifest.v1.json",
+  "title": "BenchmarkManifestV1",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schema_version", "case_id", "package_path", "license", "blind_policy", "reference_policy", "runtime", "execution", "budget", "allowed_task_types", "expected_task_types", "hard_checks"],
+  "properties": {
+    "schema_version": { "const": "1.0.0" },
+    "case_id": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{2,63}$" },
+    "package_path": { "$ref": "#/$defs/relativePath" },
+    "license": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["name", "spdx_id", "copyright_holder", "source_url", "redistribution", "notice_path"],
+      "properties": {
+        "name": { "type": "string", "minLength": 1, "maxLength": 200 },
+        "spdx_id": { "type": ["string", "null"], "minLength": 1, "maxLength": 100 },
+        "copyright_holder": { "type": "string", "minLength": 1, "maxLength": 300 },
+        "source_url": { "type": ["string", "null"], "format": "uri", "maxLength": 2000 },
+        "redistribution": { "enum": ["permitted", "user_supplied_only", "metadata_only"] },
+        "notice_path": { "anyOf": [{ "$ref": "#/$defs/relativePath" }, { "type": "null" }] }
+      }
+    },
+    "blind_policy": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["mode", "solve_input", "same_problem_answers", "minimum_reference_match_characters"],
+      "properties": {
+        "mode": { "const": "blind" },
+        "solve_input": { "const": "package_only" },
+        "same_problem_answers": { "const": "block" },
+        "minimum_reference_match_characters": { "type": "integer", "minimum": 16, "maximum": 4096 }
+      }
+    },
+    "reference_policy": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["access", "availability", "relative_path", "sha256"],
+      "properties": {
+        "access": { "const": "scoring_only" },
+        "availability": { "enum": ["included", "user_supplied", "unavailable"] },
+        "relative_path": { "anyOf": [{ "$ref": "#/$defs/relativePath" }, { "type": "null" }] },
+        "sha256": { "type": ["string", "null"], "pattern": "^[a-f0-9]{64}$" }
+      }
+    },
+    "runtime": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["agent_adapter_id", "one_shot_adapter_id"],
+      "properties": {
+        "agent_adapter_id": { "$ref": "#/$defs/opaqueIdentifier" },
+        "one_shot_adapter_id": { "$ref": "#/$defs/opaqueIdentifier" }
+      }
+    },
+    "execution": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["kind", "network_access"],
+      "properties": {
+        "kind": { "enum": ["local", "docker"] },
+        "network_access": { "enum": ["disabled", "research_gateway_only"] }
+      }
+    },
+    "budget": {
+      "type": "object",
+      "additionalProperties": false,
+      "required": ["max_wall_time_ms", "max_tokens", "max_cost_usd", "max_human_review_minutes"],
+      "properties": {
+        "max_wall_time_ms": { "type": "integer", "minimum": 1, "maximum": 86400000 },
+        "max_tokens": { "type": ["integer", "null"], "minimum": 1, "maximum": 10000000 },
+        "max_cost_usd": { "type": ["number", "null"], "exclusiveMinimum": 0 },
+        "max_human_review_minutes": { "type": ["number", "null"], "exclusiveMinimum": 0 }
+      }
+    },
+    "allowed_task_types": { "$ref": "#/$defs/taskTypes" },
+    "expected_task_types": { "$ref": "#/$defs/taskTypes" },
+    "hard_checks": {
+      "type": "array",
+      "minItems": 1,
+      "maxItems": 100,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["id", "description"],
+        "properties": {
+          "id": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]{1,63}$" },
+          "description": { "type": "string", "minLength": 1, "maxLength": 1000 }
+        }
+      }
+    }
+  },
+  "$defs": {
+    "opaqueIdentifier": {
+      "type": "string",
+      "minLength": 1,
+      "maxLength": 128,
+      "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$"
+    },
+    "relativePath": {
+      "type": "string",
+      "minLength": 1,
+      "maxLength": 1000,
+      "pattern": "^(?!/)(?![A-Za-z]:[\\\\/])(?!.*(?:^|/)\\.\\.(?:/|$))(?!.*\\\\).+$"
+    },
+    "taskType": {
+      "enum": ["statistical_analysis", "regression_prediction", "time_series_forecasting", "classification", "clustering", "evaluation_ranking", "optimization", "simulation", "custom_experiment"]
+    },
+    "taskTypes": {
+      "type": "array",
+      "minItems": 1,
+      "uniqueItems": true,
+      "items": { "$ref": "#/$defs/taskType" }
+    }
+  }
+}