session: add restricted Python DSL extraction protocol and make it the default

[chenjw]

What

Introduces a restricted Python memory SDK as an alternative output protocol for the memory extraction loop, and makes it the default (memory.extraction_output_format: json -> python).

The model emits restricted Python function calls (sdk.create_events(...), obj.content.edit(...), obj.delete(replacement=...), obj.link(...)) that an AST compiler resolves into the exact same ResolvedOperations structure as the JSON protocol. Both protocols share one ExtractionOutputProtocol abstraction, so schema rules, patch-repair, URI resolution, and streaming merge are identical.

Why

The JSON protocol struggles with transactional reorganization (create N files + delete source + inherit links), and long Markdown content frequently breaks JSON parsing. The Python DSL expresses references naturally via variables/object methods and parses via ast.parse, giving precise error locations.

Results — LoCoMo full eval (1540 Q, doubao-seed-2-0-code-preview)

Metric	Python	JSON
Overall accuracy	82.99% (1278/1540)	81.75% (1259/1540)
Cat 1 · single-hop (valid)	75.11%	77.29%
Cat 2 · temporal (valid)	90.53%	90.18%
Cat 3 · multi-hop (valid)	63.95%	63.95%
Cat 4 · open-domain (valid)	91.58%	89.67%
Avg tokens / Q	17,339	19,908
Avg iteration	1.34	1.44
One-shot rate (cache baseline proxy)	72.4% (197/272)	57.0% (155/272)
Total memory size	3,854 KB	4,625 KB

Results — memory-organization A/B (controlled, 3 cases × 20 runs each, zh output)

Two independent metrics: 成功拆分合并 (was the required merge/migrate/split actually performed) and 信息完整 (every expected fact present exactly once in the final tree).

Case	Metric	JSON	Python
1 · case-insensitive duplicate entity merge	split/merge done	1/20 (5%)	18/20 (90%)
	facts intact	11/20 (55%)	20/20 (100%)
2 · migrate preferences out of an oversized Profile	split/merge done	12/20 (60%)	20/20 (100%)
	facts intact	20/20	20/20
3 · split an oversized Preference file	split/merge done	1/20 (5%)	18/20 (90%)
	facts intact	12/20 (60%)	19/20 (95%)

Case 1 — entities/Projects/atlas.md + entities/projects/atlas.md point to the same Atlas project with different facts; must merge into one canonical entity and remove/replacement the duplicate. Python's edge is both the multi-object create/delete/replacement organization and fact conservation during the merge.
Case 2 — an oversized Profile mixes stable personal facts with several preference groups; preferences must be migrated into Preferences without touching stable facts.
Case 3 — an oversized Preference file mixes two dimensions and must split into ≥2 files with the oversized source deleted, preserving every fact exactly once.

The A/B harness lives in benchmark/memory_organization/ (run_ab.py, cases under cases/); it uses the same production prompt/Schema as the live pipeline.

Key changes

• extraction_output_protocol/{base,json,python}.py — protocol abstraction + implementations. Python compiles a restricted AST into the same operations model as JSON.
• Default flips to python (memory_config.py, extract_loop.py, ov.conf.example).
• Python syntax errors surface the offending source line + caret; string-literal breaks get targeted triple-quote retry guidance.
• Canonical merges must preserve every distinct fact before deleting a duplicate.
• Removed hardcoded memory-type names from prompts so custom memory_types render dynamically.
• Benign batch-delete link-inheritance read failures downgraded ERROR -> WARNING.
• Added benchmark/memory_organization/ A/B harness and openviking/utils/message_format.py pretty-printer.

How to reproduce

LoCoMo full eval

Select the protocol via memory.extraction_output_format (python | json) in ov.conf,
start the server, then run the driver (imports conversations, waits for extraction, answers
QA, LLM-judges):

# from benchmark/locomo/vikingbot/
./run_full_eval.sh                      # eval all samples (import + QA + judge)
./run_full_eval.sh --skip-import        # reuse already-imported memories
./run_full_eval.sh --parallel-import-sessions 50 --parallel-run-eval 100 --parallel-judge 100
./run_full_eval.sh --keep-runs 10       # keep the last N run dirs (default 10)

Each run writes an isolated dir result/locomo/runs/<timestamp>/ with locomo_result.csv,
import_success.csv, summary.txt, bot logs, and a snapshot of the produced memory files.

Memory-organization A/B (JSON vs Python)

Same production prompt/Schema for both protocols; only the output protocol differs.

# smoke: one case, one repeat
python -m benchmark.memory_organization.run_ab --case merge_travel_aliases \
  --repeat 1 --output benchmark/memory_organization/result/smoke.jsonl
python -m benchmark.memory_organization.report \
  benchmark/memory_organization/result/smoke.jsonl

# paired experiment (3 cases x 20 repeats x 2 protocols)
python -m benchmark.memory_organization.run_ab --repeat 20 \
  --output benchmark/memory_organization/result/ab_repeat20.jsonl
python -m benchmark.memory_organization.report \
  benchmark/memory_organization/result/ab_repeat20.jsonl \
  --output benchmark/memory_organization/result/ab_repeat20.summary.json

# autonomous suite (expected organization lives only in the grader, not shown to the model)
python -m benchmark.memory_organization.run_autonomous_ab --repeat 20 --parallel 6 \
  --output benchmark/memory_organization/result/core_three_repeat20.jsonl
python -m benchmark.memory_organization.report_autonomous \
  benchmark/memory_organization/result/core_three_repeat20.jsonl \
  --output benchmark/memory_organization/result/core_three_repeat20.summary.json

Tests

ruff check / ruff format --check clean on changed files. Extraction protocol, config loader, memory react, patch-merge, and extract-loop match-text suites pass (173 targeted tests). Merged latest origin/main and resolved conflicts in extract_loop.py / patch_merge_context_provider.py. Six failures in the broader memory suite are pre-existing and unrelated to this PR (environment-dependent language detection that reads local ov.conf output_language_override, plus graph_view / schema_models tests untouched here).