Persist agent memory and isolate prompt evaluation

.github/workflows/ci.yml

@@ -0,0 +1,22 @@
+name: Offline regressions
+
+on: [push, pull_request]
+
+jobs:
+  tests:
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
+    env:
+      LITELLM_LOCAL_MODEL_COST_MAP: "True"
+      DO_NOT_TRACK: "1"
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python -m pip install -e ".[dev]"
+      - run: python -m ruff check --select E9,F63,F7,F82 src tests examples/07_generate_demo_data.py
+      - run: python -m pytest tests -q
+      - run: python examples/07_generate_demo_data.py --db-path .data/ci-demo.db --benchmark-dir .data/ci-benchmarks

README.md

@@ -75,6 +75,9 @@ src/self_evolving/
 - Retrieval now uses vector similarity as the primary score.
 - Default setup uses a lightweight local hashing embedder so the project works without extra model downloads.
 - The embedder backend is pluggable, so this can later be swapped to `sentence-transformers` or an external embedding model.
+- Retrieval is `cosine × importance + 0.15 × lexical overlap`, with a linear scan of the retained entries. Access count is tracked, not used as a ranking factor.
+- Empty memory now accumulates its first lesson. Completed episodes checkpoint the **full** memory: summaries, evictions, access counts and the stored-episode count survive reconstruction with the same database and `agent_id`.
+- Retrieval outside `agent.run` requires an explicit `agent.save_memory()` to checkpoint its updated access counts. Failed episodes that raise before checkpoint do not save these changes.
 
 ### Embedders
 - `src/self_evolving/evolution/memory/embedders.py`
@@ -87,7 +90,10 @@ src/self_evolving/
 - Persists:
   - runs
   - steps
-  - episodic memory entries
+- episodic memory entries
+- Memory checkpoints replace one agent's snapshot atomically with an optimistic revision check. Concurrent stale writers fail with `MemoryConflictError` instead of overwriting a newer snapshot; API jobs expose this as `failed` with an error message.
+- `save_memory_entries` and `list_memory` retain their append/query interfaces. Internal snapshot loads preserve oldest-first order; legacy entries migrate automatically, but their unknown historical store count starts at zero.
+- For read-only QA, reload with `agent.load_memory()` and rerun after a conflict. For tools with external side effects, reconcile those effects before retrying. Memory and trajectory writes are separate transactions, not an exactly-once workflow.
 
 ### API service
 - `src/self_evolving/service/api.py`
@@ -109,6 +115,8 @@ src/self_evolving/
 ### Reflection
 - `src/self_evolving/mechanisms/reflection/reflexion.py`
 - Adds post-failure reflection and retry behavior.
+- Retries use a fresh conversation and the previous attempt's reflection. The original prompt and reflector are restored on success, exhaustion, or exceptions.
+- Each attempt is persisted separately; the final run also records attempt IDs/counts and total attempt steps. Reflection text is included as unverified context when distilling a lesson, not treated as verified knowledge or a model-weight update.
 
 ### Reward scoring
 - `src/self_evolving/mechanisms/reward/scorer.py`
@@ -130,6 +138,9 @@ src/self_evolving/
   - reflexion
   - prompt optimization
 - Writes JSON artifacts for each variant plus a session summary.
+- Explicit `tuning_tasks` are used only to score OPRO prompt candidates; the selected prompt is then evaluated on `tasks`. Exact normalized goal overlap is rejected. Without `tuning_tasks`, the protocol is labeled `resubstitution`, never held-out evaluation.
+- Sessions have unique directories and agent IDs, so concurrent sessions do not overwrite artifacts or inherit a previous benchmark's memories. Summaries include the task manifest, protocol and data source.
+- The QA scorer is a case-insensitive reference substring smoke test, not exact-match accuracy. Memory uses sequential online adaptation; Reflexion allows up to two attempts. Final-attempt mean steps and total retry steps are both identified, but reflection/OPRO token cost and latency are **not** measured. These variants do not represent equal-cost quality comparisons.
 
 ### Dashboard
 - `app.py`
@@ -166,8 +177,8 @@ That is not yet enough for:
 
 ### Engineering gaps
 - no container delivery files
-- no CI workflow
-- no background job execution for long-running runs and benchmarks
+- CI runs the offline regression suite and synthetic demo on Windows and Linux
+- background jobs exist, but job state is in memory and is lost on API restart
 
 ### Systems / platform gaps
 - no safe tool sandbox
@@ -212,6 +223,32 @@ cp .env.example .env
 
 ## Quick Start
 
+For a reproducible offline acceptance run, no API keys or `.env` file are needed:
+
+```bash
+python -m pip install -e ".[dev]"
+python -m pytest tests -q
+python examples/07_generate_demo_data.py --db-path .data/offline-demo.db --benchmark-dir .data/offline-benchmarks
+```
+
+The tests block provider calls and socket connections (except Windows' internal asyncio socket-pair setup). The demo uses explicit deterministic agent/memory/reflection/optimizer implementations, never global patches; its scores are **synthetic fixtures**, not real model results. Package imports use LiteLLM's bundled cost map by default to avoid a metadata download.
+
+For an actual model-backed prompt experiment, provide independent tuning and evaluation tasks:
+
+```python
+from self_evolving.evaluation.benchmark import BenchmarkRunner
+
+runner = BenchmarkRunner(
+    tuning_tasks=[("What is 3 * 7?", "21"), ("Who wrote Hamlet?", "Shakespeare")],
+    tasks=[("What is 8 * 9?", "72"), ("What is the capital of France?", "Paris")],
+    model="your/configured-model",
+)
+summary = runner.run(["baseline", "prompt_optimization"])
+print(summary["evaluation_protocol"])
+```
+
+These tiny examples demonstrate split wiring only. Disjoint question strings do not establish semantic independence; review task families and answer leakage before reporting a real benchmark result. Model-backed examples below require your own provider configuration and may incur API charges.
+
 ```python
 from dotenv import load_dotenv
 load_dotenv()
@@ -263,12 +300,17 @@ curl -X POST http://127.0.0.1:8000/benchmarks/qa \
       {"goal": "What is the capital of France?", "reference_answer": "Paris"},
       {"goal": "What is 12 * 7?", "reference_answer": "84"}
     ],
-    "variants": ["baseline", "memory", "reflexion"]
+    "tuning_tasks": [
+      {"goal": "What is 3 * 7?", "reference_answer": "21"}
+    ],
+    "variants": ["baseline", "memory", "reflexion", "prompt_optimization"]
   }'
 ```
 
 Inspect recent jobs:
 
+Completed benchmark jobs return `evaluation_protocol` and `data_source` alongside their variant results; the dashboard's session summary displays the same fields. If `tuning_tasks` is omitted, OPRO output is explicitly marked `resubstitution`.
+
 ```bash
 curl http://127.0.0.1:8000/jobs
 ```
@@ -334,6 +376,8 @@ This populates:
 
 Use it when you want the dashboard to have data immediately without calling a real external model.
 
+The generated runs use model label `offline/deterministic-fixture` and benchmark JSON uses `data_source=synthetic_deterministic_fixture`. No learned-performance claim can be inferred from those values.
+
 ## Next Step
 
 The next practical step after the current background-job control plane is:
@@ -354,6 +398,10 @@ In other words, the next upgrade is turning the current local control plane into
 pytest tests/ -v
 ```
 
+Regression coverage includes empty-memory cold start; app/agent reconstruction and memory injection; summary/eviction/counter restoration; snapshot rollback and stale-writer conflicts; actual Reflexion failure/retry/cleanup and persisted attempt evidence; OPRO tuning/evaluation separation; unique same-second artifacts; and the fully offline demo.
+
+Remaining limits: the default hashing vectors are not a transformer semantic model; embedding model/version migration is not automatic. Memory lacks trusted-source filtering, semantic duplicate detection and tenant authorization. Tool learning is an experimental Python execution path, not a safe sandbox. Background job state remains process-local. No real-model improvement, training result, throughput or token-cost benchmark has been measured by the offline suite.
+
 ## Project Direction
 
 The target outcome for this repo is no longer just:

examples/07_generate_demo_data.py

@@ -9,15 +9,21 @@
     python examples/07_generate_demo_data.py
 """
 
-from dotenv import load_dotenv
+import argparse
+import os
 
-load_dotenv()
+# An offline CLI must also avoid import-time provider metadata refreshes.
+os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
 
 from self_evolving.dashboard.demo_data import generate_demo_data
 
 
 def main():
-    result = generate_demo_data()
+    parser = argparse.ArgumentParser(description="Generate synthetic offline data, not model scores")
+    parser.add_argument("--db-path", default=".data/sea.db")
+    parser.add_argument("--benchmark-dir", default="runs/benchmarks")
+    args = parser.parse_args()
+    result = generate_demo_data(args.db_path, args.benchmark_dir)
     print(result)
 
 

src/self_evolving/__init__.py

@@ -3,6 +3,12 @@
 Based on: "A Survey of Self-Evolving Agents" (arXiv:2507.21046)
       and "A Comprehensive Survey of Self-Evolving AI Agents" (arXiv:2508.07407)
 """
+import os
+
+# Importing the framework should not fetch provider metadata. Real completions
+# still require an explicit call; users can opt into a refreshed cost map.
+os.environ.setdefault("LITELLM_LOCAL_MODEL_COST_MAP", "True")
+
 from self_evolving.core.agent import BaseAgent
 from self_evolving.core.environment import Environment
 from self_evolving.core.types import AgentState, Trajectory, Feedback

src/self_evolving/core/agent.py

@@ -8,14 +8,15 @@
 import os
 import uuid
 import logging
+from copy import deepcopy
 from typing import Any, Callable, Optional, TYPE_CHECKING
 
 import litellm
 from tenacity import retry, stop_after_attempt, wait_exponential
 
 from self_evolving.core.types import (
     AgentState, Trajectory, Step, Feedback, FeedbackType, Message,
-    EvolutionRecord, EvolutionTarget, EvolutionStage,
+    EvolutionRecord,
 )
 from self_evolving.core.environment import Environment
 
@@ -37,7 +38,7 @@ class BaseAgent:
             action = agent.act(obs)
             obs, feedback, done = env.step(action)
             agent.observe(feedback)
-        agent.evolve(trajectory)   ← inter-test-time evolution hook
+        optionally reflect, distil memory, and persist the episode
     """
 
     DEFAULT_SYSTEM = (
@@ -52,6 +53,8 @@ def __init__(
         max_steps: int = 20,
         agent_id: Optional[str] = None,
     ):
+        if max_steps < 1:
+            raise ValueError("max_steps must be positive")
         self.agent_id = agent_id or str(uuid.uuid4())[:8]
         self.model = model or os.getenv("SEA_MODEL", "deepseek/deepseek-chat")
         self.state = AgentState(
@@ -67,6 +70,7 @@ def __init__(
 
         self._conversation: list[Message] = []
         self._current_trajectory: Optional[Trajectory] = None
+        self._memory_revision: Optional[int] = None
 
     # ------------------------------------------------------------------
     # Public API
@@ -80,6 +84,8 @@ def run(
         progress_callback: Optional[Callable[[float, str, dict[str, Any]], None]] = None,
     ) -> Trajectory:
         """Execute a full task episode and return the trajectory."""
+        if self.memory is not None and self.store is not None and self._memory_revision is None:
+            self.load_memory(preserve_initial=True)
         task_id = task_id or str(uuid.uuid4())[:8]
         obs = env.reset(goal)
         trajectory = Trajectory(task_id=task_id, goal=goal)
@@ -124,10 +130,9 @@ def run(
             trajectory = self.reflector.reflect(trajectory)
 
         # Store episode in memory
-        if self.memory:
-            memory_entries = self.memory.store(trajectory)
-            if self.store:
-                self.store.save_memory_entries(self.agent_id, memory_entries)
+        if self.memory is not None:
+            self.memory.store(trajectory)
+            self.save_memory()
 
         if self.store:
             trajectory.metadata["run_id"] = self.store.save_trajectory(
@@ -218,13 +223,43 @@ def _call_llm(self, messages: list[dict]) -> str:
     # State management
     # ------------------------------------------------------------------
 
+    def load_memory(self, *, preserve_initial: bool = False) -> None:
+        """Reload a persisted snapshot before running; safe for an empty memory."""
+        if self.memory is None or self.store is None:
+            return
+        snapshot = self.store.load_memory_snapshot(self.agent_id)
+        if not preserve_initial or snapshot["revision"] or snapshot["entries"]:
+            self.memory.load(snapshot["entries"], stored_count=snapshot["stored_count"])
+        self._memory_revision = snapshot["revision"]
+
+    def save_memory(self) -> None:
+        """Checkpoint the complete memory, including retrieval access counts.
+
+        Concurrent stale snapshots raise MemoryConflictError; they are never
+        merged blindly. Call load_memory and rerun the task after a conflict.
+        """
+        if self.memory is None or self.store is None:
+            return
+        expected = self._memory_revision if self._memory_revision is not None else 0
+        self._memory_revision = self.store.save_memory_snapshot(
+            self.agent_id, self.memory.dump(), stored_count=self.memory.stored_count,
+            expected_revision=expected,
+        )
+
     def get_state(self) -> AgentState:
-        return self.state
+        state = deepcopy(self.state)
+        if self.memory is not None:
+            state.memory_entries = self.memory.dump()
+            state.metadata["memory_stored_count"] = self.memory.stored_count
+        return state
 
     def load_state(self, state: AgentState) -> None:
-        self.state = state
-        if self.memory and state.memory_entries:
-            self.memory.load(state.memory_entries)
+        self.state = deepcopy(state)
+        self.agent_id = state.agent_id
+        self._memory_revision = None
+        if self.memory is not None:
+            self.memory.load(state.memory_entries,
+                             stored_count=state.metadata.get("memory_stored_count", 0))
 
     def __repr__(self) -> str:
         return f"BaseAgent(id={self.agent_id}, model={self.model})"

src/self_evolving/core/environment.py

@@ -53,6 +53,12 @@ class SimpleQAEnvironment(Environment):
 
     def __init__(self, qa_pairs: list[tuple[str, str]]):
         super().__init__("simple_qa")
+        if not qa_pairs or any(not question.strip() or not answer.strip()
+                               for question, answer in qa_pairs):
+            raise ValueError("QA pairs require nonempty questions and reference answers")
+        questions = [question.strip().casefold() for question, _ in qa_pairs]
+        if len(set(questions)) != len(questions):
+            raise ValueError("QA questions must be unique")
         self._qa_pairs = qa_pairs
         self._current_answer: str = ""
         self._done: bool = False
@@ -65,9 +71,13 @@ def reset(self, goal: str) -> str:
             if question.strip().lower() == goal.strip().lower():
                 self._current_answer = answer
                 break
+        else:
+            raise ValueError(f"No reference answer for goal: {goal!r}")
         return f"Question: {goal}"
 
     def step(self, action: str) -> tuple[str, Feedback, bool]:
+        if not self._current_answer:
+            raise RuntimeError("reset must select a valid QA goal before step")
         self._step_count += 1
         correct = self._current_answer.lower() in action.lower()
         self._done = True