feat: consolidate Experience and evaluate Skill revisions (WikiSkill)

[Teingi]

Feature description

Let PowerContext accumulate evidence about recurring task failures, consolidate that evidence into existing Experience, and evaluate a proposed Skill revision before an operator decides whether to approve and publish it.

This is a concrete application of the broader evaluation and improvement loop in #1422, informed by WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution. Its specific scope is cross-iteration Experience consolidation and reuse of previous Skill evaluation results.

User-visible example

A coding agent repeatedly changes HTTP APIs. Some tasks miss generated client updates; others run expensive checks that do not address the actual failure. PowerContext should:

1. Connect the new Task Outcomes to an existing Experience about API contract changes.
2. Propose an evidence-backed revision that refines the lesson and its applicability, preserving counterexamples.
3. Use approved Experience, exact Skill usage evidence, and previous unsuccessful interventions to propose one Skill change.
4. Compare the existing Skill and candidate on separate, reproducible tasks.
5. Show the operator the diff, evidence, regressions, and cost changes before Review and explicit publication.

If an intervention does not help, the current published Skill remains in place and the evaluation remains available to inform the next proposal.

Problem and proposed solution

Research basis

Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, and Tu Vu. WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution, arXiv:2608.27454v1, August 27, 2026.

• Abstract and publication metadata
• Full paper
• PDF

WikiSkill separates immutable task traces, a persistent knowledge wiki, and active Skills. It consolidates successes and failures across iterations, proposes one Skill change at a time, and accepts only changes that strictly improve validation performance. Rejected Skill changes are rolled back while knowledge and intervention history persist. The evaluation harness records proposal diffs and measured outcomes.

In the paper's Gemini ablation, giving the proposer persistent wiki access increases average performance across four benchmarks from 48.7 to 63.7. This supports investigating accumulated knowledge as an input to Skill improvement.

These are author-reported results, not reproduced PowerContext results. The study fully injects active Skills, uses validation sets of 10–40 tasks, and does not establish real-host triggering or very long-task effectiveness. Cross-model transfer can also hurt performance. Those limits motivate independent validation in PowerContext.

Existing foundations and precise gaps

Code references below are pinned to source master at 80d618bd, inspected on September 8, 2026.

Existing capability	Gap addressed by this proposal
ExperienceIncubationInput contains the current Task Outcome evidence window. The automatic writer proposes with target=None.	Automatic incubation does not supply related existing Experience or propose replacements for it. Similar lessons can accumulate independently across windows.
Explicit Experience and Skill generation resolves caller-selected exact evidence and creates pending Candidates.	No built-in orchestration assembles recurring patterns and previous intervention results for the next proposal.
SkillUsageCapture records an exact Skill/package, selection, invocation, validation, outcome, and an optional environment fingerprint.	Observed success after use does not establish improvement caused by a revision. A paired comparison and its durable evidence are still needed.
Candidate Review validates evidence, package content, and revision consistency before committing.	Review does not execute efficacy tests. Skill validation instructions are not measured validation results.
The evaluation service provides SWE-bench Pro execution and reporting; #1397 delivered the standard Skill lifecycle.	Reuse these foundations, adding exact candidate-package evaluation and Skill-version comparisons rather than treating PowerContext OFF/ON as the same experiment.

Proposed minimal implementation

1. Consolidate related Experience. Before generating from new Task Outcomes, retrieve a bounded set of relevant Experience in the authorized scope. Provide their exact revisions alongside the new evidence. Support three outcomes: a new Candidate, a replacement Candidate for an existing Experience, or no change. Preserve contradictory evidence and applicability limits rather than merging solely by textual similarity. Continue using existing Candidate/Review, lineage, and revision-conflict handling.

2. Persist intervention evidence. Begin with a versioned structured payload in a Content Source, with a stable run identity. Record the baseline Skill reference, exact Candidate ID/version and package digest, motivating Experience references, change description, task-set/version, model/Host/environment configuration, comparison results, and report references. Keep task failures separate from infrastructure failures, timeouts, cancellation, and insufficient evidence. The evaluator writes observed results; the proposer must not invent them. Candidate decisions remain authoritative in Review.

3. Assemble the next proposal from that history. Select relevant approved Experience, bounded usage evidence, previous intervention results, and the exact current Skill. Reuse existing origin=usage generation for replacement: the current contract requires the target Skill and Source evidence. origin=experience is the creation path and currently rejects a replacement target. Generate a complete replacement with a reviewable diff; do not discard scripts or other assets from an existing package. Start with instruction-only Skills, and defer automated edits of multi-file packages until exact asset preservation is supported.

4. Add a manually triggered evaluation path. Materialize the exact pending Candidate into an isolated evaluation workspace through a dedicated runner adapter. This does not make it approved or publishable. Compare it with the frozen baseline, persist results, and present a report alongside the existing Review workflow. A changed Candidate version invalidates the earlier evaluation recommendation. Evaluation must not automatically approve or publish. The workflow's proposer/evaluator must not receive approval or production-publication authority; align this with #1391 rather than assuming a pending state enforces human-only review.

For this first slice, use existing Source, Experience, Candidate, Skill, and publication identities. A manually invoked runner plus a stable evaluation record is sufficient to test the idea. A new Wiki Artifact family, general agent runtime, or autonomous background evolution service is not required.

Evaluation plan

Choose one recurring coding-task family with independently checkable results. Separate discovery, validation, and final test tasks, including duplicate/near-duplicate checks across splits. Compare:

Arm	Purpose
Current published Skill	Establish the operational baseline.
Skill modified from new task evidence only	Measure ordinary feedback-driven modification.
Skill modified with consolidated Experience and prior intervention history	Measure the additional value of persistent knowledge.

Freeze the model, Host, tools, task environment, Memory/Experience recall policy, and execution/optimization budgets. Use paired task reporting, vary execution order where appropriate, and repeat discordant cases. Report task success, regressions, latency, tokens/cost, actual Skill invocation when observable, and inconclusive/infrastructure outcomes. The final acceptance run must exercise the real Host's Skill discovery/loading path; unknown invocation must remain unknown.

Acceptance criteria

• A recurring task pattern across separate ingestion windows can revise one existing Experience through Review, with exact evidence and conflict handling; duplicate evidence can produce no change.
• A rejected or ineffective intervention remains retrievable by the next proposal workflow without making rejected Candidate content an approved Artifact or ordinary recalled instruction.
• Baseline and candidate evaluations identify exact content, Candidate version, workload, and environment; results cannot silently attach to a changed candidate.
• The runner distinguishes lack of evidence and infrastructure failure from measured task regression, and preserves the existing published Skill after an unsuccessful experiment.
• Passing evaluation neither approves nor publishes a Candidate; real deployment uses the existing explicit publication flow.
• At least one real coding-agent experiment reports all three arms and their limitations, including a no-improvement result if that is what the evidence shows.

Alternatives considered

• Keep appending standalone Experience: retains observations but does not refine a shared lesson across windows.
• Regenerate Skills from only the latest failure: loses counterexamples and previous unsuccessful interventions.
• Introduce a separate Markdown Wiki as authoritative storage: duplicates existing Artifact revision, evidence, scope, and Review semantics. A generated browsing view can be considered later.
• Use Skill usage success or Review approval as proof of efficacy: neither establishes a causal improvement over the baseline.
• Copy full Skill injection or automatic acceptance into normal operation: real-host discovery and operator-controlled publication need their own validation and authority boundaries. Normal Memory/Experience recall should remain available; its configuration is held fixed during the experiment.

Additional context

• Parent tracking scope: Tracking Issue: build a governed evaluation and improvement loop #1422.
• Existing Skill infrastructure: feat: complete the standard Skill package lifecycle #1397 and Standard Skill Package Lifecycle RFC.
• Related authority discussion: Review authority: approve_artifact_candidate is an agent-facing MCP tool, contradicting "a model … cannot approve … itself" (README.md:79, RFC 0051:35) #1391.
• Broader workload coverage: Tracking Issue: expand benchmark coverage #1263.

AI assistance: prepared with Codex from the cited paper and repository source. This issue proposes an implementation and evaluation plan; it does not claim a reproduced benchmark result.

[thunguo]

Plz assign to me, i'd like to do this.