Skip to content

Add multi-SKIP case to good-first-issue-sweep step-3 evals - #1072

Open
deepusnath wants to merge 1 commit into
apache:mainfrom
deepusnath:add-multi-skip-gfi-sweep-eval-case
Open

Add multi-SKIP case to good-first-issue-sweep step-3 evals#1072
deepusnath wants to merge 1 commit into
apache:mainfrom
deepusnath:add-multi-skip-gfi-sweep-eval-case

Conversation

@deepusnath

Copy link
Copy Markdown
Contributor

The step-3-present-proposals fixtures only ever exercised a single SKIP issue at a time, so the "summary count only, grouped by reason" rule (issue #1001) couldn't actually fail: a model that just prints the one issue's title/reason still looks like a valid one-line summary.

case-5-multi-skip adds 4 SKIP issues spanning all three skip reasons (2 security, 1 architectural, 1 deprecation) alongside READY/NEAR-MISS issues, so a regression that lists individual SKIP titles/reasons or miscounts a category is actually caught.

Summary

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other:

RFC-AI-0004 compliance

  • HITL — any new mutation is gated on explicit user confirmation
  • Sandbox — no new unrestricted host access; network reach declared in the adapter
  • Vendor neutrality — placeholders (<PROJECT>, <tracker>, <upstream>, <security-list>) used in all skill / tool prose (the check-placeholders prek hook is the mechanical gate)
  • Conversational + correctable — agentic-override path documented if behaviour is adopter-tunable
  • Write-access discipline — no autonomous outbound messages; drafts only, sent on confirmation
  • Privacy LLM — private content does not reach a non-approved LLM; redactor invoked where needed

Linked issues

Notes for reviewers (optional)

The step-3-present-proposals fixtures only ever exercised a single SKIP
issue at a time, so the "summary count only, grouped by reason" rule
(issue apache#1001) couldn't actually fail: a model that just prints the one
issue's title/reason still looks like a valid one-line summary.

case-5-multi-skip adds 4 SKIP issues spanning all three skip reasons
(2 security, 1 architectural, 1 deprecation) alongside READY/NEAR-MISS
issues, so a regression that lists individual SKIP titles/reasons or
miscounts a category is actually caught.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants