Skip to content

needs-human-run: commit one sanitised evaluation run per track (garak / pyrit / laaf) #39

Description

@emmanuelgjr

What

evals/ ships runnable security-test profiles for three tools (Garak, PyRIT, LAAF) mapped to OWASP GenAI entries, but no committed run evidence — nothing in the repo demonstrates that a single profile has ever executed end to end. T-ENG05 (#38) parameterized every target and documented the procedure; this issue is the part only a human can do.

Why an agent can't do it

Producing a real run requires all three, none available to an automated contributor here:

  1. Tools installedpip install garak pyrit + the LAAF repo (git+https://github.com/qorvexconsulting1/laaf-V2.0.git).
  2. A credential + written authorisation to run adversarial prompts against a model deployment (see the authorisation notice atop evals/README.md).
  3. Budget — roughly $0.10–1.00 per track with a small hosted model.

Fabricating or hand-editing results is explicitly out of bounds (evals/samples/README.md, hard rule "never fabricate results").

Definition of done

Follow evals/samples/README.md and commit one sanitised run per track into evals/samples/{garak,pyrit,laaf}/, each with a RUN.md provenance block (run date, tool version, target label, authorisation, outcome, sanitisation done). Then:

  • evals/samples/garak/report.jsonl per profile + SUMMARY.txt (no hit logs)
  • evals/samples/pyrit/ — per-script summary blocks (no echoed model output)
  • evals/samples/laaf/crosswalk-report.md + per-stage JSON with payload/response arrays stripped
  • Update the "Committed run evidence" row in evals/README.md
  • Revisit evals/THRESHOLDS.md — the first real pass rates are the moment to review the DRAFT thresholds (SME sign-off)

The evals.test.mjs guard already requires a RUN.md beside any committed sample, so a results dir without provenance fails CI.

Context

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-human-runRequires a human to execute (credential/authorisation/budget); agents must not fabricate the result

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions