What
evals/ ships runnable security-test profiles for three tools (Garak, PyRIT, LAAF) mapped to OWASP GenAI entries, but no committed run evidence — nothing in the repo demonstrates that a single profile has ever executed end to end. T-ENG05 (#38) parameterized every target and documented the procedure; this issue is the part only a human can do.
Why an agent can't do it
Producing a real run requires all three, none available to an automated contributor here:
- Tools installed —
pip install garak pyrit + the LAAF repo (git+https://github.com/qorvexconsulting1/laaf-V2.0.git).
- A credential + written authorisation to run adversarial prompts against a model deployment (see the authorisation notice atop
evals/README.md).
- Budget — roughly $0.10–1.00 per track with a small hosted model.
Fabricating or hand-editing results is explicitly out of bounds (evals/samples/README.md, hard rule "never fabricate results").
Definition of done
Follow evals/samples/README.md and commit one sanitised run per track into evals/samples/{garak,pyrit,laaf}/, each with a RUN.md provenance block (run date, tool version, target label, authorisation, outcome, sanitisation done). Then:
The evals.test.mjs guard already requires a RUN.md beside any committed sample, so a results dir without provenance fails CI.
Context
What
evals/ships runnable security-test profiles for three tools (Garak, PyRIT, LAAF) mapped to OWASP GenAI entries, but no committed run evidence — nothing in the repo demonstrates that a single profile has ever executed end to end. T-ENG05 (#38) parameterized every target and documented the procedure; this issue is the part only a human can do.Why an agent can't do it
Producing a real run requires all three, none available to an automated contributor here:
pip install garak pyrit+ the LAAF repo (git+https://github.com/qorvexconsulting1/laaf-V2.0.git).evals/README.md).Fabricating or hand-editing results is explicitly out of bounds (
evals/samples/README.md, hard rule "never fabricate results").Definition of done
Follow
evals/samples/README.mdand commit one sanitised run per track intoevals/samples/{garak,pyrit,laaf}/, each with aRUN.mdprovenance block (run date, tool version, target label, authorisation, outcome, sanitisation done). Then:evals/samples/garak/—report.jsonlper profile +SUMMARY.txt(no hit logs)evals/samples/pyrit/— per-script summary blocks (no echoed model output)evals/samples/laaf/—crosswalk-report.md+ per-stage JSON with payload/response arrays strippedevals/README.mdevals/THRESHOLDS.md— the first real pass rates are the moment to review the DRAFT thresholds (SME sign-off)The
evals.test.mjsguard already requires aRUN.mdbeside any committed sample, so a results dir without provenance fails CI.Context
evals/THRESHOLDS.md(25 rows, all DRAFT)