add/testing tools orchestration

[kdenney]

🎟️ Tracking

📔 Objective

[github-actions]

Claude Code validation

Result: Issues found

Validated the bitwarden-testing-tools plugin as changed by this pull request, against baseline origin/main (2d8a75b) at head 3503de2. Coverage: one plugin validation, nine SKILL.md reviews, six agent definitions, and the plugin's script/reference/template/eval support files. No secrets, no committed settings.local.json, and no prompt-injection or review-directing content were found — the injection-shaped strings in the eval fixtures and the guardrail prose in the agents are the plugin defending itself, correctly classified and not reported as attacks.

The verdict is Issues found on one clause only: the agent Bash tool grants are wider than the changeset justifies. Everything else below is a warning that does not block.

Critical

None.

Major

Tool grants and permissions

• plugins/bitwarden-testing-tools/agents/playwright-test-runner/AGENT.md:10 — Four script grants are unanchored leading-wildcard suffix globs (Bash(*/bitwarden-testing-tools/skills/using-stripe-cli/scripts/stripe_cli.py *) and three siblings); agents/localhost-web-health-checker/AGENT.md:9 carries the same shape. Nothing anchors the * to the plugin's install directory, so a stale checkout, vendored copy, or any writable directory reproducing the trailing segments is auto-approved with no prompt. stripe_cli.py and external_trigger.py are the sharp ends — the POST-only, host-allowlist, and read-only-Stripe properties the pipeline relies on live inside those specific files. Fix: anchor the grants to ${CLAUDE_PLUGIN_ROOT}, or implement the PreToolUse hook that references/playwright-tool-policy.md:82 already names as the fix and records as "not yet implemented." The acknowledgement in the policy doc is not the mitigation. (This is the finding that sets the verdict.)
• plugins/bitwarden-testing-tools/agents/playwright-test-runner/AGENT.md:10 — The same patterns require bitwarden-testing-tools/skills/… to be adjacent, but a marketplace install puts a version segment between them (…/bitwarden-testing-tools/1.6.0/skills/…), so no rule matches on an installed plugin — only in a local checkout. Because line 20 forbids requesting out-of-allowlist tools, every mailcatcher read, external trigger, and Stripe query becomes a reported obstacle instead of a prompt. Fix: add a wildcard for the version segment, or move enforcement to the hook above. Same at agents/localhost-web-health-checker/AGENT.md:9.
• plugins/bitwarden-testing-tools/agents/playwright-test-case-scoper/AGENT.md:9 — Bash(git log:*) is granted but git log appears nowhere else in the plugin; the agent's skill declares only Bash(git diff:*). Fix: drop the unused grant.
• plugins/bitwarden-testing-tools/agents/localhost-web-health-checker/AGENT.md:9 — Grant omits Bash(playwright-cli:*), yet the agent makes playwright-cli a hard prerequisite (line 23) and its skill's step 3 drives the browser through it. A skill invoked inside a subagent runs under the subagent's grant, so the render-verify gate can never execute. Fix: add Bash(playwright-cli:*), matching playwright-test-runner.
• plugins/bitwarden-testing-tools/skills/start-playwright-test/SKILL.md:5 — allowed-tools omits Skill, but lines 221 and 270 instruct the orchestrator to invoke Skill(compiling-playwright-report) as the sole mechanism for obtaining the merge/render grants, so the report pipeline is unreachable. Fix: add Skill(bitwarden-testing-tools:compiling-playwright-report).
• plugins/bitwarden-testing-tools/skills/running-playwright-tests/SKILL.md:4 and plugins/bitwarden-testing-tools/skills/checking-localhost-web-health/SKILL.md:4 — Both restrict allowed-tools to their co-located scripts while their bodies route every browser action through Skill(playwright-cli). Fix: add Skill(playwright-cli) to each.

Versioning

• plugins/bitwarden-testing-tools/agents/playwright-test-case-scoper/AGENT.md:3 (also playwright-test-context-gatherer/AGENT.md:3, services-under-test-mapper/AGENT.md:3) — These carry version: 1.3.0 while plugin.json, marketplace.json, and the root README.md all read 1.6.0. The remaining three agents (localhost-web-health-checker, playwright-test-case-writer, playwright-test-runner) carry no version key at all, so a field-updating bump script skips them silently and the drift becomes permanent. .claude/CLAUDE.md lists agents/*/AGENT.md among the four files a bump must update. Fix: set all six to 1.6.0, or drop the field from all six and stop claiming agent-level versioning.

Correctness in skill procedures

• plugins/bitwarden-testing-tools/skills/mapping-services-under-test/SKILL.md:18 and plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/SKILL.md:24 — Both document git diff origin/main...HEAD -- <repo-path> without naming a working directory, but clients/ and server/ are separate checkouts under a non-repo container (per services.md:5 and playwright-test-case-scoper/AGENT.md:40), so the command either exits "not a git repository" or silently returns empty and drops a required service. The git -C form that would work is not covered by the Bash(git diff:*) grant. Fix: specify the per-repo working directory, widen the grant to the form actually used, and report a repo that could not be diffed rather than treating it as "no changes."
• plugins/bitwarden-testing-tools/skills/mapping-services-under-test/SKILL.md:23 — Step 6 leaves the primary test URL undefined when the route list is empty, which is exactly the state step 5 exists to handle; localhost-web-health-checker/AGENT.md:37 parses that marker and stalls without it. Fix: default to the web vault (https://localhost:8080) when no routes are present.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/SKILL.md:38 — The "target state iff post-condition of a change-driven flow" rule is narrower than the artifact needs and forces the skill's own eval case 2 (past-due banner, reached via a Stripe test clock) to classify as setup, discarding its verification points. Fix: define target by assertion role rather than producer provenance.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/SKILL.md:97 — Produced by: none is bound to one meaning ("reachable out-of-band"), but line 36 mints route-only setup states with no producer that are playwright-reachable, so they have no valid encoding and the author is pushed toward a fabricated Reach via: recipe and a spurious [HUMAN] pause. Fix: allow none for both cases and disambiguate on the Reachable by playwright: line.
• plugins/bitwarden-testing-tools/skills/running-playwright-tests/SKILL.md:68 — Two rules in the same section prescribe different outputs for the same event: the exit-4 handler says "mark the affected test case FAIL," but Step 2 is pre-first-case setup where no test case exists, and line 74 requires an abort with no cases. Fix: split the handling by context.
• plugins/bitwarden-testing-tools/skills/start-playwright-test/SKILL.md:58-63 — The Task 2 dispatch block passes only Input type: / Input value:, but playwright-test-context-gatherer/AGENT.md:77 requires the fence token to be named in the task prompt; it currently arrives only incidentally, embedded in guardrail prose at line 47. Picking the wrong token hard-halts the run at line 73. Fix: add an explicit Fence token: <nonce> line.
• plugins/bitwarden-testing-tools/skills/using-stripe-cli/SKILL.md:21 — "This is enforcement in code, not an instruction" overstates what the code delivers: the only guard is a sk_live_/rk_live_ prefix check on STRIPE_API_KEY plus never passing --live. A live key in the CLI's own profile (~/.config/stripe/config.toml) is never inspected, so reads would route to live data while the text says that is impossible. The skill's whole safety story rests on that sentence. Fix: qualify the claim to what is actually enforced and tell the operator to confirm the linked account and mode before first use.

Descriptions and stale references

• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:3 — Advertises "screenshot checkpoints" as an output, but the skill never mentions screenshots and its mandated output block has no screenshot field; that naming is owned by running-playwright-tests. Fix: drop the clause or replace it with what the skill actually emits.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:3 — States neither the hard precondition (an ## Application Context section from scoping-playwright-test-cases, per line 10) nor that it is pipeline-dispatched, so a direct request dead-ends in the line-16 error path instead of routing to start-playwright-test. Every sibling carries that guard. Fix: append a precondition and non-trigger clause.
• plugins/bitwarden-testing-tools/skills/start-playwright-test/evals/README.md:9 and plugins/bitwarden-testing-tools/CHANGELOG.md:12 — Both reference a qa-testing-notes skill. No skill by that name exists anywhere in the repo, on this branch or on origin/main; the near-miss query at evals/trigger-eval.json:20 actually targets writing-manual-test-cases, a same-plugin sibling. Fix: replace the name in both places and drop "is a separate skill."

Minor

Security-adjacent hardening

• plugins/bitwarden-testing-tools/skills/using-stripe-cli/scripts/stripe_cli.py:94 — check_path rejects a non-/v1/ prefix, whitespace, and flag-like segments but not .., so check_path("/v1/../v2/accounts") passes. The sibling check_clock_id docstring calls out exactly this traversal risk for a value with the same untrusted provenance, and tests it. Impact is bounded (always a GET, no caller flags forwarded), but it lets model-constructed argv reach paths outside the audited /v1/ surface. Fix: reject .. path segments and add the mirror-image test.
• plugins/bitwarden-testing-tools/skills/running-playwright-tests/playwright.config.json:4 — ignoreHTTPSErrors: true disables certificate validation for every origin the test browser reaches, not just the self-signed dev hosts. The same PR takes the opposite approach in external_trigger.py:124-139, deliberately narrowing its TLS bypass to an allowlist, and playwright-tool-policy.md:80 states the navigation allowlist is an unenforced instruction. Fix: scope it to the dev origins, or record the gap in the policy's "Known limits."
• plugins/bitwarden-testing-tools/skills/reading-mailcatcher-api/scripts/read_mailcatcher.py:205 — --link-filter is a caller-supplied regex with no runtime bound, so a backtracking pattern hangs the script; re.error is caught but timeouts are not. Fix: constrain the filter to a literal alternation, or cap candidate length.
• plugins/bitwarden-testing-tools/scripts/running-evals.md:33 — The documented export PATH="$SHIM:$PATH" silently applies --setting-sources project to every later claude invocation in that shell, dropping user-level hooks until cleanup. Fix: scope the shim to a subshell, or warn at the export line.

Consistency and duplication

• plugins/bitwarden-testing-tools/CHANGELOG.md:7 — 1.6.0, 1.5.0, and 1.4.0 are all dated 2026-08-24, earlier than 1.3.0 at 2026-09-03, so four consecutive releases run backwards. Fix: correct the three dates.
• plugins/bitwarden-testing-tools/skills/reading-mailcatcher-api/scripts/get_admin_email.py:1 — Duplicates running-playwright-tests/scripts/read_admin_email.py (both added here) with a divergent contract: --secrets-file vs --secrets-path, and exit 3 vs 4 for a missing secrets file. Two skills in one plugin now document contradictory failure contracts for the same operation. Fix: keep one implementation, or align flags and exit codes and note the shared contract in both.
• plugins/bitwarden-testing-tools/skills/running-playwright-tests/SKILL.md:171-174 — Cites the tool policy as holding the canonical script path, then hardcodes that path inline, which the policy explicitly forbids. Fix: reference the policy's canonical path instead of restating it.
• plugins/bitwarden-testing-tools/skills/running-playwright-tests/SKILL.md:229-250 — Step 4 restates both abort shapes verbatim from Steps 2 and 3, and the case/step schema at lines 102-139 duplicates compiling-playwright-report/references/results-schema.md field for field, giving the JSON contract four copies to edit in lockstep. Fix: collapse to pointers.
• plugins/bitwarden-testing-tools/skills/checking-localhost-web-health/SKILL.md:37 — Inlines a third copy of the accepted-service-names list, which the sync comment in scripts/health-check.sh:18-21 does not cover (it names only two places). Line 15 already defers correctly. Fix: drop the inline list, or extend the sync comment.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:13 — Field named Post-condition state: but the producing skill emits Post-condition state(s):, and line 38 of the same file uses the plural form. Fix: align on the canonical name.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:34 — Third copy of test-master-password-12, this one without the "local dev fixture, never a real account credential" disclaimer the canonical definition carries. Fix: add the disclaimer clause.
• plugins/bitwarden-testing-tools/agents/services-under-test-mapper/AGENT.md:36 — Points at references/services.md as a bare path that resolves neither from the agent directory nor the plugin root; the file is at skills/mapping-services-under-test/references/services.md. Fix: use the fully qualified ${CLAUDE_PLUGIN_ROOT} form.
• plugins/bitwarden-testing-tools/skills/using-stripe-cli/SKILL.md:110 and plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:108 — Bare or mixed ${CLAUDE_SKILL_DIR}/${CLAUDE_PLUGIN_ROOT} reference paths where sibling lines use the qualified form. Fix: use one form consistently per file.
• plugins/bitwarden-testing-tools/skills/reading-mailcatcher-api/SKILL.md:5 — Grants restate the skill's own directory name inside a ${CLAUDE_PLUGIN_ROOT} path, so renaming the directory silently breaks auto-approval; siblings use the rename-proof ${CLAUDE_SKILL_DIR} form. Fix: switch, or record why the longer form is needed.

Triggering and description quality

• plugins/bitwarden-testing-tools/skills/checking-localhost-web-health/SKILL.md:3, running-playwright-tests/SKILL.md:3, scoping-playwright-test-cases/SKILL.md:3, start-playwright-test/SKILL.md:3 — None carries a negative-scope clause, though each requires inputs only the orchestrator supplies; every user-facing sibling in this plugin ends with a "Do NOT use it to…" clause. This PR adds eight new same-plugin siblings with heavily overlapping vocabulary. Fix: append explicit scoping and exclusion clauses; re-run the trigger evals afterward.
• plugins/bitwarden-testing-tools/skills/compiling-playwright-report/SKILL.md:3 — Description is an inventory of contents with no "use when" clause, giving the selector no activation signal beyond the name. Fix: lead with a usage condition.
• plugins/bitwarden-testing-tools/skills/mapping-services-under-test/SKILL.md:3 — Names only routes as input while the body makes affected repos a second required input with no fallback if absent. Fix: name both inputs and add a contingency.

Procedure gaps

• plugins/bitwarden-testing-tools/skills/checking-localhost-web-health/SKILL.md:43 — Requires a YYYYMMDD-HHmm timestamp, but neither the skill nor its dispatching agent can reach a clock (Bash(date:*) is not granted), so the minute component must be fabricated or the run halts on a tool obstacle. Fix: grant Bash(date:*), or pass the orchestrator's run timestamp in as a documented input.
• plugins/bitwarden-testing-tools/skills/checking-localhost-web-health/SKILL.md:17 — <artifacts-output-dir>/screenshots/ is never created by anything in the pipeline, and this skill is its first writer. Fix: have start-playwright-test/SKILL.md:85 create it (it already holds Bash(mkdir *)) and state the precondition.
• plugins/bitwarden-testing-tools/skills/checking-localhost-web-health/SKILL.md:37 — The documented HEALTH_CHECK_TIMEOUT=<seconds> invocation form does not prefix-match the skill's own Bash(${CLAUDE_SKILL_DIR}/scripts/health-check.sh *) grant, so it prompts mid-run. Fix: parse the timeout from argv instead.
• plugins/bitwarden-testing-tools/skills/start-playwright-test/SKILL.md:85 — mkdir without -p fails on the first run in any repo, since the parent .playwright-testing-artifacts does not exist. The grant already permits the flag. Fix: state mkdir -p.
• plugins/bitwarden-testing-tools/skills/start-playwright-test/SKILL.md:14 — Directs the orchestrator to show "the usage line from this skill's argument-hint," a frontmatter field it may not have available in the body. Fix: inline the usage line.
• plugins/bitwarden-testing-tools/skills/running-playwright-tests/SKILL.md:68 — Only exit 4 of read_admin_email.py is documented; EXIT_USAGE = 2 is reachable and unspecified, unlike the mailcatcher block which branches all three codes. Fix: document exit 2.
• plugins/bitwarden-testing-tools/skills/using-stripe-cli/SKILL.md:32 — The exit-21 row says "report as an obstacle; don't work around it" while the wrapper's own stderr says "Unset it and retry," giving two conflicting remediations. Fix: align the cell with the script's message.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/SKILL.md:140 — Self-review check 1 requires every Produced by: slug to exist as a flow heading, with no exemption for the literal none that line 97 sanctions, and line 145 forbids self-fixing. Fix: exempt none.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/SKILL.md:42 — Pruning removes a state and its producing flow but not flows naming it as Precondition state:, leaving dangling references that fail check 1. Fix: extend the pruning rule.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/SKILL.md:73 — "Every step must be a real user interaction" bans the Category 2 and 3 non-UI steps the tool policy sanctions and the copied catalog flows already use. Fix: allow sanctioned non-UI steps, marked as such.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/SKILL.md:22 — The empty-union example contradicts references/services.md:16, under which clients/apps/web/** always yields Web, so the stated case can never reach the fallback. Fix: use a case that genuinely matches no Required by: clause.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/SKILL.md:54 — One ~250-word bullet carries the entire selector-grounding doctrine. Fix: move it to references/selector-grounding.md and leave a pointer.

Eval documentation

• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/evals/README.md:31 and :35 — Claim "24 expectations"; behavior-eval.json contains 22 (4/3/3/4/4/4). 24 is the count from the running-playwright-tests suite. Fix: correct to 22, or add the two missing expectations.
• plugins/bitwarden-testing-tools/skills/start-playwright-test/evals/README.md:22-33 — The inventory is introduced as "the ten sibling skills the orchestrator competes with" but lists ten entries including itself while omitting writing-manual-test-cases; the plugin ships eleven skills. The README says the inventory is what makes the readings meaningful. Fix: correct the list and the wording.
• plugins/bitwarden-testing-tools/skills/running-playwright-tests/evals/README.md:22 — Attributes the exit-1-versus-exit-3 distinction to "the tool policy's own words" and quotes a sentence absent from playwright-tool-policy.md; the distinction actually lives in SKILL.md:177-193. Fix: correct the attribution.

Cosmetic

• plugins/bitwarden-testing-tools/agents/localhost-web-health-checker/AGENT.md:8 — color: purple (and magenta at playwright-test-case-scoper/AGENT.md:8) sits outside the set validate-agent.sh recognizes, emitting an "Unknown color" warning. Fix: use a recognized value.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:1-4 — The only skill in the plugin with no allowed-tools, though its procedure performs two Read calls; it works only because the dispatching agent grants Read. Fix: add allowed-tools: "Read" for parity.
• plugins/bitwarden-testing-tools/skills/start-playwright-test/SKILL.md:165 — "## Optional review gate" sits unnumbered between Task 6 and Task 7, undercutting the Task 1-9 numbering for the step most likely to be referenced by number. Fix: number or retitle it.
• plugins/bitwarden-testing-tools/skills/mapping-services-under-test/SKILL.md:33 — Inline illustration writes the URL unbackticked while the worked example backticks every URL. Fix: align.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/SKILL.md:120 — The ## Flows schema writes field keys unbulleted while the catalog flows it says to copy "verbatim" write them bulleted, so verbatim copying yields two formats. Fix: align the schema with the catalog.

Excluded as pre-existing

Reported here for transparency, not counted in the severities above:

• .claude-plugin/marketplace.json:39, root README.md:19, and plugins/bitwarden-product-analyst/.claude-plugin/plugin.json show bitwarden-product-analyst at 0.1.6 where origin/main has 0.1.7. Verified this is main advancing (PR Update release notes skill #223) after the branch was cut, not a regression this branch introduced: the version is 0.1.6 both at the branch point (9060e9f) and at head, and no commit on this branch touched it. Merging without rebasing would nonetheless revert that plugin to 0.1.6 — rebase on origin/main before merge.
• plugins/bitwarden-testing-tools/scripts/eval_harness.py:52 — the shell_chains tuple omits >/<, so a redirect slips through the read-only carve-out. Byte-identical to the constant this PR relocated from assessing-test-coverage/evals/run_real_eval.py; neither introduced nor worsened here.

Checks run

Check	Status
Plugin structure	Skipped here — run by the workflow as a dedicated step; see the job log and check status
Marketplace	Skipped here — run by the workflow as a dedicated step; see the job log and check status
Version bump	Skipped here — run by the workflow as a dedicated step; see the job log and check status
Plugin validation (AI)	Ran — 1 plugin (bitwarden-testing-tools); findings above
Skill review (AI)	Ran — all 9 changed SKILL.md files reviewed individually; findings above
Configuration & security	Ran — 6 agent definitions plus plugin scripts, references, templates, and evals; findings above
Secret scan	Ran — clean; no credentials, no settings.local.json in the changeset
Prompt-injection (CWE-1427)	Ran — clean; injection-shaped strings are defensive eval fixtures and guardrail prose

The three script checks are not run by this review by design. Their absence here is not a pass — read their results from the workflow's own steps.