add/testing tools test case writing

[kdenney]

🎟️ Tracking

📔 Objective

[github-actions]

Claude Code validation

Result: Issues found

Validated the bitwarden-testing-tools plugin changes in PR #211. This is a stacked pull request — its base is add/testing-tools-context-and-scoping, not main, so findings are scoped to this PR's own changeset (merge base 22c883d → cbe8d0e, 12 files: the new writing-playwright-test-cases skill and playwright-test-case-writer agent, the Category 3 addition to references/playwright-tool-policy.md, and pointer edits to known-flows/billing.md). The 47-file list supplied to this run is the whole stack measured against main; the other 35 files belong to earlier layers and were checked only for breakage this PR could cause — none was found.

No secrets and no prompt injection. Nothing in the changeset attempts to direct this review (no CWE-1427). test-master-password-12 is a documented local-dev fixture, server/dev/secrets.json is named as a path with no value, and the Stripe card numbers are public test-mode values. The new agent grants only Read, Skill — no Bash — and opens with an explicit untrusted-content preamble.

Version bump is complete and correct. 1.3.0 → 1.4.0 (MINOR, for a new skill + agent) is consistent across .claude-plugin/marketplace.json:105, plugins/bitwarden-testing-tools/.claude-plugin/plugin.json:3, the root README.md:22 catalog row, and the plugin README, with a Keep a Changelog entry.

The verdict is Issues found on the security clause below, not on a critical finding. There are no critical findings.

Critical

None.

Major

• plugins/bitwarden-testing-tools/references/playwright-tool-policy.md:30-34 — new Category 3 authorizes HTTP POSTs with no origin constraint. (security) Category 1 explicitly fences navigation to localhost, 127.0.0.1, ::1, or bitwarden.test (line 22). Category 3, added by this PR, names no such limit — it authorizes a POST to whatever endpoint a plan step names, and the pipeline's own inputs (Jira tickets, comments, linked issues) are explicitly untrusted per agents/playwright-test-case-writer/AGENT.md:11. Every example given is local, so the granted capability is wider than the changeset justifies. Exposure today is deferred rather than live: no agent in this PR holds Bash, so the planning layer cannot execute the POST — but the policy text addresses an execution log, and the executor lands in a later branch of this stack. Fix: carry Category 1's origin constraint into Category 3 — restrict external-trigger endpoints to the same four origins and state that any other origin is an obstacle to report, not a step to execute.
• plugins/bitwarden-testing-tools/references/playwright-tool-policy.md:28-46 — Category 3 is permitted but has no sanctioned transport. Categories 1, 2, and 4 each name an owning skill or script; Category 3 names none, and "Never Permitted" (line 57, untouched) bans "CLI tools not related to service startup" with the using-stripe-cli wrapper as its only exception — which forecloses curl. The Category 3 block is the edit that worsened this: it turned a heading with no body into an actively-instructed category while the exclusion stayed as-is. Net effect: a correctly classified Category 3 step is un-executable under the policy that authorizes it. Fix: name the mechanism in the Category 3 section (a canonical script under "Canonical script paths", or an explicit carve-out at line 57 covering the Category 2 mailcatcher reader and the Category 3 external-trigger call alongside the Stripe wrapper).
• plugins/bitwarden-testing-tools/references/playwright-tool-policy.md:38 contradicts plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/references/billing-test-data.md:42. Policy line 38 says applying a coupon to a subscription should "use the Admin portal or the web vault purchase flow"; billing-test-data.md:42 says discounts are "never added to a Stripe customer or subscription directly via the Stripe API, the Admin portal, or any other mechanism". Both lines are new in this PR. A discount test case can be emitted as SETUP: In the Admin portal, apply coupon <id> to the subscription — sanctioned by one file, impossible per the other. Fix: restate line 38 as the mechanism billing-test-data.md:44-47 documents — import the coupon in the Admin portal, then complete the web vault purchase flow as an eligible user, where checkout applies it automatically.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/references/known-flows/billing.md:161 uses a different label format from the one the policy mandates. playwright-tool-policy.md:44 requires "this exact form": EXTERNAL TRIGGER: POST <endpoint> — <rationale>. The catalog's only external-trigger entry reads **EXTERNAL TRIGGER**: POST to \http://localhost:33656/...\`. This simulates...— bolded, "POST to", period separator, no em dash.scoping-playwright-test-cases/SKILL.md:14instructs copying catalog entries **verbatim**, so the two rules collide on catalog-derived output. **Fix:** updatebilling.md:161` to the mandated form, or relax the policy from "this exact form" to a required-elements description.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/evals/behavior-eval.json:14 (and :10) — eval grades for a wrapper script that does not exist. Case 1 expects the model to "route the step through the external-trigger wrapper script rather than raw curl". No such script exists anywhere in the plugin (verified: the only scripts are read_mailcatcher.py, get_admin_email.py, stripe_cli.py, and eval harnesses), and "Canonical script paths" lists only the mailcatcher reader. The case cannot pass for a reason unrelated to skill quality. Fix: drop the wrapper expectation from both :10 and :14, or add the script and register it under "Canonical script paths".
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/evals/README.md:24-25, :31, :35 — grading notes are stale relative to the suite. Lines 24 and 25 quote expectations ("States the reasoning for the classification…", "States that direct database access is never permitted") that appear nowhere in behavior-eval.json — verified by grep, zero matches. Lines 31 and 35 both say the suite has "24 expectations"; the actual count is 22 (4+3+3+4+4+4). Fix: requote lines 22-27 against the current expectations or delete that block, and correct both counts to 22.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:3 — description advertises "screenshot checkpoints", which the skill never produces. The word "screenshot" appears nowhere in the plugin except this description and CHANGELOG.md:11. The output template (SKILL.md:124-140) is Starting URL / Precondition / Setup Steps / Test Steps / Notes, with no screenshot element. Descriptions drive skill selection, so this advertises a deliverable that does not exist. Fix: drop the phrase from the description and the changelog bullet, or add a screenshot-checkpoint step form to the output template.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:3 — description omits a hard precondition. It says "Use when you have plan context (file paths, acceptance criteria, UI flows)", but line 16 makes the skill return an error unless an ## Application Context section produced by scoping-playwright-test-cases is present. A caller matching the description exactly gets a refusal. Fix: fold the precondition into the trigger, e.g. "Use after scoping-playwright-test-cases has produced an ## Application Context, when plan context needs to become concrete test cases."

Minor

• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:20 — "no step may write to Stripe" over-constrains relative to playwright-tool-policy.md:50, which permits exactly one write (advancing an already-attached test clock). The skill's own template at line 130 uses a test clock, so as written it would refuse a legitimate renewal-test step. Fix: "…beyond the single permitted test-clock advance."
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:132 — the template's SETUP: Inspect <path> for <pattern> step form is a filesystem read matching none of the four categories that line 20 says every generated step must fall into. Fix: add a carve-out near line 20 for planning/inspection steps, or remove line 132.
• plugins/bitwarden-testing-tools/skills/scoping-playwright-test-cases/references/known-flows/billing.md:117 — a backticked file path was added inside the **Parameters:** field, where this catalog's notation is backticked parameter identifiers (compare line 133: **Parameters:** `orgName`, `billingEmail`, `planTier`). Since scoping-playwright-test-cases/SKILL.md:14 copies entries verbatim and its self-review checks that every declared parameter appears as a placeholder in Steps, the path can read as a fourth declared parameter with no match. Fix: **Parameters:** none — line 116 already carries the same pointer one line above.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:108 vs known-flows/billing.md:116-117,136 — the same new file is referenced as ${CLAUDE_SKILL_DIR}/references/billing-test-data.md in one place and ${CLAUDE_PLUGIN_ROOT}/skills/writing-playwright-test-cases/references/billing-test-data.md in the other. Both resolve; the inconsistency was introduced within a single PR. Fix: pick one form.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/evals/behavior-eval.json:72 — case 6 grades refusals recorded "in the test case's Notes: line", but SKILL.md documents Notes: only for coverage gaps and free-form notes, and the policy's Stop Condition (playwright-tool-policy.md:61-63) says to STOP and report instead. evals/README.md flags this silent-compliance tension for cases 2 and 3 but not case 6. Fix: document Notes: as the refusal channel, or add case 6 to the grading-caveat list.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/references/billing-test-data.md:42-49 restates the "never attach a coupon via the Stripe API" rule that using-stripe-cli already owns at SKILL.md:40 and references/redirecting-writes.md:7. That rule now lives in four places, and the drift is what produced the major finding above. Fix: trim to the app-specific mechanism and point at ${CLAUDE_PLUGIN_ROOT}/skills/using-stripe-cli/references/redirecting-writes.md for the redirect rule.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md:3 — no negative boundary, unlike every sibling skill in this plugin (assessing-test-coverage, writing-manual-test-cases, reading-mailcatcher-api, using-stripe-cli all carry a "Do NOT use it to…" clause). The plugin now ships three similarly named test-authoring skills. Fix: append a boundary naming scoping-playwright-test-cases and writing-manual-test-cases.
• plugins/bitwarden-testing-tools/skills/writing-playwright-test-cases/SKILL.md — no untrusted-data line, where assessing-test-coverage:12 and writing-manual-test-cases:12 both carry a CWE-1427 clause. The warning exists at the agent layer (AGENT.md:11), but the skill is directly invocable by any caller pasting ticket text. Fix: mirror the sibling one-liner.
• plugins/bitwarden-testing-tools/agents/playwright-test-case-writer/AGENT.md:3 — the description says "dispatched by the start-playwright-test skill", but no start-playwright-test skill or command exists in the repository. The three sibling agents carry the same dangling reference from the baseline, so this is a fourth instance of an established pattern rather than a regression; noting it so the orchestrator skill lands before or with the agents that name it. No <example> blocks in the description, which .claude/CLAUDE.md asks for — defensible here, since the agent is explicitly dispatch-only and must not be auto-delegated, and consistent with all three siblings.
• plugins/bitwarden-testing-tools/CHANGELOG.md:7 — ## [1.4.0] - 2026-08-24 carries the same date as ## [1.3.0] - 2026-08-24 (line 16); today is 2026-09-01. Fix: set the actual release date.

Checks run

Check	Status
Plugin structure	Skipped — run as a dedicated workflow step before this review; see the job log and check status
Marketplace	Skipped — run as a dedicated workflow step before this review; see the job log and check status
Version bump	Skipped — as above; separately verified by hand here as consistent at 1.4.0 across all four required files
Plugin validation (AI)	Passed with findings — plugin-dev:plugin-validator over plugins/bitwarden-testing-tools
Skill review (AI)	Passed with findings — plugin-dev:skill-reviewer over all 5 listed skills (using-stripe-cli needed a retry after an API 529; the retry completed)
Configuration & security	Passed with findings — reviewing-claude-config → reviewing-agent-definitions over the new agent, plus the changed skill support files; secret scan clean, no settings.local.json in the changeset