Create tooling that supports streamlined setup of the QA Manual Regression Suites

[kendratodd]

🎟️ Tracking

📔 Objective

The objective of this PR is to automate the creation of our Partial and Full Regression test runs in Testmo, the tool we use for test case execution.

[github-actions]

Claude Code validation

Result: Issues found

Validated the new bitwarden-testmo-tools plugin (26 files, all additions) against merge base f855dfc, at head d912cb0. Coverage: plugin manifest and marketplace consistency, the creating-regression-runs skill, both bundled Python scripts, release-profiles.json, and all 16 spec files. Version parity is clean at 0.4.0 across plugin.json, marketplace.json, the root README.md catalog row, and the changelog. No hardcoded credentials and no prompt-injection content (CWE-1427) were found in any of the 23 plugin files — all _comment fields, docstrings, and the SKILL.md body were read in full.

The verdict is Issues found on the security clause: the first major finding introduces a new credential-disclosure path, and three files affirmatively document the opposite guarantee.

Critical

None.

Major

• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/scripts/testmo_create_run.py:32 — The Testmo API key is interpolated into curl's argv: "-H", f"Authorization: Bearer {KEY}". Command-line arguments are world-readable via ps auxww and /proc/<pid>/cmdline on Linux (no hidepid by default) and via ps on macOS, for the duration of every request — and setup_release_runs.py issues one such call per page per spec, so the exposure window is wide. This directly contradicts three guarantees shipped in the same changeset: the module docstring at :5 ("Never prints the API key"), plugins/bitwarden-testmo-tools/README.md:32 ("never printed or committed"), and SKILL.md:16-17 / SKILL.md:169 ("never print, echo, log" / "Never expose the key"). Fix: drop the curl subprocess for urllib.request with the header set in-process — this also removes the curl PATH dependency documented at SKILL.md:18, improving Windows portability. If curl must stay, pass the header via curl --config - on stdin, or -H @<file> with a 0600 temp file.

• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/scripts/testmo_create_run.py:39 — HTTP error responses are treated as success. curl is invoked without -f/--fail-with-body and exits 0 on 4xx/5xx, so the only guard (if out.returncode != 0) never fires. Two concrete failures: an expired TESTMO_API_KEY returning 401 makes fetch_all return [], so the dry-run prints Project 1: 0 total cases -> 0 match filter — indistinguishable from a spec that legitimately matches nothing, which is the exact signal the review workflow depends on. A 403 on the POST makes res.get("result", {}).get("id") return None, so :217 prints CREATED run id=None https://bitwarden.testmo.net/run/None — a rejected create reported as a success. Same defect at setup_release_runs.py:165-167, where the loop then continues through the remaining specs and ends with Done. All runs linked to milestone .... Additionally json.loads(out.stdout) at :41 is unguarded, so a proxy HTML error page or empty body raises a bare JSONDecodeError traceback. Fix: add -w "\n%{http_code}" (or --fail-with-body), split the status from the body, and sys.exit with both on non-2xx; wrap json.loads in try/except. In setup_release_runs.py, track per-run success and print a final tally of what was and was not created.

• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/SKILL.md:39 — The "sandbox first" guardrail cannot be executed as documented. Step 2 says "Dry-run against the sandbox (project 2) first" and gives python3 scripts/testmo_create_run.py --spec specs/<run>.json, but that script has no --project flag: the project comes solely from spec["project_id"] (testmo_create_run.py:154), and all 15 shipped specs plus specs/regression-run.template.json pin "project_id": 1. The documented command therefore hits live project 1, never the sandbox. SKILL.md:166 ("Sandbox first — validate new specs against project 2") and the "Test against the sandbox first" bullet in plugins/bitwarden-testmo-tools/README.md repeat the same unreachable instruction. Fix: add a --project override to testmo_create_run.py mirroring the one in setup_release_runs.py, or reword step 2 to say the spec's project_id must be temporarily set to 2.

• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/scripts/setup_release_runs.py:67 — The orchestrator ignores each spec's project_id, so the two entrypoints resolve the target project differently for the same spec. --project defaults to 1 and is used at :85, :98-99, :113-114, and :165, while testmo_create_run.py:154 treats spec["project_id"] as authoritative. A sandbox spec written with "project_id": 2 (per the guardrail above) and fed to setup_release_runs.py without an explicit --project 2 silently POSTs into project 1, the live Bitwarden repository. All shipped specs are project_id: 1 today, so the divergence is latent — but the template invites new specs and the documentation actively tells operators to make sandbox ones. Fix: read project_id from each spec and error out when it disagrees with an explicitly passed --project or with its sibling specs, rather than silently overriding.

• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/SKILL.md:60 — The schema example places "tags": ["regression"] at the top level (a run-payload key, consumed at testmo_create_run.py:184), but the prose immediately below — :71 "Every key under filters is optional" followed by the key list — documents tags as a case filter (:74, consumed at testmo_create_run.py:55). The example's filters block contains no tags key, so the two readings conflict. A reader following the prose to write a tag-only spec puts tags at the top level and leaves filters empty; resolve_folders then returns None, matches() returns True for every case, and the run is created containing all ~13.7k project-1 cases. The zero-case guard at :213 catches only the opposite error. Fix: split the schema section into "top-level keys" and "keys under filters", and show the tag filter as "filters": {"tags": ["oldnew"]} — which is exactly how the shipped specs/old-client-new-server-regression.json correctly does it, setting both.

• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/SKILL.md:41 — Every documented command uses a cwd-relative path (python3 scripts/testmo_create_run.py --spec specs/<run>.json), at :41, :49, :200, :202, and :215. An installed plugin lives in the plugin cache while Claude's working directory is the user's own repository, so both the script path and --spec specs/... resolve to nonexistent files. ${CLAUDE_PLUGIN_ROOT} is the supported mechanism and the established convention here — bitwarden-devops-engineer, bitwarden-security-engineer, bitwarden-delivery-tools, bitwarden-init, and claude-config-validator all use it; this plugin contains zero occurrences. Fix: prefix each invocation with ${CLAUDE_PLUGIN_ROOT}/skills/creating-regression-runs/. (setup_release_runs.py:22-24 already self-locates its specs via Path(__file__) and needs no change — only the documented paths do.)

• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/scripts/setup_release_runs.py:156 — The idempotency guardrail has no coverage on the path the skill calls "recommended". SKILL.md:167 states "Idempotent — do not create a duplicate run for a period/milestone," but the manual check is workflow step 4 (SKILL.md:45-46), which appears only in the single-spec workflow. The "Setting up a whole release (recommended)" section never repeats it, and the orchestrator performs no existing-run lookup, so re-running --create with the same --milestone-name silently creates a second full set of 14 runs against the live instance. Fix: have setup_release_runs.py fetch runs for the resolved milestone_id and refuse or warn on collision, or add an explicit "confirm no runs are already attached to this milestone (GET /projects/1/runs)" step before SKILL.md:202.

Minor

• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/scripts/testmo_create_run.py:56 — Tag names are interpolated into the query string unencoded ("tags=" + ",".join(...)), so a tag containing a space, &, #, or + produces a malformed URL. Combined with the missing status check above, the server's rejection surfaces as a silently empty case set — the exact failure SKILL.md:139-141 warns about. Fix: from urllib.parse import quote and encode each tag with safe="".
• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/scripts/setup_release_runs.py:33 — collect() does p = profiles[pname] on the extends target without validating it, so a typo'd parent raises an unhandled KeyError — inconsistent with the friendly sys.exit for an unknown top-level profile at :30. Two profiles extending each other recurse to RecursionError. Separately, --release common runs successfully despite release-profiles.json:2 and SKILL.md:193 stating it is "not meant to be run directly". Fix: validate parent in profiles with a sys.exit, track a visiting-set for cycles, and mark common abstract if it should be refused.
• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/SKILL.md:3 — The description carries no trigger vocabulary for the release-orchestration workflow, even though that workflow is labeled "(recommended)" at :184 and accounts for roughly a fifth of the body. "Set up all the regression runs for the 2026.8.0 full release" or "link this cycle's runs to the milestone" matches nothing in it, and bimonthly is narrow for the only cadence word present. Over-triggering risk is low — "Testmo" is specific. Fix: add release/milestone/profile vocabulary, e.g. "…set up a release's regression runs (full or partial), create or link runs to a Testmo milestone, select cases for a run…".
• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/SKILL.md:143 — Four lines document "config_id": null as "a deliberate placeholder", with matching refusal logic in both scripts, but no shipped spec can produce that state: nine specs carry real config ids and the rest omit the key entirely, so the --create refusal never fires for them. specs/regression-run.template.json has no config_id key either, so a user starting from the template will never encounter it. Fix: compress to one sentence (or leave it to the script docstrings), and add a commented config_id line to the template if the convention is meant to be discoverable.
• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/SKILL.md:98 — --period is documented as retained "for any spec that still uses the placeholder", but no file under specs/ contains the literal <period> (the template uses <Domain>). It is documented again at :225. Fix: drop the parenthetical and the :225 mention, or say plainly that no shipped spec uses it and it exists for custom specs.
• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/SKILL.md:232 — The "Not yet implemented (next steps)" section asks to "Seed the full profile's additional specs as they are captured", but all eight full-only specs (CLI, three Desktop, four Extension) ship in this same changeset and are listed in release-profiles.json. Shipping an already-done to-do invites duplicated work. Fix: delete the section.
• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/SKILL.md:20 — Progressive disclosure: the body is 1,792 words, comfortably inside the 1,000–3,000 target, but roughly 600 of those are pure lookup material the model does not need loaded to perform the task — "API reference" (:20-32), "Project 1 field reference" (:148-161), and "Multi-configuration runs" (:103-146), the last largely restated in each spec's _comment. Fix: move them to references/testmo-api.md and references/multi-config-runs.md with one-line pointers, leaving a core of roughly 1,100 workflow-focused words.
• plugins/bitwarden-testmo-tools/CHANGELOG.md:55 — The version sequence runs 0.1.0 → 0.2.0 → 0.3.1 → 0.4.0: there is no 0.3.0, and 0.3.1 is a ### Fixed-only entry that from 0.2.0 should have been 0.2.1. The whole file is introduced by this PR, so the history can be made consistent before it is published. Fix: renumber 0.3.1 to 0.2.1, or add the missing 0.3.0 entry.
• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/scripts/testmo_create_run.py:128 — if "automation_type_ids" in spec and auto_id not in spec["automation_type_ids"] guards on key presence rather than value, so "automation_type_ids": null in a spec raises TypeError: argument of type 'NoneType' is not iterable. Every sibling filter uses spec.get(...), which tolerates null. No shipped spec triggers it. Fix: guard with spec.get("automation_type_ids") is not None, matching the surrounding style.
• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/scripts/setup_release_runs.py:135 — results is appended at :127 for NEEDS CONFIG_ID rows as well as ready ones, so the footer {len(results)}/{len(spec_names)} runs ready and the TOTAL case count both include specs that :145-153 will refuse to create. Fix: count only rows whose status is ready, and report placeholder-blocked specs on a separate line.
• plugins/bitwarden-testmo-tools/skills/creating-regression-runs/scripts/testmo_create_run.py:152 — spec["run_name"].replace(...) raises KeyError when --period is passed for a spec without run_name, where the sibling at setup_release_runs.py:110 defensively uses spec.get("run_name", name). Fix: use .get() with a fallback for consistency.
• README.md:23 — The root catalog description ends mid-phrase: "…from reviewable filter specs with dry-run". Both the manifest and the marketplace entry end "with dry-run and milestone linkage." Fix: complete or trim the clause.

Notes

• Prompt injection (CWE-1427): none found. SKILL.md, both README files, the changelog, release-profiles.json, all 16 spec _comment fields, and both script docstrings were read in full. All content is legitimate Testmo domain documentation; nothing addresses a reviewer, claims repository policy, or attempts to direct this validation.
• No secrets committed. The key is read only from os.environ (testmo_create_run.py:27). The major finding above is a runtime disclosure path, not a committed credential.
• Verified consistent: version 0.4.0 across all four required files; the seven Desktop/Extension config_ids and the two *-essential tag ids in SKILL.md:118-122 match the shipped specs exactly; profile resolution yields full → 14 specs and partial → 7, and every one of the 15 referenced spec names resolves to a real file (the 16th file is the template, correctly not in any profile); all JSON parses; both scripts compile; prettier --check passes on the plugin's .md and .json files.
• Not applicable: the plugin ships no agents, commands, hooks, or MCP servers, so those component checks had nothing to validate. No CLAUDE.md or .claude/ path changed in this PR, so the .claude-pr/ snapshot mapping was not needed.

Checks run

Check	Status
Plugin structure	Skipped — run by the workflow as a dedicated step; see the job log and check status
Marketplace	Skipped — run by the workflow as a dedicated step; see the job log and check status
Version bump	Skipped — run by the workflow as a dedicated step; see the job log and check status
Plugin validation (AI)	Passed with findings — plugin-dev:plugin-validator over plugins/bitwarden-testmo-tools
Skill review (AI)	Passed with findings — plugin-dev:skill-reviewer over creating-regression-runs/SKILL.md
Configuration & security	Passed with findings — scripts, specs, and release-profiles.json; no config paths changed