fix: apply 2026-09-05 skill-audit findings and release five plugins

.github/workflows/validate.yml

@@ -8,6 +8,7 @@ on:
       - ".claude-plugin/marketplace.json"
       - ".github/workflows/validate.yml"
       - "plugins/**"
+      - "scripts/**"
 
 # Cancel superseded runs on the same PR; never cancel default-branch pushes.
 concurrency:
@@ -95,110 +96,33 @@ jobs:
     steps:
       - uses: actions/checkout@v7
 
+      - uses: actions/setup-python@v7
+        with:
+          python-version: '3.11'
+
+      - name: Install audit validation dependencies
+        run: python3 -m pip install -r scripts/requirements-audit.txt
+
       - name: Validate in-repo plugin structure
+        run: python3 scripts/validate_plugins.py
+
+      - name: Test audit tooling and hook regressions
+        run: python3 -m unittest discover -s scripts -p 'test_*.py'
+
+      - uses: actions/setup-node@v7
+        with:
+          node-version: '22'
+
+      - name: Install official Claude plugin validator
+        run: npm install -g @anthropic-ai/claude-code@2.1.258
+
+      - name: Validate plugins with the target host
+        env:
+          CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1'
         run: |
-          python3 - <<'PY'
-          import json, os, re, stat, sys
-
-          errors = []
-          plugin_dirs = sorted(
-              d for d in (os.path.join("plugins", n) for n in os.listdir("plugins"))
-              if os.path.isdir(d)
-          )
-          if not plugin_dirs:
-              sys.exit("FAIL: no plugin directories under plugins/")
-
-          for pdir in plugin_dirs:
-              tag = os.path.basename(pdir)
-              mpath = os.path.join(pdir, ".claude-plugin", "plugin.json")
-              try:
-                  with open(mpath) as f:
-                      manifest = json.load(f)
-              except FileNotFoundError:
-                  errors.append(f"{tag}: missing {mpath}")
-                  continue
-              except json.JSONDecodeError as e:
-                  errors.append(f"{tag}: {mpath} invalid JSON — {e}")
-                  continue
-
-              name = manifest.get("name", "")
-              if not re.fullmatch(r"[a-z0-9-]+", name):
-                  errors.append(f"{tag}: plugin.json 'name' must be kebab-case, got {name!r}")
-              if not re.fullmatch(r"\d+\.\d+\.\d+", manifest.get("version", "")):
-                  errors.append(f"{tag}: plugin.json 'version' must be semver, got {manifest.get('version')!r}")
-              if not manifest.get("description"):
-                  errors.append(f"{tag}: plugin.json 'description' is empty")
-
-              hooks_ref = manifest.get("hooks")
-              hooks_path = None
-              if isinstance(hooks_ref, str):
-                  hooks_path = os.path.normpath(os.path.join(pdir, hooks_ref))
-                  if not os.path.isfile(hooks_path):
-                      errors.append(f"{tag}: plugin.json 'hooks' points at missing file {hooks_ref}")
-                      hooks_path = None
-              elif os.path.isfile(os.path.join(pdir, "hooks", "hooks.json")):
-                  hooks_path = os.path.join(pdir, "hooks", "hooks.json")
-
-              if hooks_path:
-                  try:
-                      with open(hooks_path) as f:
-                          hooks_cfg = json.load(f)
-                  except json.JSONDecodeError as e:
-                      errors.append(f"{tag}: {hooks_path} invalid JSON — {e}")
-                      hooks_cfg = {}
-                  events = hooks_cfg.get("hooks")
-                  if not isinstance(events, dict) or not events:
-                      errors.append(f"{tag}: {hooks_path} must have a non-empty top-level 'hooks' object")
-                  else:
-                      for event, matchers in events.items():
-                          if not isinstance(matchers, list):
-                              errors.append(f"{tag}: hooks.{event} must be an array")
-                              continue
-                          for m in matchers:
-                              for h in m.get("hooks", []):
-                                  if h.get("type") != "command":
-                                      continue
-                                  cmd = h.get("command", "")
-                                  for rel in re.findall(r"\$\{CLAUDE_PLUGIN_ROOT\}\"?(/[^\s\"']+)", cmd):
-                                      script = os.path.normpath(pdir + rel)
-                                      if not os.path.isfile(script):
-                                          errors.append(f"{tag}: hook command references missing file {rel}")
-                                      elif not os.stat(script).st_mode & stat.S_IXUSR:
-                                          errors.append(f"{tag}: hook script {rel} is not executable (chmod +x)")
-
-              skills_dir = os.path.join(pdir, "skills")
-              if os.path.isdir(skills_dir):
-                  for sname in sorted(os.listdir(skills_dir)):
-                      spath = os.path.join(skills_dir, sname, "SKILL.md")
-                      if not os.path.isdir(os.path.join(skills_dir, sname)):
-                          continue
-                      if not os.path.isfile(spath):
-                          errors.append(f"{tag}: skills/{sname}/ has no SKILL.md")
-                          continue
-                      with open(spath) as f:
-                          text = f.read()
-                      fm = re.match(r"\A---\n(.*?)\n---\n", text, re.DOTALL)
-                      if not fm:
-                          errors.append(f"{tag}: skills/{sname}/SKILL.md missing frontmatter block")
-                          continue
-                      for field in ("name", "description"):
-                          if not re.search(rf"^{field}:\s*\S", fm.group(1), re.MULTILINE):
-                              errors.append(f"{tag}: skills/{sname}/SKILL.md frontmatter missing '{field}'")
-
-          with open(".claude-plugin/marketplace.json") as f:
-              mp = json.load(f)
-          for entry in mp.get("plugins", []):
-              src = entry.get("source")
-              if isinstance(src, str) and not os.path.isdir(src):
-                  errors.append(f"catalog: {entry.get('name')} relative source {src} does not exist")
-
-          if errors:
-              print("plugin validation FAILED:")
-              for e in errors:
-                  print(f"  - {e}")
-              sys.exit(1)
-          print(f"OK: {len(plugin_dirs)} in-repo plugin(s) valid")
-          PY
+          for plugin in plugins/*; do
+            claude plugin validate "$plugin"
+          done
 
       - name: Shellcheck hook scripts
         run: |

README.md

@@ -30,7 +30,7 @@ ranked, `file:line`-cited findings and leaves fixes to you.
 | `/perf-audit` | Budgets declared vs enforced, sizes vs stated limits, shipped-asset hygiene |
 | `/devops-audit` | CI/CD + deploy configs: action pinning, injection, secret handling, release/migration safety, local-vs-CI gate drift |
 | `/audit-fix` | Applies an audit report: mechanical fixes with exact-match edits, re-verified by the report's own methods; decisions stay decisions |
-| `/plugin-release` | Cuts an in-repo plugin release: semver bump, three-surface description sync, JSON validation, conventional commit; --dry-run prints the diff |
+| `/plugin-release` | Cuts an in-repo plugin release: semver bump, three-surface description sync, JSON validation, conventional commit; --dry-run previews the diff from a temporary copy and leaves the tree untouched |
 | `/skill-validate` | Blind-agent validation harness: pinned ground truth, reproduction/discrimination/fidelity scoring |
 | `/adr` | Architecture Decision Records, numbered and templated from the repo's own conventions |
 | `/pr-description` | PR descriptions from the branch diff, template-aware |

docs/MAINTAINING.md

@@ -109,7 +109,7 @@ absence produced a real, verified bug — none are style preferences.
    miss, and an unrelated directory. Zero blank outputs, zero stderr leaks.
 
 3. **Descriptions carry trigger phrases, not just capability summaries** — skills
-   undertrigger by default. Name the stack when the skill is stack-specific, so it
+   need positive and negative discovery tests on the supported host/model. Name the stack when the skill is stack-specific, so it
    doesn't fire (or collide with a sibling) in the wrong repo.
 
 4. **Audit skills are assessment-only.** They report ranked findings cited `file:line`,

docs/SKILL_AUDITS.md

@@ -0,0 +1,81 @@
+# Skill audits in Ops
+
+The first owned-skill audit is [2026-09-05](audits/2026-09-05/REPORT.md).
+Its JSON record defines the rubric, scope, findings, source hashes, and per-entity
+scores. The Markdown report is the readable companion. Scoring is a reviewed
+judgment, not an automated lint score or a measured model success rate.
+
+The post-fix verification, run against the release commits that closed all eleven
+findings, is [2026-09-05-post-fix](audits/2026-09-05-post-fix/REPORT.md). A
+subsequent complete audit like this one clears resolved findings in ops by sending
+an empty findings summary at severity 0 for every entity it re-reviews.
+
+## Publish a reviewed audit
+
+Generate and inspect the payload first:
+
+```sh
+python3 scripts/publish_skill_audit.py docs/audits/2026-09-05/audit.json \
+  --output /tmp/skill-audit-payload.json
+```
+
+Set `OPS_URL` to the deployment URL and supply `OPS_INGEST_TOKEN` through your
+secret manager, then submit the same reviewed record:
+
+```sh
+python3 scripts/publish_skill_audit.py docs/audits/2026-09-05/audit.json --publish
+```
+
+For 1Password CLI, a local env file can contain the secret reference
+`OPS_INGEST_TOKEN=op://Personal/OPS_INGEST_TOKEN/credential`; use that file with
+`op run --env-file <local-env-file> -- python3 scripts/publish_skill_audit.py ... --publish`.
+Never put the resolved token in a report, command argument, or committed file.
+The publisher refuses HTTP and redirects, uses a 30-second timeout, and verifies
+the response counts. It never prints the token or server error body.
+
+No recurring audit job is installed. Run and review audits explicitly; publishing
+does not run skill instructions or alter the audited packages.
+
+## Signal contract
+
+All five metrics are state snapshots. Each record is associated with the existing
+`skill:<plugin>:<skill>` or `plugin:clownware/<plugin>` inventory ID:
+
+| Metric | Meaning |
+|---|---|
+| `skill_audit.review_score` | Review index, 0–100; never a capability percentage |
+| `skill_audit.findings` | Full finding summary and count; maximum finding severity drives triage |
+| `skill_audit.provenance` | Run, rubric, reviewer, source revision, content hash, and coverage |
+| `skill_audit.behavior` | Explicit description of tested and untested behavior |
+| `skill_audit.portability` | Cross-host validation status and outstanding adapter review |
+
+Do not mark absent behavioral tests as zero failures or 100% passed. An unknown
+status carries no numeric success value. Plugin packaging scores and average skill
+scores answer different questions and must remain separate.
+
+The dedupe key combines run ID and the full report hash. An exact retry is
+idempotent; changed evidence becomes a new record. Preserve observation time on
+retries. A subsequent complete audit sends an empty findings summary at severity
+0 for resolved entities. Do not use a partial audit to clear unreviewed findings.
+
+The publisher omits entity metadata because the inventory poller owns it. Source
+revision and hash are stored in the provenance signal instead. No dashboard
+deployment or schema migration is required; `skill_audit` appears as a findings
+domain. `/health` continues to describe pollers, not skill effectiveness.
+
+## Repeat the review
+
+1. Pin the source revision and content hash before reviewing. Inventory installed
+   copies separately; local source versions do not prove release availability.
+2. Read every entrypoint in scope, parse real YAML, run the host plugin validator,
+   and inspect references and executable dependencies relevant to the task.
+3. Execute reviewed probes in empty, matching, and unrelated contexts. Check their
+   meaning as well as exit status. Never automatically execute arbitrary Markdown.
+4. Record findings with exact locations, reproduction evidence, and proposed fixes.
+5. Use isolated blind fixtures for behavioral samples. Record host/model identity,
+   artifact criteria, tool availability, and any baseline comparison. Never claim
+   broad compatibility from one successful sample.
+6. Review the complete JSON record and generated payload, publish, and retain the
+   server acknowledgement beside the report.
+
+Validate the publisher with `python3 -m unittest discover -s scripts -p 'test_*.py'`.