Skip to content

test(ci): run example plugin suites after standalone core tests - #633

Open
be-student wants to merge 1 commit into
DobermanCore:mainfrom
be-student:chore/ci/example-plugin-suites
Open

test(ci): run example plugin suites after standalone core tests#633
be-student wants to merge 1 commit into
DobermanCore:mainfrom
be-student:chore/ci/example-plugin-suites

Conversation

@be-student

Copy link
Copy Markdown
Contributor

Pull Request

Slice

What this PR does

Fixes #595. Root pytest discovery currently excludes the example plugins, so their own tests can regress while CI remains green. Add a Linux/Python 3.12 step that installs and tests each examples/plugin-* package after the standalone core suite. Root testpaths remains unchanged, and installation or pytest failures stop the step immediately. Future example packages are included by the same glob.

README and CONTRIBUTING describe the sequence and recommend a separate local environment for example tests.

Tests added (run in CI)

  • tests/unit/test_example_plugin_ci.py executes the actual CI shell with a local command shim: discovers a future example, preserves install-before-test order, and propagates failures from both commands. It also guards standalone-before-examples ordering and exclusion from root discovery. This shell-specific check skips Windows; the production step runs only on Linux.
  • Actual workflow step executed locally in a separate environment: 12 audit-sink, 9 detector, and 12 guardrail tests passed.
  • A temporary failing audit-sink test made the actual step exit 1; the probe was restored, then all 33 tests passed. No failing probe is committed.
  • Static checks: ruff, format, import boundaries, offline Markdown links, changelog validation, and parity generation passed.
  • Full core suite: 5,401 passed, 4 skipped; 92.54% coverage (90% gate), as an unprivileged user in Linux/Python 3.12.14 with Xvfb. The same process then executed the actual CI example step: all 33 tests passed..

Final Linux validation used the official python:3.12-bookworm image, Xvfb, and an unprivileged runner user. Earlier macOS runs exposed environment differences (temporary-path entropy, color settings, nested venv inheritance, and Tk threading), and an initial root container run bypassed the unreadable-directory fixture's permissions. The final run corrects those execution conditions without changing application code or existing assertions.

Changelog

  • CI plumbing has no user-visible behavior change; no fragment required under changelog.d/README.md.

Public-release safety (doberman-core only)

  • No enterprise/hosted code, proprietary detection, customer data, secrets, or commercial-license code added.
  • Core validation runs with no enterprise or example plugin packages installed.

Security checklist

  • Install/test errors fail the CI step; core decision behavior is unchanged.
  • No secret, private file, or unredacted prompt logged or committed.
  • Guardrail, learning, BLOCK/AUTH explanations, and enterprise import boundaries are unchanged.

Edge cases covered / Deviations from plan / Risks introduced

  • Future example discovery, install failure, test failure, and standalone isolation covered.
  • No production-code changes. No new dependencies or workflow permissions.
  • macOS local verification is not a substitute for the requested Linux CI run; upstream CI must confirm that leg.

Implemented with OpenAI Codex assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI never runs the examples/ plugin test suites

2 participants