test(ci): add JSONL audit compat checks for OpenSearch and Splunk - #1131
test(ci): add JSONL audit compat checks for OpenSearch and Splunk#1131chokoblitz wants to merge 3 commits into
Conversation
HatemMn
left a comment
There was a problem hiding this comment.
@chokoblitz thanks for the PR but in the current state something needs to be changed or it should be closed and kept as artifact/proof (i'll explain)
the two workflows test against a static fixture. The fixture was indeed been generated by the KMS, but if later on we change something in the KMS, those tests won't catch it, so there is no goal of the CI (for now)
Worse,
Images pinned (OpenSearch 3.8.0, Splunk 10.4.2), not :latest —
reproducibility over drift, per team discussio
Since they are pinned, when can't even (empiracally) discover a breaking change that Splunk or OpenSearch produce and makes them incompatible with our system (it's actually the one use case where a static fixture might be an acceptable idea)
Anyway, two paths:
- Save and archive this PR, save it to the internal docs (Notion) with any other useful info like configuring opensearch & splunk
- Make that file sample-audit.jsonL generated by a live KMS: that's the path I would prefer we take
feel free to give me your opinion on the matter
What
Two ephemeral (
--rm, no volume) CI checks validating that the KMS JSONL auditformat ingests cleanly into OpenSearch (bulk API, strict mapping) and Splunk
(HEC), with a shared Python validation script and versioned schema files
(
schemas/kms-audit-*.json). Follow-up to #934.Finding
resultfield is polymorphic ("Success"|{"Failure": "..."}) and silentlydrops from Splunk's default
_jsonfield extraction when it's an object. Bothbackends now get a normalized
result_status/result_detailpair instead, sothe comparison is apples-to-apples. Worth discussing upstream whether the audit
schema itself should avoid polymorphic fields — not blocking here.
Security note — CI-only relaxations
Both backends run with security features disabled, strictly scoped to the
ephemeral test container, never touching real KMS data or a production system:
DISABLE_SECURITY_PLUGIN=true— auth/TLS off, container is--rm, no volume, torn down at job end.script skips cert verification only when
--insecureis explicitly passed bythe task, never a script default.
Testing
Both pass locally (
mise run test:audit-compat-{opensearch,splunk}, 8/8 eventseach). Images pinned (OpenSearch 3.8.0, Splunk 10.4.2), not
:latest—reproducibility over drift, per team discussion.
Unrelated fixes bundled (pre-existing, blocked all commits on
develop)deny.toml: ignoreRUSTSEC-2026-0258(h2, low severity, real upgrade tracked separately)lychee.toml: excludesekretz.com(dead placeholder domain indeny.toml's own template comment)