Skip to content

feat(nist-ir): emit ai_reasoning rows and canonicalise AI verdict fields - #1051

Merged
scottbrumley merged 1 commit into
mainfrom
fix/ai-reasoning-telemetry
Aug 18, 2026
Merged

feat(nist-ir): emit ai_reasoning rows and canonicalise AI verdict fields#1051
scottbrumley merged 1 commit into
mainfrom
fix/ai-reasoning-telemetry

Conversation

@scottbrumley

Copy link
Copy Markdown
Contributor

SOCPromoteAIPhaseOutput was the only place that saw a parsed AI result, and it wrote nothing anywhere. A prompt that ran and returned nothing looked identical to a prompt that never ran, because continueonerror: true on the aiTask means neither fails the phase. War room entries are the only other evidence and every entry endpoint returns 303 on this tenant.

ai_reasoning rows

One row to xsiam_socfw_ir_execution_raw per AI execution, via the same socfw-post-to-dataset / socfw_ir_execution_writer path SOCCommandWrapper uses. reasoning_status separates the cases: promoted, parsed_not_promoted, parse_error, no_output.

Structured verdict fields only — the narrative stays out; it is large, duplicates the layout, and is not what anyone queries on. verdict_field_count makes a thin result visible without unpacking the row.

The write cannot fail the phase; a failure goes to demisto.debug.

Canonical vocabulary

The model is not constrained to an enum, so it invents spellings. Observed on deathstar in one run: spread_level as multi-host, multiple_hosts, multi_host and lateral; compromise_level as host, host_and_identity, host_and_user, host_and_lateral, endpoint and confirmed; primary_entity_type including WindowsServer2019. Anything keyed on these silently misses most rows — spread_level = "multi-host" matched about a third.

Values are now snapped to canonical form before promotion, so context, layout and dataset share one vocabulary. An unrecognised value is not silently rewritten: the field becomes unknown, the original is kept at <field>_raw, and contract_unmapped names it. confirmed in compromise_level is a certainty claim rather than a scope, so it has no correct target and is flagged rather than mapped.

Verified on deathstar

Four replays. Before: four spellings of spread_level, six values of compromise_level. After: two and two, with 2 of 37 rows correctly flagged. The telemetry also caught a real parse_error (1 in ~120) that was previously invisible.

Prompts are tenant-local and versioned nowhere, so enforcement belongs here, in content that ships through the SDK and applies identically everywhere.

Labelled version:patch → 1.8.2.

@scottbrumley scottbrumley added the version:patch Bug fix or hotfix → x.x.N label Aug 17, 2026
@scottbrumley
scottbrumley merged commit 841fcaa into main Aug 18, 2026
14 of 24 checks passed
@scottbrumley
scottbrumley deleted the fix/ai-reasoning-telemetry branch August 18, 2026 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

version:patch Bug fix or hotfix → x.x.N

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants