KQL docs: plain-language rewrite, collector attribution, silent-collector triage #25
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,22 +17,48 @@ configures: a | |
| [Data Collection Rule](https://learn.microsoft.com/azure/azure-monitor/agents/data-collection-windows-events) | ||
| whose XPath list reads the `ForwardedEvents` channel. | ||
|
|
||
| An analogy that holds up well: the workspace is a filing cabinet and each | ||
| table is a drawer. | ||
|
|
||
| - **Two drawers look similar.** `SecurityEvent` is a *different* drawer | ||
| with its own clerk: the *Windows Security Events via AMA* connector | ||
| files a machine's **own** Security log into `SecurityEvent`. Put that | ||
| clerk on a collector and it files the collector's own activity - the | ||
| thousands of forwarded events sitting in its ForwardedEvents log are | ||
| ignored. Practical consequence: a detection that only searches | ||
| `SecurityEvent` never sees anything that travelled via WEF (ASIM | ||
| parsers union both drawers - see | ||
| [normalisation](https://learn.microsoft.com/azure/sentinel/normalization)). | ||
| - **Every document keeps its original letterhead.** Everything physically | ||
| passed through the collector, but each event still records the machine | ||
| that created it: `Computer` is the **original source**, not the | ||
| collector. That is what makes fleet verification possible from the | ||
| workspace end - you can list exactly which servers are represented | ||
| without logging into anything. | ||
| - **The envelope is thrown away; the letter is kept.** ForwardedEvents | ||
| was only the transport envelope. Once filed, each event shows its | ||
| *original* log name (`Channel` = `Security` and so on), so "everything | ||
| that came via forwarding" cannot be filtered for directly - it is | ||
| inferred from the `Computer` names instead. And the event's details are | ||
| not split into neat named columns the way `SecurityEvent`'s are; they | ||
| sit bundled in one `EventData` field that queries unpack | ||
| (`EventData.CommandLine`). | ||
| - **There is also a stamp saying which clerk filed it.** Every row | ||
| carries | ||
| [`_ResourceId`](https://learn.microsoft.com/azure/azure-monitor/logs/log-standard-columns#_resourceid) | ||
| - the Azure resource the record is associated with, which for | ||
| agent-collected data is the machine running the agent, i.e. the | ||
| **collector** - while `Computer` stays the end device. That pair | ||
| (collector stamp + original letterhead) powers the | ||
| collector-attribution queries below; sanity-check the mapping in your | ||
| own workspace by comparing against `Heartbeat._ResourceId`. | ||
|
|
||
| One real-world wrinkle: a single DCR can carry **two data sources** - a | ||
| Custom XPath one reading `ForwardedEvents!*` (those rows go to | ||
| `WindowsEvent`) *and* a Basic one collecting the collector's own | ||
| Application/Security/System logs (those rows go to the `Event` table). | ||
| Finding the collectors' own noise in `Event` rather than `WindowsEvent` | ||
| is the second source doing exactly what its checkboxes say, not a fault. | ||
|
|
||
| ## Confirming AMA actually collects ForwardedEvents | ||
|
|
||
|
|
@@ -112,25 +138,151 @@ WindowsEvent | |
| | order by Events desc | ||
| ``` | ||
|
|
||
| **Collection method map** - for every source, *how* its events reached | ||
| the workspace: direct AMA on the machine itself, or WEF via a named | ||
| collector. The row's `_ResourceId` is the machine whose agent shipped it, | ||
| so when the source's own name matches that resource, the machine shipped | ||
| its own events (direct); when it differs, the events rode a subscription | ||
| through that collector. Joining Heartbeat on the lowercased full | ||
| `_ResourceId` (never on computer names, whose short/FQDN forms differ | ||
| between tables) adds each shipping agent's health. Field-tested against a | ||
| mixed direct-and-forwarded estate: | ||
|
|
||
| ```kusto | ||
| let Lookback = 24h; | ||
| let ActiveAgents = | ||
| Heartbeat | ||
| | where TimeGenerated > ago(Lookback) and Category == "Azure Monitor Agent" | ||
| | extend AgentResourceId = tolower(_ResourceId) | ||
| | summarize arg_max(TimeGenerated, Version, OSType) by AgentResourceId | ||
| | project AgentResourceId, LastHeartbeat = TimeGenerated, AgentVersion = Version, OSType; | ||
| WindowsEvent | ||
| | where TimeGenerated > ago(Lookback) | ||
| | extend SourceComputer = tostring(Computer) | ||
| | extend SourceShortName = tolower(tostring(split(Computer, ".")[0])) | ||
| | extend AgentResourceId = tolower(tostring(_ResourceId)) | ||
| | extend Collector = extract(@"([^/]+)$", 1, AgentResourceId) | ||
| | extend CollectorShortName = tolower(tostring(split(Collector, ".")[0])) | ||
| | summarize | ||
| Events = count(), | ||
| Channels = dcount(Channel), | ||
| FirstEvent = min(TimeGenerated), | ||
| LastEvent = max(TimeGenerated) | ||
| by SourceComputer, SourceShortName, Collector, CollectorShortName, AgentResourceId | ||
| | join kind=leftouter ActiveAgents on AgentResourceId | ||
| | extend CollectionMethod = case( | ||
| isempty(AgentResourceId), "Unknown - Resource ID unavailable", | ||
| SourceShortName == CollectorShortName, "Direct AMA", | ||
| strcat("WEF via collector: ", Collector)) | ||
| | extend CollectorHeartbeatStatus = | ||
| iff(isnotempty(LastHeartbeat), "Active", "No heartbeat in last 24h") | ||
| | project | ||
| SourceComputer, CollectionMethod, Collector, CollectorHeartbeatStatus, | ||
| LastHeartbeat, AgentVersion, Events, Channels, FirstEvent, LastEvent | ||
| | order by CollectionMethod asc, Events desc | ||
| ``` | ||
|
|
||
| One row per source, and the `CollectionMethod` column answers the | ||
| question directly; `CollectorHeartbeatStatus` flags a shipping agent that | ||
| has since gone quiet. (Absence of a heartbeat is evidence within the | ||
| window, not proof the machine is down - see the | ||
| [agent troubleshooting](https://learn.microsoft.com/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-windows-vm).) | ||
|
|
||
| **Per-collector rollup of observed events** - how balanced the shipping | ||
| collectors are. There is no "DCR name" column in the table, so scope by | ||
| the machines listed on the DCR's **Resources** tab (add | ||
| `| where Collector in ("wec01", "wec02", ...)` when other machines also | ||
| write to `WindowsEvent`). `TimeGenerated` is when an event happened on | ||
| the source; `ingestion_time()` is when the workspace received it - this | ||
| query windows and reports on the latter, since delivery freshness is the | ||
| claim being made | ||
| ([standard columns](https://learn.microsoft.com/azure/azure-monitor/logs/log-standard-columns)). | ||
| A collector that shipped nothing cannot appear here; the next section | ||
| finds those: | ||
|
|
||
| ```kusto | ||
| WindowsEvent | ||
| | where ingestion_time() > ago(24h) | ||
| | extend Collector = tolower(tostring(split(_ResourceId, "/")[-1])) | ||
| | summarize Events = count(), EndDevices = dcount(Computer), LastIngested = max(ingestion_time()) by Collector | ||
| | order by Events desc | ||
| ``` | ||
|
|
||
| And if the DCR also carries a Basic data source for the collectors' own | ||
| Application/Security/System logs, those rows are in the `Event` table: | ||
|
|
||
| ```kusto | ||
| Event | ||
| | where TimeGenerated > ago(24h) | ||
| | extend Collector = tolower(tostring(split(_ResourceId, "/")[-1])) | ||
| | summarize Events = count() by Collector, EventLog | ||
| | order by Collector asc, Events desc | ||
| ``` | ||
|
|
||
| **Silent collectors: attached to the DCR but forwarding nothing.** The | ||
| rollup above only shows collectors that shipped at least one row - a dead | ||
| collector is invisible in it. This version starts from the machines that | ||
| *should* be shipping (their AMA heartbeats) and left-joins what actually | ||
| arrived, so the silent ones surface with zero counts. Replace the list | ||
| with the names from the DCR's Resources tab: | ||
|
|
||
| Both sides derive the collector name from `_ResourceId` (present on | ||
| [both tables](https://learn.microsoft.com/azure/azure-monitor/logs/log-standard-columns#_resourceid)) | ||
| so the join key cannot disagree on short name vs FQDN; only the | ||
| `expectedCollectors` list needs to match the resource names from the | ||
| Resources tab (lowercase, to match the `tolower` normalisation): | ||
|
|
||
| ```kusto | ||
| let window = 24h; | ||
| let expectedCollectors = dynamic(["wec01", "wec02", "wec03", "wec04", "wec05"]); | ||
| let shipping = WindowsEvent | ||
| | where ingestion_time() > ago(window) | ||
| | extend Collector = tolower(tostring(split(_ResourceId, "/")[-1])) | ||
| | summarize Events = count(), EndDevices = dcount(Computer), LastIngested = max(ingestion_time()) by Collector; | ||
| let alive = Heartbeat | ||
| | where TimeGenerated > ago(window) and Category == "Azure Monitor Agent" | ||
| | extend Collector = tolower(tostring(split(_ResourceId, "/")[-1])) | ||
| | summarize LastHeartbeat = max(TimeGenerated) by Collector; | ||
| print Collector = expectedCollectors | ||
| | mv-expand Collector to typeof(string) | ||
| | join kind=leftouter alive on Collector | ||
| | join kind=leftouter shipping on Collector | ||
| | project Collector, LastHeartbeat, Events = coalesce(Events, 0), EndDevices = coalesce(EndDevices, 0), LastIngested | ||
| | order by Events asc | ||
| ``` | ||
|
|
||
| Reading the result rows for a silent collector, in order (heartbeat | ||
| presence or absence here means *within this query's window and filters* - | ||
| it is evidence, not proof, of a machine's state): | ||
|
|
||
| 1. **No matching heartbeat** - the machine, its agent, or heartbeat | ||
| ingestion is not working (or the name in `expectedCollectors` does not | ||
| match the resource name); nothing about WEF yet. Start with the | ||
| [agent troubleshooting](https://learn.microsoft.com/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-windows-vm). | ||
| 2. **Heartbeat present, events zero** - the agent reports in but ships no | ||
| forwarded events; the question becomes *which side of the collector is | ||
| broken*. On that collector, | ||
| check whether ForwardedEvents itself has recent events: | ||
|
|
||
| ```powershell | ||
| Get-WinEvent -LogName ForwardedEvents -MaxEvents 5 | Select-Object TimeCreated, MachineName, Id | ||
| ``` | ||
|
|
||
| - **ForwardedEvents has recent events** -> the WEF half works; the | ||
| workspace hop is broken *for this machine*. Verify the DCR | ||
| association actually includes it (a five-collector estate where | ||
| only three were ever associated looks exactly like this) and grep | ||
| the local config cache for `ForwardedEvents` as in the four-layer | ||
| check above. | ||
| - **ForwardedEvents is empty or stale** -> the WEF half is broken: | ||
| run `wecutil es` / `wecutil gr` on that collector. No subscriptions | ||
| = it was never set up; subscriptions with zero or Inactive sources | ||
| = work the [WEC page's](wec.md) reconciliation and silent-failures | ||
| table (GPO scope, WinRM, the Security-log permission). It is | ||
| entirely possible for some collectors in an estate to have | ||
| subscriptions and others none - each collector's subscription store | ||
| is local to it. | ||
|
|
||
| **Channel and event mix** - compare against the subscription query and the | ||
| source baseline (the [Reference page](reference.md) lists what each kit | ||
| setting emits): | ||
|
|
@@ -220,6 +372,72 @@ WindowsEvent | |
| | take 20 | ||
| ``` | ||
|
|
||
| ## Domain controllers: which path are they on? | ||
|
|
||
| DCs are usually the highest-value sources and often the messiest to | ||
| trace, because one DC's telemetry can arrive over **three separate | ||
| paths** into **three separate tables**: | ||
|
|
||
| | DC telemetry | Path | Table | | ||
| |---|---|---| | ||
| | Security / directory events via WEF | DC -> collector -> AMA | `WindowsEvent` | | ||
| | Security events via direct AMA (the [Security Events connector](https://learn.microsoft.com/azure/sentinel/connect-services-windows-based) on the DC itself) | DC -> AMA | `SecurityEvent` | | ||
| | DNS server activity (the [ASIM DNS via AMA connector](https://learn.microsoft.com/azure/sentinel/dns-normalization-schema)) | DC -> AMA | `ASimDnsActivityLogs` | | ||
|
|
||
| The DNS path can never ride WEF - that connector's DCR runs on the DNS | ||
| server (typically the DCs) itself - so DNS rows are always evidence of a | ||
| working *direct* agent on that DC. | ||
|
|
||
| Which tables each DC is actually landing in (short names, lowercase): | ||
|
|
||
| ```kusto | ||
| let DCs = dynamic(["dc01", "dc02"]); | ||
| union isfuzzy=true | ||
| (WindowsEvent | where TimeGenerated > ago(24h) | extend Table = "WindowsEvent", Host = tolower(tostring(split(Computer, ".")[0]))), | ||
| (SecurityEvent | where TimeGenerated > ago(24h) | extend Table = "SecurityEvent", Host = tolower(tostring(split(Computer, ".")[0]))), | ||
| (ASimDnsActivityLogs | where TimeGenerated > ago(24h) | extend Table = "ASimDnsActivityLogs", Host = tolower(tostring(split(coalesce(DvcHostname, Dvc), ".")[0]))) | ||
| | where Host in (DCs) | ||
| | summarize Events = count(), LastIngested = max(ingestion_time()) by Host, Table | ||
| | order by Host asc, Table asc | ||
| ``` | ||
|
|
||
| A DC missing a row for an expected table means **no matching rows were | ||
| observed in the window** - strong evidence, not proof, that the path is | ||
| broken: the path may be deliberately unconfigured for that DC, or a | ||
| [DCR XPath filter](https://learn.microsoft.com/azure/azure-monitor/vm/data-collection-windows-events) | ||
| may exclude the events. Interpret against the intended design, then | ||
| confirm with the tracer-event and configuration checks above before | ||
| declaring it broken. For the `WindowsEvent` rows, *how* each DC arrives | ||
| (WEF via which collector, or direct) is the collection method map above - | ||
| insert `| where SourceShortName in (DCs)` before its `project`. | ||
|
|
||
| (`Host` in the DNS leg prefers `DvcHostname` and falls back to `Dvc`, | ||
| which per the | ||
| [ASIM device schema](https://learn.microsoft.com/azure/sentinel/normalization-entity-device) | ||
| can also carry an IP or device ID - rows where the fallback is not a | ||
| hostname will not match the `DCs` list.) | ||
|
|
||
| And which machines are shipping DNS activity at all (field-tested; the | ||
| resource ID also says whether each is an Arc-enabled server or an Azure | ||
| VM): | ||
|
|
||
| ```kusto | ||
| ASimDnsActivityLogs | ||
| | where TimeGenerated > ago(24h) | ||
| | summarize Events = count(), LastIngested = max(ingestion_time()) by _ResourceId | ||
| | extend Machine = tolower(tostring(split(trim_end(@"/", _ResourceId), "/")[-1])) | ||
| | extend HostType = case( | ||
| _ResourceId has "/microsoft.hybridcompute/machines/", "Arc-enabled server", | ||
| _ResourceId has "/microsoft.compute/virtualmachines/", "Azure VM", | ||
|
Comment on lines
+430
to
+431
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: printf '%s\n' '--- applicable repository guidance ---'
find /tmp/coderabbit-repo-knowledge/spydisec-winlogkit-eceb4215 -type f -name '*.md' -print
printf '%s\n' '--- target file context ---'
sed -n '390,435p' docs/kql.md
printf '%s\n' '--- related definitions and uses ---'
rg -n -C 3 'HostType|_ResourceId has|Arc-enabled server|Azure VM' docs/kql.mdRepository: spydisec/WinLogKit Length of output: 4958 🏁 Script executed: printf '%s\n' '--- repository-wide guidance ---'
cat /tmp/coderabbit-repo-knowledge/spydisec-winlogkit-eceb4215/conventions/repo-wide.md
printf '%s\n' '--- scoped architecture guidance ---'
cat /tmp/coderabbit-repo-knowledge/spydisec-winlogkit-eceb4215/architecture/infrastructure.mdRepository: spydisec/WinLogKit Length of output: 3487 🌐 Web query:
💡 Result: In the Kusto Query Language (KQL), the forward slash (/) character is treated as a delimiter rather than a single atomic token when processing strings for search or index operations [1]. When you use a term like "microsoft.hybridcompute/machines/" in a KQL query, the Kusto engine performs tokenization based on non-alphanumeric characters, including forward slashes, dots, and hyphens [1]. These characters act as delimiters, breaking the string into separate, searchable tokens (e.g., "microsoft", "hybridcompute", "machines") [1]. Key technical considerations regarding this behavior include: 1. Tokenization and Indexing: Kusto indexes alphanumeric terms [1]. During ingestion or search, non-alphanumeric characters are treated as breaks, meaning the full path "microsoft.hybridcompute/machines/" is decomposed into its constituent parts [1]. 2. String Literal Handling: When this path is used as a string literal within a query, it must be enclosed in quotes (e.g., 'microsoft.hybridcompute/machines/') [2][3]. Because the slash is not a special escape character in standard KQL string literals, it does not require special escaping [2][3]. However, if you are working with file paths or patterns where backslashes are used, using a verbatim string literal (prefixed with @) is recommended to treat the backslash as a literal character rather than an escape character [3]. 3. Context of Use: The string "microsoft.hybridcompute/machines/" is commonly used as a value for the 'type' column when querying the Azure Resource Graph [4][5][6]. In this context, it is treated as a single, exact-match string value, and the tokenization rules do not prevent it from being correctly identified when filtered using equality operators like == or =~ [4][5][7]. In summary, the slash in this path does not define a unique punctuation operator in KQL, but rather serves as a delimiter during tokenization and full-text indexing, while remaining part of the literal string value when used in equality comparisons [1][4]. Citations:
🌐 Web query:
💡 Result: In Kusto Query Language (KQL), the 'has' operator is specifically designed to search for a whole term within a string [1][2]. A 'whole term' is defined by how the data is indexed; specifically, Kusto indexes terms of three or more characters [1][3]. When you use the 'has' operator: - It checks if the right-hand-side (RHS) expression exists as a distinct term within the left-hand-side (LHS) column [1][2]. - It is a case-insensitive operator [1]. - If you require a case-sensitive version, you should use 'has_cs' [1][2]. - Because it utilizes a term index, 'has' is generally more performant than operators like 'contains', which perform a more general substring search and may require a full column scan [2]. If you search for a term shorter than three characters, the 'has' operator will still function, but it will automatically fall back to scanning the column values instead of using the term index, which is slower [1][3]. Citations:
🏁 Script executed: python3 - <<'PY'
from pathlib import Path
p = Path("docs/kql.md")
lines = p.read_text().splitlines()
for n in range(408, 422):
print("%4d %s" % (n + 1, lines[n]))
PYRepository: spydisec/WinLogKit Length of output: 808 🏁 Script executed: python3 - <<'PY'
import urllib.request
url = "https://learn.microsoft.com/en-us/kusto/query/has-operator?view=microsoft-fabric"
try:
text = urllib.request.urlopen(url, timeout=10).read().decode("utf-8", "replace")
for needle in ("whole term", "term", "punctuation", "multiple", "has_all"):
print("=== %s ===" % needle)
lower = text.lower()
start = 0
count = 0
while count < 5:
pos = lower.find(needle.lower(), start)
if pos < 0:
break
print(" ".join(text[max(0, pos-180):pos+320].split()))
start = pos + len(needle)
count += 1
except Exception as e:
print(type(e).__name__ + ": " + str(e))
PYRepository: spydisec/WinLogKit Length of output: 301 🌐 Web query:
💡 Result: In the context of the Kusto Query Language (KQL), the term North appears as an example value used to illustrate the behavior of the has string operator [1]. The has operator filters a record set by checking if a specified Right-Hand Side (RHS) expression is a whole term within the Left-Hand Side (LHS) column [1]. The documentation uses the example "North America" to demonstrate that "North America" has "america" evaluates to true because "america" is treated as a distinct term within the string [1]. Regarding punctuation, the KQL has operator is designed to search for indexed terms [1]. When using this operator, symbols, numbers, and punctuation are typically treated as term delimiters [1]. Consequently, if a string contains punctuation, it is split into separate terms based on those boundaries; a search for a specific term will only return a match if that term exists as a whole, standalone word within the data, rather than as a substring [1]. Citations: Use
🤖 Prompt for AI Agents |
||
| "Other") | ||
| | project Machine, HostType, Events, LastIngested, _ResourceId | ||
| | order by Machine asc | ||
| ``` | ||
|
|
||
| An on-prem DC expected here but absent shipped no matching DNS rows in | ||
| the window - triage it like any silent direct-AMA machine (heartbeat, | ||
| DCR association, config cache), not like a WEF problem. | ||
|
|
||
| ## The reconciliation that matters | ||
|
|
||
| The single most useful standing assertion is a three-list comparison, and | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.