-
Notifications
You must be signed in to change notification settings - Fork 238
Add KQL string matching rules to the coding guidelines #2223
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,7 @@ On this page: | |
|
|
||
| - [βΉοΈ General guidelines](#βΉοΈ-general-guidelines) | ||
| - [π€ Content (strings and microcopy)](#-content-strings-and-microcopy) | ||
| - [β‘ KQL](#-kql) | ||
| - [π Changelog](#-changelog) | ||
|
|
||
| --- | ||
|
|
@@ -23,6 +24,7 @@ Here's a quick run-down of the main points: | |
| - Follow standard language conventions: | ||
| - [PowerShell guidelines](https://learn.microsoft.com/powershell/scripting/developer/cmdlet/cmdlet-development-guidelines) | ||
| - [Bicep lint rules](https://learn.microsoft.com/azure/azure-resource-manager/bicep/linter) | ||
| - [KQL best practices](https://learn.microsoft.com/azure/data-explorer/kusto/query/best-practices) β see [KQL](#-kql) below for the project-specific rules | ||
|
|
||
| <br> | ||
|
|
||
|
|
@@ -45,6 +47,47 @@ We adhere to the [Microsoft style guide](https://docs.microsoft.com/style-guide/ | |
|
|
||
| <br> | ||
|
|
||
| ## β‘ KQL | ||
|
|
||
| Hub KQL runs over every ingested row, so string matching choices show up directly in ingestion cost and query latency. Two rules cover most of it: **never wrap a column in `tolower()` to compare it**, and **reach for `has` before `contains`**. | ||
|
|
||
| ### String comparison | ||
|
|
||
| Every KQL string operator is already case-insensitive (`has`, `contains`, `startswith`, `endswith`, `=~`, `in~`). The `_cs` suffixed forms (and `==`, `in`) are the case-sensitive ones. Wrapping a column in `tolower()` therefore adds a per-row function call and prevents the engine from using the term index, with no behavioral benefit. | ||
|
|
||
| | β Avoid | β Prefer | Why | | ||
| | --- | --- | --- | | ||
| | `tolower(Col) contains 'term'` | `Col has 'term'` | `has` matches whole terms using the term index; `tolower()` disables it | | ||
| | `tolower(Col) == 'value'` | `Col =~ 'value'` | `=~` is the case-insensitive equality operator | | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: Does |
||
| | `tolower(a) != tolower(b)` | `a !~ b` | One comparison instead of two per-row allocations | | ||
| | `Col contains 'Windows'` | `Col has 'Windows'` | Whole-word needle β see below for when `contains` is right | | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As far as I know, this can be a tokenization trap, aside from the case-sensitivity. |
||
| | `indexof(Col, 'x') >= 0` | `Col has 'x'` *or* `Col contains 'x'` | Don't compute a position you don't need β but pick the operator by intent, not mechanically: `indexof()` is case-**sensitive** and substring-based, so it is equivalent to `contains_cs`, not to `has` | | ||
| | `tostring(Dyn.Field) =~ 'true'` | `Dyn.Field =~ 'true'` | These operators accept a *scalar* dynamic operand directly β see the note below | | ||
|
|
||
| Prefer `in~` over chained `=~` comparisons, and `has_any` / `has_all` over chained `has`. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: Why not work these into the table like the others for consistency? |
||
|
|
||
| > [!IMPORTANT] | ||
| > Only compare a dynamic field this way when it holds a **scalar** (string, bool, or number), as `x_SkuDetails.AHB` does. If the field can hold an object or an array, the comparison runs against its JSON serialization, which is rarely what you want: `has` matches a value nested anywhere inside the JSON text, so `{"nested":"true"} has 'true'` is `true` while `=~ 'true'` is `false`. Adding `tostring()` does not change this β it produces the same JSON text and the same result. Extract the value you actually mean instead (for example `Dyn.Field.nested`), or compare with `array_index_of()` / `set_has_element()` for arrays. | ||
|
|
||
| ### When `contains` is required | ||
|
|
||
| ADX tokenizes string columns at ingestion: runs of alphanumeric characters become *terms*, and punctuation (`/`, `.`, `-`, space) are separators. `has` matches whole terms and can use the term index; `contains` scans for an arbitrary substring. That makes them **semantically different**, not just faster and slower. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: I'd prefer to see this baked into the guidance above. I'd rather see 2 lines, one for each with the valid cases for each so all rules are cleanly listed together. Not a blocker, but this merged with what's above would tell the whole story. What's above alone can lead to bugs. This applies to both H3s here. |
||
|
|
||
| Use `contains` only when the text you're matching can be fused inside a larger token and you want to match anyway β for example `ConsumedUnit contains 'MB'`, which also matches `Mbps`. Word-stem matching (`'Trial'` inside `'Trials'`) and fragments that never form a whole term are the other legitimate cases. | ||
|
|
||
| For a multi-word or path-shaped needle, use the full phrase with `has` rather than falling back to `contains`: `ResourceId has '/microsoft.capacity/reservationorders/'` respects the surrounding separators exactly like `contains` does, and still uses the index. | ||
|
|
||
| > [!NOTE] | ||
| > Needles that are pure punctuation (`has '/'`) or shorter than three characters can't use the term index and fall back to a scan β same speed as `contains`, but still term-bounded semantics. Prefer `has` anyway for consistency. | ||
|
|
||
| ### Verify before you swap | ||
|
|
||
| Switching `contains` to `has` changes matching semantics, so treat it as a behavioral change until proven otherwise. Cross-tabulate both directions on real data (`countif(old != new)` must be `0` β equal aggregate counts can hide offsetting false positives and negatives), and add fixtures to the executable harness at `src/powershell/Tests/assets/StringOperatorEquivalence.kql`, which runs on any Kusto database and returns zero rows when it passes. | ||
|
|
||
| These rules are enforced on every pull request by `src/powershell/Tests/Unit/HubsKqlOperators.Tests.ps1`: `tolower()` in comparison position fails the build, and each `contains` usage must be listed in that test's allowlist with a justification. | ||
|
|
||
| <br> | ||
|
|
||
| ## π Changelog | ||
|
|
||
| The [changelog](../docs-mslearn/toolkit/changelog.md) documents user-facing changes for each release. It follows [Keep a Changelog](https://keepachangelog.com) conventions adapted for this project's multi-tool structure. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought
haswas tokenized and case-sensitive. Is it not?