Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,5 +212,6 @@ This repository supports production infrastructure managing significant revenue.
- Content (text strings): Follow the Microsoft style guide and always use sentence casing except for proper nouns
- Bicep: Follow Azure Bicep style guide
- PowerShell: Use PowerShell best practices and approved verbs
- KQL: Never use `tolower()`/`toupper()` in comparison position β€” KQL string operators are already case-insensitive (`_cs` variants are the case-sensitive ones). Use `has` for whole terms and path phrases, `=~`/`!~` for equality, `in~`/`has_any` for sets. Reserve `contains` for genuine substring matching (needle fused inside a larger token) and justify it in the allowlist in `src/powershell/Tests/Unit/HubsKqlOperators.Tests.ps1`, which enforces both rules on every PR. See the KQL section of `docs-wiki/Coding-guidelines.md`
- Documentation: Use markdown with consistent formatting
- Commit messages: Use conventional commit format
43 changes: 43 additions & 0 deletions docs-wiki/Coding-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ On this page:

- [ℹ️ General guidelines](#ℹ️-general-guidelines)
- [πŸ”€ Content (strings and microcopy)](#-content-strings-and-microcopy)
- [⚑ KQL](#-kql)
- [πŸ“‹ Changelog](#-changelog)

---
Expand All @@ -23,6 +24,7 @@ Here's a quick run-down of the main points:
- Follow standard language conventions:
- [PowerShell guidelines](https://learn.microsoft.com/powershell/scripting/developer/cmdlet/cmdlet-development-guidelines)
- [Bicep lint rules](https://learn.microsoft.com/azure/azure-resource-manager/bicep/linter)
- [KQL best practices](https://learn.microsoft.com/azure/data-explorer/kusto/query/best-practices) – see [KQL](#-kql) below for the project-specific rules

<br>

Expand All @@ -45,6 +47,47 @@ We adhere to the [Microsoft style guide](https://docs.microsoft.com/style-guide/

<br>

## ⚑ KQL

Hub KQL runs over every ingested row, so string matching choices show up directly in ingestion cost and query latency. Two rules cover most of it: **never wrap a column in `tolower()` to compare it**, and **reach for `has` before `contains`**.

### String comparison

Every KQL string operator is already case-insensitive (`has`, `contains`, `startswith`, `endswith`, `=~`, `in~`). The `_cs` suffixed forms (and `==`, `in`) are the case-sensitive ones. Wrapping a column in `tolower()` therefore adds a per-row function call and prevents the engine from using the term index, with no behavioral benefit.

| ❌ Avoid | βœ… Prefer | Why |
| --- | --- | --- |
| `tolower(Col) contains 'term'` | `Col has 'term'` | `has` matches whole terms using the term index; `tolower()` disables it |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought has was tokenized and case-sensitive. Is it not?

| `tolower(Col) == 'value'` | `Col =~ 'value'` | `=~` is the case-insensitive equality operator |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Does =~ do a tolower on both values? I thought tolower(Col) == '{lowercase-literal}' was actually faster. Do we know?

| `tolower(a) != tolower(b)` | `a !~ b` | One comparison instead of two per-row allocations |
| `Col contains 'Windows'` | `Col has 'Windows'` | Whole-word needle – see below for when `contains` is right |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know, this can be a tokenization trap, aside from the case-sensitivity.

| `indexof(Col, 'x') >= 0` | `Col has 'x'` *or* `Col contains 'x'` | Don't compute a position you don't need – but pick the operator by intent, not mechanically: `indexof()` is case-**sensitive** and substring-based, so it is equivalent to `contains_cs`, not to `has` |
| `tostring(Dyn.Field) =~ 'true'` | `Dyn.Field =~ 'true'` | These operators accept a *scalar* dynamic operand directly – see the note below |

Prefer `in~` over chained `=~` comparisons, and `has_any` / `has_all` over chained `has`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Why not work these into the table like the others for consistency?


> [!IMPORTANT]
> Only compare a dynamic field this way when it holds a **scalar** (string, bool, or number), as `x_SkuDetails.AHB` does. If the field can hold an object or an array, the comparison runs against its JSON serialization, which is rarely what you want: `has` matches a value nested anywhere inside the JSON text, so `{"nested":"true"} has 'true'` is `true` while `=~ 'true'` is `false`. Adding `tostring()` does not change this β€” it produces the same JSON text and the same result. Extract the value you actually mean instead (for example `Dyn.Field.nested`), or compare with `array_index_of()` / `set_has_element()` for arrays.

### When `contains` is required

ADX tokenizes string columns at ingestion: runs of alphanumeric characters become *terms*, and punctuation (`/`, `.`, `-`, space) are separators. `has` matches whole terms and can use the term index; `contains` scans for an arbitrary substring. That makes them **semantically different**, not just faster and slower.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I'd prefer to see this baked into the guidance above. I'd rather see 2 lines, one for each with the valid cases for each so all rules are cleanly listed together. Not a blocker, but this merged with what's above would tell the whole story. What's above alone can lead to bugs. This applies to both H3s here.


Use `contains` only when the text you're matching can be fused inside a larger token and you want to match anyway – for example `ConsumedUnit contains 'MB'`, which also matches `Mbps`. Word-stem matching (`'Trial'` inside `'Trials'`) and fragments that never form a whole term are the other legitimate cases.

For a multi-word or path-shaped needle, use the full phrase with `has` rather than falling back to `contains`: `ResourceId has '/microsoft.capacity/reservationorders/'` respects the surrounding separators exactly like `contains` does, and still uses the index.

> [!NOTE]
> Needles that are pure punctuation (`has '/'`) or shorter than three characters can't use the term index and fall back to a scan – same speed as `contains`, but still term-bounded semantics. Prefer `has` anyway for consistency.

### Verify before you swap

Switching `contains` to `has` changes matching semantics, so treat it as a behavioral change until proven otherwise. Cross-tabulate both directions on real data (`countif(old != new)` must be `0` – equal aggregate counts can hide offsetting false positives and negatives), and add fixtures to the executable harness at `src/powershell/Tests/assets/StringOperatorEquivalence.kql`, which runs on any Kusto database and returns zero rows when it passes.

These rules are enforced on every pull request by `src/powershell/Tests/Unit/HubsKqlOperators.Tests.ps1`: `tolower()` in comparison position fails the build, and each `contains` usage must be listed in that test's allowlist with a justification.

<br>

## πŸ“‹ Changelog

The [changelog](../docs-mslearn/toolkit/changelog.md) documents user-facing changes for each release. It follows [Keep a Changelog](https://keepachangelog.com) conventions adapted for this project's multi-tool structure.
Expand Down