From ceaaa339e2e4bb28ee0b4d2cb40bf54f0da382b0 Mon Sep 17 00:00:00 2001 From: Claude Fable 5 Date: Mon, 31 Aug 2026 08:42:43 +0000 Subject: [PATCH] =?UTF-8?q?docs(skills):=20objectstack-query=20factual=20s?= =?UTF-8?q?weep=20=E2=80=94=2017=20false=20behavioral=20claims=20corrected?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Flight (3) of the published-skills factual sweep. Every behavioral claim in skills/objectstack-query/** verified against the implementation with executed probes; corrections are byte-shrinking against the pinned token ratchets. Net: -33 lines, -255 tokens across the package. No ceiling raised. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EnE7G31tqbxN1rqpQmzurT --- skills/objectstack-query/SKILL.md | 123 ++++++++---------- skills/objectstack-query/evals/README.md | 17 ++- skills/objectstack-query/rules/aggregation.md | 60 ++++----- skills/objectstack-query/rules/filters.md | 24 ++-- skills/objectstack-query/rules/pagination.md | 11 +- 5 files changed, 101 insertions(+), 134 deletions(-) diff --git a/skills/objectstack-query/SKILL.md b/skills/objectstack-query/SKILL.md index 34e0d0142d..473407c189 100644 --- a/skills/objectstack-query/SKILL.md +++ b/skills/objectstack-query/SKILL.md @@ -22,14 +22,10 @@ Expert instructions for constructing data queries using the ObjectStack Query DSL. This skill covers filter expressions, sorting, pagination, aggregation, full-text search, and the expand system for related records. -**Schema vs. runtime:** the `QueryAST` schema declares more than the engine -currently executes. Sections below marked - -> ⚠️ **Schema-reserved — NOT executed by the engine yet.** - -describe properties that validate against the schema but are silently -ignored (or rejected) at runtime. Never emit them in production queries — -each caveat shows the working alternative. +**Schema vs. runtime:** every callout below says which side a property is on — +⛔ **REMOVED** (tombstoned; a query carrying it fails to parse), ⚠️ **not +enforced** (validates, then silently ignored — never emit it), ✅ **Enforced**. +Each removal callout names the live replacement. --- @@ -217,13 +213,12 @@ Filter through relationships without an explicit join: ### Field References (Cross-Field Comparisons) -> ⚠️ **Schema-reserved — NOT executed by the engine yet.** `$field` exists -> only in the filter schema. No engine or driver code interprets it — the -> `{ $field: '...' }` object binds as a **literal value**, so the query -> silently returns zero rows. Do not use it. +> ✅ **Enforced.** A `{ $field: '...' }` comparand compares two columns of the +> same row. The in-memory evaluator resolves the reference against the record; +> `driver-sql` pushes it down as a column-to-column predicate. Same rows. ```typescript -// ❌ Schema-valid but NOT executed — matches nothing +// ✅ Accounts whose actual revenue beat the estimate { where: { actual_revenue: { $gt: { $field: 'estimated_revenue' } } @@ -231,11 +226,9 @@ Filter through relationships without an explicit join: } ``` -**Working alternatives:** -- Define a **formula field** on the object that computes the comparison - (e.g. `exceeds_estimate` as a boolean), then filter on it: - `{ where: { exceeds_estimate: true } }` (see **objectstack-data**). -- Fetch both fields and compare in **application code**. +Legal in a **comparison** position only. As an `$in` / `$nin` member or a +`$between` endpoint it is refused at parse — no evaluation path resolves a +reference there. --- @@ -331,12 +324,11 @@ unique or near-unique column such as `created_at` or `id`) so | `max` | Maximum | `MAX(field)` | | `count_distinct` | Unique count | `COUNT(DISTINCT field)` | -> ⚠️ **Driver support varies.** On SQL datasources the driver executes only -> `count` / `sum` / `avg` / `min` / `max` and **throws** on `count_distinct`; -> the per-aggregation `distinct: true` flag is also ignored there. The -> in-memory fallback path (driver-rest, driver-memory, timezone/date-bucket -> fallbacks) supports all six functions plus `distinct`. For portable queries, -> stick to the first five. +> ✅ **All six are portable.** `count_distinct` lowers to `COUNT(DISTINCT x)` +> on every SQL face and computes identically on the in-memory path, so the +> declared-but-uncompiled set is empty. The per-aggregation `distinct: true` +> flag went the other way — **removed in 17**, refused at parse; the live +> spelling for a deduplicated count is `count_distinct`. > **Removed in 17.** `array_agg` and `string_agg` left this vocabulary: > declared but lowered by no SQL backend, so whether they worked depended on @@ -387,25 +379,23 @@ const rows = await engine.aggregate('deal', { ### Filtered Aggregation -> ⚠️ **Per-aggregation `filter` is schema-reserved — NOT executed by the -> engine yet.** The SQL driver ignores it and the in-memory path ignores it -> too, so a `filter`-carrying aggregation returns the **unfiltered** number — -> silently wrong results. **Working alternative:** issue one aggregate call -> per condition, moving the condition into the query-level `where`: +> ✅ **Enforced.** A per-aggregation `filter` scopes that one measure, so a +> total and a conditional count share one call. Any aggregation carrying a +> non-empty `filter` forces the in-memory path — no driver compiles a +> conditional aggregate, and one reached directly refuses `NOT_IMPLEMENTED`; +> unfiltered aggregations keep native push-down. ```typescript -// ❌ filter on the aggregation is silently ignored -// { function: 'count', alias: 'high_value_orders', -// filter: { amount: { $gt: 1000 } } } - -// ✅ Separate aggregate calls, condition in `where` -const [totals] = await engine.aggregate('order', { - aggregations: [{ function: 'count', alias: 'total_orders' }], -}); -const [highValue] = await engine.aggregate('order', { - where: { amount: { $gt: 1000 } }, - aggregations: [{ function: 'count', alias: 'high_value_orders' }], +// ✅ Total and conditional counts in ONE call +const [kpis] = await engine.aggregate('order', { + aggregations: [ + { function: 'count', alias: 'total_orders' }, + { function: 'count', alias: 'high_value_orders', + filter: { amount: { $gt: 1000 } } }, + ], }); +// An unknown operator inside `filter` refuses INVALID_FILTER/400 — it never +// silently answers the unfiltered number. ``` --- @@ -475,10 +465,10 @@ Load related records through lookup/master_detail fields: Only the **`query` + `fields`** subset of the search schema executes. The engine expands the search string into a driver-agnostic filter: each term -becomes an `$or` of `$contains` predicates across the resolved searchable -fields, and multiple whitespace-separated terms are **AND-ed** (every term -must hit some field). Matching is case-insensitive; `select`/`status` -fields match by option *label*, mapped to stored values. +becomes an `$or` of `$icontains` predicates (the case-INSENSITIVE twin of the +case-sensitive `$contains`) across the resolved searchable fields, and multiple +whitespace-separated terms are **AND-ed** (every term must hit some field). +`select`/`status` fields match by option *label*, mapped to stored values. ```typescript { @@ -491,8 +481,8 @@ fields match by option *label*, mapped to stored values. } // Executes as: // { $and: [ -// { $or: [{ title: { $contains: 'machine' } }, { content: { $contains: 'machine' } }] }, -// { $or: [{ title: { $contains: 'learning' } }, { content: { $contains: 'learning' } }] }, +// { $or: [{ title: { $icontains: 'machine' } }, { content: { $icontains: 'machine' } }] }, +// { $or: [{ title: { $icontains: 'learning' } }, { content: { $icontains: 'learning' } }] }, // ]} ``` @@ -529,19 +519,18 @@ maintained on write and listed in `task.searchableFields`: limit: 20, } // Expands to a single-table scan — no traversal, every driver: -// { $and: [{ $or: [ -// { name: { $contains: 'apollo' } }, -// { project_name: { $contains: 'apollo' } }, -// ]}]} +// { $or: [ +// { name: { $icontains: 'apollo' } }, +// { project_name: { $icontains: 'apollo' } }, +// ]} ``` ❌ The mirror must be a **stored** field — a `formula` field is virtual, no -driver materializes a column for it, so a `$contains` predicate against one has -nothing to scan. Nothing rejects the mistake for you: `searchableFields` admits -any field the object declares, so a formula entry clears both lint and the -ingress gate and then never matches. The trade-off is mirror maintenance — hooks -on both write paths (child re-parented, parent renamed) plus a backfill for rows -written around the hooks. +driver materializes a column for it, so a search predicate against one has +nothing to scan. Two guards catch that: lint errors on a virtual +`searchableFields` entry, and the ingress gate refuses one by name. The +trade-off is mirror maintenance — hooks on both write paths (child re-parented, +parent renamed) plus a backfill for rows written around the hooks. Cross-object search paths are rejected by design, not pending. Modelling side of this (the field, the hooks, the lint wording): **objectstack-data → Search Fields @@ -606,25 +595,21 @@ use [`expand`](#expand-related-records). ### Dashboard Aggregation Pattern -Unconditional KPIs can share one aggregate call; a KPI with its own -condition needs a **separate call** with the condition in `where` -(per-aggregation `filter` is schema-reserved — see Filtered Aggregation): +Every KPI on a dashboard shares **one** aggregate call — unconditional +measures plain, conditional ones carrying their own `filter`. `where` scopes +the whole call, so reach for it only when every measure wants the same scope: ```typescript -// KPI dashboard: unconditional aggregations share one call +// KPI dashboard: one call, conditional measures scoped per aggregation const [kpis] = await engine.aggregate('deal', { aggregations: [ { function: 'count', alias: 'total_deals' }, { function: 'sum', field: 'amount', alias: 'pipeline_value' }, { function: 'avg', field: 'amount', alias: 'avg_deal_size' }, + { function: 'count', alias: 'won_deals', + filter: { stage: 'closed_won' } }, ], }); - -// Conditional KPI: separate call, condition in `where` -const [won] = await engine.aggregate('deal', { - where: { stage: 'closed_won' }, - aggregations: [{ function: 'count', alias: 'won_deals' }], -}); ``` --- @@ -636,9 +621,9 @@ code — the renderer issues the queries for you: | Query Need | Pattern | |:--|:--| -| KPI widgets | Aggregates (`sum`, `count`, `avg`) over the object, each conditional KPI scoped by the widget/dataset filter. Add `compareTo: 'previousPeriod' \| 'previousYear'` on the widget for a one-line period-over-period delta. | -| Time-series chart | Date filters + `categoryGranularity: 'day' \| 'week' \| 'month' \| 'quarter' \| 'year'` for server-side bucketing — never bucket by hand on the client. Pair with `compareTo` for an aligned YoY overlay. | -| Matrix report | `groupingsDown` + `groupingsAcross` + `dateGranularity: 'quarter'` | +| KPI widgets | Aggregates (`sum`, `count`, `avg`) over the object, each conditional KPI scoped by the widget/dataset filter. Add `compareTo: { kind: 'previousPeriod' \| 'previousYear' }` on the widget for a one-line period-over-period delta (the bare string form was removed in 17). | +| Time-series chart | Date filters + `dateGranularity: 'day' \| 'week' \| 'month' \| 'quarter' \| 'year'` on the widget's dataset selection for server-side bucketing — never bucket by hand on the client. Pair with `compareTo` for an aligned YoY overlay. | +| Matrix report | Dataset-bound `rows` (down) + `columns` (across) + a `dateGranularity` dimension | | Funnel summary | Multi-level grouping (`owner -> stage`) + aggregated measures | | Operational filter | Prefer declarative operators (`$ne`, `$nin`, `$gte`) over hardcoded SQL | diff --git a/skills/objectstack-query/evals/README.md b/skills/objectstack-query/evals/README.md index 70966a0a21..58f0f20c4a 100644 --- a/skills/objectstack-query/evals/README.md +++ b/skills/objectstack-query/evals/README.md @@ -17,18 +17,17 @@ subset the engine actually executes. manual keyset pagination (`where` on the sort key + `orderBy` + `limit`); fail if the answer uses the removed `cursor` property. 4. **Aggregation correctness** — "Count deals by region and show total - revenue." Expect `groupBy` + `count`/`sum` with aliases; on SQL targets - the answer must stay within `count`/`sum`/`avg`/`min`/`max`. -5. **The FILTER-WHERE trap** — "One call: total orders and count of orders - over $1000." The correct answer is **two** aggregate calls with the - condition in `where`; fail if the answer puts `filter` on an aggregation - (silently returns the unfiltered number). + revenue." Expect `groupBy` + `count`/`sum` with aliases; fail on a missing + `alias`, or on `array_agg`/`string_agg` (removed in protocol 17). +5. **Filtered aggregation** — "One call: total orders and count of orders + over $1000." Expect one call with a per-aggregation `filter` on the + conditional measure; fail on `where`, which scopes every measure. 6. **Post-aggregation filtering** — "Customers with more than 5 orders." - Expect aggregate + app-code filter of the group rows; fail on `having` - (schema-reserved, silently dropped). + Expect `having` on the aggregation alias; fail on `where`, which filters + input rows before the alias exists. 7. **Date-bucketed time series** — "Monthly revenue for the last year." Expect structured `groupBy` with `dateGranularity: 'month'`, not - client-side bucketing and not window functions (schema-reserved). + client-side bucketing and not `windowFunctions` (removed in protocol 17). 8. **Expand vs direct query** — "Show a task list with assignee names; page through one project's tasks." Expect `expand` for the lookup display and a direct query on the related object for pagination (nested diff --git a/skills/objectstack-query/rules/aggregation.md b/skills/objectstack-query/rules/aggregation.md index d532ca116a..088a15df72 100644 --- a/skills/objectstack-query/rules/aggregation.md +++ b/skills/objectstack-query/rules/aggregation.md @@ -13,12 +13,11 @@ Guide for building ObjectStack aggregation queries. | `max` | `MAX(field)` | Maximum value | Yes | | `count_distinct` | `COUNT(DISTINCT field)` | Count unique values | Yes | -> ⚠️ **Driver support varies.** On SQL datasources the driver executes only -> `count` / `sum` / `avg` / `min` / `max` and **throws** (`Unsupported -> aggregate function`) on `count_distinct`; the per-aggregation -> `distinct: true` flag is also ignored there. The in-memory aggregation path -> (driver-rest, driver-memory, timezone/date-bucket fallbacks) supports all six -> functions plus `distinct`. For portable queries, stick to the first five. +> ✅ **All six are portable.** `count_distinct` lowers to `COUNT(DISTINCT x)` +> on `driver-sql` and turso's remote transport, and `driver-mongodb` / +> `driver-memory` compute it too, so the declared-but-uncompiled set is empty. +> The per-aggregation `distinct: true` flag went the other way — **removed in +> 17**, refused at parse. For a deduplicated count, use `count_distinct`. > **Removed in 17.** `array_agg` and `string_agg` are no longer part of > the vocabulary — they were declared and lowered by no SQL backend, so a query @@ -118,38 +117,33 @@ use `where` to shrink the scan, `having` to threshold the aggregates. ## Filtered Aggregation (FILTER WHERE) -> ⚠️ **Per-aggregation `filter` is schema-reserved — NOT executed by the -> engine yet.** The SQL driver never reads it and the in-memory path ignores -> it, so the aggregation returns the **unfiltered** number — silently wrong -> results. **Working alternative:** one aggregate call per condition, with -> the condition in the query-level `where`: +> ✅ **Enforced.** A per-aggregation `filter` scopes that one measure, so a +> total and a conditional count share ONE call. Any aggregation carrying a +> non-empty `filter` forces the in-memory path — no driver compiles a +> conditional aggregate, and one reached directly refuses `NOT_IMPLEMENTED`; +> unfiltered aggregations keep native push-down. ```typescript -// ❌ filter on the aggregation is silently ignored — active_count -// would equal total! -// { function: 'count', alias: 'active_count', filter: { status: 'active' } } - -// ✅ Separate aggregate calls, condition in `where` -const [totals] = await engine.aggregate('user', { - aggregations: [{ function: 'count', alias: 'total' }], -}); -const [active] = await engine.aggregate('user', { - where: { status: 'active' }, - aggregations: [{ function: 'count', alias: 'active_count' }], +// ✅ Total and conditional count in one call +const [row] = await engine.aggregate('user', { + aggregations: [ + { function: 'count', alias: 'total' }, + { function: 'count', alias: 'active_count', filter: { status: 'active' } }, + ], }); +// An unknown operator inside `filter` refuses INVALID_FILTER/400 — it never +// silently answers the unfiltered number. ``` ## DISTINCT Aggregation -> ⚠️ **Not available on SQL datasources.** `count_distinct` **throws** on the -> SQL driver, and the `distinct: true` flag is silently ignored there (see -> the driver-support caveat above). Both forms work only on the in-memory -> aggregation path. On SQL, get a distinct count by grouping on the field -> and counting the result rows in app code: -> `(await engine.aggregate('employee', { groupBy: ['department'], aggregations: [{ function: 'count', alias: 'n' }] })).length`. +> ✅ **`count_distinct` runs everywhere** — `COUNT(DISTINCT field)` on the SQL +> faces, the same answer in memory. `field` is REQUIRED; there is no +> `COUNT(DISTINCT *)`. The per-aggregation `distinct: true` flag is NOT its +> equivalent: **removed in 17**, refused at parse, because exactly one of the +> six backends that read an aggregation ever honoured it. ```typescript -// In-memory drivers only: // SQL: SELECT COUNT(DISTINCT department) FROM employee { object: 'employee', @@ -157,14 +151,6 @@ const [active] = await engine.aggregate('user', { { function: 'count_distinct', field: 'department', alias: 'dept_count' } ] } - -// Alternative (also in-memory only): use distinct flag -{ - object: 'employee', - aggregations: [ - { function: 'count', field: 'department', alias: 'dept_count', distinct: true } - ] -} ``` ## Window Functions diff --git a/skills/objectstack-query/rules/filters.md b/skills/objectstack-query/rules/filters.md index 30db6c49a7..1818f6a700 100644 --- a/skills/objectstack-query/rules/filters.md +++ b/skills/objectstack-query/rules/filters.md @@ -99,23 +99,20 @@ where: { ## Field References -> ⚠️ **`$field` is schema-reserved — NOT executed by the engine yet.** It -> exists only in the filter schema; no engine or driver code interprets it, -> so the `{ $field: '...' }` object binds as a **literal value** and the -> query silently returns zero rows. +> ✅ **Enforced.** `{ $field: '...' }` compares two columns of the same row. +> The in-memory evaluator resolves the reference against the record; the SQL +> driver pushes it down as a column-to-column predicate. Same rows either way. ```typescript -// ❌ Schema-valid but NOT executed — matches nothing +// ✅ Projects that ran over budget where: { actual_cost: { $gt: { $field: 'budget' } } } ``` -**Working alternatives:** -- Define a **formula field** that computes the cross-field comparison - (e.g. a boolean `over_budget`), then filter on it: - `where: { over_budget: true }` (see **objectstack-data**). -- Fetch both fields and compare in **application code**. +Legal in a **comparison** position only. As an `$in` / `$nin` member or a +`$between` endpoint it is refused at parse — no evaluation path resolves a +reference there. ## Nested Relation Filters @@ -188,15 +185,16 @@ where: { } ``` -### ❌ Wrong: Null check with equality +### ⚠️ Prefer `$null` to a bare `null` comparand ```typescript -// ❌ Don't use equality to check for null +// ⚠️ Works — a bare null lowers to IS NULL on both paths — but it reads +// as "equals null" and has no IS NOT NULL spelling where: { deleted_at: null } -// ✅ Use $null operator +// ✅ Explicit, and `$null: false` is IS NOT NULL where: { deleted_at: { $null: true } } diff --git a/skills/objectstack-query/rules/pagination.md b/skills/objectstack-query/rules/pagination.md index ea1f40c4f4..9b7f8176d7 100644 --- a/skills/objectstack-query/rules/pagination.md +++ b/skills/objectstack-query/rules/pagination.md @@ -215,14 +215,13 @@ When building paginated REST endpoints: ## DISTINCT Queries -> ⚠️ **The top-level `distinct: true` flag is schema-reserved — NOT executed -> by the engine yet.** Neither the engine nor the SQL driver reads it from a -> `QueryAST`; the query returns duplicate rows as if the flag were absent. -> **Working alternative:** group by the fields — each unique combination -> becomes one result row: +> ⛔ **`query.distinct` was REMOVED in `@objectstack/spec` 17.** No driver ever +> rendered `SELECT DISTINCT`. The key is tombstoned — a query carrying it fails +> to parse with the prescription — and `QueryBuilder.distinct()` is gone. Group +> by the fields instead: each unique combination becomes one result row. ```typescript -// ❌ distinct is silently ignored +// ❌ tombstoned — this query is refused at parse // { object: 'order', fields: ['customer_id', 'product_category'], distinct: true } // ✅ groupBy collapses duplicates