fix(drivers): $exists means HAS A VALUE on the live mingo path, the analytics face and translateFilter

.changeset/exists-has-value-three-exits.md

@@ -0,0 +1,66 @@
+---
+"@objectstack/driver-memory": minor
+"@objectstack/driver-mongodb": minor
+---
+
+fix(drivers): `$exists` means HAS A VALUE on the live mingo path, the analytics face and `translateFilter` (#13195)
+
+The platform's settled semantic is that `$exists` means "the field has a value"
+(`!= null`), never key-presence — #5298 leg ③ / #5369, landed in PR #5962. Three
+exits still read key-presence; the maintainer ruled on 2026-08-30 that all three
+align. They now do:
+
+- `driver-memory`'s **live mingo query path** (`InMemoryDriver.find()`) — the
+  operator went to mingo under its own name, and mingo tests key presence;
+- `driver-memory`'s **analytics execution face** — it built its own
+  `{$exists: <bool>}`, so it inherited key-presence independently;
+- `driver-mongodb`'s **`translateFilter`** — it passed the operator through, and
+  MongoDB's `$exists` is key-presence at the wire level.
+
+Nothing was invented. All three lower to `{$ne: null}` / `{$eq: null}` — the
+spelling the same files already emit for `$null` — which answers has-value on
+**both** readings of "no value": a stored `null` and an absent key.
+
+**Grading, argued from what was measured rather than from custom.** This is
+`minor`, not `patch`, and the sibling card #13166 is why the distinction is
+worth stating: that one graded `patch` on the explicit ground that
+`InMemoryDriver.find()` was unaffected and only the non-exported reference
+matcher moved. Here the opposite is true — the live query path callers actually
+reach changes on **two published drivers**, on a filter operator in the public
+Filter Protocol. Measured on a 3-row fixture where one row stores `name: null`:
+
+| filter | before | after |
+|:--|:--|:--|
+| `{name: {$exists: true}}` | `['1','2','3']` | `['1','2']` |
+| `{name: {$exists: false}}` | `[]` | `['3']` |
+| `{$not: {name: {$exists: true}}}` | `[]` | `['3']` |
+
+The middle row is the harm the ruling's record calls the hardest live one: a
+caller asking for the rows with **no value** got an empty result — silent
+absence, with nothing to narrow — on three of the four exits. A caller who was
+getting nothing starts getting rows, which is a behaviour change however welcome
+it is.
+
+The **key-absent** reading is unchanged on every exit, by construction and by
+test: `{$ne: null}` already answers has-value there, so the column that agreed
+with the ruling before still agrees. It is kept in the suites as the control
+that the alignment moved only what it was meant to.
+
+**One thing the ruled lowering needed that the ruling did not name.** `{$ne:
+null}` / `{$eq: null}` reuse keys an author can write on the same field, so
+`{name: {$exists: true, $ne: 'b'}}` would assign `$ne` twice into one object and
+one of the two constraints would vanish — with *which* one decided by the
+author's key order. Measured unguarded: that filter answered `['1','3']` and its
+key-swapped twin answered `['1','2']`, where the reference matcher says `['1']`
+for both. Four composed cells that agreed with the reference matcher on `main`
+would have started disagreeing. So a lowered `$exists` whose key is already
+taken is promoted to its own `$and` branch instead of merged; a free key still
+merges inline. Both key orders now emit one document, and every composed cell
+measured agrees with the reference matcher — including two that did **not**
+agree before this change.
+
+⛔ Not included, deliberately: no `FILTER_LOGIC_CASES` enrolment and no
+`packages/spec` edit (the backends had to move first — that is the card's own
+step 4, and it is the next card), and nothing retires or discourages `$exists`
+in favour of `$null`. Whether one predicate should keep two authorable spellings
+is the consumer census, #13492.

packages/drivers/driver-memory/src/memory-analytics-echo-operator-coverage.test.ts

@@ -74,7 +74,7 @@
  * |:--|:--|:--|:--|
  * | `{name: {$in: [a, b]}}` | both rows | `name = a` — one row | `name IN (a, b)` |
  * | `{name: {$nin: [a]}}` | the other four | `name = a` — the **complement** | `(name IS NULL OR name NOT IN (a))` |
- * | `{name: {$exists: true}}` | four rows | `name = 1` — **no** rows | `name IS NOT NULL` |
+ * | `{name: {$exists: true}}` | five rows | `name = 1` — **no** rows | `name IS NOT NULL` |
  *
  * # Reverse verification
  *
@@ -347,22 +347,41 @@ describe('[#7117] the analytics echo renders the query it describes', () => {
     });
 
     /**
-     * The one cell where SQL cannot say what mingo says, asserted as an
-     * INEQUALITY so it cannot be closed in silence.
+     * [#13195] The one cell where SQL could not say what mingo said — CLOSED,
+     * and closed from the mingo side.
      *
-     * mingo's `$exists` tests KEY PRESENCE; a relational column always has it.
-     * A row storing an explicit `null` therefore satisfies `$exists: true` on
-     * `query()` and fails `IS NOT NULL` in the echo. `IS NOT NULL` is
-     * nonetheless the spelling both of this repo's other SQL lowerings use
-     * (`read-scope-sql.ts`'s `$exists` arm; `driver-sql`'s "a present field is a
-     * non-null column in SQL"), and it is a far smaller gap than the `name = 1`
-     * it replaces, which matched nothing at all.
+     * This assertion used to be an INEQUALITY, kept so the split could not be
+     * repaired in silence. The split was: mingo's `$exists` tests KEY PRESENCE
+     * and a relational column always has a key, so row 6 — which stores an
+     * explicit `null` — satisfied `$exists: true` on `query()` and failed
+     * `IS NOT NULL` in the echo. The chart and the statement drawn beside it
+     * answered the same query differently.
+     *
+     * The maintainer ruled on 2026-08-30 that `$exists` means HAS A VALUE
+     * (`!= null`) on every exit — #5298 leg 3 / #5369, shipped in PR #5962 and
+     * until then unmet on this face. `IS NOT NULL` was already the ruled
+     * answer, so the ECHO half is untouched and the executed half moved to meet
+     * it: `CUBE_OPERATOR_TO_MONGO_PREDICATE`'s `set` row now emits
+     * `{$ne: null}` / `{$eq: null}` instead of `{$exists: <bool>}`.
+     *
+     * ⛔ It was not re-baselined onto whatever the new pipeline printed. The
+     * target is the ECHO's pre-existing row set, which this note named as the
+     * ruled answer before the repair existed, and the `$exists` entry in the
+     * enumeration below is no longer skipped — it is asserted by the same rule
+     * as every other operator.
      */
-    it('documents the one `$exists` cell SQL cannot translate exactly', async () => {
+    it('the `$exists` cell SQL could not translate exactly is now exact', async () => {
       const where = { name: { $exists: true } };
-      // Row 6 stores an explicit `null`: present to mingo, NULL to SQL.
-      expect(await executedIds(where)).toEqual(['1', '2', '3', '4', '5', '6']);
+      // Row 6 stores an explicit `null`: no longer a value to either engine.
+      expect(await executedIds(where)).toEqual(['1', '2', '3', '4', '5']);
       expect(await echoIds(where)).toEqual(['1', '2', '3', '4', '5']);
+      expect(await executedIds(where)).toEqual(await echoIds(where));
+
+      // The other direction, which the old split hid entirely: asking for the
+      // rows with NO value used to return none of them on the executed side.
+      const none = { name: { $exists: false } };
+      expect(await executedIds(none)).toEqual(['6']);
+      expect(await echoIds(none)).toEqual(['6']);
     });
   });
 
@@ -432,11 +451,10 @@ describe('[#7117] the analytics echo renders the query it describes', () => {
       it(`${op}: running the echo returns exactly the rows the query returns`, async () => {
         const executed = await executedIds(where);
         const echoed = await echoIds(where);
-        if (op === '$exists') {
-          // The documented residue above — asserted there, skipped here so this
-          // loop stays a statement about every OTHER operator.
-          return;
-        }
+        // [#13195] `$exists` used to return early here — the documented residue
+        // above was asserted there and skipped in this loop, so the loop was a
+        // statement about every OTHER operator. The residue is gone, the skip
+        // with it, and this loop is now total over the face's vocabulary.
         expect(echoed, `${op}: the echo describes a different row set`).toEqual(executed);
       });
 

packages/drivers/driver-memory/src/memory-analytics.ts

@@ -262,9 +262,18 @@ const CUBE_OPERATOR_TO_MONGO_PREDICATE: Readonly<Record<CubeOperator, MongoPredi
   // negation has to wrap a pattern, which is exactly what the live query path
   // builds for `$notContains` (`memory-driver.ts` `normalizeFieldOperators`).
   notContains: ({ raw, substring }) => ({ $not: { $regex: substring(raw[0]) } }),
-  // A presence flag, not a comparand. The `raw.length === 0` arm keeps the old
-  // call site's reading of a valueless `set` ("does it exist" → true).
-  set: ({ raw }) => ({ $exists: raw.length > 0 ? Boolean(raw[0]) : true }),
+  // [#13195] A presence flag, not a comparand — and "present" means HAS A
+  // VALUE (`!= null`), never key presence: #5298 leg 3 / #5369, landed in PR
+  // #5962, ruled onto this face 2026-08-30. It used to emit `{$exists: <bool>}`
+  // and hand it to mingo, which reads key presence, so this exit EXECUTED the
+  // key-presence answer while {@link CUBE_OPERATOR_TO_SQL_PREDICATE} ECHOED
+  // `IS NOT NULL` beside it — the rows a chart was drawn from and the statement
+  // shown next to it answered the same query differently. The two now agree,
+  // and the residue that disagreement left in
+  // `memory-analytics-echo-operator-coverage.test.ts` is gone rather than
+  // documented. The `raw.length === 0` arm keeps the old call site's reading of
+  // a valueless `set` ("does it exist" → true).
+  set: ({ raw }) => ((raw.length === 0 || Boolean(raw[0])) ? { $ne: null } : { $eq: null }),
 });
 
 /**
@@ -381,18 +390,29 @@ function globSubstringPattern(value: unknown): string {
  * the new operator's SQL spelling. The totality is proven rather than defended,
  * so no future operator can silently render as an equality nobody wrote.
  *
- * # The one cell where SQL cannot say what mingo says
+ * # The `set` cell — once the one place SQL could not say what mingo said
  *
  * `set` renders `IS NOT NULL` / `IS NULL` — the spelling this repo's other two
  * SQL lowerings already use (`read-scope-sql.ts`'s `$exists` arm, `driver-sql`'s
- * "a present field is a non-null column in SQL"). It is not an exact
- * translation, and cannot be: mingo's `$exists` tests KEY PRESENCE, which a
- * relational column always has. A row storing an explicit `null` therefore
- * satisfies `$exists: true` on `query()` and fails `IS NOT NULL` in the echo.
- * That residue is inherent to describing a document store in SQL, it is pinned
- * as an explicit inequality in `memory-analytics-echo-operator-coverage.test.ts`
- * so it cannot be "fixed" in silence, and it is a far smaller gap than the
- * `name = 1` it replaces — which matched nothing at all.
+ * "a present field is a non-null column in SQL"). This row is UNCHANGED, and it
+ * is worth saying why it is now an exact translation rather than a documented
+ * residue.
+ *
+ * It used to be inexact in one direction only: the mingo twin emitted
+ * `{$exists: <bool>}`, mingo reads that as KEY PRESENCE, and a relational
+ * column always has a key — so a row storing an explicit `null` satisfied
+ * `$exists: true` on `query()` and failed `IS NOT NULL` in the echo. The chart
+ * and the statement drawn beside it answered the same query differently, and
+ * the gap was pinned as an explicit INEQUALITY in
+ * `memory-analytics-echo-operator-coverage.test.ts` so it could not be closed
+ * in silence.
+ *
+ * [#13195] It was not closed in silence: the maintainer ruled on 2026-08-30
+ * that `$exists` means HAS A VALUE (`!= null`) on every exit — #5298 leg 3 /
+ * #5369, shipped in PR #5962 and until then still unmet here — so the mingo
+ * twin now emits `{$ne: null}` / `{$eq: null}`. SQL's `IS NOT NULL` was already
+ * the ruled answer; it is the OTHER exit that moved to meet it. The inequality
+ * pin is now an equality, and this face no longer contradicts itself.
  */
 const CUBE_OPERATOR_TO_SQL_PREDICATE: Readonly<Record<CubeOperator, SqlPredicateBuilder>> = Object.freeze({
   // [#5373] A null comparand is a NULLNESS test, not a comparison. SQL's

packages/drivers/driver-memory/src/memory-driver-document-not.test.ts

@@ -208,7 +208,11 @@ describe('[#5324] InMemoryDriver.find compiles a document-level $not', () => {
    *
    *   `$exists`      REFERENCE is correct. `$exists` means "has a value"
    *                  (#5298 ③ / #5369, PR #5962), so mingo's key-presence
-   *                  reading is the divergent one. STILL OPEN — #13195.
+   *                  reading was the divergent one. CLOSED by #13195: the live
+   *                  path stopped handing `$exists` to mingo under its own name
+   *                  and lowers it to `{$ne: null}` / `{$eq: null}` — the
+   *                  spelling `$null` in the same method already used — so the
+   *                  two faces agree. Ruled 2026-08-30.
    *   `$nin`         LIVE was correct. Negative operators MATCH no-value rows —
    *                  #5146, extended by #5298, re-affirmed 2026-08-10 — so a
    *                  missing key satisfying `$nin` is the affirmed answer, and
@@ -235,26 +239,46 @@ describe('[#5324] InMemoryDriver.find compiles a document-level $not', () => {
    * matcher began printing — the target was the live path's pre-existing
    * answer, named as correct in this very note before the fix existed.
    *
-   * ⛔ The `$exists` row is untouched and stays a pinned divergence. It is a
-   * different cell with a different backend list (`driver-mongodb` reads
-   * key-presence too), and it belongs to #13195. What that row still shows is
-   * why this pin exists — this package answers with two faces, so a statement
-   * like "driver-memory already reads has-value" is true of the reference
-   * matcher and FALSE of the live query path users actually reach.
+   * ⚠️ [#13195, ruled 2026-08-30] The third cell has now converged too, and by
+   * the same discipline: the target was the REFERENCE column, which this note
+   * named correct before the fix existed, not whatever the live path began
+   * printing. `driver-mongodb` — which read key-presence for its own,
+   * wire-level reason — moved in the same change, so the statement this row
+   * used to disprove is finally true of the whole package AND of the other
+   * document-shaped backend.
+   *
+   * ⛔ What the row still shows, and why the pin stays: this package answers
+   * with two faces. "driver-memory reads has-value" was true of the reference
+   * matcher and FALSE of the live query path users actually reach, for the
+   * three months between #5962 and #13195. Asserting the two columns against
+   * each other — rather than each against a literal — is what makes a future
+   * one-sided edit fail here.
    */
-  describe('[#5299] the settled cells, live vs reference — $nin / $notContains converged (#13166), $exists still open (#13195)', () => {
+  describe('[#5299] the settled cells, live vs reference — $nin / $notContains converged (#13166), $exists converged (#13195)', () => {
     const liveVsReference = async (where: unknown) => ({
       live: await idsFrom(nulled, where),
       reference: NULLED.filter((r) => match(r, where)).map((r) => r.id),
     });
 
-    it('$exists on a present-but-null field: mingo says "the key is there", the matcher says "no value"', async () => {
+    it('$exists on a present-but-null field: the two faces now AGREE (#13195)', async () => {
+      // Was `live: ['1','2','3','4']` — mingo said "the key is there" while the
+      // matcher said "no value". The REFERENCE column is unchanged, and it is
+      // the column this note already named correct.
       expect(await liveVsReference({ stage: { $exists: true } })).toEqual({
-        live: ['1', '2', '3', '4'],
+        live: ['1', '2'],
         reference: ['1', '2'],
       });
     });
 
+    it('$exists: false on a present-but-null field: the two faces agree there too (#13195)', async () => {
+      // The direction the old pin never recorded, and the worse one: the live
+      // path returned NOTHING for the query asking for the rows with no value.
+      expect(await liveVsReference({ stage: { $exists: false } })).toEqual({
+        live: ['3', '4'],
+        reference: ['3', '4'],
+      });
+    });
+
     it('$nin on an ABSENT field: the two faces now AGREE (#13166)', async () => {
       // Was `reference: ['2']` — the matcher's `value === undefined` guard
       // short-circuited before the `$nin` arm ran. The LIVE column is unchanged,