Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
runtime `AreRecordsEncrypted` gate (so default-config plaintext databases benefit without
`NoEncryptMode`). Fixed-size hash-indexed SET columns are re-pointed with one lock per index
(`HashIndex.RemoveBatchKeys`/`AddBatchKeys`).
- **FW contiguous path: per-key B-tree probe replaced by decode verification + `DeleteBulk` (M3)** -
the shared B8/B9 probe resolved every key with one `Index.Search` (~10K per batch) and the FW
DELETE removed the PK entries one `Delete` at a time. The probe now locates only the FIRST key
through the tree, computes the remaining positions by the fixed-width stride, reads the
contiguous span once and verifies every record by its length prefix AND its decoded fixed-width
PK slot (equal to the batch key); the DELETE removes the PK entries with one `DeleteBulk` pass.
The same batches are rejected as before (gaps/tombstones surface as prefix mismatches; a
differing PK falls back like a tree miss). Fair-PK median-of-3 (sequential, uncontended):
fixed-width UPDATE ~245K and DELETE ~172K ops/s on this branch (master baseline reported in the PR).
- **Sequential ascending-PK batch resolution for legacy DELETE (Fase B: legacy fast paths)** -
`DeleteMultipleKeys` on a legacy (variable-length, plaintext, non-fixed-width) Columnar table now
resolves a strictly-ascending INTEGER-PK literal batch with a single sequential decode pass that
Expand Down
1 change: 1 addition & 0 deletions docs/performance/EXECUTION_PLAN_UPDATE_DELETE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
| #377 | **Commit-marker writes gebatcht** (`TombstoneRecords` patcht alle markers eerst in het whole-file snapshot — markers mogen page-grenzen kruisen — en flusht elke geraakte storage-page één keer i.p.v. één 4B-pwrite per marker) | fair-PK (median, zelfde machine): legacy ~70K → **~81K ops/s** (+16%); fixed-width ~97K → **~141K ops/s** (+45%) — DELETE-gap vs SQLite op FW → ~2,4x |
| P5 (deze branch) | **Duplicate-key hash-removal O(m·n) → O(n) per key** (`RemoveBatchKeys`/`RemoveBatch`: directe allocatie-vrije pad voor single-row keys; gedupliceerde keys gedeferred in set + één O(list)-compaction) | correct (regressietests op volle + partiële duplicate-groepen, beide backends, incl. reopen); benchmark-neutraal op unieke keys |
| C6 (deze branch) | **In-place UPDATE-overwrites per storage-page geflusht** (`FlushBufferedOverwrites`: page-content één keer lezen, payloads patchen, één write per page; cross-page records direct) | fair-PK: FW-UPDATE ~88K → **~153-164K ops/s** (+75-85%, gap → ~1,7x SQLite); legacy-UPDATE ~53-56K → ~64-70K; DELETE binnen ruis; full suite 1765/0 |
| M3 (deze branch) | **FW contigu pad zonder per-key B-tree probe + `DeleteBulk`** (probe: 1 tree-search voor eerste key, stride-posities, span één keer lezen & per record prefix+PK-slot verifiëren) | fair-PK median-of-3 (sequenced): FW-DELETE gap vs SQLite ~2,6x → **~2,1x**; FW-UPDATE ~0,78x → ~0,82x van SQLite; legacy-DELETE gap ~4,3x → ~3,0x; full suite 1766/0 |

**Root cause (niet in Grok-doc):** `ExecuteBatchSQL` draait elke batch in een storage-transactie; zonder #368 deed batch-DELETE nog steeds de #366 full-file compactie (~690 ms in `tableFlushLoop`). Daardoor waren eerdere “winst”-metingen niet-duurzaam/logisch-only.

Expand Down
58 changes: 34 additions & 24 deletions src/SharpCoreDB/DataStructures/Table.CRUD.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3592,11 +3592,16 @@ this.storage is null ||
}

/// <summary>
/// Shared B8/B9 probe: resolves each key's record position through the PK B-tree, requires the
/// positions to be physically adjacent at the fixed-width stride, reads the whole contiguous
/// span through the storage layer's cached handle and verifies every 4-byte length prefix.
/// Returns the raw span bytes, or <see langword="null"/> so the caller falls back to the generic
/// per-row loop — nothing is modified before this succeeds.
/// Shared B8/B9 probe: locates the FIRST key through the PK B-tree, requires the records to be
/// physically adjacent at the fixed-width stride, reads the whole contiguous span through the
/// storage layer's cached handle and verifies every record by its 4-byte length prefix AND its
/// decoded PK slot (equal to the batch key). It deliberately avoids one B-tree search per key:
/// with the whole span already in memory, decoding each record's fixed-width PK slot (~100 ns)
/// is two orders of magnitude cheaper than ~10K tree searches and rejects the same batches
/// (gaps/tombstones surface as prefix mismatches; a record whose PK differs from the requested
/// key falls back like the old per-key search miss). Returns the raw span bytes, or
/// <see langword="null"/> so the caller falls back to the generic per-row loop — nothing is
/// modified before this succeeds.
/// </summary>
private byte[]? TryReadContiguousFixedWidthRecords(
string[] keys,
Expand All @@ -3605,6 +3610,10 @@ this.storage is null ||
long[] positions)
{
int count = keys.Length;
if (this.PrimaryKeyIndex < 0)
{
return null;
}

var first = this.Index.Search(keys[0]);
if (!first.Found)
Expand All @@ -3613,18 +3622,11 @@ this.storage is null ||
}

long basePosition = first.Value;
positions[0] = basePosition;
long expected = basePosition;
for (int i = 1; i < count; i++)
for (int i = 0; i < count; i++)
{
expected += stride;
var search = this.Index.Search(keys[i]);
if (!search.Found || search.Value != expected)
{
return null;
}

positions[i] = expected;
expected += stride;
}

long totalBytes = stride * count;
Expand All @@ -3639,12 +3641,23 @@ this.storage is null ||
return null;
}

int pkIdx = this.PrimaryKeyIndex;
var pkType = this.ColumnTypes[pkIdx];
for (int i = 0; i < count; i++)
{
int prefix = BinaryPrimitives.ReadInt32LittleEndian(raw.AsSpan((int)(i * stride), 4));
var record = raw.AsSpan((int)(i * stride), 4 + layout.FixedSize);
int prefix = BinaryPrimitives.ReadInt32LittleEndian(record);
if (prefix != layout.FixedSize)
{
return null;
return null; // tombstoned slot or layout mismatch -> generic per-row path
}

var pkSlot = record.Slice(4 + layout.Offsets[pkIdx], layout.SlotSizes[pkIdx]);
var pkValue = ReadTypedValueFromSpan(pkSlot, pkType, out _);
if (pkValue is null || pkValue is System.DBNull ||
!string.Equals(pkValue.ToString(), keys[i], StringComparison.Ordinal))
{
return null; // record PK differs from the requested key -> generic per-row path
}
}

Expand Down Expand Up @@ -3733,14 +3746,11 @@ this.storage is null ||
return false;
}

// Remove PK entries (the keys are the WHERE literals) and then every loaded hash-index
// entry, decoding only the indexed columns from the raw fixed-width records (no full-row
// deserialization). Variable values resolve through the overflow arena, mirroring the
// fixed-width codec used by the generic path.
for (int i = 0; i < count; i++)
{
this.Index.Delete(keys[i]);
}
// Remove PK entries (the keys are the WHERE literals) in one sorted bulk pass and then every
// loaded hash-index entry, decoding only the indexed columns from the raw fixed-width records
// (no full-row deserialization). Variable values resolve through the overflow arena, mirroring
// the fixed-width codec used by the generic path.
this.Index.DeleteBulk(keys);

var arena = GetOverflowArena();
foreach (var (colName, hashIdx) in this.hashIndexes)
Expand Down
Loading