perf: batch tombstone writes (C4) + structured batch DELETE (B3)

docs/CHANGELOG.md

@@ -42,17 +42,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   overflow arena is reclaimed on the next explicit VACUUM/compaction. Regression: delete half the
   rows, `Flush`, reopen ÔÇö exactly the remaining rows come back. Measured DELETE in the `--pk`
   harness now includes this durability rewrite (~18.6K ops/s when deleting 10K of 100K rows).
-- **Durable DELETE is now in-place via tombstones (no rewrite)** ÔÇö non-transactional Columnar
-  deletes write a tombstone marker (the record's 4-byte length prefix is replaced by the NEGATIVE
-  slot size) instead of queueing a flush-time rewrite, and every raw record enumerator/compactor
-  skips the slot, so DELETE survives a reopen in O(delete). Flush/dispose compaction now only
-  covers transactional deletes (a marker written before commit would survive a rollback that
-  restores the row in the PK index; those deletes are compacted against the current PK after commit
-  ÔÇö rollback-safe). Tombstoned space is reclaimed by the tombstone-aware `CompactTable`, including
-  the ULID-migration compaction (which previously produced an empty file when tombstones were
-  present because `CompactTable` broke on a negative prefix). Measured `--pk` DELETE (10K of 100K
-  rows): ~0.54s/18.6K ops/s (flush rewrite)  **~0.16s/~64K ops/s (legacy)** and
-  **~0.12s/~81K ops/s (fixed-width)** ÔÇö DELETE is back on par with UPDATE.
+- **Durable DELETE is now in-place via tombstones (no rewrite)** ÔÇö Columnar deletes write a
+  tombstone marker (the record's 4-byte length prefix is replaced by the NEGATIVE slot size)
+  instead of queueing a flush-time rewrite, and every raw record enumerator/compactor skips the
+  slot, so DELETE survives a reopen in O(delete). Non-transactional deletes write the marker at
+  delete time; transactional deletes (e.g. any `ExecuteBatchSQL` batch, which runs inside a storage
+  transaction) buffer the offsets and apply the markers at COMMIT ÔÇö rollback discards the buffer,
+  so a rolled-back delete keeps its row, and the flush-time full-file rewrite (`CompactPendingDeletes`)
+  is no longer on the batch-DELETE path. Tombstoned space is reclaimed by the tombstone-aware
+  `CompactTable`, including the ULID-migration compaction (which previously produced an empty file
+  when tombstones were present because `CompactTable` broke on a negative prefix). Measured `--pk`
+  DELETE (10K of 100K rows): ~0.54s/18.6K ops/s (flush rewrite)  **~0.16s/~64K ops/s (legacy)** and
+  **~0.13s/~78K ops/s (fixed-width)**; comparative-harness DELETE (docs table): SQL ~0.82s/~12K 
+  **~0.24s/~41K ops/s**, Direct ~0.63s/~16K  **~0.17s/~58K ops/s**  DELETE is back on par with UPDATE.
 - **Dedicated SQL batch-INSERT fast path (WP14)** ÔÇö `ExecuteBatchSQL` INSERTs no longer build a
   per-row `Dictionary<string, object>`; VALUES clauses are parsed directly into column-ordered
   `object[]` rows (`PreparedInsertStatement.ParseValuesToArray`) and inserted via the new

src/SharpCoreDB/DataStructures/Table.CRUD.cs

@@ -2655,11 +2655,16 @@
 
             if (this.storage is { IsInTransaction: true })
             {
-                // Transactional delete: writing the tombstone now would survive a rollback that
-                // restores the row in the PK index, so defer the physical removal to the post-commit
-                // flush compaction (rollback-safe: that rewrite is driven by the CURRENT PK, which
-                // still contains the row after a rollback).
-                Interlocked.Add(ref _pendingLogicalDeletes, recordsToDelete.Count);
+                // Transactional delete: buffer the physical offsets so the in-place marker is
+                // applied at COMMIT (rollback discards the buffer). Durable in O(delete) — the
+                // flush-time full-file rewrite is no longer needed for transactional deletes.
+                foreach (var position in positions)
+                {
+                    if (position >= 0)
+                    {
+                        this.storage.BufferTombstoneForCommit(DataFile, position);
+                    }
+                }
             }
             else
             {
@@ -3016,6 +3021,160 @@
         }
     }
 
+    /// <summary>
+    /// Structured variant of <see cref="DeleteMultiple"/> used by the SQL batch dispatcher for
+    /// canonical single-row DELETE statements (`DELETE FROM t WHERE col = literal`, as detected by
+    /// the canonical-DML scanner). The WHERE column and raw literal are already known, so the
+    /// per-statement `col = literal` string rebuild and the second <see cref="TryParseSimpleWhereClause"/>
+    /// pass are skipped on the hot path. Semantics mirror <see cref="DeleteMultiple"/> exactly
+    /// (PK fast path → hash-index fast path → generic fallback with a lazily rebuilt WHERE string,
+    /// only reached for unusual shapes).
+    /// </summary>
+    /// <param name="conditions">Column name and RAW literal value (quotes as written in SQL).</param>
+    [MethodImpl(MethodImplOptions.AggressiveOptimization)]
+    internal void DeleteMultipleKeys(List<(string Column, string Literal)> conditions)
+    {
+        if (this.isReadOnly) throw new InvalidOperationException(ReadOnlyDeleteError);
+        if (conditions.Count == 0) return;
+
+        this.rwLock.EnterWriteLock();
+        try
+        {
+            var engine = GetOrCreateStorageEngine();
+            EnsureAllRegisteredIndexesLoaded();
+
+            // B9 single-pass contiguous DELETE consumes WHERE strings; attempt it (without touching
+            // anything) only when every condition is a literal on the PK column, like the caller.
+            if (this.PrimaryKeyIndex >= 0 && AllPkLiteralConditions(conditions))
+            {
+                var wheres = new List<string>(conditions.Count);
+                foreach (var (col, literal) in conditions)
+                {
+                    wheres.Add(col + " = " + literal);
+                }
+
+                if (TryBulkDeleteContiguousFixedWidth(wheres))
+                {
+                    return;
+                }
+            }
+
+            var recordsToDelete = new List<(long storagePosition, Dictionary<string, object> row)>();
+
+            foreach (var (col, literal) in conditions)
+            {
+                string value = UnquoteSqlLiteral(literal);
+
+                // Issue #7 PK fast path (structured: no WHERE-string re-parse).
+                if (StorageMode != StorageMode.PageBased &&
+                    this.PrimaryKeyIndex >= 0 &&
+                    string.Equals(col, this.Columns[this.PrimaryKeyIndex], StringComparison.OrdinalIgnoreCase))
+                {
+                    var fastSearch = this.Index.Search(value);
+                    if (fastSearch.Found)
+                    {
+                        var fastData = engine.Read(Name, fastSearch.Value);
+                        if (fastData != null)
+                        {
+                            var fastRow = DeserializeRowFromSpan(fastData);
+                            if (fastRow != null)
+                            {
+                                recordsToDelete.Add((fastSearch.Value, fastRow));
+                            }
+                        }
+
+                        continue;
+                    }
+                }
+                // Hash index fast path.
+                if (this.registeredIndexes.ContainsKey(col))
+                {
+                    EnsureIndexLoaded(col);
+                    if (this.hashIndexes.TryGetValue(col, out var hashIndex))
+                    {
+                        var colIdx = this.Columns.IndexOf(col);
+                        if (colIdx >= 0)
+                        {
+                            var key = ParseValueForHashLookup(value, this.ColumnTypes[colIdx]);
+                            if (key != null)
+                            {
+                                foreach (var pos in hashIndex.LookupPositionsUnsafe(key))
+                                {
+                                    var data = engine.Read(Name, pos);
+                                    if (data != null)
+                                    {
+                                        var row = DeserializeRowFromSpan(data);
+                                        if (row != null) recordsToDelete.Add((pos, row));
+                                    }
+                                }
+
+                                continue;
+                            }
+                        }
+                    }
+                }
+
+                // Generic fallback (rare for canonical shapes): rebuild the WHERE string exactly as
+                // the caller would have and mirror DeleteMultiple.
+                string where = col + " = " + literal;
+                if (this.PrimaryKeyIndex >= 0)
+                {
+                    var rows = SelectInternal(where, orderBy: null, asc: true, noEncrypt: false);
+                    var pkCol = this.Columns[this.PrimaryKeyIndex];
+                    foreach (var row in rows)
+                    {
+                        if (row.TryGetValue(pkCol, out var pkValue) && pkValue != null)
+                        {
+                            var searchResult = this.Index.Search(pkValue.ToString() ?? string.Empty);
+                            if (searchResult.Found)
+                                recordsToDelete.Add((searchResult.Value, row));
+                        }
+                    }
+                }
+                else
+                {
+                    foreach (var (storageRef, data) in engine.GetAllRecords(Name))
+                    {
+                        var row = DeserializeRowFromSpan(data);
+                        if (row != null && (string.IsNullOrEmpty(where) || EvaluateSimpleWhere(row, where)))
+                            recordsToDelete.Add((storageRef, row));
+                    }
+                }
+            }
+
+            if (recordsToDelete.Count == 0) return;
+
+            DeleteRecordsCore(recordsToDelete);
+        }
+        finally
+        {
+            this.rwLock.ExitWriteLock();
+        }
+    }
+
+    private bool AllPkLiteralConditions(List<(string Column, string Literal)> conditions)
+    {
+        var pkCol = this.Columns[this.PrimaryKeyIndex];
+        foreach (var (col, _) in conditions)
+        {
+            if (!string.Equals(col, pkCol, StringComparison.OrdinalIgnoreCase))
+            {
+                return false;
+            }
+        }
+
+        return true;
+    }
+
+    /// <summary>
+    /// Mirrors the value normalization of <see cref="TryParseSimpleWhereClause"/> (trim whitespace,
+    /// then trim enclosing <c>'</c>/<c>"</c> quotes) on a raw SQL literal.
+    /// </summary>
+    private static string UnquoteSqlLiteral(string raw)
+    {
+        return raw.AsSpan().Trim().Trim("'\"".AsSpan()).ToString();
+    }
+
     /// <summary>
     /// Shared B8/B9 probe: resolves each key's record position through the PK B-tree, requires the
     /// positions to be physically adjacent at the fixed-width stride, reads the whole contiguous
@@ -3214,9 +3373,15 @@
 
         if (this.storage is { IsInTransaction: true })
         {
-            // Transactional delete: defer physical removal to the post-commit flush compaction
-            // (see DeleteRecordsCore — a tombstone would survive a rollback that restores the row).
-            Interlocked.Add(ref _pendingLogicalDeletes, count);
+            // Transactional delete: buffer the physical offsets so the in-place marker is applied
+            // at COMMIT (see DeleteRecordsCore — rollback discards the buffer).
+            foreach (var position in positions)
+            {
+                if (position >= 0)
+                {
+                    this.storage.BufferTombstoneForCommit(DataFile, position);
+                }
+            }
         }
         else
         {

src/SharpCoreDB/DataStructures/Table.Compaction.cs

@@ -8,7 +8,6 @@ namespace SharpCoreDB.DataStructures;
 using SharpCoreDB.Storage.Engines;
 using System;
 using System.Collections.Generic;
-using System.IO;
 using System.Linq;
 using System.Threading.Tasks;
 
@@ -113,7 +112,8 @@ public void CompactPendingDeletes()
     /// present in the data file, making the Columnar logical delete durable across reopen (readers
     /// skip the marker) without rewriting the remaining rows. Rows appended inside an uncommitted
     /// transaction (positions beyond the current file length) are skipped — their delete commits
-    /// with the rest of the transaction.
+    /// with the rest of the transaction. The storage layer batches the in-place marker writes and
+    /// de-duplicates page-cache evictions per page.
     /// </summary>
     private void TombstoneDeletedPositions(long[] positions)
     {
@@ -122,14 +122,7 @@ private void TombstoneDeletedPositions(long[] positions)
             return;
         }
 
-        long fileLength = File.Exists(DataFile) ? new FileInfo(DataFile).Length : 0;
-        foreach (var position in positions)
-        {
-            if (position >= 0 && position + 4 <= fileLength)
-            {
-                this.storage.TombstoneRecord(DataFile, position);
-            }
-        }
+        this.storage.TombstoneRecords(DataFile, positions);
     }
 
     /// <summary>

src/SharpCoreDB/Database/Execution/Database.Batch.cs

@@ -923,10 +923,12 @@
     /// <param name="tableName">The parsed table name.</param>
     /// <param name="where">The parsed WHERE clause.</param>
     /// <returns>True if the statement was successfully parsed as a DELETE.</returns>
-    private bool TryParseDeleteForBatch(string sql, out string tableName, out string where)
+    private bool TryParseDeleteForBatch(string sql, out string tableName, out string where, out string? whereColumn, out string? whereLiteral)
     {
         tableName = string.Empty;
         where = string.Empty;
+        whereColumn = null;
+        whereLiteral = null;
 
         // Phase-2 fast path: canonical single-row shape
         // `DELETE FROM <table> WHERE <col> = <literal>` — regex-free.
@@ -938,7 +940,10 @@
             }
 
             tableName = fastTable;
-            where = whereCol + " = " + whereValRaw;
+            whereColumn = whereCol;
+            whereLiteral = whereValRaw;
+            // Where stays empty for canonical statements — the table layer reconstructs it only
+            // when a fallback actually needs it (B3: no per-statement string allocation).
             return true;
         }
 
@@ -997,7 +1002,9 @@
 
         // ✅ PERF: Group UPDATE/DELETE by table for single-lock batch execution
         Dictionary<string, List<(string where, Dictionary<string, object> updates)>> updatesByTable = [];
-        Dictionary<string, List<string>> deletesByTable = [];
+        // Canonical DELETE statements carry the pre-parsed column + raw literal so the table layer
+        // can skip the per-statement WHERE rebuild/re-parse (B3); Where stays empty for those.
+        Dictionary<string, List<(string Where, string? Column, string? Literal)>> deletesByTable = [];
 
         foreach (var sql in statements)
         {
@@ -1031,14 +1038,14 @@
                 }
                 updList.Add((updWhere, updSets));
             }
-            else if (TryParseDeleteForBatch(sql, out var delTableName, out var delWhere))
+            else if (TryParseDeleteForBatch(sql, out var delTableName, out var delWhere, out var delCol, out var delLiteral))
             {
                 if (!deletesByTable.TryGetValue(delTableName, out var delList))
                 {
                     delList = [];
                     deletesByTable[delTableName] = delList;
                 }
-                delList.Add(delWhere);
+                delList.Add((delWhere, delCol, delLiteral));
             }
             else
             {
@@ -1097,11 +1104,42 @@
                 }
 
                 // ✅ PERF: Batch DELETE — single lock per table instead of per-statement
-                foreach (var (tableName, wheres) in deletesByTable)
+                foreach (var (tableName, deletes) in deletesByTable)
                 {
                     if (tables.TryGetValue(tableName, out var tbl) && tbl is DataStructures.Table concreteDelete)
                     {
-                        concreteDelete.DeleteMultiple(wheres);
+                        // Canonical batches go through the structured path (no WHERE rebuild/re-parse);
+                        // any non-canonical statement forces the string form for the whole table.
+                        bool allCanonical = true;
+                        foreach (var (_, column, _) in deletes)
+                        {
+                            if (column is null)
+                            {
+                                allCanonical = false;
+                                break;
+                            }
+                        }
+
+                        if (allCanonical)
+                        {
+                            var keys = new List<(string Column, string Literal)>(deletes.Count);
+                            foreach (var (_, column, literal) in deletes)
+                            {
+                                keys.Add((column!, literal!));
+                            }
+
+                            concreteDelete.DeleteMultipleKeys(keys);
+                        }
+                        else
+                        {
+                            var wheres = new List<string>(deletes.Count);
+                            foreach (var (where, column, literal) in deletes)
+                            {
+                                wheres.Add(where.Length > 0 ? where : column + " = " + literal);
+                            }
+
+                            concreteDelete.DeleteMultiple(wheres);
+                        }
                     }
                 }