From e95e7aa4e89d6653ad8948f6e00714a6a8f103dd Mon Sep 17 00:00:00 2001 From: MPCoreDeveloper Date: Fri, 4 Sep 2026 20:13:06 +0200 Subject: [PATCH] docs(2.0.0.2): finalize release documentation - 2.0.0.2_WHAT_CHANGED: update quality gate to 1777 core + CI-filter suites, PR range #376-#390, add reopen data-integrity fixes and legacy-delete purge to the DELETE-durability row - INDEX: status 2.0.0.2 + list 2.0.0.2 release notes under 0b - NuGet.README (packed as package readme): add the reopen data-integrity fixes to What is New in 2.0.0.2 - upgrade-and-downgrade: legacy-table UPDATE/DELETE is safe in 2.0.0.2 (delete purge); refresh test count --- docs/2.0.0.2_WHAT_CHANGED.md | 24 +++++++++++++++++++----- docs/INDEX.md | 4 +++- docs/manual/upgrade-and-downgrade.md | 7 ++++--- src/SharpCoreDB/NuGet.README.md | 6 ++++++ 4 files changed, 32 insertions(+), 9 deletions(-) diff --git a/docs/2.0.0.2_WHAT_CHANGED.md b/docs/2.0.0.2_WHAT_CHANGED.md index b9b9fdd3..57978a65 100644 --- a/docs/2.0.0.2_WHAT_CHANGED.md +++ b/docs/2.0.0.2_WHAT_CHANGED.md @@ -1,10 +1,12 @@ # SharpCoreDB 2.0.0.2 — What changed & the 2.x engine vs the 1.9.x line - **Release:** `2.0.0.2` (2026-09-04) — 4-part patch release on the 2.0.0.x line. -- **Quality gate:** full suite **1770 tests / 0 failed**, CI green on Windows/Ubuntu/macOS + - CodeQL + SonarCloud + Server Compatibility Smoke for every merged PR. -- **Contents:** everything merged since 2.0.0.1 — the perf/hardening batch of PRs #376-#388 — - plus the consolidated 2.x-vs-1.9.x overview below. +- **Quality gate:** full core suite **1777 tests / 0 failed**; CI-filtered core suite **1674 / 0**, + plus SharpCoreDB.CQRS.Tests 64/64, VectorSearch 143, EF Core 116 and Functional.Linq2DB 24 — + validated on Windows/Ubuntu/macOS + CodeQL + SonarCloud + Server Compatibility Smoke. +- **Contents:** everything merged since 2.0.0.1 — the perf/hardening batch (PRs #376-#388), the + release documentation PR #389, and the reopen **data-integrity hardening** PR #390 described + below. - See `docs/CHANGELOG.md` → `[2.0.0.2]` for the detailed change list and `docs/2.0.0.0_WHAT_CHANGED.md` for the full 2.0.0.0 report (this document is the short, forward-looking summary). @@ -15,7 +17,7 @@ | Step | 1.9.x | 2.x | |---|---|---| | Record layout | variable-length length-prefixed rows only | **Fixed-width record layout** (constant stride + out-of-line `.ovf` overflow arena) for columnar PK tables; **default for new PK tables** since B7; per-table flag **persisted in metadata** (authoritative on reopen) with **1.x → 2.0 auto-migration** (opt-in via `FixedWidthRecordLayout`) | -| DELETE durability | logical delete + full-file rewrite at flush | **Commit-time tombstone markers** (in-place negative length-prefix) → durable in O(delete), no full-file rewrite | +| DELETE durability | logical delete + full-file rewrite at flush | **Commit-time tombstone markers** (in-place negative length-prefix) → durable in O(delete), no full-file rewrite; legacy (1.x) variable-length tables additionally purge remaining versions of a deleted key so stale rows cannot resurrect after reopen | | In-place writes | only same-length patches, buffered later | **Buffered in-place overwrites flushed per storage page** (C6); markers also batched per page (C5) | | Single-file (`.scdb`) | fixed-layout registry, compression bugs | **Format v2** (block registry/FSM/table directory), crash-safe auto-migration, block-level Brotli/GZip/Zstd, envelope encryption, concurrent-write corruption fix (2.0.0.1) | @@ -46,6 +48,18 @@ PageBased (not yet OLTP-ready: ~26K UPDATE ops/s vs ~245K on the fixed-width Columnar path); PageBased remains explicit opt-in. - 4-part patch versioning (`n.n.n.n`) since 2.0.0.0. +- **Reopen data-integrity fixes (PR #390, found by the new reopen round-trip matrix):** + - **Overflow-arena reload** — a valid zero-length record (empty TEXT/BLOB value) was treated as + end-of-file by the `.ovf`/data-file loader, silently dropping every later row/block after a + reopen; length-0 records are now parsed as valid empty payloads. + - **Single-file fixed-width overflow arena** — freed blocks were serialized as zero-filled gaps + that misaligned the sequential loader; dead slots are now persisted as negative-length + tombstone markers tracked across sessions. + - **Legacy (1.x) variable-length DELETE** — only the newest row version was tombstoned, so an + older stale UPDATE version won the reopen index rebuild and resurrected the deleted row; + DELETE now purges every remaining record of a deleted key (1.x-upgrade path made safe). + - Locked in by `ReopenRoundTripMatrixTests` (four storage variants × insert/update/delete × + three reopen rounds) and dedicated empty-value/delete regressions. ## Benchmark snapshot (fair-PK, tuned harness config, median-of-3, same machine) diff --git a/docs/INDEX.md b/docs/INDEX.md index bcada197..d2b855d6 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -1,6 +1,6 @@ # SharpCoreDB Documentation Index -**Status:** Active documentation set (`2.0.0.0`) +**Status:** Active documentation set (`2.0.0.2`) ## 0. Manual (start here) @@ -23,6 +23,8 @@ guide explaining when SharpCoreDB is fastest. ## 0b. Release notes +- `2.0.0.2_WHAT_CHANGED.md` — **what's new in 2.0.0.2** (2.x-vs-1.9.x major steps + the hardening + and data-integrity batch) - `2.0.0.0_WHAT_CHANGED.md` — **what's new in v2.0 vs v1.9** (features + benchmarks vs SQLite/LiteDB) ## 1. Project Entry Points diff --git a/docs/manual/upgrade-and-downgrade.md b/docs/manual/upgrade-and-downgrade.md index 13daa28d..15a0df68 100644 --- a/docs/manual/upgrade-and-downgrade.md +++ b/docs/manual/upgrade-and-downgrade.md @@ -7,7 +7,7 @@ Downgrade below this line is **not supported** after the database contains certa | Scenario | Supported? | Notes | |---|---|---| -| Open a legacy (pre-fixed-width, variable-length) database with the current version | ✅ Yes | Plaintext + encrypted-header legacy files are read natively; a table without `IsFixedWidthRecords` stays variable-length (the persisted flag is authoritative). | +| Open a legacy (pre-fixed-width, variable-length) database with the current version | ✅ Yes | Plaintext + encrypted-header legacy files are read natively; a table without `IsFixedWidthRecords` stays variable-length (the persisted flag is authoritative). UPDATE/DELETE on such tables is safe in 2.0.0.2: DELETE purges older stale row versions of a deleted key so deleted rows cannot resurrect after a reopen. | | Open a current fixed-width Columnar database with the current version after reopen cycles | ✅ Yes | Covered by `FormatCompatPolicyTests` / `FixedWidthMigrationTests`; per-table flag persisted. | | Migrate a legacy table to fixed-width | ✅ Yes (opt-in) | `DatabaseConfig.FixedWidthRecordLayout = true` triggers `MigrateToFixedWidth()` on open (never on read-only opens). | | Open a database written by the **current** version (contains commit-time tombstone markers) with an **older** version that predates markers | ❌ **Not supported** | Deleted rows are stored as **negative length-prefix markers** (introduced with commit-time tombstones). An older binary that only knows positive length prefixes cannot skip these records and must not be pointed at such a file. | @@ -32,8 +32,9 @@ Recommended upgrade order: ## Verification - In-repo: `FormatCompatPolicyTests`, `FixedWidthMigrationTests`, - `DefaultEngineSelectionTests`, and the full suite (currently 1768+ tests) cover legacy reads, - opt-in migration, marker durability across reopen cycles, and the default fast-path engine. + `DefaultEngineSelectionTests`, `ReopenRoundTripMatrixTests` and the full suite (currently 1777+ + tests) cover legacy reads, opt-in migration, marker durability across reopen cycles, and the + default fast-path engine. - Planned (CI): a true **cross-version** job that writes a database with a pinned older commit and reads it with `master` (requires an old-binary generator; tracked as follow-up). diff --git a/src/SharpCoreDB/NuGet.README.md b/src/SharpCoreDB/NuGet.README.md index ffa2ce86..ec999c7a 100644 --- a/src/SharpCoreDB/NuGet.README.md +++ b/src/SharpCoreDB/NuGet.README.md @@ -22,6 +22,12 @@ SharpCoreDB is a modern, encrypted, file-based database engine with SQL support, OLTP-ready); upgrade/downgrade policy documented with format-compat regression tests; a pure-default-config benchmark arm (`--pk-default`) and a same-window interleaved A/B mode (`--pk-ab`) now guard the numbers. +- **Reopen data-integrity fixes** — a reopen round-trip hardening sweep fixed three silent-data-loss + edge cases: empty TEXT/BLOB values no longer truncate the overflow-arena reload; the single-file + fixed-width overflow arena persists freed slots as markers (no misaligned zero gaps); and legacy + (1.x-upgrade) variable-length tables purge older row versions when a row is deleted, so deleted + rows can no longer resurrect after a reopen. Locked in by the new `ReopenRoundTripMatrixTests` + and outbox/CQRS integration coverage. - Full report (incl. the 2.x-vs-1.9.x major steps): `docs/2.0.0.2_WHAT_CHANGED.md`. ## What's New in 2.0.0.1