fix!: skip non-Merk descendants during recursive discovery

[QuantumExplorer]

Issue being fixed or feature implemented

Closes #890. Recursive discovery tries to decode populated non-Merk descendants as Merk nodes, preventing discovery and deletion of an otherwise healthy ancestor tree. Reproduced independently for MMR, bulk append, dense, commitment, and private document storage, plus a wrapped MMR.

What was done?

Use separate find_subtrees_v0 and find_subtrees_v1 functions in version-specific modules, selected by a dispatcher. V0 preserves the original implementation. V1 classifies descendants using their already-decoded elements, keeping every tree namespace in the cleanup result but only enqueueing Merk-backed descendants for traversal. The shared cleanup still clears non-Merk payloads and all indexed-axis namespaces.

Enable the fix through operations.non_merk_tree.subtree_discovery on Grove V4. V1–V3 retain historical traversal, costs, and failures. No additional storage reads are needed.

How Has This Been Tested?

• Confirmed six regression tests fail on the unmodified implementation with Merk decoding errors.
• New tests cover all five families, empty and populated descendants, a NonCounted wrapper, direct/full-batch/partial-batch deletion, transaction visibility, persisted namespace cleanup, all three indexed axes, retained sibling data, and legacy/unknown version behavior.
• Initial fix: cargo test -p grovedb -p grovedb-version --lib --offline: 3,506 passed, eight existing ignored tests. Includes existing versioned deletion-cost regressions.
• After extracting separate V0/V1 functions: 95 subtree tests and 10 indexed-secondary cleanup tests passed. Source comparisons confirm the V0 body matches the original implementation and the V1 body retains the fix without the shared version branch.
• cargo fmt -p grovedb -p grovedb-version --check and git diff --check passed.
• cargo clippy -p grovedb -p grovedb-version --lib --tests --no-deps --offline -- -D warnings passed.

Breaking Changes

Adds subtree_discovery to the public GroveDBOperationsNonMerkTreeVersions struct. Custom struct literals must initialize the new version slot. Grove V1–V3 behavior remains unchanged.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation

For repository code-owners and collaborators only

• I have assigned this pull request to a milestone

Summary by CodeRabbit

• New Features

  • Recursive subtree discovery now recognizes non-Merk descendants in protocol version 4 and later.
  • Cleanup operations remove namespaces associated with MMR, bulk, dense, commitment, and private document store trees while preserving surviving data.

• Bug Fixes

  • Deleting ancestors no longer misinterprets non-Merk records as Merk nodes.
  • Discovery remains compatible with legacy protocol versions and fails safely for unsupported version configurations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 2737a454-9fe8-4583-81a5-a8217b94dc2e

📥 Commits

Reviewing files that changed from the base of the PR and between 8be4cf2 and e2d86f9.

📒 Files selected for processing (5)

• AGENTS.md
• grovedb/src/operations/auxiliary.rs
• grovedb/src/operations/auxiliary/find_subtrees/mod.rs
• grovedb/src/operations/auxiliary/find_subtrees/v0.rs
• grovedb/src/operations/auxiliary/find_subtrees/v1.rs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The PR adds a versioned subtree-discovery setting. GROVE_V4+ includes non-Merk descendants in cleanup without traversing their records as Merk nodes. GROVE_V1–V3 retain legacy behavior. Tests cover storage families, deletion routes, cleanup, and version errors.

Changes

Non-Merk subtree discovery

Layer / File(s)	Summary
Subtree discovery version contract grovedb-version/src/version/grovedb_versions.rs, grovedb-version/src/version/v1.rs, grovedb-version/src/version/v2.rs, grovedb-version/src/version/v3.rs, grovedb-version/src/version/v4.rs, AGENTS.md	Adds the subtree_discovery version field. Versions 1–3 use 0; version 4 uses 1 and documents the new behavior. Version-specific implementations must remain separate.
Version-gated subtree discovery grovedb/src/operations/auxiliary.rs, grovedb/src/operations/auxiliary/find_subtrees/*	Moves find_subtrees into a dispatcher with separate legacy and V4+ implementations. V4+ skips traversal into non-Merk data-storage namespaces while retaining legacy traversal for earlier versions.
Discovery and cleanup validation grovedb/src/tests/mod.rs, grovedb/src/tests/non_merk_subtree_discovery_tests.rs	Tests cover five non-Merk storage families, wrapped elements, direct and batch deletion routes, namespace cleanup, legacy failures, and unknown-version rejection.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Severity of issue fixed: Low

Merge Risk: ⚪ Minimal · up to e2d86

V4 cleanup can now remove populated non-Merk descendant namespaces without decoding their payloads as Merk nodes, while prior versions retain historical behavior. The covered cleanup and compatibility paths show no remaining merge-blocking risk.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 58.82% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 11 files. (1 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes satisfy issue #890 by separating versioned discovery behavior, recognizing non-Merk descendants, including their namespaces in cleanup, preserving indexed-axis cleanup, and adding coverage…
Out of Scope Changes check	✅ Passed	The changes remain within scope. The version configuration, implementation split, documentation, and tests directly support the recursive subtree discovery fix and its versioned behavior.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: skipping non-Merk descendants during recursive subtree discovery.

Full details: Docstring Coverage

Explanation

Docstring coverage is 58.82% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 17 functions across 11 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-890-non-merk-subtree-discovery

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.00%. Comparing base (985ece6) to head (e2d86f9).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop     #949   +/-   ##
========================================
  Coverage    92.99%   93.00%           
========================================
  Files          327      330    +3     
  Lines       103297   103349   +52     
========================================
+ Hits         96065    96117   +52     
  Misses        7232     7232           

Components	Coverage Δ
grovedb-core	91.33% <100.00%> (+<0.01%)	⬆️
merk	93.93% <ø> (ø)
storage	91.86% <ø> (ø)
commitment-tree	95.62% <ø> (ø)
mmr	95.11% <ø> (ø)
bulk-append-tree	92.78% <ø> (ø)
element	97.18% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[QuantumExplorer]

Reviewed