perf(lmdb): batch radix DFS reads + cache probe DBI handle - #36
Merged
Merged
Conversation
Search song song tren LMDB chạm gioi han reader-slot (`MDB_BAD_RSLOT`): moi `get_node`/`get_children` la mot `begin_ro_txn` rieng, mot lan search tao O(nodes) read-txn, nhieu request thi nhan lai gap gioi han. Tang 1 - cache DBI handle trong probe_version: `probe_env` giu them `Database` handle (Copy) cho `sg_meta` thay vi `open_db` lai moi request. `SharedGraphIndex::ensure_fresh` chay `current_version` truoc moi request nen moi lan deu ton mot `open_db` (tha bang `Environment`) va mot `begin_ro_txn`; gio con 1 read-txn. Tang 2 - batch read tren CategoryStorage: them `get_nodes`/`get_childrens` doc nhieu id trong MOT read-txn. Default impl goi lai ham don le nen 6 backend nen (in-memory/sqlite/redis/pg/mysql/ cached) khong doi. LMDB override de dung 1 txn cho ca danh sach. `Radix::search_dfs` chuyen sang goy cac lan doc: doc node cua frame + tat ca child chua duyet trong 1 lenh goi, thay vi mot lenh goi cho tung node. So read-txn moi vong DFS giam tu ~2*children xuong 1. `base` giu nguyen thu tu duyet nen hanh vi DFS khong doi. Test: - batch khop `get_node`/`get_children` tuang tung, thu tu va sort giu nguyen - id khong ton tai van tra `BranchOutOfRange` nhu ban don le - regression: 8 task doc song song khong loi - `probe_version` qua cache handle 100 lan van dung version Benchmark moi `lmdb_batch_read` (feature `lmdb`) do doc 1/16/64 node don lue vs batch, ca voi `get_children`: cargo bench -p codegraph-graph --bench lmdb_batch_read --features lmdb Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- `lmdb::Error::Other` nhận `c_int` (mã lỗi C), không phải `String`. `open_db` đã trả `lmdb::Result<Database>` nên chuyển thẳng với `?`. - `children` trong `search_dfs` (Collect state) cần `mut` để gọi `.pop()` — tương tự `node` ngay dòng trên. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Test viết tay sai cú pháp: - `storage::EMPTY` không resolve trong `mod tests` (chỉ có `EMPTY` từ `use super::*`) → E0433 - `.map(|i| s.new_node(..).await)` — `.await` trong closure không async → E0728. Vòng for thay thế. - `b.new_node(..)` gọi method trên `usize` do nhầm biến `b` (id) với `s` (storage) → E0599 - `assert_eq!(batch[0], kids)` so `Vec` với array → so với bản sort Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #36 +/- ##
==========================================
+ Coverage 78.35% 78.44% +0.09%
==========================================
Files 88 88
Lines 20718 20819 +101
==========================================
+ Hits 16233 16331 +98
- Misses 4485 4488 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
clippy::needless_borrow: `Self::to_vec(&cp_bytes)` trong khi `cp_bytes` đã là `&Vec<u8>` từ `&child_nodes[rel]` → bỏ dấu & thừa. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Merging this PR will degrade performance by 17.05%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | sample |
81.5 ms | 103.3 ms | -21.12% |
| ❌ | sample |
81.5 ms | 103.3 ms | -21.11% |
| ❌ | sample |
39.4 ms | 49.3 ms | -20.06% |
| ❌ | sample |
39.4 ms | 49.2 ms | -19.97% |
| ❌ | sample |
193.8 ms | 241.5 ms | -19.77% |
| ❌ | sample |
193.8 ms | 241.5 ms | -19.74% |
| ❌ | sample |
6.2 ms | 7.1 ms | -12.5% |
| ❌ | sample |
6.2 ms | 7.1 ms | -12.21% |
| ❌ | warm_broad |
2.1 ms | 2.4 ms | -11.63% |
| ❌ | warm_depth2 |
3.5 ms | 3.9 ms | -11.33% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing perf/lmdb-batch-read (6a80b83) with main (e3daee0)
clippy 1.99 (rust-toolchain.toml dùng channel = stable) bật lint double_must_use: async_tair tự sinh must_use cho future, trùng với kiểu future boxed vốn đã must_use. Lỗi nằm trong macro của async_trait nên allow tại chỗ sinh ra, không đổi hành vi. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vấn đề
Search song song trên LMDB chạm giới hạn reader-slot →
MDB_BAD_RSLOT.Nguyên nhân: mỗi
get_node/get_childrenlà mộtbegin_ro_txnriêng. Một lầnDFS duyệt ~N node thì tạo ~2N read-txn. Nhiều request chạy song song (MCP/GraphQL
dùng chung
Arc<GraphIndex>) thì số read-txn đồng thời nhân lên.Thêm nữa,
SharedGraphIndex::ensure_fresh→current_version→probe_versionchạy trước mọi request, mỗi lần lại
open_db+begin_ro_txn.Thay đổi
Tầng 1 — cache DBI handle trong
probe_versionprobe_envgiữ thêmDatabasehandle (Copy) chosg_meta, thay vìopen_dblại mỗi request. Mỗi request còn đúng 1 read-txn + 1
get.Tầng 2 — batch read trên
CategoryStorageThêm
get_nodes(&[usize])/get_childrens(&[usize])đọc nhiều id trong mộtread-txn. Default impl gọi lại hàm đơn lẻ nên 6 backend nền (in-memory, sqlite,
redis, postgres, mysql, cached) không đổi. LMDB override.
Radix::search_dfsgom các lần đọc: đọc node của frame + toàn bộ child chưa duyệttrong một lệnh gọi, thay vì một lệnh gọi cho từng node. Số read-txn mỗi vòng DFS
giảm từ ~2×children xuống 1. Biến
basegiữ nguyên thứ tự duyệt nên hành vi DFSkhông đổi.
Test
get_node/get_childrentừng cái — thứ tự và sort giữ nguyênBranchOutOfRangenhư bản đơn lẻprobe_versionqua cache handle 100 lần vẫn đúng versionBenchmark
lmdb_batch_read(featurelmdb) — đọc 1/16/64 node đơn lẻ vs batch, cả vớiget_children:Ghi chú
_bulksong song với bản cũ — thay đổi input trực tiếp trêntrait, backend nền dùng default impl.