Explicit index persistence via save/load commands - #51
Merged
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
vectorlite's BestIndex rejects queries without a knn_search/rowid constraint, so verify the freshly created table is empty with a knn_search returning no rows instead of select count(*). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The two-arg hnswlib load constructor defaults allow_replace_deleted to false, and the flag is runtime-only (not serialized), so LoadFrom was silently disabling deleted-slot reuse for tables created with the default true. Retain the configured value on VirtualTable and reapply it to the freshly loaded index. Add a regression test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoadFrom now passes the table's configured max_elements and allow_replace_deleted into the loading constructor, restoring the ability to reload a saved index into a larger-capacity table. Previously the default-argument constructor forced max_elements to the file's saved value, silently dropping the grow-on-load behavior that README, api.md and examples/index_serde.py still documented. Also document the load semantics (dimension and element type must match; distance type may differ; max_elements may grow) and add tests for grow-on-load and element-type mismatch rejection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the schema-baked index file path with explicit, user-driven persistence. The index path is no longer stored anywhere, so a vectorlite table is fully relocatable.
CREATE VIRTUAL TABLEis now 2-arg (vector space + index options). The optional 3rd path argument is removed; the 3-arg form is rejected with a helpful error. Breaking change → version bumped0.2.0→0.3.0.xUpdate(which has theVirtualTable*directly), so there's no table-name registry. Works on empty tables, runs exactly once.VirtualTable::SaveTo/LoadFromreplace the old auto load/save/delete.LoadFrombuilds a new index and swaps it in only on success, validates the file's per-vector data size (label_offset_ - offsetData_) against the table's vector space, and preserves the runtime-onlyallow_replace_deletedflag (not serialized by hnswlib).README.md,doc/markdown/api.md, andexamples/index_serde.py.operation,path, anddistanceare reserved column names. The in-memory index is lost on connection close unless explicitly saved.Design spec and implementation plan live under
docs/superpowers/specs/anddocs/superpowers/plans/.Test Plan
ctest --test-dir build/dev/vectorlite— 47/47 passpytest bindings/python/vectorlite_py/test— 13/13 pass, including new tests for save/load round-trip, load-replaces-contents, dimension-mismatch rejection, missing-file rejection, unknown-operation rejection, 3-arg-create rejection, hidden command columns, andallow_replace_deletedpreservation across loadpython examples/index_serde.pyruns clean (exit 0)