Skip to content

Explicit index persistence via save/load commands - #51

Merged
1yefuwang1 merged 13 commits into
mainfrom
1yefuwang1/explicit-index-persistence
Jun 14, 2026
Merged

Explicit index persistence via save/load commands#51
1yefuwang1 merged 13 commits into
mainfrom
1yefuwang1/explicit-index-persistence

Conversation

@1yefuwang1

Copy link
Copy Markdown
Owner

Summary

Replaces the schema-baked index file path with explicit, user-driven persistence. The index path is no longer stored anywhere, so a vectorlite table is fully relocatable.

  • CREATE VIRTUAL TABLE is now 2-arg (vector space + index options). The optional 3rd path argument is removed; the 3-arg form is rejected with a helpful error. Breaking change → version bumped 0.2.00.3.0.
  • Persistence is explicit and in-memory-only, driven by INSERT-commands on two hidden columns:
    insert into t(operation, path) values ('save', '/path/index.bin');
    insert into t(operation, path) values ('load', '/path/index.bin');
    These route through xUpdate (which has the VirtualTable* directly), so there's no table-name registry. Works on empty tables, runs exactly once.
  • VirtualTable::SaveTo/LoadFrom replace the old auto load/save/delete. LoadFrom builds a new index and swaps it in only on success, validates the file's per-vector data size (label_offset_ - offsetData_) against the table's vector space, and preserves the runtime-only allow_replace_deleted flag (not serialized by hnswlib).
  • Removed auto-load on connect, auto-save on disconnect, and file deletion on drop.
  • Updated README.md, doc/markdown/api.md, and examples/index_serde.py.

operation, path, and distance are reserved column names. The in-memory index is lost on connection close unless explicitly saved.

Design spec and implementation plan live under docs/superpowers/specs/ and docs/superpowers/plans/.

Test Plan

  • C++ unit tests: ctest --test-dir build/dev/vectorlite — 47/47 pass
  • Python integration tests: pytest bindings/python/vectorlite_py/test — 13/13 pass, including new tests for save/load round-trip, load-replaces-contents, dimension-mismatch rejection, missing-file rejection, unknown-operation rejection, 3-arg-create rejection, hidden command columns, and allow_replace_deleted preservation across load
  • python examples/index_serde.py runs clean (exit 0)

1yefuwang1 and others added 13 commits June 14, 2026 12:55
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
vectorlite's BestIndex rejects queries without a knn_search/rowid
constraint, so verify the freshly created table is empty with a
knn_search returning no rows instead of select count(*).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The two-arg hnswlib load constructor defaults allow_replace_deleted to
false, and the flag is runtime-only (not serialized), so LoadFrom was
silently disabling deleted-slot reuse for tables created with the
default true. Retain the configured value on VirtualTable and reapply
it to the freshly loaded index. Add a regression test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LoadFrom now passes the table's configured max_elements and
allow_replace_deleted into the loading constructor, restoring the
ability to reload a saved index into a larger-capacity table. Previously
the default-argument constructor forced max_elements to the file's saved
value, silently dropping the grow-on-load behavior that README, api.md
and examples/index_serde.py still documented.

Also document the load semantics (dimension and element type must match;
distance type may differ; max_elements may grow) and add tests for
grow-on-load and element-type mismatch rejection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@1yefuwang1
1yefuwang1 merged commit 89f250f into main Jun 14, 2026
7 checks passed
@1yefuwang1
1yefuwang1 deleted the 1yefuwang1/explicit-index-persistence branch June 14, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant