feat(rust): port vectorlite to Rust (vtab in Rust; FFI only to hnswlib + ops)

[1yefuwang1]

Summary

Adds a Rust port of the vectorlite SQLite extension under rust/. All virtual-table logic lives in Rust; C++ is reached only via FFI, and only for two things: hnswlib and the SIMD ops (Google Highway). The port is a drop-in replacement for the C++ extension and passes the existing Python integration suite (112 tests) unchanged.

What is in Rust vs C++

Rust (this crate) — the entire virtual table + all policy:

• SQLite glue: xCreate/xConnect/xBestIndex/xFilter/xUpdate/xColumn/xRowid/xRename/xFindFunction/...
• Scalar functions: vector_distance, vector_from_json, vector_to_json, knn_search, knn_param, vectorlite_info
• Constraint handling and rowid IN (...) pushdown (sqlite3_vtab_in)
• Per-connection index registry (survives reparse/vacuum/rename)
• Vector-space and index-option parsing, JSON/blob serde
• All numeric policy: which distance/space to use, quantization + normalization for float32/bfloat16/float16, the rowid filter predicate, per-query ef save/restore, the load data-size mismatch check, and save/load orchestration

C++ (reached only through the C ABI) — nothing but hnswlib + ops:

• cpp/core_shim.cpp is generic, policy-free glue: a SpaceInterface adapter around a Rust distance callback, a BaseFilterFunctor adapter around a Rust predicate, thin HierarchicalNSW wrappers, and forwarders to ops.
• vectorlite/ops/ops.cpp — the existing, un-ported Highway SIMD kernels.
• Only these two .cpp files are compiled; no C++ virtual-table sources are used.

The distance function hnswlib calls is a Rust extern "C" callback that forwards to ops; the rowid filter is a Rust predicate invoked via a trampoline. So the only C++ is hnswlib itself, the ops kernels, and the minimal adapters required to expose those through a C ABI.

Highlights

• Faithful behavior — mirrors the C++ paths: knn with ef/rowid filters, knn_param pointer passing, persistence with the data-size check, quantization/normalization across all element types and l2/ip/cosine, and the registry that keeps the index alive across schema reparses.
• No libclang for normal builds — SQLite extension-API bindings are pre-generated and committed in a vendored vectorlite-sqlite-sys crate (the libsqlite3-sys pattern). Refresh with cargo build -p vectorlite-sqlite-sys --features regenerate.
• Static SQLite — the full SQLite amalgamation is statically embedded (~3.2 MB), mirroring the C++ build; embedded symbols aren't exported, so they can't interpose the host's SQLite.
• Cross-platform — build.rs discovers the vcpkg triplet by scanning build/*/vcpkg_installed/*/, picks the right static-archive names (.a/.lib), system libs, and per-linker flags, so it builds on Linux, macOS and Windows. Only hwy + sqlite3 are linked.

Testing

sh build.sh                 # repo root: build C++/vcpkg deps once
sh rust/build.sh            # build + deploy the Rust extension
PYTHONPATH=bindings/python python -m pytest bindings/python/vectorlite_py/test
# 112 passed

Notes

• This PR only adds the rust/ tree; no existing C++ sources or build files are modified.
• Cargo.lock is committed for reproducible builds; target/, .libclang/ are git-ignored.