extending bruteforce and lbbruteforce with streaming

CONTINUITY.md

@@ -65,7 +65,10 @@ These algorithms do not all share the same dependency profile or feature set.
 
 - Top-k search is the baseline capability.
 - Range search is modeled through `SearchConfig` and is not universally implemented.
-- Streaming insert is not a generic capability; default behavior throws.
+- Streaming insert is implemented by `BruteForceSearch`, `LbBruteforce`, and `Coconut`;
+  the base implementation still throws for algorithms that do not support it.
+- Bruteforce streaming grows the owned in-memory database incrementally. LbBruteforce also
+  computes SAX summaries incrementally, using the breakpoint set fixed by the initial build.
 - `setNormalized(bool)` is a declaration about the input data, not a preprocessing step.
 - Data can come from in-memory arrays or file-backed sources via `DataSource`.
 

README.md

@@ -46,8 +46,8 @@ The following table summarizes the key features of each algorithm:
 
 | Algorithm | Description |
 |-----------|-------------|
-| **Bruteforce** | Naive parallel similarity search implementation |
-| **Lower Bound Bruteforce** | Optimized bruteforce with lower bounding for the distance calculations |
+| **Bruteforce** | Naive parallel similarity search implementation with incremental streaming inserts |
+| **Lower Bound Bruteforce** | Optimized bruteforce with lower bounding and incremental streaming inserts |
 | **[MESSI](https://helios2.mi.parisdescartes.fr/~themisp/messi/)** | In-memory parallel similarity search |
 | **[PARIS](https://helios2.mi.parisdescartes.fr/~themisp/paris/)** | Disk-based parallel similarity search |
 | **[SING](https://helios2.mi.parisdescartes.fr/~themisp/sing/)** | GPU-accelerated in-memory parallel similarity search |
@@ -58,6 +58,23 @@ The following table summarizes the key features of each algorithm:
 | **[FreSH](http://publications.ics.forth.gr/tech-reports/2023/2023.TR489_FreSh_A_LockFree_Data_Series_Index.pdf)** | In-memory lock-free parallel similarity search using an iSAX index (SRDS 2023) |
 | **[COCONUT](http://www.vldb.org/pvldb/vol11/p677-kondylakis.pdf)** | Sortable-SAX index built bottom-up; supports both static datasets and **streaming** (incremental) inserts (PVLDB 2018) |
 
+### Incremental streaming inserts
+
+`BruteForceSearch`, `LbBruteforce`, and `Coconut` implement the common streaming API. Build
+the initial index once, then append one series or a contiguous batch without rebuilding:
+
+```cpp
+daisy::BruteForceSearch search(daisy::DistanceType::L2_SQUARED);
+search.buildIndex(initial_data, initial_size, dim);
+search.insert(one_series);
+search.insertBatch(batch_data, batch_size);
+```
+
+Inserted series receive consecutive IDs beginning at the size of the initial database and are
+immediately visible to top-k and range searches. `LbBruteforce` computes a SAX summary for each
+insert using the breakpoints established during the initial build. Inserts can reallocate the
+owned database, so callers should not retain a pointer returned by `getDatabase()` across them.
+
 
 
 ## Quickstart
@@ -206,5 +223,3 @@ For questions and suggestions through mail, you can contact us at [manos.chatzak
 
 
 
-
-

demos/CMakeLists.txt

@@ -37,6 +37,13 @@ if(BUILD_DEMO)
         message(STATUS "Include directories added for demo_bruteforce_L2Square.")
     endif()
 
+    add_executable(demo_Bruteforce_Streaming demo_Bruteforce_Streaming.cpp)
+    target_link_libraries(demo_Bruteforce_Streaming PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_Bruteforce_Streaming PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
     # ////// COCONUT (static + streaming) //////
     if(BUILD_COCONUT)
         if(DEBUG_MSG)
@@ -124,6 +131,13 @@ if(BUILD_DEMO)
         message(STATUS "Include directories added for demo_LbBruteforce_L2Square.")
     endif()
 
+    add_executable(demo_LbBruteforce_Streaming demo_LbBruteforce_Streaming.cpp)
+    target_link_libraries(demo_LbBruteforce_Streaming PRIVATE dino_lib commons_lib)
+    target_include_directories(demo_LbBruteforce_Streaming PRIVATE
+        ${CMAKE_CURRENT_SOURCE_DIR}/../lib
+        ${CMAKE_CURRENT_SOURCE_DIR}/../commons
+    )
+
     # ////// LBBRUTEFORCE DTW //////
     if(DEBUG_MSG)
         message(STATUS "---")

demos/demo_Bruteforce_Streaming.cpp

@@ -0,0 +1,40 @@
+// Bruteforce streaming: build once, then append new series without rebuilding.
+
+#include "../commons/dataloaders.hpp"
+#include "../lib/daisy.hpp"
+
+#include <cstdio>
+
+int main()
+{
+    const daisy::idx_t dim = 96;
+    const daisy::idx_t initial = 50000;
+    const daisy::idx_t batch = 25000;
+    const daisy::idx_t n_query = 5;
+    const daisy::idx_t k = 5;
+
+    float *stream = loadRandomData(initial + batch, dim, 100, true);
+    float *query = loadRandomData(n_query, dim, 50, true);
+
+    daisy::BruteForceSearch search(daisy::DistanceType::L2_SQUARED);
+    search.buildIndex(stream, initial, dim);
+    search.insert(stream + initial * dim);
+    search.insertBatch(stream + (initial + 1) * dim, batch - 1);
+
+    daisy::idx_t *indices = new daisy::idx_t[n_query * k];
+    float *distances = new float[n_query * k];
+    search.searchIndex(query, n_query, k, indices, distances);
+
+    std::printf("Bruteforce now contains %llu series. Query 0 kNN: ",
+                search.getNDatabase());
+    for (daisy::idx_t j = 0; j < k; ++j)
+        std::printf("%llu(%.3f) ", indices[j], distances[j]);
+    std::printf("\n");
+
+    delete[] stream;
+    delete[] query;
+    delete[] indices;
+    delete[] distances;
+    return 0;
+}
+

demos/demo_Bruteforce_Streaming.py

@@ -0,0 +1,40 @@
+import numpy as np
+
+from daisy import BruteForceSearch, DistanceType, LbBruteforce
+
+
+def run_streaming(index, name, initial, single, batch, query):
+    index.buildIndex(initial)
+    index.insert(single)
+    index.insertBatch(batch)
+    indices, distances = index.searchIndex(query, 3)
+    print(f"{name} query 0 indices:", indices[0])
+    print(f"{name} query 0 distances:", distances[0])
+
+
+def main():
+    rng = np.random.default_rng(100)
+    stream = rng.normal(size=(1000, 32)).astype(np.float32)
+    query = rng.normal(size=(5, 32)).astype(np.float32)
+
+    run_streaming(
+        BruteForceSearch(DistanceType.L2_SQUARED),
+        "BruteForceSearch",
+        stream[:500],
+        stream[500],
+        stream[501:],
+        query,
+    )
+    run_streaming(
+        LbBruteforce(DistanceType.L2_SQUARED),
+        "LbBruteforce",
+        stream[:500],
+        stream[500],
+        stream[501:],
+        query,
+    )
+
+
+if __name__ == "__main__":
+    main()
+

demos/demo_LbBruteforce_Streaming.cpp

@@ -0,0 +1,40 @@
+// Lower-bound Bruteforce streaming: raw series and their SAX summaries are appended together.
+
+#include "../commons/dataloaders.hpp"
+#include "../lib/daisy.hpp"
+
+#include <cstdio>
+
+int main()
+{
+    const daisy::idx_t dim = 96;
+    const daisy::idx_t initial = 50000;
+    const daisy::idx_t batch = 25000;
+    const daisy::idx_t n_query = 5;
+    const daisy::idx_t k = 5;
+
+    float *stream = loadRandomData(initial + batch, dim, 100, true);
+    float *query = loadRandomData(n_query, dim, 50, true);
+
+    daisy::LbBruteforce search(daisy::DistanceType::L2_SQUARED);
+    search.buildIndex(stream, initial, dim);
+    search.insert(stream + initial * dim);
+    search.insertBatch(stream + (initial + 1) * dim, batch - 1);
+
+    daisy::idx_t *indices = new daisy::idx_t[n_query * k];
+    float *distances = new float[n_query * k];
+    search.searchIndex(query, n_query, k, indices, distances);
+
+    std::printf("LbBruteforce now contains %llu series. Query 0 kNN: ",
+                search.getNDatabase());
+    for (daisy::idx_t j = 0; j < k; ++j)
+        std::printf("%llu(%.3f) ", indices[j], distances[j]);
+    std::printf("\n");
+
+    delete[] stream;
+    delete[] query;
+    delete[] indices;
+    delete[] distances;
+    return 0;
+}
+

docs/demos-guide.md

@@ -4,9 +4,9 @@ The demos module provides practical examples of how to use the DaiSy library's a
 Each demo illustrates a specific algorithm or specific distance metric. This module includes both C++ and Python implementations for various algorithms and use cases.
 
 Most demos follow the same batch pattern: `buildIndex(...)` once, then `searchIndex(...)`.
-The **Coconut** algorithm additionally supports **streaming**: `demo_Coconut_L2Square` shows
-the static (batch) build, while `demo_Coconut_Streaming` builds on an initial batch and then
-`insert`/`insertBatch`es new series into the live index, querying after each step.
+**Bruteforce**, **LbBruteforce**, and **Coconut** additionally support streaming through
+`insert(...)` and `insertBatch(...)`. See `demo_Bruteforce_Streaming`,
+`demo_LbBruteforce_Streaming`, and `demo_Coconut_Streaming` for live-index examples.
 
 ## Demo Program Structure
 

docs/how-to-contribute.md

@@ -12,7 +12,7 @@ Here is a (non-exhaustive) mockup of our future and ongoing goals:
 
 - Extension of DaiSy for subsequence similarity search
 - Extension of DaiSy for more algorithms (e.g., SFA, Hercules, Dumpy, etc.)
-- Streaming / updatable indexing for more algorithms (currently supported by Coconut)
+- Streaming / updatable indexing for more algorithms (currently supported by Bruteforce, LbBruteforce, and Coconut)
 - Implementation of a DaiSy autotuner to automatically optimize indexing and search parameters
 - Extension of DaiSy to support learned optimization approaches, e.g., LeaFi and ProS