Communication-Efficient Distributed Vector Memory

Investigating Adaptive Routing Across Partitioned Semantic Memories

This repository contains the code and data for our paper on adaptive routing strategies for distributed vector databases.

Overview

We propose four routing strategies that reduce communication costs in distributed vector search while maintaining retrieval quality:

1. Confidence Threshold — Expand search when confidence is low
2. Top-N Neighbors — Always query N closest partitions
3. Progressive Expansion — Expand until confidence target met
4. Budgeted Communication — Hard limit on partitions contacted

Key Results

• 42% communication reduction with 100% recall retention at K=16
• KMeans outperforms random partitioning by 8-12%
• Adaptive routing matches global recall with sub-linear communication

Communication-Quality Tradeoff

Adaptive strategies achieve comparable recall with significantly less communication than broadcasting.

Recall vs Number of Partitions

While local-only search degrades with more partitions, adaptive strategies maintain stable recall.

Results (K=16, 100 queries)

Strategy	Recall@10	Partitions Contacted	Communication Savings
Global (baseline)	0.939	16/16	0%
Local (baseline)	0.826	1/16	93.8%
Threshold	0.939	9.3/16	42%
Top-N	0.923	4/16	75%
Progressive	0.939	9.3/16	42%
Budgeted	0.940	8/16	50%

Repository Structure

├── paper/
│   ├── main.tex                    # LaTeX source
│   ├── main.pdf                    # Compiled paper
│   ├── figure1_*.png               # Publication figures
│   ├── figure2_*.png
│   ├── figure3_*.png
│   └── figure4_*.png
├── notebook/
│   └── dbpedia14_distributed_memory.ipynb  # Kaggle notebook
├── README.md
└── LICENSE

Dataset

DBpedia 14 — 100K documents across 14 semantic classes:

• Company, EducationalInstitution, Artist, Athlete
• OfficeHolder, MeanOfTransportation, Building
• NaturalPlace, Village, Animal, Plant
• Album, Film, WrittenWork

Requirements

pip install turbovec sentence-transformers scikit-learn matplotlib seaborn datasets

Usage

Run on Kaggle

1. Upload notebook/dbpedia14_distributed_memory.ipynb to Kaggle
2. Enable internet access
3. Run all cells

Run locally

cd notebook
jupyter notebook dbpedia14_distributed_memory.ipynb

Citation

@article{silva2026distributed,
  title={Communication-Efficient Distributed Vector Memory: Investigating Adaptive Routing Across Partitioned Semantic Memories},
  author={Silva, Agostina},
  year={2026}
}

License

• Code (notebook, scripts): MIT License
• Paper and figures: CC BY 4.0