add_items: insert in a random permutation by default

[SilvioM97]

Summary

HierarchicalNSW randomises each element's level (getRandomLevel) but inserts elements in whatever order the caller supplies. HNSW's analysis is average-case over a random insertion order and the average case may not hold for a specific provided order, especially when the provided order is not random but given by some similarity factor (e.g. clustered documents from real world collections).

On a tested 8.8M-passage text embedding collection it is worth ~2 accuracy points at fixed ef, and the graph needs 1.4×–2.6× the distance computations for equal recall.

This PR makes add_items insert in a random permutation by default, with shuffle=False to opt out.

Why the order matters

At the moment an element is inserted, its neighbours are selected from the elements already in the graph. It can acquire a later element only if that element subsequently selects it, and getNeighborsByHeuristic2 is not symmetric, so often it does not. What the insertion order governs is therefore what each element is allowed to choose from, and the two orders differ in the shape of that candidate pool, not its size:

• Random order: the already-inserted prefix is a uniform sample of the collection. It covers the whole space at reduced density, so the candidate set the heuristic prunes is representative and the surviving neighbourhood approximates what the finished collection would give.
• Caller's order on a sorted collection: the prefix is a contiguous block. It covers part of the space rather than all of it, so through most of the build an element's neighbours are the best available within one region rather than the best overall.

The heuristic diversifies whatever pool it is handed; it cannot supply directions the pool does not contain. This is why the effect does not wash out as the graph grows.

Evidence

Two arms differing only in insertion order, same unmodified library at d9b3608, same getRandomLevel, same addPoint, same ParallelFor. M=32, ef_construction=200, 64 build threads, k=10, accuracy@10 against exact ground truth. Machine: Intel Xeon Silver 4314 (2×16 cores, AVX-512, 503 GB RAM), g++ -O3 -march=native -std=c++17.

1. Synthetic, reproducible in ~2 minutes with no download

1M × 128 points drawn from 2,000 Gaussian clusters and written cluster by cluster, which manufactures the one property the effect needs — a file stored in a meaningful order. L2, 1,000 queries. Three independent builds per arm, mean ± sd:

ef_search	file order	shuffled	delta
15	81.49 ± 0.78	89.48 ± 0.83	+7.99
24	87.90 ± 1.23	95.82 ± 0.37	+7.91
32	90.55 ± 1.10	98.14 ± 0.18	+7.59
45	92.79 ± 0.93	99.45 ± 0.07	+6.66
90	95.47 ± 0.45	99.94 ± 0.07	+4.46
150	96.77 ± 0.53	100.00 ± 0.00	+3.23
200	97.47 ± 0.27	100.00 ± 0.00	+2.53

Mean +6.03 points over a 13-point ef ladder. The file-order arm never exceeds 97.6% accuracy@10 at any beam width tested, while the shuffled arm saturates at 100%. File order is also markedly less stable build-to-build (sd up to 1.4, against ≤0.8 shuffled).

The generator is at the bottom of this description, numpy only, no download.

2. A real collection

MS MARCO v1 passage / Dragon embeddings, 8,841,823 × 768, inner product, 6,980 queries. One build per arm, three search repetitions each (search is deterministic, so the repetitions bound timing noise, not build noise; the build-noise estimate comes from SIFT1M below).

ef_search	file order	shuffled	delta
25	88.785	92.490	+3.71
45	92.845	95.493	+2.65
90	95.517	97.279	+1.76
160	96.951	98.337	+1.39
300	97.855	98.894	+1.04
520	98.414	99.242	+0.83

Mean over the full 13-point ladder: +1.98 accuracy points.

Work-normalised, which is the stronger statement, distance computations needed to reach the same accuracy, both arms measured on the same two indexes:

accuracy@10 target	shuffled, dist/query	file order, dist/query	ratio
92.49	1020.7	1387.4	1.36×
95.49	1474.9	2408.9	1.63×
97.28	2457.6	4806.2	1.96×
98.34	3953.5	10426.4	2.64×

(file-order column interpolated across its own measured ladder.) Counting distance computations requires a small fix to metric_distance_computations, which does not fire at ground level as shipped; that is a separate issue and I will open it separately.

3. Control: the effect tracks corpus orderedness, as predicted

The effect requires the input file to be ordered. Measured directly on 4,000-row samples, we measure how similar (cosine for Dragon, L2 distance for SIFT1M) are consecutive vectors and random ones:

collection	consecutive rows	random pairs
Dragon (cosine similarity)	0.478	0.180	2.7× more similar
SIFT1M (L2 distance)	479.4	525.8	0.91 — no ordering structure

And the effect follows it: on SIFT1M, where the file is unordered, shuffling is worth only +0.11 accuracy points on average (three independent builds per arm, per-point sd 0.03–0.12), against +1.98 on Dragon.

What this PR changes

• add_items gains shuffle=True. It permutes the batch and inserts in that order.
• Labels are unaffected: only the order of the addPoint calls changes, not which label a vector is stored under. Covered by a test.
• The permutation is drawn from a generator seeded with init_index's random_seed, so builds stay reproducible to the extent they already were.
• Docs: one line in the add_items API description, one entry in ALGO_PARAMS.md.
• New test tests/python/bindings_test_shuffle.py: label integrity under shuffling, and the recall effect on a small ordered corpus (0.960 → 0.990 at the tested configuration, runs in ~1 s).

All existing Python test modules pass: bindings_test, _labels, _getdata, _metadata, _pickle, _filter, _recall, _replace, _resize, _spaces.

Trade-offs, stated up front

• Shuffling costs build time, because file order gives the insertion searches artificial locality: Dragon 1498 s → 1726 s, SIFT1M 25.6 s → 28.0 s, synthetic 9.85 s → 14.03 s. So the current default is faster and worse. If a ~15% build-time regression on ordered data is unacceptable as a default, flipping the default to shuffle=False is a one-line change and users still get the switch.
• add_items can only permute within the batch it is given, so a user inserting in chunks of 1,000 gets little benefit. A single large call randomises thoroughly; this is a partial fix by construction.
• Index contents change for users who rebuild with this version. Results stay correct and labels are unchanged, but an index built now will not be identical to one built before.
• The C++ API is unaffected, because it has no batch insert. addPoint sees one point at a time and the caller has already fixed the order. C++ users have to permute at their own call site.

Reproducing the synthetic result

python3 make_ordered_synthetic.py .

then build twice with M=32, ef_construction=200, once in array order, once with the rows permuted, and compare accuracy@10 across an ef ladder.

make_ordered_synthetic.py

#!/usr/bin/env python3
"""Build a clustered, cluster-ordered synthetic collection that reproduces the
hnswlib insertion-order effect with no proprietary data.

The only property that matters is that the file is stored in a semantically
meaningful order, which is what a document-ordered text corpus gives you. Here
that is manufactured by generating points cluster by cluster and writing them out
in cluster order.
"""
import numpy as np, sys, os

out = sys.argv[1] if len(sys.argv) > 1 else "."
N, d, C, NQ, K = 1_000_000, 128, 2_000, 1_000, 10
rng = np.random.default_rng(42)

centers = rng.normal(0, 1.0, size=(C, d)).astype(np.float32)
per = N // C
# Points are emitted cluster by cluster, so the file is ORDERED.
data = np.empty((N, d), dtype=np.float32)
for c in range(C):
    data[c*per:(c+1)*per] = centers[c] + rng.normal(0, 0.35, size=(per, d)).astype(np.float32)

qc = rng.integers(0, C, size=NQ)
queries = (centers[qc] + rng.normal(0, 0.35, size=(NQ, d))).astype(np.float32)

# Exact top-K by L2, in chunks.
dn = (data * data).sum(1)
best_d = np.full((NQ, K), np.inf, np.float32); best_i = np.zeros((NQ, K), np.uint32)
CH = 50_000
for s in range(0, N, CH):
    e = min(s + CH, N)
    dist = dn[s:e][None, :] - 2.0 * (queries @ data[s:e].T)
    idx = np.argpartition(dist, K, axis=1)[:, :K]
    cand_d = np.take_along_axis(dist, idx, 1); cand_i = (idx + s).astype(np.uint32)
    alld = np.concatenate([best_d, cand_d], 1); alli = np.concatenate([best_i, cand_i], 1)
    keep = np.argpartition(alld, K, axis=1)[:, :K]
    best_d = np.take_along_axis(alld, keep, 1); best_i = np.take_along_axis(alli, keep, 1)
order = np.argsort(best_d, axis=1)
gt = np.take_along_axis(best_i, order, 1).astype(np.uint32)

np.save(os.path.join(out, "dataset.npy"), data)
np.save(os.path.join(out, "queries.npy"), queries)
np.save(os.path.join(out, "groundtruth.npy"), gt)

# Report the ordering structure, the property the effect depends on.
a = data[:4000].astype(np.float64)
cons = np.linalg.norm(a[:-1] - a[1:], axis=1).mean()
r = np.sort(rng.choice(N, 4000, replace=False)); b = data[r].astype(np.float64)
rnd = np.linalg.norm(b[:-1:2] - b[1::2], axis=1).mean()
print(f"N={N} d={d} clusters={C} queries={NQ} k={K}")
print(f"mean L2, consecutive rows {cons:.2f} | random pairs {rnd:.2f} | ratio {cons/rnd:.3f}")
print("(ratio well below 1.0 = the file is ordered, which is what the effect needs)")

Both real collections used above are public: Dragon (27 GB) and SIFT1M (0.5 GB).