perf(simd): explore multiple accumulators in SIMD kernels

[bennibbelink]

Context

The SIMD arithmetic kernels introduced in PR #72 use a single accumulator per loop iteration (one vector register accumulating the result for each step). #72 (review)

Suggested Fix

Restructure the inner loop to use multiple independent accumulator chains. For example:

// Instead of:
for i := 0; i+8 <= n; i += 8 {
    acc = simd.AddFloat32x8(a[i:], b[i:])
    ...
}

// Use:
for i := 0; i+16 <= n; i += 16 {
    acc0 = simd.AddFloat32x8(a[i:],   b[i:])
    acc1 = simd.AddFloat32x8(a[i+8:], b[i+8:])
    ...
}

[kolkov]

Good question. Here's what we found after researching how Burn and other frameworks handle this.

TL;DR

Multiple accumulators are essential for reductions (sum, dot product, GEMM) but provide marginal gain for element-wise ops. Born's GEMM kernel (#96) already uses them correctly. The element-wise ops (#72) are fine as-is.

Why element-wise ops are different

Element-wise binary ops have no inter-iteration dependency:

load A[i:i+8]    // independent
load B[i:i+8]    // independent
C = A + B         // depends only on this iteration's loads
store C[i:i+8]   // depends only on this iteration's result

Modern OoO CPUs overlap these iterations automatically. The bottleneck is memory bandwidth, not compute pipeline.

What Burn does

Burn's production code (burn-flex/src/simd/portable.rs) uses the exact same single-vector-per-iteration pattern as Born for element-wise ops. No unrolling, no multiple accumulators.

For reductions (burn-flex/src/simd/kernels.rs), Burn uses 8 accumulators — that's where the technique shines (3-6x on top of SIMD).

Expected gain

Optimization	Gain	Status
SIMD element-wise (PR #72)	3-5x vs scalar	✅ Done
SIMD GEMM with 12 accumulators (#96)	10-29x vs scalar	✅ Done
Multi-accumulator element-wise	0-15% on top of current	Low priority
Multi-accumulator reduction (sum)	3-6x on top of basic SIMD	Worth doing

What would be worth doing instead

1. SIMD-accelerated reduction ops (Sum, Mean, Norm) with 4-8 accumulators — this is where the Burn pattern applies
2. Thread-level parallelism for large tensors — Burn uses rayon at >4M elements, Born could use goroutines

Keeping this issue open for reduction ops. The element-wise ops are at their practical ceiling for single-threaded SIMD.

[kolkov]

Status update: the reduction accumulator win from this issue is shipped — Sum with 8 accumulators landed in #115 (by you, @bennibbelink). The companion issue #90 (SIMD min threshold) is also done in v0.9.23.

For the remaining element-wise ops (Add, Mul, Sub, Div): after studying the pipeline, the expected gain is marginal. These ops are memory-bandwidth-bound on modern CPUs — out-of-order execution already hides the single-accumulator latency since there are no data dependencies between loop iterations. Multiple accumulators help reductions (where iteration N depends on N-1) but not independent element-wise ops.

That said, if someone wants to benchmark a dual-accumulator variant on a specific microarchitecture and show a measurable improvement — contributions are very welcome! The SIMD kernels live in internal/backend/cpu/ops_float32.go (dispatch) and the vendored assembly in *_amd64.s / *_arm64.s. The existing benchmark suite (simd_threshold_bench_test.go) provides the comparison framework. Leaving this open for now.