Performance: stream-head background updates schedule a Task.Run per commit under write load

[AGiorgetti]

Summary

When snapshot support is enabled and background stream-head updates are on, each successful commit calls UpdateStreamHeadInBackgroundThread(...) via Task.Run. This schedules a new thread-pool work item per commit, adding task-scheduling overhead and GC pressure at high write rates.

Why this matters

• Task-scheduling overhead accumulates at high write rates.
• Fire-and-forget work can add GC pressure and increase tail latency.
• Benchmarks show a 2.5× write-time increase with background updates enabled vs disabled.

Baseline (net10, 2026-05-20)

Benchmark	Mean
WriteToStream(CommitsToWrite=1000, DisableSnapshotSupport=False, PersistStreamHeadsOnBackgroundThread=True)	2,631.6 ms
WriteToStream(CommitsToWrite=1000, DisableSnapshotSupport=False, PersistStreamHeadsOnBackgroundThread=False)	1,059.4 ms

Delta: +148% overhead with background updates enabled.

Proposed investigation

• Determine whether the 2.5× overhead is from task-scheduling cost or actual MongoDB write contention from background updates.
• Evaluate batching or debouncing stream-head updates rather than scheduling one Task.Run per commit.
• Consider a bounded channel / dedicated background worker to coalesce stream-head updates.

Reference

See docs/Performance-Investigation.md → Finding #6 for full context.

[AGiorgetti]

Write-side follow-up for #78:

I rechecked the stream-head background-update question with SnapshotOverheadBenchmarks on 2026-06-10 after removing the pinned InvocationCount=1 from the benchmark job. BenchmarkDotNet now chooses invocation counts through its pilot phase for this slice.

What was tested:

• DisableSnapshotSupport=false, PersistStreamHeadsOnBackgroundThread=false: snapshot support enabled, stream-head update inline.
• DisableSnapshotSupport=false, PersistStreamHeadsOnBackgroundThread=true: snapshot support enabled, stream-head update scheduled on the background path.
• The same background on/off matrix was also run with DisableSnapshotSupport=true for comparison.

Relevant #78 results with snapshot support enabled:

Commits	Background stream-head update	Mean	StdDev	Allocated
100	false	206.2 ms	47.78 ms	22.69 MB
100	true	220.3 ms	140.86 ms	22.70 MB
1,000	false	1,724.0 ms	362.08 ms	110.82 MB
1,000	true	1,676.5 ms	477.40 ms	110.96 MB

Interpretation:

• Background stream-head updates do not show a clear repeatable wall-clock win.
• Allocation is effectively unchanged when snapshot support is enabled.
• The 1,000-commit background=true mean is slightly lower, but variance is high enough that it should not drive an engine change.
• Keeping the current implementation is preferable for now because it is simple and the benchmark signal is weak.

Decision: put #78 in standby. Reopen only if write throughput becomes the target again and a stronger benchmark shows this path is material.