Performance: default checkpoint generator issues one extra DB read per commit (AlwaysQueryDb)

[AGiorgetti]

Summary

MongoPersistenceEngine.Initialize() defaults to AlwaysQueryDbForNextValueCheckpointGenerator. That generator calls GetLastValue() on every Next() invocation — one extra MongoDB round-trip before every insert.

Why this matters

• Write-heavy workloads pay an extra database read per commit on top of the insert.
• The cost scales directly with commit volume.
• Benchmarks show the always-query generator is ~47% slower than the in-memory generator for the same workload.

Baseline (net10, 2026-05-20)

Benchmark	Mean
WriteWithCheckpointGenerator(CommitsToWrite=1000, GeneratorType=Always)	2,908.6 ms
WriteWithCheckpointGenerator(CommitsToWrite=1000, GeneratorType=InMemory)	1,541.0 ms

Delta: +89% overhead with the default always-query generator.

Proposed investigation

• Confirm the round-trip overhead is the GetLastValue() call (not write contention).
• Evaluate making InMemoryCheckpointGenerator the default, or using an atomic-increment strategy to avoid per-commit reads.
• Verify concurrency correctness of any alternative approach (no checkpoint collisions under concurrent writers).

Reference

See docs/Performance-Investigation.md → Finding #4 for full context.

[AGiorgetti]

Write-side follow-up for #76:

I rechecked the checkpoint-generator comparison with CheckpointGeneratorBenchmarks on 2026-06-10 after removing the pinned InvocationCount=1 from the benchmark job. BenchmarkDotNet now chooses invocation counts through its pilot phase for this slice.

What was tested:

• Always: current engine default, AlwaysQueryDbForNextValueCheckpointGenerator, which rereads MongoDB for the next checkpoint value.
• InMemory: explicit InMemoryCheckpointGenerator, which initializes from MongoDB and then increments in memory, resyncing on duplicate checkpoint signal.

Stabilized net10 results:

Commits	Generator	Mean	StdDev	Allocated
100	Always	244.3 ms	14.29 ms	16.88 MB
100	InMemory	147.7 ms	6.87 ms	28.24 MB
1,000	Always	2,506.2 ms	238.76 ms	110.96 MB
1,000	InMemory	1,322.9 ms	274.52 ms	95.54 MB

Interpretation:

• InMemoryCheckpointGenerator is materially faster for write throughput.
• It is not a safe default swap in this pass: the existing acceptance tests document the default no-hole behavior after concurrency exceptions, while the in-memory generator can leave checkpoint holes under the Continue strategy.
• So this should be treated as an explicit tuning option for callers that accept that trade-off, not as an engine default change.

Decision: keep #76 measured/opt-in. No runtime engine change for now.