Skip to content

Fix multi-GB memory consumption when zooming into smooth solids - #340

Draft
dsn27 wants to merge 2 commits into
masterfrom
triangulation-memory-fixes
Draft

Fix multi-GB memory consumption when zooming into smooth solids#340
dsn27 wants to merge 2 commits into
masterfrom
triangulation-memory-fixes

Conversation

@dsn27

@dsn27 dsn27 commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

Zooming deeply into a single sphere drove the process from ~70 MB to 2.3 GB, and the memory never came back. The investigation (driven by per-second heap instrumentation over ten diagnostic rounds) showed the memory is not held by rendering — it is triangulation churn and GC bookkeeping in the core, and it behaves the same with the old OpenGL renderer as with the Silk.NET branch. These fixes are therefore independent of the renderer migration (#339).

Root causes and fixes

1. Background re-triangulation abort cascade (Model.RecalcDisplayLists)
Every zoom notch aborted the running background re-triangulation thread and respawned it at the next finer precision. A fast zoom gesture produced a cascade of aborted rounds, each abandoning its partial triangulation results as garbage (~280 MB/s of heap growth was measured with zero user interaction and zero painting). A running round now finishes; the next finer level starts on a later paint, so levels are computed sequentially and each at most once. Aborts on object add/remove are unchanged.

2. Speculative half-precision rounds (Model.RecalcDisplayLists)
The background round triangulated at Precision / 2.0 ("damit es nicht sooft drankommt"): after every zoom pause, a full re-triangulation with 4× the displayed triangle count ran for many seconds, and its completion forced another complete display list rebuild to swap in detail nobody had asked for. Rounds now compute exactly the displayed precision, clamped to the finest precision ModelView can ever request (Extent.MaxSide / HighestDisplayPrecision).

3. Committed heap segments never returned to the OS
The triangulation transients survived into gen2/LOH, and an ordinary forced compacting collection cleans the heap but never decommits: the process was measured at 55 MB of live objects inside a 1042 MB committed heap. A collection helper using GCCollectionMode.Aggressive (.NET 7+, value 4, with a compacting fallback for older runtimes so the netstandard2.0 target keeps working) now runs when a background round completes — before NewDisplaylistAvailableEvent, so it cannot race the UI rebuild that event triggers — and after any display list rebuild that ends with the heap above 300 MB (an absolute gate; a delta gate missed rebuilds entered with an already-ballooned heap).

4. Normals recomputed on every paint (Face.PaintFaceTo3D)
All vertex normals were recomputed and reallocated on every paint call — for a zoomed sphere a fresh multi-MB GeoVector[] plus hundreds of thousands of surface.GetNormal evaluations per frame or display list recording. The normals only depend on the triangulation and are now cached alongside it, invalidated by reference identity of the trianglePoint array.

Measurements

Scenario Before After
Sphere, zoom to precision floor, then wait 2.3 GB, permanent 12.9 MB live managed heap; committed segments returned to the OS
Post-zoom idle +280 MB/s growth for ~20 s flat
2D-only models unaffected unaffected (collection gates never reached)

Notes

  • The transient peak during a heavy triangulation round (~1–2 GB for a few seconds while two ~120k-triangle faces compute in parallel) is the triangulation algorithm's inherent working memory and is unchanged — it is now reliably reclaimed afterwards.
  • Verified by compilation on Linux (netstandard2.0/net8) and by repeated instrumented runs on Windows; the instrumentation itself is not part of this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_014HjM12HeHU7a7n9DJx91Y4

Generated by Claude Code

claude added 2 commits July 22, 2026 14:26
@claude
Cache face triangulation normals instead of recomputing per paint
6d083b9 
Face.PaintFaceTo3D recomputed every vertex normal on every paint call:
for a finely tessellated face (e.g. a zoomed sphere with hundreds of
thousands of vertices) that was a fresh multi-MB GeoVector array and as
many surface.GetNormal evaluations per frame or display list recording.
The normals only depend on the triangulation, so cache them alongside
it, invalidated by reference identity of the trianglePoint array.
@claude
Tame background triangulation: sequential rounds, displayed precision…
5e6fd84 
…, return memory

Investigating multi-GB memory consumption when zooming into a sphere
(70 MB -> 2.3 GB) revealed several compounding problems in the display
list regeneration pipeline, none of them renderer-specific:

- Every zoom notch aborted the running background re-triangulation
  thread and respawned it finer; a fast zoom gesture produced a cascade
  of aborted rounds, each leaving its partial results as garbage.
  A running round now finishes and the next finer level starts on a
  later paint — levels are computed sequentially, each at most once
  (aborts on object add/remove are unchanged).
- The background round triangulated at HALF the requested precision
  ('damit es nicht sooft drankommt'): after every zoom pause a full
  re-triangulation with four times the displayed triangle count ran for
  many seconds, and its completion forced another complete display list
  rebuild for detail nobody had asked for. Rounds now compute exactly
  the displayed precision, clamped to the finest precision ModelView
  can ever request (Extent.MaxSide/HighestDisplayPrecision).
- The triangulation churn left the process at its peak footprint: an
  ordinary compacting collection cleans the heap but never returns the
  committed segments (measured: 55 MB live objects in a 1 GB working
  set). A collection helper using GCCollectionMode.Aggressive (.NET 7+,
  with a compacting fallback for older runtimes) now runs when a
  background round completes — before NewDisplaylistAvailableEvent, so
  it does not race the UI rebuild it triggers — and after any display
  list rebuild that ends with the heap above 300 MB.

Final measurement: a fully zoomed sphere costs 12.9 MB of live managed
heap at rest, with committed segments returned to the OS.
@davidebazzi

davidebazzi commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Hi, I did a test, draw a sphere at location 0,0,0 and radius 20, the used memory jumped from 206MB to 252MB, then I zoomed a little and it increased to 485MB.
I continued to zoom and I reached 3.1GB, then it stopped.

@dsn27

dsn27 commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator Author

Yes, you are right.
The code the AI produced is not really better.
I will convert this to a draft and work on the problem later.

@dsn27
dsn27 marked this pull request as draft July 29, 2026 09:01
@davidebazzi

davidebazzi commented Jul 29, 2026 via email

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dsn27 @davidebazzi @claude