Interactivity on 100GB+ datasets: O(change) view sync and ledger writes by AlexGrayBox · Pull Request #303 · GrayboxTech/weightslab

AlexGrayBox · 2026-09-03T10:01:29Z

Backend half of the 100GB-interactivity work. Companion to weights_studio#144, which
carries the UI half; neither is much use without the other.

Validated on the UltraEdit InstructPix2Pix harness: 3,959,093 train samples,
859M params, batch 24, A10G.

The problem

Every path that kept the served view in sync with the ledger ran O(dataset)
rather than O(change): a full materialized-view rebuild on each signal tick, and
a read-modify-append of the whole H5 table per upsert. At 4M rows that is a 670s
startup and lock holds long enough that the UI looks hung while training.

Worse, several of the failures reported themselves as success — the differential
sync wrote nothing and returned True, view-build errors were swallowed at debug
level, and per-signal exceptions were caught by wrappered_fwd. A signal could
fail on every step for hours and the only symptom was a column that stayed empty.

What is in here

O(change) view + storage (999e859b)

data_service: differential view refresh (_fastUpdateInternals) writing
value-only deltas through a sample_id → row-position map. Falls back to the
full rebuild on any structural change, so correctness never depends on the fast
path being right about a schema change. Off with WL_FAST_VIEW=0.
Sort restructured to run off-lock — ops and pos-map rebuild on a shallow copy,
only the pointer swap under the lock.
h5_dataframe_store: in-place row updates via modify_coordinates with a
cached coordinate map. Index construction split from storage layout, so the
flush path stops rebuilding an index no hot-path read uses.
dataframe_manager: column-wise vectorised passes replacing the per-row
iterrows() scans that dominated startup.

Correctness fixes (cf0397df, 9466dc13, 0ef0a1ce, 2c37d714)

Differential sync addressed rows by get_level_values(0) — which is origin,
not sample_id — so the intersection was always empty.
Rebuild when the ledger gains columns the view lacks (signal columns are created
on first write, so a fresh view could never gain them).
View-dirty backlog kept until a rebuild actually lands, not discarded on overflow.
Named image views: probe extra_images() on the unwrapped dataset (WL's tracking
wrapper does not forward it), and advertise filtered-out views with an empty
thumbnail so their toggles do not vanish.
Histogram bins over equal-population boundaries on the finite subset, so the bars
stay usable click-targets instead of four fat bars plus 500 slivers.
trainer_tools.process_sample unpacked exactly 3 values from _getitem_raw,
whose contract is (data, id, target, *metadata) — any dataset with metadata
raised "too many values to unpack" and every grid cell came back with no image.
Positional NB_SEEN lookup and a cached level-0 index for sample-id coercion.

Post-merge repairs (53ac28a2, ac608f69, 70caddee, da353661)

Four fixes for things a wholesale take of dev's version reverted — dev carries no
equivalent, so the merge silently dropped them. All four were invisible at runtime:

deque import missing → NameError on every per-sample write, swallowed; the
history tail just stayed empty.
Label cache dropped from write_signal_shapes → an incremental pass no longer
returned a whole-dataset distribution.
inputs= dropped from the batched subscribe_to path → sig/loss_debiased
raised KeyError 12,079 times in one five-hour run, once per step, with no
visible error.
Generated protos regenerated with package-relative imports (dev's flat
import experiment_service_pb2 only resolves with weightslab/proto on
sys.path, which is not how the trainer loads it).

Docs (b26f0109, 83a67e25) — the O(data) register and the triage of all 18
_slowUpdateInternals call sites, including why the ApplyDataQuery filter paths
deliberately stay on the rebuild.

Removed (a360234d) — optrace and all 82 call sites. Pure deletion, 437 lines
out, 0 in. It found the sample_id level bug, the NB_SEEN lookup and the flush
accounting; it has served its purpose.

Measured

	before	after
startup	670s	250s
H5 upsert (24 rows)	127.9s	~16ms
snapshot flush	338s	0 samples
max lock hold	129,440ms	none over 1s
throughput under UI	13%	45–51% of idle
signals / step	1230ms	28–41ms

End-to-end WL overhead on this harness: 1171ms → 1406ms/step, +20.1% median,
fully attributed — sig_loss 65.15ms (73%), sig_x0 22.00ms (25%), sig_save
1.20ms (1.3%); 88.35ms accounted against 89.00ms measured.

The residual loss under UI load is CPU/GIL contention (8 vCPUs shared by 6
dataloader workers, training and image encode), not lock waiting.

UI contract suite: 21/21 on this build.

Known gaps, deliberately left for review

ensure_index() has no caller and conflicts with _try_inplace, which refuses
indexed tables. It documents the deliberate-index path but is dead code as
committed.
_POSMAP_CACHE is class-level and never evicts (~400–600MB at 4M rows).
Three ApplyDataQuery sites still force a full rebuild, pending a decision on
_is_filtered semantics.
The branch is 32 commits behind dev as of this PR; it needs one more merge
before landing. The last two merges each dropped a fix (see above), so that
merge wants reviewing rather than accepting.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RHV8zUd5aCtHtQourmwUJC

Storage and serving paths whose cost scales with dataset size rather than with what changed. Storage findings re-verified against dev; two serving costs noted as already fixed upstream so they are not re-claimed as wins. No code changes - baseline and measurement protocol only.

16 of 18 sites only need fresh values for dirty rows (O(change)); only first build and schema change need a full reconstruction. Records why ApplyDataQuery filter paths stay on the rebuild (_is_filtered semantics) and why a no-client benchmark cannot show the difference.

Every path that kept the served view in sync ran O(dataset): a full materialized-view rebuild on each signal tick, and a read-modify-append of the whole H5 table per upsert. At 3.96M rows that meant a 670s startup and lock holds long enough that the UI looked hung while training. Three changes, each turning a whole-dataset pass into a per-change one: data_service: differential view refresh (_fastUpdateInternals). The materialized view only ever recomputes index-derived state, so a value-only delta can be written straight into the existing view through a sample_id -> row-position map instead of rebuilding it. Falls back to the full path on any structural change (unknown sample_id, missing pos map, backlog over max_dirty), so correctness never depends on the fast path being right about a schema change. 9 call sites routed here; the 10 that genuinely change shape still force a rebuild. Off with WL_FAST_VIEW=0. The sort path is restructured to do its work off-lock: ops and the pos-map rebuild both run on a shallow copy, and only the pointer swap happens under the lock. Skipping the pos-map rebuild after a sort would have been a data corruption bug -- sorting reorders the view, so stale positions send differential writes to the wrong rows. h5_dataframe_store: in-place row updates via modify_coordinates, with a cached sample_id -> coordinate map (stable row positions come free from the no-row-loss invariant). PyTables cannot invalidate a column index during modify_coordinates, so _try_inplace refuses indexed tables and the caller falls back to the append path. Index construction is also split from storage layout: data_columns=True keeps the on-disk layout queryable while index=False keeps the flush path from rebuilding an index no hot-path read uses (92.7s -> 6.9s per upsert at 4M rows). dataframe_manager: replaces the per-row iterrows() scans that dominated startup with column-wise vectorised passes, and adds the dirty-row/source-row accessors the differential refresh needs. Also fixes an unrelated thumbnail bug in trainer_tools.process_sample: it unpacked exactly 3 values from _getitem_raw, whose contract is (data, id, target, *metadata). Any dataset implementing get_items() with metadata raised "too many values to unpack" and every cell in the grid came back with no image. Now unpacked positionally. Measured on 3.96M-row UltraEdit, A10G: startup 670s -> 250s H5 upsert (24 rows) 127.9s -> ~16ms snapshot flush 338s -> 0 samples max lock hold 129,440ms -> none over 1s throughput under UI 13% -> 45-51% of idle The residual loss under load is CPU/GIL contention (8 vCPUs shared by 6 dataloader workers, training, and image encode), not lock waiting. Known gaps, deliberately left for review: - ensure_index() has no caller yet, and conflicts with _try_inplace, which refuses indexed tables. It documents the deliberate-index path but is dead code as committed. - _POSMAP_CACHE is class-level and never evicts (~400-600MB at 4M rows). - Three ApplyDataQuery sites still force a full rebuild pending a decision on _is_filtered semantics. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The ledger was always correct; the view readers see was not, and every failure mode reported itself as success. On a 3.96M-sample run the UI showed ~2k samples with loss data at 34k steps, and toggling image modalities showed the source image twice. View correctness: * Address differential-sync rows by the SAMPLE_ID index level. The view is indexed (origin, sample_id), so get_level_values(0) returned origin and its intersection with the dirty sample_ids was always empty -- the sync wrote nothing and returned True, which suppressed the rebuild that would have repaired it. Positions now come from Index.get_indexer (cached hash engine), so it stays vectorised. * Rebuild when the ledger gains columns the view lacks. Per-sample signal columns are created on their first write, so on a fresh ledger the view predates them and could never gain them: sorting and histogramming failed with "column not in view" and last_seen served -1 forever. Checked before the dirty-set drain, since a schema gain is independent of dirty rows. * Keep the view-dirty backlog until a rebuild actually lands. It was discarded on overflow assuming the caller would rebuild, but the force path returns early on a contended lock -- those ids were then lost with nothing left to re-mark them. Cleared at the atomic view swap instead. * Log view-build failures as errors. They were swallowed at debug level and returned the previous view, so a broken build was indistinguishable from "no new data". Named image views: * Probe extra_images() on the unwrapped dataset -- WL's tracking wrapper does not forward it, so every named view was silently dropped. * Honour stats_to_retrieve for image views, but still advertise filtered-out views with an empty thumbnail so their toggles do not vanish from the panel. Cost: * Loss-shape autotagging runs on its own interval (WL_LOSS_SHAPE_INTERVAL_SECONDS, default 60s) instead of the 2s flush tick, where each pass cost ~990ms of GIL-held pandas work. * Signal-DAG history reads a bounded in-memory tail (WL_HISTORY_TAIL, default 16) instead of scanning per_sample -- 140ms per step at 20M rows, growing without bound. Neither an index nor a rewritten IN clause helped (1.1x/1.4x). * Skip array normalisation for columns the H5 write excludes: with predictions off it rasterised via get_mask, which reads the source image, for data that is never persisted. * Close inherited HDF5 fds in forked dataloader workers; they made HDF5 refuse the parent's read-write open and killed ledger persistence for 12 hours. Measured on the UltraEdit harness (859M params, batch 24, A10G) against an identical run with weightslab stubbed out: 1574ms -> ~1290ms/step versus a 1171ms baseline, i.e. +34% -> ~+10%. optrace.py is included: the @traced/hit markers the other files import are what located the sample_id level bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…e view The numeric path cut bin boundaries by row position across the WHOLE view and dropped non-finite values only afterwards, so every bucket spanned len(view)/max_bins rows regardless of the data. On a 3,963,189-row view with 512 bins that is 7,740 rows per bucket -- so a column where only ~33k samples carry a value (any signal early in a run) collapsed into the first four buckets, and the remaining 500 sliced empty space into 1-6 sample slivers. Visible as four fat bars followed by a long tail of random-looking spikes, with the same 7740/7741 counts appearing on unrelated columns because the number came from the row count, not the data. These bars are a search surface over the loss landscape: each should be a click-target holding a comparable number of samples. Mask first, then cut equal-population boundaries over the finite subset. Row ORDER is untouched, so this is still "bin the current view by row order" -- it just stops counting rows that have nothing to show. Also fixes the per-(origin, discarded) sub-bars, which were grouped by the same positional bins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three things, all needed to make the branch runnable on top of dev at 4M rows. 1. Regenerated experiment_service_pb2{,_grpc}.py dev's .proto declares AnnotationExportFormat / EXPORT_FORMAT_CVAT but the committed gencode predates it, so a clean checkout of dev does not import at all: AttributeError: module 'weightslab.proto.experiment_service_pb2' has no attribute 'EXPORT_FORMAT_CVAT' Regenerated with grpcio-tools 1.68.1 (protoc 5.28.1), matching the runtime version already pinned in the file, so the gencode major does not move. 2. Cache the level-0 index for sample-id coercion _coerce_sample_id_for_index() called index.get_level_values(0) on every invocation. That materialises a fresh Index over all rows, and a fresh Index carries a fresh hash engine, so each `sid in level_0_values` paid a full engine build -- twice per sample when the int probe missed. enqueue_batch does that per sample, 24x a step: training sat at 0 iterations with the main thread pinned at 100% CPU inside pandas __contains__ (py-spy: active+gil). Cached on the index object's identity. pandas Index is immutable, so any reindex or rebuild yields a new object and invalidates it; membership semantics are unchanged, the engine is simply reused. 3. Positional NB_SEEN lookup get_sample_column_values() then still materialised both index levels, ran isin over every row and copied a boolean-masked frame -- a full pass plus a copy over 3.96M rows to read 24 integers, ~1.2s/step (signals 6ms -> 1230ms, total 1290ms -> 2500ms). The wanted rows are exactly (sample_id, 0), so resolve their positions with Index.get_indexer instead. Falls back to the original scan when the index is not unique. Measured on the UltraEdit harness (859M params, batch 24, A10G, 3.96M samples): signals 1230ms -> 28-41ms total 2500ms -> 1310-1338ms (1171ms with weightslab stubbed out) NB_SEEN now actually increments (it was stuck at 0 before dev's fix), verified against the ledger: rows with nb_seen>0 equals rows with last_seen>=0. The UI contract suite passes 21/21 on this build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Each bar must cover total_rows / max_bins samples so the chart carries density: a column that is only 0.2% populated should show a few filled bars and the rest empty. Binning over just the rows that carry a value made the chart look equally full at any coverage, which reads as "every sample already has a loss". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…gb-datasets # Conflicts: # weightslab/trainer/services/data_service.py

dev checks in generated code that does a flat 'import experiment_service_pb2', which only resolves if weightslab/proto is itself on sys.path. Imported as a package -- which is how the trainer loads it -- startup dies with ModuleNotFoundError. Regenerated from the merged .proto at the repo root so dev's new RPCs are kept and the import is package-relative again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dev reimplemented the O(change) loss-shape autotag independently, and its version is a superset of ours: write_signal_shapes gained exp_hash scoping, and the dirty set is keyed (graph, exp_hash) with a difference_update that does not drop ids staged DURING the classify call. Took dev's logger.py and src.py whole rather than hand-merging two implementations of the same idea. Kept on top of it the one thing dev has no equivalent for: the bounded in-memory history tail (_default_history_tail / WL_HISTORY_TAIL / _recent_tail / recent_per_sample) and history()'s fast path in src.py. Those make b.history() O(batch) instead of scanning per_sample every step, and sig/loss_shape reads through them -- dropping them costs ~140ms/step and leaves loss_shape with no history to classify. Protos regenerated from the merged .proto so dev's new fields are present and the import stays package-relative.

The merge re-applied our in-memory history tail onto dev logger.py, which imports only defaultdict. Every per-sample write then raised NameError inside _stage_sample_row. The caller swallows per-signal exceptions, so nothing crashed: the tail just stayed empty, sig/loss_debiased failed every step, and loss_shape had no history to classify. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dev rewrite keeps the O(change) read and adds exp_hash scoping, both kept. What it dropped is the label cache, which two behaviours depended on: - an incremental pass still returns a distribution over the WHOLE dataset, not just the samples it happened to touch; - a sample whose label did not change is not re-written to the ledger. Both are asserted by e2e_autotag (distribution_covers_dataset, incremental_writes_bounded), which failed on the merge until this went back. The test also moves to dev parameter name, sample_ids. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

On the subscribe_to path BatchSignalContext was built without inputs=, so b.inputs was {} and any signal declaring inputs=[...] raised KeyError on every call. sig/loss_debiased does exactly that: it failed 12,079 times in one five hour run -- once per step -- and because wrappered_fwd swallows per-signal exceptions nothing crashed, the column just silently never got values. We had already fixed this; taking dev src.py whole during the merge reverted it, since dev never carried the fix. Same class as the deque import and the label cache: dev has no equivalent, so a wholesale take drops it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@Traced

Drops the optrace module and every call site: 64 @Traced decorators, 13 hit() markers and 5 imports across the data stores, the dataframe manager and the two services. Pure deletion -- 437 lines out, 0 in. Every hit() was verified to be a bare statement rather than an expression, so removing it cannot change a value, and removal was parenthesis-balanced because several spanned three lines. Each file is compiled after editing, which is what would catch a removal that left an empty block. The tracing was built to find where interactivity time went on a 100GB dataset. It has served that purpose: the O(change) view sync, the O(batch) NB_SEEN lookup and the flush accounting all came out of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three fixes, one per failing check. ruff F841, trainer_tools.process_sample: the positional unpack of _getitem_raw bound _res[1] to idx, which nothing reads -- the function returns sid. Dropped. ruff F401, examples/.../wl-video-generation/utils/data.py: unused `os` import. Pre-existing on dev and untouched by this branch; it only surfaces here because the lint step appends ./weightslab to the changed-file list, so ruff scans the whole package on any PR that touches it. Removing it is what unblocks the gate. AttributeError in tests/gRPC/test_grpc_user_actions.py: _fastUpdateInternals duck-types take_view_dirty and get_source_rows on the df manager. Both are new on this branch, so _FakeDFManager -- and any third-party manager -- raised AttributeError instead of taking the fallback. Guarded: a manager without dirty tracking cannot serve a delta, which is the same "structural change" case the method already falls back on, and the caller then runs _slowUpdateInternals exactly as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RHV8zUd5aCtHtQourmwUJC

Alexandru Rotaru and others added 15 commits August 19, 2026 10:29

Merge remote-tracking branch 'origin/dev' into feat/interactivity-100…

8944304

…gb-datasets # Conflicts: # weightslab/trainer/services/data_service.py

Navigation Menu

Uh oh!

Interactivity on 100GB+ datasets: O(change) view sync and ledger writes - #303

Interactivity on 100GB+ datasets: O(change) view sync and ledger writes#303
AlexGrayBox wants to merge 15 commits into
devfrom
feat/interactivity-100gb-datasets

AlexGrayBox commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

AlexGrayBox commented Sep 3, 2026

The problem

What is in here

Measured

Known gaps, deliberately left for review

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant