Fit streaming retrains on the current window of stream items - #51
Merged
Conversation
Two problems with the re-train data handling: Each streamed item carries a single frame's sparse contact map, a variable-length array of concatenated row and column indices. np.concatenate flattened every frame in the batch into one array, so frame boundaries were lost before the data reached model.fit. Build an object array instead, one entry per frame, which is the ragged layout the collector produces. The batch was also appended to contact_map_history and pcoord_history, which were never trimmed. Memory grew for the life of the run and every retrain re-consumed all data seen so far, making successive retrains progressively slower. Fit on the current batch only. How much history to retain is still open (see the existing TODO). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two problems in
run_stream_train's handling of re-train data.Contact maps were flattened across frames. Each streamed item carries a single frame's sparse contact map —
np.concatenate([row, col]), so a variable-length 1-D array whose length depends on the number of contacts in that frame. The batch was combined with:which concatenates every frame in the batch into one flat array, destroying frame boundaries before the data reaches
model.fit. Replaced with an object array holding one entry per frame, matching the ragged layoutContactMapCollectorproduces elsewhere:History grew without bound. Each batch was appended to
contact_map_history/pcoord_history, which were never trimmed. Memory climbed for the life of the run, and every retrain re-consumed all data seen so far, so successive retrains got progressively slower.Each retrain now fits on the current window of stream items only. How much history to retain is still an open question — the existing
TODOis left in place.Extracted from #40 so it can land on its own.
🤖 Generated with Claude Code