Test streaming - #40
Open
braceal wants to merge 26 commits into
Open
Conversation
This was referenced Aug 11, 2026
# Conflicts: # deepdrivewe/examples/openmm_ntl9_ddwe/train.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gets the streaming DDWE workflow (
DDWEStreamThinker) running end-to-end on Vista, and adds a complete streaming example plus setup docs.Four bugs found while debugging this branch were split out and merged separately: #48 (
WeightedEnsemble.metadatadefault), #49 (inverteduse_stale_modelgating), #50 (print_errorson nullfailure_info), #51 (unbounded re-train history).mainhas been merged back in, so what remains here is only the streaming feature work.Stream configuration
ProxyStreamConfiggains two validators (workflows/stream.py):redis_hostaccepts the sentinel valuehostname, resolved toaddress_by_hostname()at load time. This is what lets one config file work unchanged across compute nodes, where the broker address is not known until the job lands.store_configrequires ahostnamewhen the connector kind isredis, and accepts the samehostnamesentinel. Previously a missing hostname surfaced later as an opaque connector error; it now fails at config load with a message that names the fix.StreamProducer(publisher, stores={topic: store})now passesstoresas a keyword, per proxystore/proxystore#641.proxystoreis installed asproxystore[redis].Proxy lifetime for streamed model weights
The training producer sends its
TrainResultwithevict=False. The default eviction-on-consume is wrong here: the thinker holds the model across iterations and hands it to every inference task until a newer one arrives, so evicting at consume time pulls the object out from under in-flight inference.Memory is still reclaimed —
train_stream_processorexplicitly evicts the previous proxy's key when it is replaced, which is now logged on both sides of the call.Example and docs
examples/openmm_ntl9_ddwe_stream/— config, CVAE config, NTL9 basis state, reference structure, and a pretrained checkpoint for a runnable streaming NTL9 DDWE run.stream_configblock with a commentedFileConnectoralternative, how to start and kill a background redis server, and how to watch its memory usage. The install instructions addconda install anaconda::redis.Notes for reviewers
print/logger.infoinstrumentation added while debugging the stream path. Worth deciding what should stay as permanent logging versus what should be dropped before merge.examples/openmm_ntl9_ddwe_stream/checkpoint-epoch-100.ptis a 13 MB binary committed directly to the repo. Flagging in case it belongs somewhere out-of-tree.errorrather thanwarning, since it stops the workflow.debug,logs,config, …) and would be better squashed on merge.