Skip to content

Fix distributed CMNRL gradient synchronization - #7

Merged
capemox merged 2 commits into
mainfrom
fix/distributed-cmnrl
Aug 17, 2026
Merged

Fix distributed CMNRL gradient synchronization#7
capemox merged 2 commits into
mainfrom
fix/distributed-cmnrl

Conversation

@capemox

@capemox capemox commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Summary

  • move CMNRL GradCache orchestration outside the DDP-wrapped model forward, following the Sentence Transformers trainer boundary
  • replay each mini-batch through the wrapped model under no_sync() and synchronize accumulated gradients on the final replay
  • reject unsafe direct distributed model calls and incompatible static-graph DDP explicitly
  • replace the finite-gradient smoke assertion with exact two-rank MNRL/CMNRL gradient equivalence, plus an end-to-end distributed Trainer step with gradient accumulation
  • document the distributed CMNRL behavior and include it in the 0.1.1 changelog

Release note

CMNRL remains directly callable in a single process. Starting in 0.1.1, directly wrapping CachedMNRLForPretraining in DDP raises an error; distributed CMNRL must use PretenseTrainer so every cached replay is synchronized correctly.

Validation

  • uv run ruff check .
  • uv run mypy --python-version 3.10 src/pretense
  • uv run pytest -q
  • two-process distributed matrix on the current lockfile: gather on/off, find-unused on/off, direct-call guard, static-graph guard, and Trainer gradient accumulation
  • full dependency-floor pytest suite and the same two-process distributed matrix on PyTorch 2.2 / the lowest supported dependencies
  • wheel and sdist build plus Twine metadata checks

@capemox
capemox merged commit db50a94 into main Aug 17, 2026
4 checks passed
@capemox
capemox deleted the fix/distributed-cmnrl branch August 17, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant