UCX-based torch.distributed backend with MPI-style tag-matched send/recv (NCCL ignores tags) and collectives on CPU & CUDA.
-
Updated
Jul 5, 2026 - C++
#
torch-distributed
Here are 3 public repositories matching this topic...
Simulated Multi-GPU inference engine implementing Megatron-style Tensor Parallelism, GPipe Pipeline Parallelism, and KV Cache Sharding from scratch. Features bit-identical fidelity validation on Qwen2-0.5B weights and analytical communication cost modeling.
distributed-systems hpc distributed-computing pytorch scaling multi-gpu inference-optimization gpipe pipeline-parallelism kv-cache megatron-lm tensor-parallelism llm qwen2 torch-distributed
-
Updated
Jul 29, 2026 - Python
Learn distributed programming and collectives in pure Python
python education distributed-systems mpi deadlock-detection collectives all-reduce torch-distributed
-
Updated
Jul 15, 2026 - Python
Improve this page
Add a description, image, and links to the torch-distributed topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the torch-distributed topic, visit your repo's landing page and select "manage topics."