Torch Distributed

Also see Collective Operations.

• https://learnopencv.com/distributed-parallel-training-pytorch-multi-gpu-setup/

Inside a node → NCCL uses NVLink
Across nodes → NCCL uses RDMA/IB

Fundamental concepts:

• DDP
• FSDP

import torch.distributed as dist

YOuu should just read this guide: https://lambda.ai/blog/multi-node-pytorch-distributed-training-guide

Some example. here:

• https://github.com/LambdaLabsML/examples/blob/main/pytorch/distributed/resnet/main.py

torchrun \
  --nnodes=2 \
  --nproc_per_node=8 \
  --rdzv_id=100 \
  --rdzv_backend=c10d \
  --rdzv_endpoint=$MASTER_ADDR:29400 \
  elastic_ddp.py

DDP under the hood

grad = sum(grad over all ranks) / world_size

• This is just an AllReduce operation!