Torch Distributed

import torch.distributed as dist

YOuu should just read this guide: https://lambda.ai/blog/multi-node-pytorch-distributed-training-guide

Some example. here:

• https://github.com/LambdaLabsML/examples/blob/main/pytorch/distributed/resnet/main.py