Collective Operations

AllReduce

AllReduce = ReduceScatter + AllGather

https://tech.preferred.jp/en/blog/technologies-behind-distributed-deep-learning-allreduce/ is really good