Collective Operations

AllReduce

All-Reduce = ReduceScatter + AllGather

https://tech.preferred.jp/en/blog/technologies-behind-distributed-deep-learning-allreduce/ is really good

Mental Model of All-Reduce

Usually, when I think of AllReduce, in my head it’s more of a Reduce + Broadcast