Collective Operations

https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html

I really understood this / internalized after watching the CS336 lecture.

• Broadcast

• Gather

• Scatter

• Reduce

• AllReduce

• AllGather

• ReduceScatter

• All-to-All