Collective Operations

Gather

Prerequisite to AllGather.

Gather pieces from all ranks to rank 0 (opposite of scatter)

# Input
rank0 = tensor([0.])
rank1 = tensor([1.])
rank2 = tensor([2.])
rank3 = tensor([3.])
# Output
rank0 = tensor([0., 1, 2, 3])