Model Parallel

Pipeline Parallelism

Use of Pipelining for GPUs to do training.

https://colossalai.org/docs/concepts/paradigms_of_parallelism/

GPipe paper:

• https://arxiv.org/abs/1811.06965