Distributed AI Training

Resources

• Data-Parallel Distributed Training of Deep Learning Models
• Pipeline-Parallelism: Distributed Training via Model Partitioning

There are 2 ways to parallelize:

https://docs.oneflow.org/en/v0.4.0/extended_topics/model_mixed_parallel.html