Fully Sharded Data Parallel (FSDP)
FSDP is just more fancy Data Parallel, where we shard many other things (parameters, gradients, optimizer state).
FSDP is an implementation of ZeRO-3.
Resources
- https://engineering.fb.com/2021/07/15/open-source/fsdp/
- PyTorch FSDP Experiences on Scaling Fully Sharded Data Parallel that you should read if you really want to understand what’s going on.
Just read https://huggingface.co/spaces/nanotron/ultrascale-playbook?section=zero-3:_adding_parameter_partitioning_(fsdp) and it explaisn very well what is happening.