Fully Sharded Data Parallel: faster AI training with fewer GPUs

Introduced Fully Sharded Data Parallel (FSDP) as a new tool for more efficient AI model training. FSDP shards model parameters, gradients, and optimizer states across data parallel workers, improving memory efficiency and computational efficiency through communication/computation overlapping. It is implemented in the FairScale library and offers simple APIs as a drop-in replacement for PyTorch's DistributedDataParallel module. The post details the technical mechanism of FSDP, explaining how it decomposes all-reduce operations into reduce-scatter and all-gather, and provides usage examples for language models (fairseq), computer vision models (VISSL), PyTorch Lightning, and direct FairScale integration.