DeepSpeed is an open source deep learning optimization library for PyTorch.[1] The library is designed to reduce computing power and memory use and to train large distributed models with better parallelism on existing computer hardware.[2][3] DeepSpeed is optimized for low latency, high throughput inference. It includes the Zero Redundancy Optimizer (ZeRO) for training models with 100 million parameters or more.[4] Features include mixed precision training, single-GPU, multi-GPU, and multi-node training as well as custom model parallelism. The DeepSpeed source code is licensed under MIT License and available on GitHub.[5]
See also
References
- ^ https://uk.pcmag.com/news-analysis/127085/microsoft-updates-windows-azure-tools-with-an-eye-on-the-future
- ^ Microsoft speeds up PyTorch with DeepSpeed | InfoWorld
- ^ Microsoft unveils “fifth most powerful” supercomputer in the world – Neowin
- ^ Microsoft trains world’s largest Transformer language model | VentureBeat
- ^ https://github.com/microsoft/DeepSpeed
Further reading
- Rajbhandari, Samyam; Rasley, Jeff; Ruwase, Olatunji; He, Yuxiong (2019). “ZeRO: Memory Optimization Towards Training A Trillion Parameter Models” (PDF). Cite journal requires
|journal=(help)
External links
- AI at Scale – Microsoft Research
- GitHub – microsoft/DeepSpeed
- ZeRO & DeepSpeed: New system optimizations enable training models with over 100 billion parameters – Microsoft Research
