NVIDIA and Microsoft collaborated to offer an end-to-end solution to training MT-NLG, combining the DeepSpeed library and the Megatron library. This enabled the model to be trained within a reasonable amount of time. The combined libraries offer an optimized system architecture with multiple GPUs, enabling faster and more efficient training of the model. The system also provides software design, hardware system, and system throughput to guarantee the best results. The model was trained with a dataset of over 45TB of text, and with a configuration of 50-billion words tokens. The results of the model have been impressive, reaching state-of-the-art performance in many natural language understanding tasks. Additionally, the team has been able to detect and mitigate bias found in the language model. The research and development team of NVIDIA and Microsoft have contributed to the development of this project, and their combined effort has led to the successful training and deployment of the world's most powerful language model.
