Title: Training of Modern Large Language Models

Abstract: The talk will showcase the development of modern large language models such as the Falcon3 series of open models, with an emphasis on the specificities of training with knowledge distillation and up-scaling techniques, which allow significant pre-training cost reduction to build the models. We will also discuss models’ performances and capabilities by providing some insights on the main components to build large efficient language models.