Keynote Talk: Collaborative Learning in the Age of LLMs: Challenges and Outlook

Abstract: Collaborative machine learning (CML) is a distributed approach to train ML models. In contrast to conventional ML training, where all model and training data is processed centrally, CML performs model update steps locally at the clients’ side on their local datasets, and data does not leave the clients’ devices. This enhances privacy since sensitive data is not shared with other parties. Federated Learning is one of the most used CML approaches and relies on a server coordinating the learning process. Alternatively, Decentralized Learning (DL) is a CML algorithm where the learning process occurs on a network of interconnected devices with no central server to supervise the training.Large Language Models (LLMs) have recently dominated the ML landscape. This is driven by the effectiveness and popularity of conversational bots such as ChatGPT and Gemini, as well as their applicability to many other tasks. LLMs are primarily trained in data centers on massive amounts of data. While LLMs conceptually can be trained using CML approaches, the vast size of LLMs, which are orders of magnitudes larger than the model sizes currently being considered by CML approaches, makes existing CML approaches infeasible. At the same time, the collaborative training of LLMs with commodity hardware, such as mobile devices, will ensure broader access to cutting-edge technology and training infrastructure, enable alternative privacy-preserving ways to train LLMs without specialized hardware and inspire further innovations in the field.
In this talk, I will present some of the technical challenges and outlook of training and fine-tuning LLMs using collaborative learning approaches such as FL and DL. These challenges mainly relate to the unreliability of participants during the learning process, dealing with the size of LLM models, and the privacy considerations of sharing model updates with other clients.