From Supercomputer to Raspberry Pi: Building Open Source Polish Language Models

The creation of Polish language models presents a unique set of challenges and opportunities in the Polish AI landscape. Through collaboration between SpeakLeash Foundation and Academic Computer Centre Cyfronet AGH, we've established Bielik - a family of open-source language models designed to democratize access to AI.

Our journey began with training larger models of 7B and 11B parameters, providing us with valuable experience and knowledge about training models in the Polish language. This experience has led us to our latest effort: developing a compact 1.5B parameter model that brings advanced language capabilities to edge devices like Raspberry Pi.

During this presentation, we'll explore the real-world challenges of training Polish language models, sharing technical insights from our transition from 11B to 1.5B parameters. We'll discuss our work with large Polish datasets, examining the intricacies of the training process for our compact model.

Our presentation will provide insights into the process of model development, from creating high-quality Polish language datasets to enhancing cooperation between an open-source foundation and academic institution. We'll also discuss the balance between model size and performance, highlighting how we make advanced language models accessible for practical use.