Skip to main content
FOSDEM PWA

Modern Python monorepo with `uv`, `workspaces`, `prek` and shared libraries

UA2.220 (Guillissen) | Day 1 | 16:00 - 16:30 | Speakers: Jarek Potiuk

Modern Python monorepo with `uv`, `workspaces`, `prek` and shared libraries
A picture of a devroom at FOSDEM 2024
Open in browser

Notes

Abstract

Apache Airflow is the most popular Data Workflow Orchestrator - developed under the Apache Software Foundation umbrella. We have 120+ Python distributions in our rep, and we often release ~ 100 of them every two week.

All those distributions are built from a single monorepo.

[jarekpotiuk:~/code/airflow] find . -name 'pyproject.toml' | wc 120 120 4248

This had always posed a lot of challenges and we had a lot of tooling to make it possible, however with the recent development of Python Packaging tools, multipel Packaging PEPs implemented, and with new wave of tools such as uv and prek, our setup is finally manageable and we removed 1000s of line of custom code we wrote before after we applied uv workspaces, switched to prek, started using inline script metadata.

It's a breeze to have monorepo now. This talk explains how.

Bonus content. If you know the differences between dynamically and statically linked libraries in C and other languages, or used NPM - you might recognise the need of being able to use different versions of the same library in the same system. It's not possible in Python. Or is it?

We've figured out a way to eat cake and have it too - and we have "statically linked" libraries in Python. How did we do it?

You will find out how from the talk.

Speakers

Jarek Potiuk

Independent Open-Source Contributor and Advisor, Committer and PMC member of Apache Airflow, Member of the Apache Software Foundation, Security Committee Member of the Apache Software Foundation. Organizer of community-focused events, speaker.

Jarek is an Engineer with a broad experience in many subjects - Open-Source, Cloud, Mobile, Robotics, AI, Backend, Developer Experience, Security, but he also had a lot of non-engineering experience - building a Software House from scratch, being CTO, organizing big, international community events, technical sales support, pr and marketing advisory but also looking at legal aspects of security, licensing, branding and building open-source communities are all under his belt.

With the experience in very small and very big companies and everything in-between, Jarek found his place in the Open Source world, where his internal individual-contributor drive can be used to the uttermost of the potential.

Notice: The placeholder video image is licensed under CC BY-SA 4.0. The original image can be found hereChanges made to the image are: Cropped the image to a new ratio, part of the image was cut off.