Skip to main content
FOSDEM PWA

Running tinygrad and ggml on microcontroller NPUs

UD2.120 (Chavanne) | Day 1 | 15:10 - 15:15 | Speakers: Roman Shaposhnik

Running tinygrad and ggml on microcontroller NPUs
A picture of a devroom at FOSDEM 2024
Open in browser

Notes

Abstract

Running various forms of inference on microcontroller NPUs is not new. Systems where machine learning is used to analyze sensor data or do light CV on microcontroller-grade systems under 1 watt, under few dozen MB of RAM and FLASH and under 10 USD bill-of-materials are being massively deployed (even if they stay in the long shadow of more flashy LLMs and GenAI). That area, however, historically has been a domain of specialized machine learning frameworks such as emlearn, LiteRT (artist formerly known as TensorFlow Lite) and a few others.

The question I will try to answer in this talk is the following: are there any benefits of trying to use more well established, but still pretty tightly optimized frameworks such as ggml and tinygrad for these types of deployments. I will share my experience with adopting these frameworks to targets such as Google Coral NPU and AI Foundry Erbium and what kind of interesting challenges it presented.

Speakers

Roman Shaposhnik

Roman Shaposhnik is the Co-founder and Chief Technology Officer of Ainekko, a company committed to making AI hardware and software fully open, modular, and community-driven. At Ainekko, Roman is leading the effort to democratize silicon by open-sourcing production-grade hardware and tooling, empowering developers to co-design AI systems from the chip up. His mission is to extend the values of open source into the core of AI hardware, enabling a new era of experimentation, accessibility, and edge innovation.

Roman is a longtime open-source advocate and contributor to the Apache Software Foundation, where he played key roles in projects like Hadoop, Bigtop, and Incubator. He also used to work on Plan9, Linux kernel, gcc and ffmpeg. He has held senior technical positions at Sun Microsystems, Pivotal, and Cloudera, and was the founding CTO of ZEDEDA. A frequent speaker and community builder, Roman’s work at the Linux Foundation and LF Edge reflects his commitment to collaborative, bottom-up innovation. He earned his Master’s in Mathematics and Computer Science from Saint Petersburg State University, graduating summa cum laude.

Notice: The placeholder video image is licensed under CC BY-SA 4.0. The original image can be found hereChanges made to the image are: Cropped the image to a new ratio, part of the image was cut off.