Self-hosted LLMs at a scale with Paddler

Paddler is an open-source llama.cpp load balancer designed to address unique challenges that Large Language Models pose.

Typical balancing algorithms like round-robin or least-connections are not the most efficient approaches.

To introduce predictability into your infrastructure, Paddler reaches for alternative solutions that account for unpredictable response times while being able to scale services up and down at any moment.

This talk will demonstrate Paddler's general design concepts (the "why") and some primary use cases (the "how").