Vulkan API for Machine Learning? Competing with CUDA and ROCm in llama.cpp

Most Machine Learning tools use CUDA for hardware acceleration, and are as a result only compatible with Nvidia GPUs. AMD has been making a lot of progress enabling simple recompilation with minimal code changes to ROCm for their hardware, but why not use an open and broadly-compatible API instead? That's where Vulkan comes in, which was built up for game development, but also allows compute-only applications, and has broad and good driver support across many hardware vendors.

As a follow-up to last year's talk about my work on the llama.cpp/GGML Vulkan backend, this talk will discuss lessons learnt from optimizations and new features that we added since, how viable Vulkan is for Machine Learning and what it is still missing.

https://github.com/ggml-org/llama.cpp https://github.com/ggml-org/ggml