Reducing observability cognitive load in KubeVirt

KubeVirt, which provides additional functionality to Kubernetes clusters, to perform virtual machine management, is a large multi-component project that, like many others, grew rapidly with a strong focus on delivering new features. In the beginning, observability was not a priority. Whenever necessary, the developers of each component would add Prometheus instrumentation in their code bases in very different ways.

As the project matured and observability became increasingly more important, we created a dedicated team. By then, there were high levels of intertwined observability and business logic code. It was challenging to keep a mental model of how different teams implemented observability across repositories and difficult to maintain them in sync with new features.

This presentation will outline and demo how we modularized the code and made it more readable, and maintainable. By encapsulating the Prometheus monitoring best practices and the common patterns into a library with a strict interface and using it as a dependency for all KubeVirt components, we reduced code complexity, made it easy to maintain and evolve, and reduced the risk of introducing errors. Our experience will give you a clear path forward on how to clean up and improve observability code in your projects.