The SELinux problem that cast a months long shadow

In February 2024, Red Hat released an update to insights-core, a package providing host data for Red Hat Insights. It slipped through our testing and caused all SELinux-enabled systems to crash the service, stopping the hosts from reporting. Even though the patch was released two days later, we couldn’t fix the hosts ourselves, and had to figure out which customers are affected and how to even contact them. It was a big lesson to both engineering and management. We’d like to share this story with the public.