How to Reliably Measure Software Performance
H.1301 (Cornil) | Day 2 | 11:50 - 12:30 | Speakers: Kemal Akkoyun, Augusto de Oliveira
Abstract
Reliable performance measurement remains an unsolved problem across most open source projects. Benchmarks are often an afterthought, and when they aren't they can be noisy, non-repeatable, and hard to act on.
This talk shares lessons learned from building a large-scale benchmarking system at Datadog and shows how small fixes can make a big difference: controlling environmental noise, designing benchmarks, interpreting results with sound statistical methods, and more.
Attendees should leave with practical principles they can apply in their own projects to make benchmarks trustworthy and actionable. We'll illustrate each principle with real data — for instance, environment tuning that cut variance by 100x, or design changes that turned a flaky benchmark into a reliable one.
Attachments
Speakers
https://github.com/kakkoyun
Software Infrastructure Engineer | System Programmer | Performance Engineer. Obsessed with Observability, Instrumentation, and low-level programming. 🤔 Slow thinker. Open Source Enthusiast. Mentor (CNCF LFX, Google Summer of Code, CommunityBridge, GoBridge). Blogger and speaker. Introverted Human (not Cylon, I guess). 😄 Pronouns: He/Him
🔭 Deep in the trenches of Go runtime and compiler internals, building AST manipulation tools for injecting instrumentation systems. Working on Go instrumentation and profiling at Datadog APM. Keeping a soft spot for profiling while tinkering with Go and its toolchain. 🌱 Exploring engineering leadership, LLM internals, and automating Personal Knowledge Management with Claude Code and MCPs.
Software engineer currently building benchmarking systems to make Datadog’s APM suite faster.
My background is in electrical engineering, mathematics, and data science, and during my studies I had the opportunity to apply these disciplines across several fields. I worked on denoising X-ray images with GE Healthcare, reconstructing of roads from radar satellite images with researchers from SONDRA/European Space Agency, speech separation for automatic speaker tracking with Orange, and automatic e-commerce product title generation for BearingPoint.
Beyond engineering, I am passionate about climbing, lifting, calisthenics, fixed-gear cycling, music production, writing, and TTRPGs. Happy to chat about any of these to break the ice :)
Links
External Links
Notice: The placeholder video image is licensed under CC BY-SA 4.0. The original image can be found hereChanges made to the image are: Cropped the image to a new ratio, part of the image was cut off.