How to Instrument Go Without Changing a Single Line of Code
UB5.132 | Day 2 | 13:30 - 14:00 | Speakers: Kemal Akkoyun, Hannah Kim
Abstract
Zero-touch observability for Go is finally becoming real. In this talk, we’ll walk through the different strategies you can use to instrument Go applications without changing a single line of code, and what they cost you in terms of overhead, stability, and security.
We’ll compare several concrete approaches and projects:
- eBPF-based auto-instrumentation, using OpenTelemetry’s Go auto-instrumentation agent:
- https://github.com/open-telemetry/opentelemetry-go-instrumentation
- https://opentelemetry.io/docs/zero-code/obi/
- Compile-time manipulation, using tools that rewrite or augment Go binaries at build time, such as:
- https://github.com/alibaba/opentelemetry-go-auto-instrumentation
- Runtime techniques, including agents, shared-library injection, and binary trampolines, as used in OpenTelemetry’s Go “Autosdk” work:
- https://opentelemetry.io/docs/zero-code/go/autosdk/
- USDT (User Statically-Defined Tracing) probes, exploring how to add or generate USDT probe points for Go services (at build time or via injection) so that external tooling (eBPF, DTrace-style tools, etc.) can consume high-level events without source changes.
Beyond what exists today, we’ll look at how ongoing work in the Go runtime and diagnostics ecosystem could unlock cleaner, safer hooks for future auto-instrumentation, including:
runtime/traceand diagnostics primitives:- https://pkg.go.dev/runtime/trace
- https://go.dev/doc/diagnostics
- Proposals such as Go “flight recording” (Issue #63185):
- https://github.com/golang/go/issues/63185
Throughout the talk, we’ll use benchmark results and small, realistic services to compare these strategies along three axes:
- Performance overhead (latency, allocations, CPU impact)
- Robustness and upgradeability across Go versions and container images
- Operational friction: rollout complexity, debugging, and failure modes
Attendees will leave with a clear mental model of when to choose eBPF, compile-time rewriting, runtime injection, or USDT-based approaches, how OpenTelemetry’s Go auto-instrumentation fits into that picture, and where upcoming runtime features might take us next. The focus is strongly practical and open-source: everything shown will be reproducible using publicly available tooling in the Go and OpenTelemetry ecosystems.
Attachments
Speakers
https://github.com/kakkoyun
Software Infrastructure Engineer | System Programmer | Performance Engineer. Obsessed with Observability, Instrumentation, and low-level programming. 🤔 Slow thinker. Open Source Enthusiast. Mentor (CNCF LFX, Google Summer of Code, CommunityBridge, GoBridge). Blogger and speaker. Introverted Human (not Cylon, I guess). 😄 Pronouns: He/Him
🔭 Deep in the trenches of Go runtime and compiler internals, building AST manipulation tools for injecting instrumentation systems. Working on Go instrumentation and profiling at Datadog APM. Keeping a soft spot for profiling while tinkering with Go and its toolchain. 🌱 Exploring engineering leadership, LLM internals, and automating Personal Knowledge Management with Claude Code and MCPs.
Links
External Links
Notice: The placeholder video image is licensed under CC BY-SA 4.0. The original image can be found hereChanges made to the image are: Cropped the image to a new ratio, part of the image was cut off.
