Hardening eBPF for runtime security: Lessons from Datadog Workload Protection | Datadog

This post distills six lessons learned from operating Datadog's Workload Protection product, powered by eBPF, at scale. Key lessons include navigating kernel version and distribution compatibility challenges (program type, helper, map, hook point, inlining, verifier compatibility), capturing and enriching data correctly (e.g., handling event ordering, context propagation), monitoring and auditing eBPF usage (e.g., detecting malicious eBPF programs, understanding eBPF attack surface), operating alongside other eBPF users (e.g., avoiding conflicts, managing shared resources), measuring and controlling performance cost (e.g., understanding eBPF overhead, optimizing program logic), and shipping changes safely with strong rollout practices (e.g., phased rollouts, canary deployments). The post highlights specific pitfalls like silent hook failures, kernel version discrepancies, and performance regressions, and outlines engineering strategies to mitigate them.