TechPrufer
Sign inGet started

Blogs›Datadog›Postgres JIT Compiler Bug Investigation

Postgres JIT Compiler Bug Investigation

Postgres JIT Compiler Bug Investigation

1
posts
2025

This post details the investigation and resolution of a critical segmentation fault in Datadog's Postgres clusters, which was traced back to a bug in the Arm64 JIT compiler (specifically within LLVM). The investigation involved isolating the issue, reproducing it on Arm64 instances, and debugging a Postgres build with JIT debugging support enabled. The root cause was identified as an infinite loop in the Arm64 JIT compilation process for certain expressions, leading to a segfault or a 100% CPU utilization stall. The team successfully identified the problematic assembly code and contributed an upstream fix to LLVM.

2025

Unraveling a Postgres segfault that uncovered an Arm64 JIT compiler bug | Datadog

1/15/2025

The post details the process of identifying and debugging a segmentation fault in Postgres JIT compilation on Arm64. It covers isolating the issue to a specific query, reproducing it by matching production environment characteristics (including Arm64 architecture), and using a debug build of Postgres with LLVM and GDB to analyze the JIT-generated assembly code. The core contribution is the identification of an infinite loop in the Arm64 JIT compiler's execution of certain expressions, leading to the segfault, and the subsequent contribution of an upstream fix to LLVM.