How we debugged a latent PgBouncer bug across four layers of the stack | Decagon

This post details the debugging process of a latent PgBouncer bug that caused connection stalls and pool exhaustion during a migration from GCP's managed connection pooler. The bug was traced across four layers: SQLAlchemy's QueuePool, PgBouncer's packet parser and event loop, OpenSSL's TLS record buffer, and the Linux kernel's epoll() interface. The root cause was identified as OpenSSL buffering decrypted data that PgBouncer's event loop could not see after a partial read, leading to a deadlock. Mitigation involved shortening the application name parameter and contributing an upstream fix to PgBouncer to check SSL_pending() before waiting on epoll().