TechPrufer
Get started
Slack
Slack logoSlack · Feature trail

Asyncio Runtime Migration

1 post · Oct 2024

What this trail covers

This thread tracks the evolution of Slack's Python runtime migration to asyncio. Initial posts detailed the project's inception in 2020, aiming to replace a custom runtime with the standard Python IO library. The migration proceeded smoothly across most environments, with one notable exception on a large single-tenant cluster where sporadic timeouts occurred during spreadsheet exports to PDF under peak load. This post details the resolution of a critical bug related to TCP connection state management, which involved identifying and fixing incorrect implementations of `closed()` and `is_connected()` methods in the `AsyncioConnection` class. These fixes addressed issues where the system failed to recognize closed TCP connections, particularly when initiated by the other side (e.g., database proxies or RPC services), leading to `EOFError` or hung requests. The resolution of these issues unblocked the final stage of the asyncio migration, allowing for the deletion of the custom runtime scheduler.

All Slack trails
  1. We’re All Just Looking for ConnectionThis post details the resolution of a critical bug that stalled the asyncio runtime migration. The bug stemmed from incorrect implementations of `closed()` and `is_connected()` in the `AsyncioConnection` class, which failed to accurately detect when TCP connections were closed by the remote end (e.g., database proxies or RPC services). This led to `EOFError` during SQL queries and hung requests during PDF export RPC calls. The investigation revealed subtle TCP state management issues, including the `CLOSE_WAIT` state and the behavior of `asyncio.StreamWriter.is_closing()`. The fixes involved correcting these methods to properly detect connection closure, which in turn unblocked the final deployment of asyncio across all services.