Cloudflare incident on January 24, 2023

This post details a critical incident on January 24, 2023, caused by a bug in the release of a new 'Last seen at' field for service tokens. The bug inadvertently overwrote the 'client_secret' metadata for service tokens, rendering them invalid and causing outages for several Cloudflare services, including Workers, Zero Trust, Cache Purge, Cache Reserve, Images, and R2. The post explains the technical details of the bug, the incident timeline, the fix implemented (manual restoration and database backup), and the root cause analysis, highlighting the importance of robust metadata handling and transactional integrity in authentication systems.