Think outside the red box

This post details the migration of Trello's delta queue storage from Redis to a Mongo TTL collection. Previously, Redis lists with a 100-item limit caused 'disconnected' errors when users were offline for extended periods. The new system uses a Mongo TTL collection to retain queue items for a week and allows querying up to 1000 items, with potential for further increase. This change has significantly reduced write-heavy operations on Redis and improved overall system headroom. Additionally, web client reconnect logic was tweaked to reduce transient connection failures by checking for offline status before incrementing reconnect rate limits, resulting in a 15% reduction in one type of red box and a 30% reduction in another.