Trello Real-time Delta Queue

2 posts · Mar 2017 – Feb 2018

What this trail covers

Trello has evolved its real-time update mechanism by migrating its delta queue storage from Redis to a Mongo TTL collection. This change allows for persistent storage of delta queue items for a week and querying up to 1000 items, significantly reducing the 'disconnected' red box errors caused by Redis memory limits and eviction. The web client's reconnect logic has also been optimized to reduce transient connection failures.

All Atlassian trails

Feb 2018Think outside the red boxThis post details the migration of Trello's delta queue storage from Redis to a Mongo TTL collection. Previously, Redis lists with a 100-item limit caused 'disconnected' errors when users were offline for extended periods. The new system uses a Mongo TTL collection to retain queue items for a week and allows querying up to 1000 items, with potential for further increase. This change has significantly reduced write-heavy operations on Redis and improved overall system headroom. Additionally, web client reconnect logic was tweaked to reduce transient connection failures by checking for offline status before incrementing reconnect rate limits, resulting in a 15% reduction in one type of red box and a 30% reduction in another.
Mar 2017Offline attachmentsThis post details the challenges of handling attachments in Trello's offline mode, specifically addressing file permission issues on Android and the blocking of the main sync queue due to long and failure-prone attachment uploads. The solution implemented a separate queue for attachment deltas, processing them after all other changes to maintain app responsiveness. It also discusses the limitations of network quality for large file uploads and the retry capping mechanism.
Advertisement