Boosting Performance: How We Scaled and Enhanced Throughput in the Jira Export Service

This post details the evolution of the Jira Cloud Data Export Service, focusing on architectural changes and code optimizations to improve export throughput. Key contributions include: adopting a hybrid architecture leveraging Lithium streaming for high-volume entities (issues, comments, change history) and HTTP Batch APIs for low-volume entities; optimizing high-volume entity extraction by splitting tasks into parallel jobs executed by multiple threads and compute nodes; and addressing N+1 query issues in key export code paths. The MVP design achieved a 5x throughput improvement, and the current design further scales horizontally by engaging multiple compute nodes based on project data size.