Revamping Confluence Cloud Search

This post details the migration of Confluence Cloud Search from a self-managed Elasticsearch cluster to a new, cross-product platform powered by AWS-managed Elasticsearch. It highlights the operational, financial, and architectural complexities of the old system, including issues with cluster health, tenant scaling, manual failovers, and licensing costs. The new architecture emphasizes multi-region availability through event multiplexing and SQS queues, efficient sharding based on tenant size to improve scalability and cost-efficiency, and an optimized permission checking mechanism that converts linear time complexity to logarithmic time complexity by transforming the 'AND of ORs' model to an 'OR of ANDs' using set theory. The post also includes architectural diagrams for both the old and new systems, as well as performance impact visualizations.