TechPrufer
Get started
Atlassian
Atlassian logoAtlassian · Feature trail

Tenant Context Service (TCS) Availability and Scalability

1 post · Aug 2022

What this trail covers

The Tenant Context Service (TCS) has evolved into a highly available and resilient infrastructure service at Atlassian, serving as a critical component for request routing and metadata provision across cloud products. Recent improvements have focused on enhancing its availability to consistently exceed six nines (99.9999%), achieving this through advanced isolation techniques in both the sidecar and server components, robust scaling of the invalidation broadcast system, and sophisticated strategies for managing outstanding requests and defending against traffic blackholes. The service now handles billions of requests daily across thousands of compute nodes, with cache hit ratios generally exceeding 99.5%.

All Atlassian trails
  1. Here’s how one of Atlassian's critical services consistently gets above 99.9999% of availabilityThis post details the technical advancements made to Atlassian's Tenant Context Service (TCS) and its sidecar to achieve consistently high availability (above 99.9999%). Key contributions include: implementing complete region isolation in sidecars through independent task queues and thread pools with dynamic sizing and load-shedding; achieving server-side isolation in the invalidation broadcast system by replicating data and processing threads into separate region-specific queues; scaling the invalidation broadcast system by batching messages to stabilize burst traffic and reduce SQS throttling, and migrating from SNS fanout to a bespoke S3-based fanout pattern to manage costs and lifecycle complexity; and introducing strategies like preemptive duplicate requests to primary and secondary TCSs, and limiting thread pools for secondary TCSs to defend against fail-slow scenarios and traffic blackholes.