Atlassian’s Inference Engine, our self-hosted AI inference service

This post details the creation and architecture of Atlassian's Inference Engine, a self-hosted AI inference service. It highlights the technical challenges of scaling AI workloads, the decision to build in-house to avoid vendor lock-in and improve control, and the resulting performance and cost benefits. Key architectural components include Kubernetes, Karpenter, ArgoCD, Helm, NVIDIA Triton Inference Server, TensorRT-LLM, and vLLM. The post elaborates on the GitOps deployment model for reliability and traceability, and the optimization stack for model inferencing. It describes the control plane for model preparation and the data plane for serving inference at scale, including details on node provisioning (Karpenter), ingress routing (Istio), autoscaling (KEDA), and observability (Prometheus/SignalFx). The post quantifies improvements in latency (up to 40% P-90 reduction for LLM) and cost savings (over 60% for LLM).