Multi-cloud Capacity Management (MCM)

6 posts · Mar 2024 – Sep 2025

What this trail covers

Baseten's Multi-Cloud Capacity Management (MCM) system unifies GPU deployments across 20+ clouds and regions into a single, elastic pool. It supports Baseten Cloud, Self-hosted, and Hybrid deployment modes, offering 99.99% uptime, low latency, and compliance with data residency requirements. The system manages autoscaling, failover, and latency-aware routing, providing a globally consistent orchestration layer for inference infrastructure. MCM enables dynamic workload placement based on geographic and compliance needs. Baseten Hybrid extends this by allowing sensitive workloads to run in a customer's VPC with optional spillover to Baseten Cloud for on-demand compute, enabling flexible use of existing cloud credits and commits.

All Baseten trails

Sep 2025How Baseten MCM, our cloud ecosystem partners, and NVIDIA drive fast, reliable inference at scaleThis post details how Baseten's MCM system orchestrates GPUs across multiple cloud providers and regions, acting as a unified, elastic resource. It highlights active-active reliability for 99.99% uptime, intelligent compute allocation for low latency, and compliance features. The post emphasizes the foundational role of cloud partners and NVIDIA's hardware in enabling this scalable inference infrastructure, and mentions the upcoming NVIDIA Blackwell GPUs.
Jun 2025How we built Multi-cloud Capacity Management (MCM)This post details the engineering effort behind Baseten's Multi-cloud Capacity Management (MCM) system. It explains the architectural shift from siloed compute to a globally fungible GPU pool, addressing challenges of inter-cloud fluidity and workload migration. Key technical contributions include the design of a global control plane and workload planes built on Kubernetes, enabling real-time placement decisions based on event streams and capacity reporting. The post highlights the implementation of intelligent routing for low latency, region/provider locking for compliance, and the benefits of deep cloud provider partnerships for unlocking capacity and enabling first-to-market GPU adoption. It also contrasts the pre-MCM manual deployment process with the current on-demand scaling and failover capabilities, emphasizing the system's role in achieving high availability and elastic scale.
Jun 2025How Baseten multi-cloud capacity management (MCM) unifies deploymentsThis post details the Baseten Multi-Cloud Capacity Management (MCM) system, which unifies GPU deployments across 20+ clouds and regions. It introduces three deployment modes: Baseten Cloud (fully managed), Self-hosted (within customer VPCs), and Hybrid (combining self-hosted with Baseten Cloud for burst capacity). The system provides 99.99% uptime, low latency, and data residency compliance by managing autoscaling, failover, and latency-aware routing. It emphasizes flexibility, vendor lock-in avoidance, and efficient utilization of compute resources across heterogeneous environments.
Sep 2024Introducing Baseten Hybrid: control and flexibility in your cloud and oursThis post introduces Baseten Hybrid, a new capability that combines Baseten Self-hosted and Baseten Cloud into a unified multi-cloud solution. It allows users to run inference workloads within their own VPC for security and compliance, with the ability to seamlessly spill over to Baseten Cloud for additional compute capacity during traffic spikes. This provides multi-cloud elasticity and agnosticism, enabling users to leverage existing cloud credits and GPU commits from providers like AWS and GCP without engineering effort. The system aims to provide ultra-low-latency inference with elastic scale, utilizing optimization techniques like TensorRT/TensorRT-LLM and vLLM, and includes improvements for cold starts and end-to-end latency. It also supports compound AI systems built with Baseten Chains.
Aug 2024Introducing Baseten Self-hostedThis post introduces Baseten Self-hosted, a new deployment option that allows AI model inference to run within a customer's own cloud environment (VPC). This offering provides enhanced control over data locality, security, compliance, and the utilization of existing cloud credits and custom hardware. It leverages Truss for packaging and deploying models, ensuring that data and compute remain within the customer's infrastructure. Baseten continues to manage application and infrastructure updates, including zero-downtime deployments and no-disruption updates, for these self-hosted instances. This expands the MCM system's capabilities to include in-VPC deployments, complementing Baseten Cloud and Hybrid modes.
Advertisement
May 2024Control plane vs workload plane in model serving infrastructureThis post introduces and details the architectural separation of control and workload planes within Baseten's multi-cloud AI model serving infrastructure. It explains how a single Kubernetes control plane manages multiple, geographically distributed Kubernetes workload planes. The control plane is responsible for user interface, API endpoints, model image building, deployment, and global load balancing. Workload planes are responsible for running model inference on GPUs, pulling images, and local load balancing. This separation addresses challenges like regional preference, data residency, GPU availability, self-hosted inference, and scaling, while allowing workload planes to adapt to environment-specific factors like GPU type, location, and cost.
Mar 2024The benefits of globally distributed infrastructure for model servingThis post details the benefits of globally distributed infrastructure for ML model serving, emphasizing multi-cloud and multi-region strategies. It explains how this approach increases compute availability by accessing diverse GPU options across providers, leads to cost savings through competitive pricing, enhances redundancy for better uptime by enabling workload shifting during outages, lowers latency by co-locating servers near users, and ensures data residency compliance. Baseten's platform abstracts these complexities, allowing users to leverage these benefits without direct operational overhead.