Private, secure DeepSeek-R1 in production in US & EU data centers

This post details the engineering challenges and solutions for deploying large open-source models like DeepSeek-R1 and DeepSeek-V3 in production. It highlights the need for multi-node and multi-cluster inference due to model size (671 billion parameters) and GPU memory constraints (e.g., H100 vs. H200 GPUs). Baseten addresses these by providing specialized hardware (H200 GPUs), supporting multi-node inference, and enabling dedicated deployments within customer VPCs for security and compliance. The post also discusses enterprise-grade performance through custom optimizations on inference frameworks like SGLang and TensorRT-LLM, and a competitive cost structure, especially for distilled models.