TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

ML Model Serving Platform Modernization

3 posts · Jul 2019 – Oct 2025

What this trail covers

Grab's ML model serving platform, Catwalk, has evolved from a multi-framework, multi-engine approach to a unified platform leveraging NVIDIA Triton Inference Server. This migration addresses technical debt, performance degradation, and cost inefficiencies. The initial phase focused on integrating Triton for ONNX models, introducing a 'Triton manager' component for seamless drop-in replacement, backward compatibility, and zero-downtime migration. Subsequent phases aim to onboard other ML frameworks. The platform has further evolved to replace Helm charts with Kubernetes CRDs for better deployment control, and then to a high-code, self-served, process-managed platform with the Catwalk Orchestrator, enabling bundled deployments and mandatory load testing. Future plans include Catwalk serverless, data serving, and LLM serving.

All Grab trails
  1. Modernising Grab’s model serving platform with NVIDIA Triton Inference ServerThis post details the migration of Grab's ML model serving platform (Catwalk) from a multi-framework, multi-engine architecture to a unified NVIDIA Triton Inference Server. It highlights the technical challenges of maintaining multiple inference engines, the evaluation and benchmarking of Triton, and the development of the 'Triton manager' component. The manager includes a Triton server manager for model preparation and launch, and a Triton proxy for backward compatibility. The post reports on the successful rollout to over 50% of online deployments, significant latency reductions (e.g., p90 latency from 120ms to 20ms for transformer models), and cost savings of ~20% for migrated services. It also discusses ongoing work to enable batching for further tail latency reduction.
  2. Evolution of Catwalk: Model serving platform at GrabThis post details the evolution of Grab's ML model serving platform, Catwalk, through four phases. Phase 1 focused on a no-code, managed platform for TensorFlow Serving models, facing scalability and admin bottlenecks. Phase 2 transitioned to a low-code, self-serving model for multiple frameworks (ONNX, PyTorch, TensorFlow) using CI/CD pipelines and introducing Catwalk proxy, transformer, and Amphawa. Phase 3 replaced Helm charts with Kubernetes CRDs for improved deployment control, enabling customisations like smart defaults, blue-green deployments, and advanced scaling. Phase 4 introduced the Catwalk Orchestrator, a high-code framework for building ML applications, supporting bundled deployments and mandatory load testing, leading to a significant increase in deployed orchestrators and models.
  3. Catwalk: Serving Machine Learning Models at ScaleThis post details the initial implementation of Catwalk, Grab's ML model serving platform. It describes the goals of abstracting infrastructure complexities for data scientists, preventing duplicated effort, and creating a performant, highly available, versioned platform. The core components discussed are TensorFlow Serving for model deployment and Kubernetes for orchestration. TensorFlow Serving is used to load models directly from S3, with automatic versioning and graceful updates. Kubernetes is employed to manage TensorFlow Serving pods, deployments, services, and ingress, enabling autoscaling and high availability. The post outlines the workflow for data scientists to export and upload models to S3, which are then automatically served by Catwalk.