Modernising Grab’s model serving platform with NVIDIA Triton Inference Server

This post details the migration of Grab's ML model serving platform (Catwalk) from a multi-framework, multi-engine architecture to a unified NVIDIA Triton Inference Server. It highlights the technical challenges of maintaining multiple inference engines, the evaluation and benchmarking of Triton, and the development of the 'Triton manager' component. The manager includes a Triton server manager for model preparation and launch, and a Triton proxy for backward compatibility. The post reports on the successful rollout to over 50% of online deployments, significant latency reductions (e.g., p90 latency from 120ms to 20ms for transformer models), and cost savings of ~20% for migrated services. It also discusses ongoing work to enable batching for further tail latency reduction.