ModelMesh and KServe bring eXtreme scale standardized model inferencing on Kubernetes

This post announces the open-sourcing of ModelMesh, a model serving management layer for Watson products, and its integration into KServe as the backend for Multi-Model Serving. It details ModelMesh's architecture, including ModelMesh Serving, runtime adapters, and integration with model servers like Triton and MLServer. Key features highlighted are distributed LRU cache management, intelligent model placement and loading, resiliency through retries and rerouting, operational simplicity with decentralized control, and scalability demonstrated by packing 20K models into two serving runtime pods with single-digit millisecond latency. The post also covers the evolution of KFServing to KServe and the collaborative development efforts.