TechPrufer
Sign inGet started

Blogs›IBM›Model Serving on Kubernetes

Model Serving on Kubernetes

Model Serving on Kubernetes

1
posts
2021

ModelMesh, a model serving management layer for Watson products, is being contributed to the open-source community as ModelMesh Serving. It is designed for high-scale, high-density, and frequently changing model use cases, intelligently loading and unloading AI models to balance responsiveness and computational footprint. ModelMesh integrates with KServe, becoming the backend for Multi-Model Serving. It features distributed LRU cache management, intelligent placement and loading of models, resiliency through retries, operational simplicity with decentralized control, and scalability supporting hundreds of thousands of models.

2021

ModelMesh and KServe bring eXtreme scale standardized model inferencing on Kubernetes

10/12/2021

This post announces the open-sourcing of ModelMesh, a model serving management layer for Watson products, and its integration with KServe. It details ModelMesh's architecture, core components (ModelMesh Serving, ModelMesh containers, runtime adapters), and supported model-serving runtimes (Triton Inference Server, Seldon MLServer). Key features highlighted include cache management and HA, intelligent placement and loading, resiliency, operational simplicity, and scalability demonstrated by packing 20K models into two serving runtime pods with single-digit millisecond latency. The post also announces the transition of KFServing to KServe and the collaborative development of a unified KServe API.