Model Serving on Kubernetes

ModelMesh, a model serving management layer for Watson products, is being contributed to the open-source community as ModelMesh Serving. It is designed for high-scale, high-density, and frequently changing model use cases, intelligently loading and unloading AI models to balance responsiveness and computational footprint. ModelMesh integrates with KServe, becoming the backend for Multi-Model Serving. It features distributed LRU cache management, intelligent placement and loading of models, resilient retries and rerouting, and seamless rolling updates. It supports hundreds of thousands of models in a single production deployment.