Troubleshooting a Search Slowdown After Upgrading Milvus: Lessons from the WPS Team

This post details the WPS engineering team's troubleshooting of a 3-5x search latency increase after upgrading Milvus from 2.2.16 to 2.5.16. It covers the motivation for the upgrade (stability issues with `planparserv2.HandleCompare`), the use of `milvus-backup` for hot backup and restore across versions, the deployment of a new Milvus 2.5.16 cluster using Helm with `mmap.enabled: false`, and the subsequent investigation into the performance degradation. The post highlights the importance of verifying backups, building new clusters for major upgrades, and keeping old clusters running during cutover.