Nemo: Data discovery at Facebook

This post introduces Nemo, an internal data discovery engine at Facebook. Nemo revamps the original data discovery solution by using Unicorn search infrastructure for scalability and more sophisticated search capabilities, including natural language queries. It incorporates personalization signals and a gradient-boosted decision tree model for post-ranking. The architecture includes bulk and instant indexing, and the serving component uses spaCy for NLP, Unicorn for retrieval and initial ranking, and an FBLearner-trained ML model for post-processing. Nemo also uses trust signals to rank results, improving data quality and recency. Lessons learned include the challenges of A/B testing and labeling in a large organization, and the importance of Nemo in maintaining data health.