Under the Hood: Building posts search

This post details the engineering challenges and solutions for building a posts search index capable of handling trillions of posts. Key contributions include: collecting data from a varied MySQL schema, indexing data in HBase using Hadoop MapReduce, updating the index via a pub/sub system (Wormhole) that bypasses caches for initial collection but hits them for updates, serving a large index from a combination of RAM and SSDs, and implementing a sophisticated ranking system with query rewriting and dynamic result scoring based on over a hundred features. The project originated from a hackathon and involved significant infrastructure development.