Anatomy of a (Postgres) search engine

This post provides a deep dive into the technical implementation of full-text search, specifically focusing on the inverted index data structure. It explains the components of an inverted index (term dictionary, postings lists, positional and frequency data), the process of tokenization (including stemming and stop words), and scoring mechanisms like BM25 for top-k queries. It also touches on index size trade-offs and the management of segments (mutable vs. immutable) for dynamic data. This article serves as a foundational explanation of how full-text search works, complementing the existing feature thread on PlanetScale's TIN extension.