TechPrufer
Get started
Discord
Discord logoDiscord · Feature trail

Message Indexing and Search

1 post · Apr 2025

What this trail covers

Discord's message search infrastructure has evolved significantly since its initial design in 2017. Initially, messages were indexed lazily into Elasticsearch, sharded by guild or direct message across two clusters. As Discord grew, this system began to show cracks, necessitating further development and optimization of message indexing and search capabilities to handle trillions of messages. This post details the evolution from indexing billions to trillions of messages, highlighting the challenges and solutions in scaling Elasticsearch for massive message volumes, including strategies for sharding, indexing, and querying.

All Discord trails
  1. How Discord Indexes Trillions of MessagesThis post details the evolution of Discord's message search infrastructure from indexing billions to trillions of messages. It describes the initial architecture using Elasticsearch sharded by guild/DM, the challenges encountered with growth, and the ongoing efforts to optimize indexing and querying for massive message volumes. The post highlights the need for continued development and optimization to handle the scale of Discord's message data.