Firefly: Instant, Full-Text Search Engine for Dropbox (Part 1)

This post introduces Firefly, Dropbox's instant, full-text search engine, detailing its goals for scale, query latency (<250ms), and indexing latency (<10s). It explains the challenges of building such a system at Dropbox's scale, particularly the operational overhead of per-user indices and the difficulty of instant updates for large indices. The core sharding strategy is presented: sharding based on 'namespaces' (which represent user files and shared folders) rather than individual users. Each token in the search index is prefixed with its namespace ID to optimize query processing for user-specific namespaces. The post also discusses the decision to build a custom solution over leveraging existing open-source search engines like SolrCloud or Elasticsearch due to scale and control requirements, while acknowledging the use of open-source components like LevelDB, HDFS, HBase, RabbitMQ, and Apache Tika.