TechPrufer
Get started
Dropbox
Dropbox logoDropbox · Feature trail

Firefly Search Engine

1 post · Mar 2015 – May 2021

What this trail covers

Dropbox developed Firefly, an instant, full-text search engine designed to scale with hundreds of billions of files. The system aims for sub-250ms query latency and sub-10-second indexing latency. It addresses the challenges of scale, query latency, and indexing latency by sharding the search index based on namespaces rather than individual users. Each token in the index is prefixed with its namespace ID to optimize query processing for user-specific namespaces. This post details the core idea behind the sharding strategy and the challenges of building a search engine at Dropbox's scale.

All Dropbox trails
  1. How image search works at DropboxThis post details the core idea behind Dropbox's image content search method, based on techniques from machine learning (image classification and word vectors), and how it was implemented on Dropbox's existing search infrastructure (Nautilus). The image search leverages image classification to understand image content and word vectors to map text queries into the image's category space, enabling relevance scoring. The production architecture approximates this method for efficiency on the Nautilus search engine, which uses forward and inverted indexes. The image search functionality is a component of the broader search engine capabilities.
  2. Firefly: Instant, Full-Text Search Engine for Dropbox (Part 1)This post introduces Firefly, Dropbox's instant, full-text search engine, detailing its goals for scale, query latency (<250ms), and indexing latency (<10s). It explains the challenges of building such a system at Dropbox's scale, particularly the operational overhead of per-user indices and the difficulty of instant updates for large indices. The core sharding strategy is presented: sharding based on 'namespaces' (which represent user files and shared folders) rather than individual users. Each token in the search index is prefixed with its namespace ID to optimize query processing for user-specific namespaces. The post also discusses the decision to build a custom solution over leveraging existing open-source search engines like SolrCloud or Elasticsearch due to scale and control requirements, while acknowledging the use of open-source components like LevelDB, HDFS, HBase, RabbitMQ, and Apache Tika.