Real-time full-text search with Luwak and Samza | Confluent

This post describes the combination of Luwak and Apache Samza for building a scalable streaming search engine. Luwak, a Lucene-based library, efficiently handles thousands of queries over a single document, while Samza, a stream processing framework based on Kafka, distributes real-time computations across a cluster. The approach involves registering queries in advance and matching them against incoming documents, optimizing the process by indexing queries themselves to quickly identify non-matching queries. This enables real-time full-text search on streams, exemplified by use cases like Twitter's firehose search and Google Alerts.