Blogs›Yahoo›Bullet Query System
The Bullet query system has been enhanced with windowing capabilities (time and record-based) to support more interactive and iterative queries. It now also supports Apache Pulsar as a native PubSub option, in addition to Kafka and REST. Furthermore, a new configuration-based data ingestion mechanism (Bullet DSL) has been introduced, allowing users to plug in data sources without writing code, abstracting connector and converter components. This post details the open-sourcing of Bullet, a forward-looking query engine for streaming data that operates on data flowing through the system after the query is submitted, rather than relying on a persistence layer. It supports powerful filtering, raw data fetching, aggregations, and distributions. The architecture is multi-tenant, linearly scalable, and has been tested to handle over 700 simultaneous queries on streams with up to 1.5 million records per second. Bullet is implemented on top of Storm and can be extended to support other stream processing engines. It is used at Yahoo for validating instrumentation, functional testing in continuous delivery pipelines, and dynamically computing cardinalities.
3/8/2019
Introduced time and record-based windowing for the Bullet query system, enabling tumbling and sliding window aggregations. Added Apache Pulsar as a supported PubSub mechanism. Implemented Bullet DSL for configuration-based data ingestion, abstracting data source connectors and converters.
6/15/2017
This post announces the open-sourcing of Bullet, Yahoo's forward-looking query engine for streaming data. It details Bullet's architecture, capabilities (filtering, aggregation, distributions), and its unique 'look-forward' approach that avoids data persistence. The post highlights its scalability, performance metrics (700+ queries, 1.5M records/sec), and its implementation on Storm, with extensibility to other stream processing engines. It also describes its usage at Yahoo for instrumentation validation, continuous delivery testing, and cardinality checks.