TechPrufer
Get started
Yahoo
Yahoo logoYahoo · Feature trail

Bullet Query System

2 posts · Jun 2017 – Mar 2019

What this trail covers

The Bullet query system has been enhanced with windowing capabilities (time and record-based) to support more interactive and iterative queries. It now also supports Apache Pulsar as a native PubSub option, in addition to Kafka and REST. Furthermore, a new configuration-based data ingestion mechanism (Bullet DSL) has been introduced, allowing users to plug in data sources without writing code, abstracting connector and converter components. This post details the open-sourcing of Bullet, a forward-looking general purpose query engine for streaming data, built on top of Storm and designed for low-latency, arbitrary queries without a persistence layer. It can scale linearly and has been tested to handle over 700 simultaneous queries on data streams with up to 1.5 million records per second. Bullet is used at Yahoo for validating application instrumentation and in continuous delivery pipelines for functional testing.

All Yahoo trails
  1. Bullet Updates - Windowing, Apache Pulsar PubSub, Configuration-based Data Ingestion, and MoreThis post details several enhancements to the Bullet query system: 1. Windowing capabilities (time and record-based) have been added to support interactive and iterative queries, including tumbling and sliding window types. 2. Apache Pulsar is now supported as a native PubSub option, alongside existing Kafka and REST support. 3. A new configuration-based data ingestion mechanism (Bullet DSL) has been introduced, allowing users to plug in data sources without writing code by abstracting connector and converter components. This DSL supports Kafka and Pulsar connectors and Avro, Maps, and POJO converters, with basic ETL capabilities.
  2. Open Sourcing Bullet, Yahoo’s Forward-Looking Query Engine for Streaming DataThis post announces the open-sourcing of Bullet, a forward-looking general purpose query engine for streaming data. It details Bullet's architecture, which operates on data flowing through the system after a query is submitted, eliminating the need for a persistence layer. This 'look-forward' approach allows for efficient querying of streaming data, scanning data only once per query. The system is multi-tenant, linearly scalable, and has been tested to handle high query volumes and data rates. It is currently implemented on Storm and can be extended to support other stream processing engines like Spark Streaming or Flink. The post also outlines its usage at Yahoo for validating application instrumentation and in continuous delivery pipelines.