Yahoo Feature Trails

The Bullet query system has been enhanced with windowing capabilities (time and record-based) to support more interactive and iterative queries. It now also supports Apache Pulsar as a native PubSub option, in addition to Kafka and REST. Furthermore, a new configuration-based data ingestion mechanism (Bullet DSL) has been introduced, allowing users to plug in data sources without writing code, abstracting connector and converter components. This post details the open-sourcing of Bullet, a forward-looking query engine for streaming data that operates on data flowing through the system after the query is submitted, rather than relying on a persistence layer. It supports powerful filtering, raw data fetching, aggregations, and distributions. The architecture is multi-tenant, linearly scalable, and has been tested to handle over 700 simultaneous queries on streams with up to 1.5 million records per second. Bullet is implemented on top of Storm and can be extended to support other stream processing engines. It is used at Yahoo for validating instrumentation, functional testing in continuous delivery pipelines, and dynamically computing cardinalities.

OpenTSDB 2.4.0 introduces Rollup and Pre-Aggregation Storage to manage growing time series data by storing lower-resolution data for longer periods and pre-aggregating data to speed up queries. This release also adds support for Histograms and Data Sketches for accurate percentile calculations across multiple series, HBase Date Tiered Compaction, a new authentication plugin interface, direct data fetching from Bigtable/HBase via search indexes, improved UID cache controls, and configurable query limits.

Vespa now supports importing ONNX models for use in ranking, expanding its AI model compatibility beyond TensorFlow. This allows for real-time model evaluation over large datasets using models trained with various tools.

Omid, an open source transaction processing platform for Big Data, has released version 1.0.0. This release introduces a re-engineered protocol (Omid LL) for ultra-low-latency OLTP applications, achieving up to 550,000 tps with low millisecond latency. It also adds ANSI SQL support for Apache Phoenix, including mechanisms to avoid recursive read-your-own-writes scenarios and extended Snapshot Isolation. Performance improvements include integration of Omid's code within HBase-resident procedures for reduced transactional read overhead.

Panoptes is a distributed system for collecting, enriching, and distributing network telemetry. It supports multiple polling formats (SNMP, vendor APIs, gNMI), uses Kafka as a metrics bus, Celery with Redis for scheduling, and integrates with InfluxDB for time-series storage. It aims to consolidate polling solutions, normalize data, and provide flexibility for evolving telemetry standards.

OakMap, a new open-source concurrent ordered key-value map, is introduced. It extends previous research (KiWi) and offers improved scalability and performance over existing solutions like ConcurrentSkipListMap, particularly for big data analytics. OakMap stores keys and values off-heap in memory buffers and uses a two-level on-heap index for efficient access. It addresses garbage collection overhead in Java applications and improves query and update speeds through data locality and in-situ computation. OakMap is being integrated into Druid's core Incremental Index component.

This post details the implementation of OAuth 2.0 Token Exchange for migrating mobile app identities and a custom device SSO mechanism using device keychain/account manager for cross-app authentication within Oath. It leverages existing standards like OAuth 2.0 and OpenID Connect, profiling them to solve specific mobile identity challenges.

This post introduces the ability for developers to access unique mail metadata like DECOS and THREADID via IMAP commands, enabling richer client-side experiences such as travel and shopping smart views and message threading.

Yahoo Mail has implemented novel revisions to its Redux architecture, introducing 'action syncers' to decouple API interactions from action creators and 'action enhancers' middleware to avoid prop drilling for action dispatching. These changes have led to significant stability and performance improvements, including reduced latencies, fewer JavaScript exceptions, better state synchronization, reduced API calls, and improved rendering times.

Yahoo Mail has implemented a secure proxy server to serve images within emails. This enhances security by preventing phishing and scams, improves user experience by automatically displaying images, and allows for accurate tracking of true email opens while respecting sender privacy. The system also advises senders to use alternative methods for IP-based location targeting.

Yahoo (Oath) successfully migrated its jobs from Apache MapReduce to Apache Tez, a DAG-based execution engine. A significant bottleneck in the Shuffle Handler Service was identified and addressed through community collaboration, leading to a new Shuffle Handler implementation. This resulted in a 100% migration score for Tez and the rollout of the new Shuffle Handler to research clusters, processing approximately 100 million Tez DAGs with 50 billion tasks over 38,000 nodes.

Yahoo Mail has significantly improved its web application's reliability and performance by adopting a new tech stack including React, Redux, and Node.js. This migration led to faster development cycles by reducing code reflection time from minutes to seconds, enhanced reliability by shifting JavaScript bundle generation from runtime to build time, and improved performance through substantial reductions in gzipped JavaScript payload size, server-side render time, and overall launch performance metrics. Key optimizations include proximity-based routing, server-side rendering with isomorphic JavaScript, efficient bundling strategies, improved CSS reuse with react-atomic-css, and the use of Mendel for bucket testing. The new stack also reduced JavaScript and CSS footprint by approximately 50%, improved memory consumption, and achieved a 99.99% success rate on core flows. Additionally, the team open-sourced Mendel, their solution for bucket testing isomorphic web applications.

Daytona is an open-source framework for automated performance testing and analysis. It provides application-agnostic capabilities for integrated performance testing, repeatable test execution, standardized reporting, and built-in profiling. The framework consists of a centralized scheduler, distributed agents, a MySQL database for metadata, and a PHP-based UI. It aims to aggregate and present application, system, and hardware performance metrics in a unified interface, enabling continuous performance analysis and optimization.