Yahoo Feature Trails

Feature trailQuiet since 2019

Bullet Query System

The Bullet query system has been enhanced with windowing capabilities (time and record-based) to support more interactive and iterative queries. It now also supports Apache Pulsar as a native PubSub option, in addition to Kafka and REST. Furthermore, a new configuration-based data ingestion mechanism (Bullet DSL) has been introduced, allowing users to plug in data sources without writing code, abstracting connector and converter components. This post details the open-sourcing of Bullet, a forward-looking general purpose query engine for streaming data, built on top of Storm and designed for low-latency, arbitrary queries without a persistence layer. It can scale linearly and has been tested to handle over 700 simultaneous queries on data streams with up to 1.5 million records per second. Bullet is used at Yahoo for validating application instrumentation and in continuous delivery pipelines for functional testing.

2 posts · 2017–2019

Timeline

20172026

Feature trailQuiet since 2018

OpenTSDB Rollup and Pre-Aggregation Storage

OpenTSDB 2.4.0 introduces Rollup and Pre-Aggregation Storage to manage growing time series data by storing lower-resolution data for longer periods and pre-aggregating data to speed up queries. This release also adds support for Histograms and Data Sketches for accurate percentile calculations across multiple series, HBase Date Tiered Compaction, a new authentication plugin interface, direct data fetching from Bigtable/HBase via search indexes, improved UID cache controls, and configurable query size and time limits.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Vespa ONNX Integration

Vespa now supports importing ONNX models for use in ranking, expanding its AI model compatibility beyond TensorFlow. This allows for real-time model evaluation over large datasets using models trained with various tools. This update also includes improvements to streaming search performance, precise transaction log pruning for faster restarts, and grouping on map attributes for structured data use cases.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Omid Transaction Processing Platform

Omid, an open source transaction processing platform for Big Data, has released version 1.0.0. This release introduces a re-engineered protocol (Omid LL) for ultra-low-latency OLTP applications, achieving up to 550,000 tps with low millisecond latency. It also adds ANSI SQL support for Apache Phoenix, including mechanisms to avoid recursive read-your-own-writes scenarios and extended Snapshot Isolation. Performance improvements include integration of Omid's code within HBase-resident procedures.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Panoptes Network Telemetry Collector

Panoptes is a distributed system for collecting, enriching, and distributing network telemetry. It supports multiple polling formats (SNMP, vendor APIs, gNMI), uses Kafka as a metrics bus, Celery with Redis for scheduling, and integrates with InfluxDB for time-series storage. This post details the open-sourcing of Panoptes, a system designed to consolidate polling solutions, normalize data, and provide flexibility for evolving telemetry standards. It currently handles hundreds of thousands of checks per second and has been tested to over one million checks per second.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

HaloDB Embedded Key-Value Storage Engine

OakMap, an open-source concurrent ordered key-value map, has been released. It extends upon earlier research and offers a Java API similar to ConcurrentNavigableMap but scales significantly better with RAM and CPU resources. OakMap stores keys and values in off-heap memory buffers, accessed via a two-level on-heap index. This design alleviates garbage collection overhead, improves data locality for faster lookups and range scans, and optimizes incremental maintenance of complex values like data sketches. Benchmarks show OakMap outperforming ConcurrentSkipListMap in memory utilization, query speed, and update speed. It is being integrated into Apache Druid's core Incremental Index component.

2 posts · 2018

Timeline

20182026

Feature trailQuiet since 2018

OAuth 2.0 Token Exchange and Device SSO

This post details the implementation of OAuth 2.0 Token Exchange for migrating mobile app identities and a custom device SSO mechanism using device keychain/account manager for cross-app authentication within Oath. It leverages existing standards like OAuth 2.0 and OpenID Connect, profiling them to solve specific mobile identity challenges.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

IMAP Mail Metadata Access

Developers can now access unique mail metadata like DECOS and THREADID via IMAP commands, enabling richer client-side experiences such as travel and shopping smart views and message threading.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Yahoo Mail Redux Architecture Enhancements

Yahoo Mail has implemented novel revisions to its Redux architecture, introducing 'action syncers' to decouple API interactions from action creators and 'action enhancers' middleware to avoid prop drilling for action dispatching. These changes have led to significant stability and performance improvements, including reduced latencies, fewer JavaScript exceptions, better state synchronization, reduced API calls, and improved rendering times. The architecture now separates API concerns from action creators and optimizes side effect handling.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Secure Image Proxy for Mail

Yahoo Mail has implemented a secure proxy server to serve images within emails. This enhances security by preventing phishing and scams, improves user experience by automatically displaying images, and allows for accurate tracking of true email opens while respecting sender privacy. The system also advises senders to use alternative methods for IP-based location targeting. This post details the implementation of the secure image proxy for mail, which serves images via proprietary secure proxy servers to increase speed and security, reduce phishing risks, and eliminate the need for "enable images" settings. It also addresses the tracking of email opens and advises senders against IP-based location targeting.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2018

Apache Tez Migration and Shuffle Handler Improvement

Yahoo (Oath) successfully migrated its jobs from Apache MapReduce to Apache Tez, a DAG-based execution engine. A significant bottleneck in the Shuffle Handler Service was identified and addressed through community collaboration, leading to a new Shuffle Handler implementation. This resulted in a 100% migration score for Tez and the rollout of the new Shuffle Handler to research clusters, processing approximately 100 million Tez DAGs with 50 billion tasks over 38,000 nodes.

1 post · 2018

Timeline

20182026

Feature trailQuiet since 2017

Yahoo Mail Web App Performance and Reliability with Webpack

Yahoo Mail has significantly improved its web application's reliability and performance by adopting a new tech stack including React, Redux, and Node.js. This migration led to faster development cycles by reducing code reflection time from minutes to seconds, enhanced reliability by shifting JavaScript bundle generation from runtime to build time, and improved performance through substantial reductions in gzipped JavaScript payload size, server-side render time, and overall launch performance. This post details the introduction of proximity-based routing, server-side rendering for isomorphic applications, efficient bundling strategies, improved CSS reuse with react-atomic-css, and the use of Mendel for bucket testing isomorphic web apps. It also highlights the reduction in JavaScript and CSS footprint by approximately 50%, faster search and message read operations, and reduced memory consumption. The new version boasts a 99.99% success rate on core flows and significantly reduced client-side errors. Product agility is enhanced through independently deployable components and a robust CI/CD pipeline enabling daily pushes. Developer effectiveness is improved through the use of open-source technologies and a consistent onboarding program. Accessibility has been state-of-the-art with enhanced visual themes and optimization for assistive technologies. Key components like Mendel have been open-sourced.

2 posts · 2017

Timeline

20172026

Feature trailQuiet since 2017

Yahoo Mail Continuous Integration and Delivery Pipeline

Yahoo Mail has implemented a forward-thinking continuous integration and delivery pipeline to ensure regular, stable product updates. Key features include a robust testing infrastructure with over 10,000 unit and 5,000 integration tests, a phased rollout strategy (alpha, beta, canary, production), a mechanism to mitigate irrational test failures by rerunning only failing tests, developer notifications for code pushes, and dedicated pipelines for hotfixes and rollbacks. The system also utilizes 'heartbeat' pushes to prompt users to refresh their browsers for important updates.

1 post · 2017

Timeline

20172026

Feature trailQuiet since 2017

Daytona Performance Analysis Framework

Daytona is an open-source framework for automated performance testing and analysis. It provides application-agnostic capabilities for integrated performance testing, repeatable test execution, standardized reporting, and built-in profiling. The framework consists of a centralized scheduler, distributed agents, a MySQL database for metadata, and a PHP-based UI. It aims to aggregate and present application, system, and hardware performance metrics in a unified interface, enabling continuous performance analysis. This post details the open-sourcing of Daytona and its architecture, including the scheduler, agents, database, and UI, and encourages community contributions.

1 post · 2017

Timeline

20172026