Feature Toggles and A/B Testing Platform

4 posts · Jul 2018 – Apr 2024

What this trail covers

Grab's Experimentation Platform (ExP) has evolved to provide a unified, scalable, and cost-effective system for running controlled experiments across the organization. It supports various assignment strategies, prevents inter-experiment interference, and automates data pipelines for real-time analysis. The platform enables teams to test new features, analyze causal relationships, and make data-driven product decisions, significantly increasing the number of concurrent experiments and computed metrics. This post details the evolution of the batch data pipeline supporting ExP, focusing on data ingestion, transformation, and storage in the data lake to support analytical queries, time series analysis, and democratized data access.

All Grab trails

Apr 2024Grab Experiment Decision Engine - a Unified Toolkit for ExperimentationIntroduced the GrabX Decision Engine, an internal open-source Python package designed to enhance experiment design and analysis. Key contributions include a Trusted Advisor for sample size estimation and best practice guidelines for between-subject and within-subject experiments, and a comprehensive post-experiment analysis toolbox offering various statistical tests and recommended regression models. This package aims to standardize experimentation practices, improve efficiency, and ensure reliable conclusions from experiments conducted on Grab's platform.
Aug 2023Streamlining Grab's Segmentation Platform with faster creation and lower latencyThis post details the migration of Grab's Segmentation Platform from a row-based storage in ScyllaDB to using Roaring Bitmaps for segment storage. This change addresses bottlenecks in segment creation by reducing the number of writes and significantly improves read latency for membership checks. A new SDK was developed to handle the retrieval, decoding, caching, and watching of Roaring Bitmaps, providing an LRU eviction policy and automatic refresh capabilities. This has enabled new use cases for the Communications Platform and Experimentation Platform.
May 2022Automated Experiment Analysis - Making experimental analysis scalableThis post introduces the Automated Experiment Analysis system, which automates the statistical analysis of experiments run on the GrabX platform. It details the architecture involving metrics configuration, a data pipeline using Spark on Databricks, a 'Decision Engine' for statistical tests, and Power BI for results visualization. The system aims to improve efficiency, quality control, and scalability of experiment analysis by standardizing processes and democratizing knowledge.
Sep 2021Automating Multi-Armed Bandit testing during feature rolloutThis post introduces a novel Multi-Armed Bandit System for Recommendation that automates MAB testing simultaneously with new feature rollouts. It details a three-building-block architecture: a stream processing framework for event pre-processing, a feedback loop that calculates goal metrics (Clickthrough Rate, Conversion Rate) using a metrics server and Spark jobs, and a dynamic value receiver. The Multi-Armed Bandit Optimiser module comprises Reward Update, Batched Multi-Armed Bandit Agent (using Beta distribution and Thompson Sampling), Monte-Carlo Simulation (with Max and Mean strategies), and Adaptive Rollout. For GrabFood recommendations, it uses Effective Conversion Rate as the optimization objective and the mean strategy for stable distribution. The rollout policy uses eater ID for unique entity assignment, and a fallback logic to a 50-50% distribution is in place for job failures.
Jan 2019A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation PlatformThis post details the evolution of Grab's batch data pipeline supporting the Experimentation Platform (ExP). It describes the design requirements driven by various user access patterns (data analysts, data scientists, QA/support engineers, monitoring systems, expert systems) and their respective SLAs. The pipeline uses Apache Spark for compute, Apache Hive for metastore, and Apache Airflow for workflow orchestration, running on AWS EMR. Key technical contributions include partitioning data by event type and ingestion time to optimize query performance and enable efficient data retrieval, and leveraging Spark's partitioning and sorting features. The system writes data to Amazon S3 and metadata to a metastore, aiming for a one-hour lag for most applications and a few hours for data analysts.
Advertisement
Nov 2018Reliable and Scalable Feature Toggles and A/B Testing SDK at GrabThis post details the evolution of Grab's feature toggling and A/B testing platform. It introduces a new SDK designed for sub-microsecond decisions, eliminating network calls for feature toggle retrieval by fetching configurations from S3 and constructing in-memory models. The SDK supports dynamic configurations (rollouts) with JSON-based definitions and experiments with advanced features like time-sliced shuffling. It also introduces the concept of 'facets' for contextual decision-making and tracking, and a Track() method for event logging directly to the data lake.
Jul 2018Building Grab’s Experimentation PlatformThis post details the creation and evolution of Grab's Experimentation Platform (ExP). It outlines the challenges of manual experimentation, the goals for building a unified platform (including power analysis, automated data pipelines, and event triggers), and the architectural approach to address network effects and experiment interference. Key components like variables, metrics, rollouts, groups, and experiments are explained, along with supported assignment strategies (randomized sampling and time-slicing). A sample use case with GrabChat is provided, highlighting the platform's impact on product development and culture.