TechPrufer
Get started
Grab
Grab logoGrab · Feature trail

API Rate Limiting

8 posts · Sep 2016 – Sep 2025

What this trail covers

Grab has implemented a highly scalable API request rate limiting solution called Quotas to mitigate service abuse and cascading failures. The system uses an asynchronous processing pipeline with Kafka at its core, allowing client services to make immediate throttling decisions without significant latency impact. It supports global rate limiting by aggregating API usage stats and making decisions periodically, with client SDKs updating local in-memory caches. The system is designed for horizontal scalability. This post details the integration of Apache Pinot for powering Partner Gateway metrics, focusing on low-latency analytical queries for API traffic status codes and response times. Optimizations include partitioning by metric name, using columns based on aggregation intervals, and implementing a Star-tree index in Apache Pinot to handle large datasets and meet SLA requirements.

All Grab trails
  1. Powering Partner Gateway metrics with Apache PinotThis post details the integration of Apache Pinot with Grab's Partner Gateway to provide low-latency analytical metrics for API traffic. It describes the data ingestion process from API calls through Datadog, Kafka, and Flink to Pinot. Key challenges included performance issues with large datasets and long query times. Solutions implemented were partitioning the Kafka topic by metric name, using columns based on aggregation intervals (e.g., Timestamp_1h) for time-based aggregations, and utilizing the Star-tree index in Apache Pinot with specific configurations for dimensionsSplitOrder, functionColumnPairs, and maxLeafRecords to optimize query performance and meet SLA requirements.
  2. Sliding window rate limits in distributed systemsThis post details the implementation of frequency capping for marketing communications to prevent user overload and spam. It introduces Roaring Bitmaps for efficient storage of user segment membership data and a distributed rate limiter using Redis. The rate limiting logic employs a sliding log algorithm implemented via a Redis Lua script to track sent communications per user, ensuring daily and weekly limits are not exceeded. The Lua script utilizes sorted sets to store timestamps and `zremrangebyscore` for efficient eviction of old entries, achieving O(log n) time complexity.
  3. Zero trust with KafkaThis post details the implementation of zero-trust security for Grab's Kafka clusters. Key contributions include: 1. **Mutual TLS (mTLS) for Authentication and Encryption:** Utilizing Hashicorp Vault's PKI engine to dynamically generate short-lived client and server certificates for authentication and encrypting all in-transit traffic. 2. **Policy-Based Access Control (PBAC) with Open Policy Agent (OPA):** Implementing OPA as the policy engine for authorization, granting access based on the principle of least privilege. Policies are managed via a GitLab repository and CI pipeline. 3. **Strimzi Integration:** Leveraging Strimzi, the Kafka on Kubernetes operator, to integrate mTLS and OPA with Kafka brokers. 4. **Client SDK Enhancements:** Upgrading the Golang Kafka SDK to provide default mTLS configuration, retry mechanisms, and automatic certificate renewal. 5. **Scalable Authorization Management:** Enabling self-service authorization through Git repository permissions for topic owners and defining authorization rules using JSON data sources.
  4. How We Prevented App Performance Degradation from Sudden Ride Demand SpikesIntroduced the 'Spampede Filter' to mitigate the impact of localized demand spikes on the overall system. This filter uses Geohash for location bucketing and Unix timestamps for time bucketing to track unallocated requests. Redis is used to store the counts of unallocated requests per bucket with a Time-To-Live (TTL) for self-cleaning. Configuration for the filter (maximum unallocated bookings) is also stored in Redis. The filter is applied early in the booking processing pipeline to short-circuit requests when adverse conditions are met, reducing system load and preventing wider user impact.
  5. Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and FallbacksThis post introduces and details three key resiliency patterns: bulkheading, load balancing, and fallbacks. Bulkheading is explained as a method of isolating failures at various levels (infrastructure, OS, process, connection/thread pools) to prevent cascading failures. The Hystrix library for Go is mentioned as supporting bulkheading via `MaxConcurrentRequests`. Load balancing strategies (proxy, client-side, lookaside, service mesh) are discussed, with Grab's internal client-side solution CSDP using etcd highlighted. Fallback strategies (fail silently, local computation, cached values, backup service) are presented, with Grab-Kit's support for cached fallbacks and a backup service for core functionality noted.
  7. Designing Resilient Systems Beyond Retries (Part 1): Rate-LimitingThis post introduces rate-limiting as a strategy for system resiliency, positioning it as a technique that goes beyond retries and circuit breakers. It details the challenges associated with retries (retry storms) and circuit breakers (client configuration dependency). The post then explores various rate-limiting strategies, including per-client, per-endpoint, and multi-level thresholds, and discusses the distinction between local and global rate-limiting. It emphasizes the importance of server-side rate-limiting and the need for a rate-limiting service that avoids becoming a single point of failure, mentioning Grab's Quotas service as an example.
  8. Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)This post delves into the technical implementation and comparison of retry mechanisms and circuit breakers for building resilient systems. It provides a deep dive into retry strategies, including their use cases, the importance of idempotency, and the concepts of backoff and jitter for managing load and recovery. It also outlines key retry settings such as maximum retries, retry filter, base and max delay, and emphasizes the need for time-boxing requests by tuning retry parameters with request timeouts and maximum delays. The post concludes by comparing the effectiveness of using retries alone versus a combined approach with circuit breakers.
  9. Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)This post introduces and explains the concept of circuit breakers as a service reliability mechanism. It details their analogy to electrical circuit breakers, how they work by blocking requests to prevent overwhelming upstream services, and their benefits such as enabling fallback strategies (e.g., using cached or estimated data) and failing fast to conserve resources. The post also discusses error tracking (focusing on network/infrastructure errors over user errors), circuit recovery via a 'sleep window', and the 'bulwark' feature for limiting concurrent requests. It outlines key Hystrix-Go circuit breaker settings: Timeout, Max Concurrent Requests, Request Volume Threshold, Sleep Window, and Error Percent Threshold.
  10. How We Designed the Quotas Microservice to Prevent Resource AbuseThis post details the design and implementation of the Quotas microservice for API rate limiting. It addresses the challenges of global rate limiting in a distributed microservice environment by using an asynchronous processing pipeline with Kafka. Client services make immediate throttling decisions based on local in-memory caches updated by Kafka streams, while the Quotas service aggregates usage stats and sends decisions asynchronously. The system employs a sliding window algorithm and optimizes Redis usage by aggregating stats in-memory before periodic writes.
  11. Round-robin in Distributed SystemsThis post details the implementation of round-robin logic for client-side load balancing between Grab's Common Data Service (CDS) clients and servers. It addresses issues encountered with AWS Elastic Load Balancers (ELB) and explores DNS discovery as an alternative. Two methods for implementing round-robin are presented: a mutex-based approach for simple cases and a channel-based approach for more complex scenarios requiring granular control over operation timeouts and centralized management. A bug in an external srv-lb package was also identified and fixed.