Problem Statement
Design a distributed metrics counter service to collect and query internal product telemetry.
Core Functionality
The service must support two primary operations:
increment(metric_name, delta, tags, timestamp): Records an update to a counter metric.get_count(metric_name, tag_filters, start_time, end_time, granularity): Retrieves aggregated counts for a specified metric within a time range, filtered by tags, and aggregated to a given granularity.
Key Requirements and Constraints
- Ingestion Rate: Handle millions of counter updates per second from numerous services.
- Near Real-time Visibility: Recent data must be queryable within a few seconds of ingestion.
- Aggregation: Support data aggregation at second, minute, and hour granularities.
- High Availability: The system must remain available and operational during machine or zone failures.
- Durability: Store data durably to support historical queries over long periods.
- Workload Handling: Effectively manage hot metrics (metrics with disproportionately high update rates), retries (ensuring idempotency and correctness), and high-cardinality tags (tags with a very large number of unique values).
- Tradeoff Analysis: Be prepared to discuss the tradeoffs between exact and approximate counting methods.
Design Aspects to Cover
Candidates should describe the following architectural components and strategies:
- API definition
- Data model
- Write path
- Read path
- Sharding strategy
- Storage choices
- Rollup mechanisms
- Failure recovery
- Scaling strategy
AI interview prep
Core dilemma
Balancing extreme write throughput and low-latency reads for recent data with durable, cost-effective storage for historical queries, all while supporting flexible aggregations and high-cardinality tags.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Problem Statement
Design a distributed metrics counter service to collect and query internal product telemetry.
Core Functionality
The service must support two primary operations:
increment(metric_name, delta, tags, timestamp): Records an update to a counter metric.get_count(metric_name, tag_filters, start_time, end_time, granularity): Retrieves aggregated counts for a specified metric within a time range, filtered by tags, and aggregated to a given granularity.
Key Requirements and Constraints
- Ingestion Rate: Handle millions of counter updates per second from numerous services.
- Near Real-time Visibility: Recent data must be queryable within a few seconds of ingestion.
- Aggregation: Support data aggregation at second, minute, and hour granularities.
- High Availability: The system must remain available and operational during machine or zone failures.
- Durability: Store data durably to support historical queries over long periods.
- Workload Handling: Effectively manage hot metrics (metrics with disproportionately high update rates), retries (ensuring idempotency and correctness), and high-cardinality tags (tags with a very large number of unique values).
- Tradeoff Analysis: Be prepared to discuss the tradeoffs between exact and approximate counting methods.
Design Aspects to Cover
Candidates should describe the following architectural components and strategies:
- API definition
- Data model
- Write path
- Read path
- Sharding strategy
- Storage choices
- Rollup mechanisms
- Failure recovery
- Scaling strategy