Problem Description
Design a near-real-time weather data platform. This service will ingest best-effort hourly weather data for approximately 2,000 distinct locations (identified by city IDs or latitude/longitude coordinates) from an external Weather Service API.
The definition of "current temperature" is the latest available hourly observation. A critical requirement is that the platform must ingest each new hourly data dump efficiently, ensuring that clients always receive data that is no more than 10 minutes stale relative to the current time.
Core Requirements
- Data Freshness: All data served to clients must be at most 10 minutes stale.
- Resilience: The system must gracefully handle slowness, partial failures, and complete outages of the upstream Weather Service API.
- Ingestion Robustness: Implement mechanisms for retries, idempotent data ingestion, and backfill capabilities for any missed hourly data points.
- API Etiquette: Strictly adhere to the external weather provider's API rate limits.
- Observability: Implement comprehensive monitoring and alerting to track and ensure compliance with data freshness Service Level Objectives (SLOs).
- Scalability: The read API component of the platform must support horizontal scaling to accommodate traffic spikes.
Deliverables
- High-Level Architecture: Outline the major components, including the ingestion pipeline, data storage, caching layer, client-facing API, and scheduling mechanisms.
- API Design: Specify the design of the public-facing API (e.g.,
GET /temperature?locationId=...), including its response schema, error handling, and metadata related to data freshness. - Data Model and Storage: Propose a data model and justify storage choices, detailing Time-To-Live (TTL) and indexing strategies.
- Caching Strategy: Develop a caching and invalidation plan specifically designed to meet the 10-minute data freshness requirement.
- Operational Considerations: Discuss deployment strategies, cost implications, and security aspects of the platform.
- Testing and Observability: Describe the overall testing approach and the observability strategy for the system.
AI interview prep
Core dilemma
Balancing the strict 10-minute data freshness SLO with the unreliability and rate limits of an external, best-effort API, while ensuring scalability and cost-efficiency for 2000 locations.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Problem Description
Design a near-real-time weather data platform. This service will ingest best-effort hourly weather data for approximately 2,000 distinct locations (identified by city IDs or latitude/longitude coordinates) from an external Weather Service API.
The definition of "current temperature" is the latest available hourly observation. A critical requirement is that the platform must ingest each new hourly data dump efficiently, ensuring that clients always receive data that is no more than 10 minutes stale relative to the current time.
Core Requirements
- Data Freshness: All data served to clients must be at most 10 minutes stale.
- Resilience: The system must gracefully handle slowness, partial failures, and complete outages of the upstream Weather Service API.
- Ingestion Robustness: Implement mechanisms for retries, idempotent data ingestion, and backfill capabilities for any missed hourly data points.
- API Etiquette: Strictly adhere to the external weather provider's API rate limits.
- Observability: Implement comprehensive monitoring and alerting to track and ensure compliance with data freshness Service Level Objectives (SLOs).
- Scalability: The read API component of the platform must support horizontal scaling to accommodate traffic spikes.
Deliverables
- High-Level Architecture: Outline the major components, including the ingestion pipeline, data storage, caching layer, client-facing API, and scheduling mechanisms.
- API Design: Specify the design of the public-facing API (e.g.,
GET /temperature?locationId=...), including its response schema, error handling, and metadata related to data freshness. - Data Model and Storage: Propose a data model and justify storage choices, detailing Time-To-Live (TTL) and indexing strategies.
- Caching Strategy: Develop a caching and invalidation plan specifically designed to meet the 10-minute data freshness requirement.
- Operational Considerations: Discuss deployment strategies, cost implications, and security aspects of the platform.
- Testing and Observability: Describe the overall testing approach and the observability strategy for the system.