Design a Centralized Logging System
Design a Centralized Logging System
Design a system that collects application logs from a large fleet of microservice instances and makes them durable and searchable for debugging and operational monitoring. Engineers should be able to find a service's logs by time range and search the log message, within seconds of the log being emitted.
The interviewer will deep-dive four areas: the ingestion pipeline, the storage and query design, and how the system stays reliable and scalable under load and failure.
Constraints & Assumptions
- ~10,000 service instances.
- Aggregate write volume on the order of ~1,000,000 log lines/sec at peak, average ~500 bytes/line, so roughly ~500 MB/s (~40 TB/day) of raw logs.
- Logs are mostly write-once, read-rarely; reads are bursty during incidents.
- Retention: a few days to weeks "hot" (fast search), older logs archived cheaply.
- Query patterns: filter by service/host/level, restrict to a time range, and full-text search on the message; target p99 search latency of a few seconds.
- Near-real-time: a log should be queryable within a few seconds of emission.
AI interview prep
Core dilemma
Balancing the high throughput, low-latency ingestion of massive log volumes against the need for durable, searchable, and cost-effective storage, while ensuring near real-time queryability and system resilience.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a Centralized Logging System
Design a Centralized Logging System
Design a system that collects application logs from a large fleet of microservice instances and makes them durable and searchable for debugging and operational monitoring. Engineers should be able to find a service's logs by time range and search the log message, within seconds of the log being emitted.
The interviewer will deep-dive four areas: the ingestion pipeline, the storage and query design, and how the system stays reliable and scalable under load and failure.
Constraints & Assumptions
- ~10,000 service instances.
- Aggregate write volume on the order of ~1,000,000 log lines/sec at peak, average ~500 bytes/line, so roughly ~500 MB/s (~40 TB/day) of raw logs.
- Logs are mostly write-once, read-rarely; reads are bursty during incidents.
- Retention: a few days to weeks "hot" (fast search), older logs archived cheaply.
- Query patterns: filter by service/host/level, restrict to a time range, and full-text search on the message; target p99 search latency of a few seconds.
- Near-real-time: a log should be queryable within a few seconds of emission.