Design a Weighted Log Distributor
Design a Weighted Log Distributor
Design a high-throughput log distributor system. This system receives log packets from agents running on various application and infrastructure hosts. Each packet may contain multiple log messages.
The distributor's primary function is to route these incoming log packets to a set of multiple analyzer services. Each analyzer service is configured with a relative weight (e.g., 0.4, 0.3, 0.1, 0.2), and it can be assumed that these weights sum to 1.0.
The core requirement is that, over time, each analyzer must process a fraction of the total log messages that is roughly proportional to its configured weight.
Your design discussion should cover:
- System Architecture
- Routing Algorithm
- Data Structures
- Scaling Strategy
- Handling of varying packet sizes
- Strategies for managing analyzer failures
- Mechanisms for backpressure handling
- Approach to updating analyzer weights dynamically
AI interview prep
Core dilemma
Balancing extreme log ingestion throughput and low latency with the statistical accuracy of weighted distribution across potentially unreliable downstream services, while maintaining system stability and allowing dynamic configuration updates.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a Weighted Log Distributor
Design a Weighted Log Distributor
Design a high-throughput log distributor system. This system receives log packets from agents running on various application and infrastructure hosts. Each packet may contain multiple log messages.
The distributor's primary function is to route these incoming log packets to a set of multiple analyzer services. Each analyzer service is configured with a relative weight (e.g., 0.4, 0.3, 0.1, 0.2), and it can be assumed that these weights sum to 1.0.
The core requirement is that, over time, each analyzer must process a fraction of the total log messages that is roughly proportional to its configured weight.
Your design discussion should cover:
- System Architecture
- Routing Algorithm
- Data Structures
- Scaling Strategy
- Handling of varying packet sizes
- Strategies for managing analyzer failures
- Mechanisms for backpressure handling
- Approach to updating analyzer weights dynamically