Problem: Designing a Fast Search System for Autonomous Vehicle Raw Logs
Autonomous vehicles continuously generate vast volumes of raw logs, encompassing sensor streams, perception outputs, and various planning and debug signals. Within these logs, engineers need to quickly identify specific situations, such as a pedestrian crossing or the appearance of a particular traffic sign (e.g., stop sign, speed limit sign).
Requirements
- Query Capabilities: Users must be able to query for predefined scenarios like "pedestrian present" or "stop sign detected."
- Filtering: Queries should optionally support filters such as time range, vehicle/software version, geographical location, specific route, and weather conditions.
- Result Granularity: Search results must precisely point to the exact time ranges (segments) within the raw logs where these situations might have occurred.
- Scale: The system must be designed to handle large-scale data, originating from numerous vehicles with continuous log uploads.
Design Considerations
Your design should explicitly address the following:
- Signal Reliance: Explain what types of signals you would rely on for detection and indexing (e.g., raw sensor data vs. derived perception outputs).
- Indexing and Search Strategy: Detail how the log data will be indexed and searched to meet the "quickly find" requirement.
- Scalability: Describe how the entire system will scale to accommodate increasing data volumes and query loads.
AI interview prep
Core dilemma
Balancing the continuous, high-volume ingestion of complex, semi-structured AV log data with the need for low-latency, flexible, and precise queries over this massive dataset, all while managing storage costs and operational complexity.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Problem: Designing a Fast Search System for Autonomous Vehicle Raw Logs
Autonomous vehicles continuously generate vast volumes of raw logs, encompassing sensor streams, perception outputs, and various planning and debug signals. Within these logs, engineers need to quickly identify specific situations, such as a pedestrian crossing or the appearance of a particular traffic sign (e.g., stop sign, speed limit sign).
Requirements
- Query Capabilities: Users must be able to query for predefined scenarios like "pedestrian present" or "stop sign detected."
- Filtering: Queries should optionally support filters such as time range, vehicle/software version, geographical location, specific route, and weather conditions.
- Result Granularity: Search results must precisely point to the exact time ranges (segments) within the raw logs where these situations might have occurred.
- Scale: The system must be designed to handle large-scale data, originating from numerous vehicles with continuous log uploads.
Design Considerations
Your design should explicitly address the following:
- Signal Reliance: Explain what types of signals you would rely on for detection and indexing (e.g., raw sensor data vs. derived perception outputs).
- Indexing and Search Strategy: Detail how the log data will be indexed and searched to meet the "quickly find" requirement.
- Scalability: Describe how the entire system will scale to accommodate increasing data volumes and query loads.