Design navigation-safety simulation parameters and experiments
Design Navigation-Safety Simulation Parameters and Experiments
You are tasked with designing a simulation-based evaluation framework for the navigation safety of an autonomous agent. Your design should be specific and cover the following aspects:
-
Scene Parameters: Enumerate and justify the scene parameters that need to be varied to ensure safety relevance. This includes, but is not limited to:
- Map topology and lane geometry
- Traffic density and composition
- Other-agent behavior models and aggressiveness
- Initial conditions and spawn rules
- Pedestrian and cyclist flows
- Occlusions
- Weather and lighting conditions
- Road friction and grade
- Sensor noise, latency, and dropouts
- Perception errors
- Rare hazards (e.g., cut-ins, jaywalks, doorings)
-
Parameter Distributions and Coverage: Define the distributions or generators for these parameters. Explain how you will cover the parameter space, considering approaches like domain randomization, curated scenario libraries, or adversarial generation. Also, describe how you will ensure the repeatability of simulations.
-
Safety Metrics: Specify the safety metrics to be used and outline their computation. Examples include collision rate per mile, near-miss counts (e.g., Time-to-Collision (TTC) less than a threshold (\tau)), intervention rate, constraint violations (e.g., using RSS or DRAC frameworks), comfort, and methods for multi-objective aggregation.
-
Rare-Event Risk Estimation Experiment Plan: Propose an experiment plan to estimate very rare-event risks without bias. This should include outlining techniques like importance sampling or CEM-based scenario weighting, writing the unbiased risk estimator with weights, and discussing strategies for variance control.
-
Sample Size Calculation and Batching: Given a baseline collision probability of approximately (10^{-4}) per mile and a target of 95% confidence with (\pm 20%) relative error, propose a sample-size calculation. Also, suggest any batching strategy that can be employed within a 10 compute-hour budget.
-
Sim-to-Real Calibration and Validation: Describe how you would calibrate and validate the simulation against real-world data. This includes parameter fitting from logs, coverage tests, sensitivity/ablation studies, and goodness-of-fit assessments. Additionally, explain how you would detect out-of-distribution (OOD) scenarios during evaluation.
Similar LeetCode problem: Design a Leaderboard
AI interview prep
Core dilemma
Balancing the breadth of simulation scenarios (to cover rare but critical safety events) against the depth of analysis required for statistical significance and computational feasibility, especially for rare event estimation.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design navigation-safety simulation parameters and experiments
Design Navigation-Safety Simulation Parameters and Experiments
You are tasked with designing a simulation-based evaluation framework for the navigation safety of an autonomous agent. Your design should be specific and cover the following aspects:
-
Scene Parameters: Enumerate and justify the scene parameters that need to be varied to ensure safety relevance. This includes, but is not limited to:
- Map topology and lane geometry
- Traffic density and composition
- Other-agent behavior models and aggressiveness
- Initial conditions and spawn rules
- Pedestrian and cyclist flows
- Occlusions
- Weather and lighting conditions
- Road friction and grade
- Sensor noise, latency, and dropouts
- Perception errors
- Rare hazards (e.g., cut-ins, jaywalks, doorings)
-
Parameter Distributions and Coverage: Define the distributions or generators for these parameters. Explain how you will cover the parameter space, considering approaches like domain randomization, curated scenario libraries, or adversarial generation. Also, describe how you will ensure the repeatability of simulations.
-
Safety Metrics: Specify the safety metrics to be used and outline their computation. Examples include collision rate per mile, near-miss counts (e.g., Time-to-Collision (TTC) less than a threshold (\tau)), intervention rate, constraint violations (e.g., using RSS or DRAC frameworks), comfort, and methods for multi-objective aggregation.
-
Rare-Event Risk Estimation Experiment Plan: Propose an experiment plan to estimate very rare-event risks without bias. This should include outlining techniques like importance sampling or CEM-based scenario weighting, writing the unbiased risk estimator with weights, and discussing strategies for variance control.
-
Sample Size Calculation and Batching: Given a baseline collision probability of approximately (10^{-4}) per mile and a target of 95% confidence with (\pm 20%) relative error, propose a sample-size calculation. Also, suggest any batching strategy that can be employed within a 10 compute-hour budget.
-
Sim-to-Real Calibration and Validation: Describe how you would calibrate and validate the simulation against real-world data. This includes parameter fitting from logs, coverage tests, sensitivity/ablation studies, and goodness-of-fit assessments. Additionally, explain how you would detect out-of-distribution (OOD) scenarios during evaluation.
Similar LeetCode problem: Design a Leaderboard