Design tests to verify shuffle randomness
Experiment Design: Is Your Array Shuffle Truly Random?
You are provided with a shuffle(arr) function that accepts an array of n distinct items and returns a random permutation of those items. Your task is to design a rigorous test plan to assess whether this shuffle function exhibits uniform randomness and independence across multiple executions.
Requirements:
- State Hypotheses: Clearly define the null and alternative hypotheses concerning both the uniformity of permutations and the independence of results across different runs.
- Specify Tests: Detail the specific statistical tests to be employed:
- Position Uniformity: Tests to verify that each item has an equal probability of appearing in any given position (e.g., using chi-square goodness-of-fit or independence tests on item-by-position counts).
- Pairwise Order/Adjacency: Tests designed to detect correlations between items, such as whether certain pairs of items tend to appear next to each other or in a specific relative order more often than expected.
- Additional Tests: Propose any other tests that could effectively catch common implementation bugs or subtle biases in shuffle algorithms.
- Choose Sample Size and Significance Level: Justify or derive the appropriate number of trials (
T), the significance level (alpha), the desired statistical power, and any necessary multiple-testing corrections. - Bias/Correlation Detection: Describe the methodology for localizing and diagnosing any detected nonuniformity or dependence within the shuffle function's output.
- Implementation and Automation (in JavaScript): Outline a strategy for implementing and automating this test plan. This should include details on data collection, statistical calculations, p-value interpretation, defining pass/fail logic, and integrating the tests into a Continuous Integration (CI) pipeline.
Assumptions:
- The
shufflefunction can be called repeatedly as a black box. - We have the flexibility to choose the array size
n(e.g., between 10 and 52 distinct items). - We can execute
Ttrials as needed.
AI interview prep
Core dilemma
Balancing the statistical rigor required to confidently assert randomness against the computational feasibility and time constraints of testing a probabilistic system, especially given the combinatorial explosion of permutations.
3 tradeoffs and 2 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design tests to verify shuffle randomness
Experiment Design: Is Your Array Shuffle Truly Random?
You are provided with a shuffle(arr) function that accepts an array of n distinct items and returns a random permutation of those items. Your task is to design a rigorous test plan to assess whether this shuffle function exhibits uniform randomness and independence across multiple executions.
Requirements:
- State Hypotheses: Clearly define the null and alternative hypotheses concerning both the uniformity of permutations and the independence of results across different runs.
- Specify Tests: Detail the specific statistical tests to be employed:
- Position Uniformity: Tests to verify that each item has an equal probability of appearing in any given position (e.g., using chi-square goodness-of-fit or independence tests on item-by-position counts).
- Pairwise Order/Adjacency: Tests designed to detect correlations between items, such as whether certain pairs of items tend to appear next to each other or in a specific relative order more often than expected.
- Additional Tests: Propose any other tests that could effectively catch common implementation bugs or subtle biases in shuffle algorithms.
- Choose Sample Size and Significance Level: Justify or derive the appropriate number of trials (
T), the significance level (alpha), the desired statistical power, and any necessary multiple-testing corrections. - Bias/Correlation Detection: Describe the methodology for localizing and diagnosing any detected nonuniformity or dependence within the shuffle function's output.
- Implementation and Automation (in JavaScript): Outline a strategy for implementing and automating this test plan. This should include details on data collection, statistical calculations, p-value interpretation, defining pass/fail logic, and integrating the tests into a Continuous Integration (CI) pipeline.
Assumptions:
- The
shufflefunction can be called repeatedly as a black box. - We have the flexibility to choose the array size
n(e.g., between 10 and 52 distinct items). - We can execute
Ttrials as needed.