Debug a GRPO training loop and explain ratios
GRPO Training Loop Debugging and Ratio Explanation
You are given a simplified implementation of a GRPO (Group Relative Policy Optimization) training step for an RLHF-style policy model. The training is supposed to be strictly on-policy, meaning rollouts are generated by the same policy that is being updated.
Tasks:
- Walk through the end-to-end GRPO training flow: sampling prompts, generating rollouts, computing group-based advantages (relative within a group of completions), computing the policy gradient loss, and updating the policy.
- You find that training is unstable due to several straightforward implementation issues. Describe three common, easy-to-miss bugs in a GRPO/PPO-like training loop that would cause incorrect learning (e.g., wrong log-prob computation, incorrect masking, advantage normalization mistakes, mixing policies, etc.). For each, explain how you would detect it and how to fix it.
- During debugging you notice the importance-sampling ratio
ratio = exp(log_pi_theta(a|s) - log_pi_old(a|s))is not always 1, even though you expected the method to be strictly on-policy. Explain why the ratio might deviate from 1 in practice. List the most likely causes and what to check in the training pipeline to confirm each cause.
AI interview prep
Core dilemma
Ensuring theoretical 'strictly on-policy' correctness in a practical, often batched or distributed, reinforcement learning training loop, while identifying and debugging subtle implementation flaws that break this assumption or lead to numerical instability.
3 tradeoffs and 2 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Debug a GRPO training loop and explain ratios
GRPO Training Loop Debugging and Ratio Explanation
You are given a simplified implementation of a GRPO (Group Relative Policy Optimization) training step for an RLHF-style policy model. The training is supposed to be strictly on-policy, meaning rollouts are generated by the same policy that is being updated.
Tasks:
- Walk through the end-to-end GRPO training flow: sampling prompts, generating rollouts, computing group-based advantages (relative within a group of completions), computing the policy gradient loss, and updating the policy.
- You find that training is unstable due to several straightforward implementation issues. Describe three common, easy-to-miss bugs in a GRPO/PPO-like training loop that would cause incorrect learning (e.g., wrong log-prob computation, incorrect masking, advantage normalization mistakes, mixing policies, etc.). For each, explain how you would detect it and how to fix it.
- During debugging you notice the importance-sampling ratio
ratio = exp(log_pi_theta(a|s) - log_pi_old(a|s))is not always 1, even though you expected the method to be strictly on-policy. Explain why the ratio might deviate from 1 in practice. List the most likely causes and what to check in the training pipeline to confirm each cause.