Forecasting rare language model behaviors

This post introduces a novel method for forecasting rare and potentially catastrophic AI behaviors by applying power law distributions to observed risk probabilities. By analyzing the relationship between the number of queries tested and the highest log risk probabilities, Anthropic can extrapolate to predict worst-case risks at much larger scales than previously feasible. This method has demonstrated significant accuracy in predicting dangerous information disclosure, misaligned AI actions (power-seeking, self-preservation, self-exfiltration), and optimizing compute budgets for automated red-teaming, offering a more efficient way to identify and mitigate rare risks before deployment.