Language Models (Mostly) Know What They Know

This post introduces the study of language models' ability to evaluate the validity of their own claims and predict which questions they can answer correctly. It demonstrates that larger models are well-calibrated on multiple-choice and true/false questions in the right format. The paper proposes a method for self-evaluation on open-ended tasks by having models first propose answers and then evaluate the probability of their correctness ('P(True)'). Encouraging performance, calibration, and scaling for 'P(True)' are observed across diverse tasks. Self-evaluation performance improves when models consider multiple samples before predicting the validity of a specific possibility. The research also investigates whether models can predict 'P(IK)' (probability of 'I know') without reference to a proposed answer. Models perform well at predicting 'P(IK)' and generalize partially across tasks, though calibration on new tasks is a challenge. Predicted 'P(IK)' increases with relevant source materials and hints, suggesting a path towards training more honest models.