Machine Learning Interpretability Methods

Two Sigma is developing capabilities to enhance the interpretability of machine learning models. This involves exploring and implementing various techniques to understand how models arrive at their predictions, moving beyond 'black box' behavior. The goal is to enable better debugging, bias detection, knowledge transfer, and assessment of trade-offs in ML systems. Methods discussed include Partial Dependence Plots (PDP), Individual Conditional Expectation (ICE), Permuted Feature Importance, Global Surrogates, and Local Surrogates (LIME). This post provides a brief survey of these methods, detailing their advantages and disadvantages.