Studying Large Language Model Generalization with Influence Functions

This post introduces the application of Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) to scale influence functions to LLMs up to 52 billion parameters, achieving similar accuracy to traditional methods with significantly faster IHVP computation. It also details algorithmic techniques like TF-IDF filtering and query batching to reduce gradient computation costs. The post uses these scaled influence functions to investigate LLM generalization patterns, including sparsity, abstraction with scale, domain-specific abilities, cross-lingual generalization, and role-playing, while identifying a limitation related to phrase order sensitivity.