Softmax Linear Units

Introduced Softmax Linear Units (SoLU) as an activation function replacement for standard MLP neurons. Demonstrated that SoLU substantially increases the fraction of interpretable neurons (responding to human-understandable concepts) with minimal impact on ML performance. Used SoLU models to gain new insights into information processing within transformers. Identified potential trade-offs where SoLU might increase interpretability of some features by hiding others, suggesting a potential 'no free lunch' scenario.