Density and Score Estimation with Transformers

Hugging Face now supports DiScoFormer, a novel transformer-based model capable of estimating both the density and score of a distribution in a single forward pass. This approach generalizes kernel density estimation (KDE) and offers significant improvements in accuracy for high-dimensional data, removing the need for retraining per distribution. DiScoFormer leverages cross-attention and a shared backbone with two output heads for density and score, utilizing a label-free consistency loss for out-of-distribution adaptation. It analytically shows that attention generalizes KDE and includes it as a special case, improving on it by learning multiple scales and adapting them to data. Trained on Gaussian Mixture Models for universal density approximation and closed-form targets, DiScoFormer outperforms KDE significantly in high dimensions and on non-Gaussian shapes, with KDE retaining an advantage in speed for small datasets. This reusable, plug-in estimator has broad applications in generative modeling, Bayesian inference, and scientific computing.