Taming Outlier Tokens in Diffusion Transformers

This post identifies the presence and impact of outlier tokens in Diffusion Transformers (DiTs) for image generation. It proposes Dual-Stage Registers (DSR) as an intervention mechanism for both the encoder and denoiser components of DiTs. DSR addresses the issue of corrupted local patch semantics associated with high-norm tokens, leading to consistent improvements in generation quality and reduction of outlier artifacts on ImageNet and large-scale text-to-image generation tasks.