TechPrufer
Get started
Hugging Face
Hugging Face logoHugging Face · Feature trail

Nyströmformer for Efficient Self-Attention

1 post · Mar 2022

What this trail covers

Introduced Nyströmformer, a method to approximate self-attention in linear time and memory using the Nyström method, enabling more efficient processing of long sequences in transformer models.

All Hugging Face trails
  1. Nyströmformer: Approximating self-attention in linear time and memory via the Nyström methodThis post details the Nyströmformer model, which approximates self-attention in linear time and memory. It leverages the Nyström method to reduce the computational complexity of self-attention mechanisms in transformer models, making them more efficient for processing long sequences. The model is available as `uw-madison/nystromformer-1024` for fill-mask tasks.