Autoregressive Transformer: Correct Attention Masking with Padding

Context

You are tasked with implementing the decoder self-attention mechanism for an autoregressive Transformer model. In this setup, input sequences within a batch are right-padded to a common maximum length.

Task

Your implementation must ensure two critical masking behaviors:

1. Causal Mask: Prevent the attention mechanism from attending to future tokens in the sequence.
2. Padding Mask: Exclude padded positions from being attended to.

Specifically, you need to:

1. Implement a causal mask that enforces the autoregressive property.
2. Implement a padding mask that correctly identifies and ignores padded positions.
3. Demonstrate how to combine these two masks and apply them within a multi-head attention layer.
4. Explain common bugs associated with attention masking, describe how these bugs typically manifest in model performance (e.g., loss or accuracy), and outline effective strategies for testing and debugging them.

Assumptions

• Tensors are batch-first, with a shape of (B, T, D), where B is batch size, T is sequence length, and D is model dimension.
• Padding is applied to the right of sequences.
• An attention_mask of shape (B, T) is provided, where 1 indicates a real token and 0 indicates a padded position. Alternatively, you may assume access to pad_token_id and input_ids.
• The PyTorch deep learning framework is available for implementation.