Debug a GPT training pipeline
Fix three bugs in a minimal GPT to meet a training-loss target
You are given a Colab notebook with a minimal GPT-style language model implemented in PyTorch (token embedding → transformer blocks → LM head), along with training and inference code. On a small toy dataset, training currently fails to reach the target loss.
Your task is to identify and fix the following three issues so that training loss drops below a specified threshold on a small dataset:
- Incorrect attention masking (causal mask and/or padding mask mishandled).
- A bug in the training loop (e.g., missing
optimizer.zero_grad(), not callingmodel.train(), misaligned input/target token shift, wrong device placement, or incorrect loss reduction). - Missing positional encoding integration.
For each fix, you must:
- Provide concrete code changes (edits or snippets).
- Provide unit tests that would have caught each bug.
- Include a brief rationale for each fix.
Assumptions
- PyTorch 2.x, CUDA if available.
- A tiny character-level dataset (or synthetic tokens) with a small vocab and fixed max sequence length (e.g., 64), with optional padding. Use
ignore_indexfor pad.
Success Criterion
- Training loss on the toy training set drops below 1.0 within a few epochs on CPU, or within 0.5 on GPU.
AI interview prep
Core dilemma
The central engineering conflict is the systematic debugging of a non-converging deep learning model, specifically a Transformer, where multiple subtle, interdependent bugs can mask each other, requiring a deep understanding of both the model's internal mechanics and the PyTorch training lifecycle.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Debug a GPT training pipeline
Fix three bugs in a minimal GPT to meet a training-loss target
You are given a Colab notebook with a minimal GPT-style language model implemented in PyTorch (token embedding → transformer blocks → LM head), along with training and inference code. On a small toy dataset, training currently fails to reach the target loss.
Your task is to identify and fix the following three issues so that training loss drops below a specified threshold on a small dataset:
- Incorrect attention masking (causal mask and/or padding mask mishandled).
- A bug in the training loop (e.g., missing
optimizer.zero_grad(), not callingmodel.train(), misaligned input/target token shift, wrong device placement, or incorrect loss reduction). - Missing positional encoding integration.
For each fix, you must:
- Provide concrete code changes (edits or snippets).
- Provide unit tests that would have caught each bug.
- Include a brief rationale for each fix.
Assumptions
- PyTorch 2.x, CUDA if available.
- A tiny character-level dataset (or synthetic tokens) with a small vocab and fixed max sequence length (e.g., 64), with optional padding. Use
ignore_indexfor pad.
Success Criterion
- Training loss on the toy training set drops below 1.0 within a few epochs on CPU, or within 0.5 on GPU.