Implement Beam Search With Length Normalization
Problem Description
You are tasked with implementing a sequence decoder for a generative model. The decoder will operate based on a provided set of parameters and a function to predict the next token probabilities.
Given Inputs
- A
start token(<bos>) to initiate sequence generation. - An
end token(<eos>) to signify the completion of a sequence. - A
maximum output length(max_len) defining the longest permissible sequence. - A
beam size(k), which determines the number of candidate sequences to maintain at each step. - A function
next_log_probs(prefix)that, given a current sequenceprefix, returns the log-probability of every possible next token in the vocabulary.
Decoder Requirements
- Greedy Decoding: When
k = 1, the decoder must perform greedy decoding, selecting the single most probable token at each step. - Beam Search: When
k > 1, the decoder must perform beam search, maintaining the topkcandidate hypotheses (sequences) at each decoding step based on their scores. - Scoring: Sequences should be scored using their cumulative log-probability.
- Output: The decoder should return the single best completed sequence. If no candidate sequence reaches the
<eos>token beforemax_lenis reached, it should return the best partial sequence found.
Explanation Task
After implementing the decoder, explain why using raw cumulative log-probability tends to penalize longer sequences. Subsequently, describe a method to reduce this bias, such as using average log-probability or a more sophisticated length normalization technique.
AI interview prep
Core dilemma
Balancing the exploration breadth of beam search with the exploitation of high-probability paths, while managing computational complexity and correcting inherent scoring biases to find the globally optimal sequence.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Implement Beam Search With Length Normalization
Problem Description
You are tasked with implementing a sequence decoder for a generative model. The decoder will operate based on a provided set of parameters and a function to predict the next token probabilities.
Given Inputs
- A
start token(<bos>) to initiate sequence generation. - An
end token(<eos>) to signify the completion of a sequence. - A
maximum output length(max_len) defining the longest permissible sequence. - A
beam size(k), which determines the number of candidate sequences to maintain at each step. - A function
next_log_probs(prefix)that, given a current sequenceprefix, returns the log-probability of every possible next token in the vocabulary.
Decoder Requirements
- Greedy Decoding: When
k = 1, the decoder must perform greedy decoding, selecting the single most probable token at each step. - Beam Search: When
k > 1, the decoder must perform beam search, maintaining the topkcandidate hypotheses (sequences) at each decoding step based on their scores. - Scoring: Sequences should be scored using their cumulative log-probability.
- Output: The decoder should return the single best completed sequence. If no candidate sequence reaches the
<eos>token beforemax_lenis reached, it should return the best partial sequence found.
Explanation Task
After implementing the decoder, explain why using raw cumulative log-probability tends to penalize longer sequences. Subsequently, describe a method to reduce this bias, such as using average log-probability or a more sophisticated length normalization technique.