Context

You are provided with a pre-trained Convolutional Neural Network (CNN) classifier for the FashionMNIST dataset. Each input image is grayscale, with dimensions 1x28x28, and has been normalized (e.g., to the range [0, 1] or using standard z-score). You also have access to an evaluation environment capable of running batched inference across the test set.

The core challenge involves a row-wise reveal protocol: at any step k (where 0 ≤ k ≤ 28), the top k rows of an image are visible, while the remaining rows (from k to 27) are replaced by a constant mask fill value m.

Your objective is to:

1. Analyze the sufficiency of early-revealed information for accurate classification.
2. Select a global mask value m that optimizes a defined reward function.
3. Propose a training augmentation policy to enhance this reward.
4. Design an early-exit policy for sequential pixel revelation at test time.

Tasks

1. Row-wise Reveal Evaluation

• For each k in the set {0, 1, …, 28}, construct masked images where rows [k, 27] are filled with a scalar m, and rows [0, k-1] retain their original pixel values.
• Record the model’s predicted class at each k and compute the classification accuracy as a function of k across the test set.
• Plot the accuracy versus k curve and interpret its implications regarding information sufficiency (how early rows contribute to classification) and the model's robustness to masking.

2. Reward-Optimal Global Mask Value m

• Define a reward R for partially revealed images: if the model's final prediction is correct when revealing stops at step k, R equals the number of pixels still masked; otherwise, R = 0.
• Using the accuracy-vs-k results, propose and implement a method to select a single global mask fill value m that maximizes the expected reward over the dataset. A suggested approach is to sweep candidate m values, estimate the expected reward for each (under a simple fixed-k stopping policy), and then select the best m.
• Discuss relevant trade-offs, including the impact of class imbalance and the distribution shift introduced by masking.

3. Training-Time Augmentation to Improve Expected Reward

• Propose a data augmentation strategy that involves masking contiguous rows or blocks of pixels during training. The goal is to train the model to maintain accuracy with limited visible pixels.
• Specify the policy details: the probability of applying the augmentation, the range of region sizes to be masked, and the fill value used for masking. Include constraints to prevent degenerate cases (e.g., masking almost all pixels).
• Describe how you would tune this augmentation policy's hyperparameters.
• If limited to only two retraining runs, state the exact two configurations you would try and specify which metrics you would compare (accuracy-vs-k and expected reward) to evaluate their effectiveness.

4. Early-Exit Policy for Sequential Pixel Reveal

• With the trained model fixed and pixels revealed sequentially at test time (one pixel at a time, up to 784 pixels), propose an early-exit policy. This policy should decide when to output a prediction to maximize the expected reward R.
• Provide a concrete strategy, such as requiring the argmax class prediction to be stable within a sliding window of the last W steps and/or exceeding a calibrated confidence threshold.
• Describe how to set the window size W and confidence thresholds through offline calibration. Also, explain how to handle scenarios involving ties or oscillations in predictions.