Explain Neural-Network Regularization and Dropout

Describe the way regularization mitigates overfitting in a neural network. Contrast L1 and L2 penalties against dropout, covering how dropout operates in training versus inference.

Constraints & Assumptions

• Describe the underlying mechanism instead of merely listing techniques.
• Use standard inverted dropout for illustration and specify the drop probability convention.
• Differentiate between a penalty appended to the objective function and modifications to the architecture, the data, or the optimization procedure.

Clarifying Questions to Ask

• Is the model exhibiting overfitting, underfitting, or rather data leakage or a distribution mismatch?
• Which layers employ dropout, and is the model currently in training mode or inference mode?
• Does the optimizer apply an L2 gradient penalty or employ decoupled weight decay?

Hint: Maintain the Expected Activation When a training activation is randomly zeroed out, figure out how to scale the surviving activations so that the conditional expectation of the activation stays unchanged.

What a Strong Answer Covers

• L1 and L2 objective penalties, along with their differing impacts on parameter values.
• Bernoulli masking and the scaling used in inverted dropout.
• The distinction between training and inference modes and why dropout does not equate to permanently removing neurons.
• Data augmentation and early stopping as separate regularization approaches.
• Diagnosing and validating the model rather than blindly assuming that more regularization invariably yields improvement.

Follow-up Questions

• Why can excessive dropout degrade both training and validation performance?
• Why are an L2 penalty and decoupled weight decay not generally equivalent when using adaptive optimizers?

Overview: Describe L1 and L2, dropout scaling, training vs inference behavior, early stopping, and the difference between gradient penalties and decoupled weight decay.

Read the full Snapchat Machine Learning Engineer interview experience where this question appeared.