Compare Batch Normalization and Layer Normalization

Compare batch normalization and layer normalization. Describe what each method normalizes, how training and inference differ, and how you would debug a normalization-related problem during training or serving.

Constraints & Assumptions

• Specify the activation shape and the axes over which normalization is applied; the technique names alone don’t define the layout of an implementation.
• Start from a minibatch of feature vectors as the default, then explain how convolutional or sequential inputs shift the normalization axes.
• Separate learnable scale and shift parameters from statistics computed from the data.

Clarifying Questions to Ask

• Does the model operate on feature vectors, convolutional activation maps, or variable-length sequences?
• What batch sizes are used during training and during serving?
• Are training/inference modes and saved normalization state kept consistent throughout?

Hint: Write down the axes Ask whether a single example’s normalized value depends on other examples in the current batch.

What a Strong Answer Covers

• The normalization equation and the axes for computing mean and variance for each method.
• How batch normalization uses batch statistics during training and running averages during inference.
• How layer normalization computes per-example statistics and handles small batch sizes.
• Learned affine parameters, numerical stability measures, and configuration-specific nuances.
• A plan for debugging when batch size changes or when training and serving produce different outputs.

Follow-up Questions

• Why does a change in batch size affect batch normalization more directly than layer normalization?
• If evaluation quality drops after switching a trained model to inference mode, what would you check?

Overview: Contrast batch and layer normalization across normalization axes, learned parameters, running statistics, sensitivity to batch size, and the transition from training to inference.

Read the full Snapchat Machine Learning Engineer interview experience this question came from.