Explain KL Divergence in Language Models

Give the definition of Kullback-Leibler divergence, discuss which direction it is computed in and what support conditions it requires, and explain the role it plays in training or post-training modern language models.

Constraints and Assumptions

• Work with probability distributions defined over the same set of events or tokens.
• KL divergence is asymmetric and does not satisfy the requirements of a metric.
• Separate a token-level estimate from the ideal objective computed over full sequences.

Questions to Clarify

• Which distribution serves as the reference, and which one is being optimized?
• Is the purpose regularization, distillation, variational inference, or monitoring?
• How do you estimate the expectation from samples?

Hint — Identify the expectation: State which distribution supplies the samples and which log-probability ratio is averaged.

What a Strong Response Should Cover

• The definition, nonnegativity, asymmetry, and the condition under which it equals zero.
• What happens when supports do not match, and how the choice of direction matters.
• Its role as a penalty from a policy to a reference distribution, or as a distillation objective, in language models.
• How it is estimated, the trade-offs involved in choosing its coefficient, and the limits of using it for monitoring.

Follow-Up Questions

1. What differs if you use reverse KL rather than forward KL?
2. Why might a sample-based KL estimate come out negative when the true KL is always nonnegative?

Overview: Go over KL divergence starting from its probability definition, then directionality, support mismatch, sample estimates, and regularization in LLMs.