Explain DPO and construct its training data

You are tasked with fine-tuning a large language model (LLM) via Direct Preference Optimization (DPO). Respond to the following:

1. Conceptual: At a high level, what is Direct Preference Optimization (DPO), and how is it conceptually different from a typical RLHF pipeline built on PPO (Proximal Policy Optimization)? Cover these points:
   • The objective that DPO optimizes.
   • Why DPO can skip training a separate reward model.
   • Practical advantages and trade-offs relative to PPO-based RLHF.
2. Dataset construction: How would you build a training dataset appropriate for DPO when fine-tuning an LLM? Describe:
   • The structure of a single training example (which fields it includes).
   • How to gather or produce the preferred and dispreferred responses.
   • How to deal with noisy labels or tied responses.
   • Any preprocessing or filtering steps you would apply to raise data quality. Assume you begin with a base SFT (supervised fine-tuned) model, and you can collect either human preference data or model-generated preference data. Overview: This question tests understanding of Direct Preference Optimization (DPO) for fine-tuning large language models, checking conceptual differences from PPO-based RLHF and the ability to design pairwise preference training datasets.