Reward Model

RLHF doesn't train a language model directly on human preferences. It trains a separate model, the reward model, to predict which of two responses a human would prefer, and then uses that prediction to guide the main model's training. The reward model is only as good as the preference data it learned from, which is usually the part teams underestimate going in.

‍

‍

‍