Training a separate model to predict which of two outputs a human would prefer, so that the preference signal can be used to tune the main model at scale.

Described as teaching a model what good looks like, when what it learns is what raters approved of, which is what people pick when asked to choose quickly between two answers.

• Preference Is Not Correctness: Raters reward the answer that reads better, and the better-reading answer is sometimes the wrong one.
• The Proxy Becomes the Target: Optimise hard against a preference model and you get output shaped to please the grader.
• House-Cleaning Reality Check: Like learning to cook from which dishes got compliments, and slowly making everything sweeter.

Love Island (Season 6): islanders adjust what they say and do for what will play well with the public vote, and the whole villa slowly shapes itself around the thing being scored.

“Reward modeling aligns the system with human values at scale.”

“We had raters pick the better of two answers 30,000 times and trained a model to predict their picks.”