ConceptsINTERNALS.md

RLHF

Reinforcement learning from human feedback. The tuning step behind InstructGPT and many chat assistants.

  • People compare two replies, and a reward model learns to predict which one they pick.
  • The language model is then trained to earn a high reward score.
  • A penalty, β times KL, keeps it close to the model it started from.
  • Set β and see which replies the tuned model gives.

Try this: drag β down from 1 to 0.1 and watch the reward score climb.

Try it

This page needs JavaScript.

What just happened

  1. A starting model answers “How do I reverse a list in Python?” in one of five ways. The blue bars are how often it picks each one.
  2. A reward model gives each reply a score. Pushing the model toward a high score alone would move all its probability onto the top-scoring reply.
  3. The penalty stops that. With a fixed set of replies the best trade-off has a closed form: new probability = starting probability × escore/β, scaled so the total is 100%. A large β changes almost nothing, and a small β lets the top score win.
  4. At β = the average reward score goes from to . The worked example grows from to , and the model has moved nats (KL) from where it started.
Where the reward model comes from, and what the real algorithm does

Labellers see two replies and pick one. The reward model gives each reply a score r, and it is trained so that the chance it picks the chosen reply is the sigmoid of the score gap, the same Bradley-Terry form as on the pairwise votes and Elo page. A gap of 1 means . The loss is −log of that chance: when the model agrees with the label, when it is equally confident the other way.

The page shows the exact optimum of expected reward minus β·KL for five replies. A real run cannot list every possible reply. InstructGPT approximates the optimum by sampling replies and updating with PPO, a policy-gradient method, with a KL penalty against the starting model. DPO reaches the same optimum straight from the comparisons, with no reinforcement learning loop.

The reward scores and the quality scores here are made up to show the shape. They are not measurements.

The five replies:
1. “I’m not sure.”
2. “Use my_list[::-1].”
3. “Use my_list.reverse() to flip it in place, or reversed(my_list) for a reversed iterator. For example, [1, 2, 3] becomes [3, 2, 1].”
4. A correct answer padded with flattery and filler.
5. “Definitely use my_list.sort(reverse=True). It is the standard, fastest way.” This sorts from largest to smallest. It does not reverse the order.

Where you meet it

Some goals are easy to judge and hard to write as a rule: be helpful, be clear, do not make things up. People can compare two replies even when they cannot define quality. RLHF turns those comparisons into a number a reinforcement learning loop can chase.

Where it breaks

The reward model is a stand-in for what people want. It is only trained on a limited set of comparisons, so it can be fooled, and a model that chases its score will find the gaps. Here the reward model likes a confident, polished wrong answer. Drag β down and compare the two lines.

At β = the reward score is , up from , while the quality people would give has fallen from to , because the wrong reply is now of the answers. Quality peaks near β = and drops below its starting value near β = . Gao and others (2022) measured this pattern with a larger reward model standing in for humans: the score of the model being optimised kept rising while the stand-in’s score eventually fell. They called it over-optimisation.

Check yourself

You set β very high. What does the tuned model do?

It stays the starting model. e to the power score/β goes to 1 for every reply, so the new probabilities equal the old ones and KL is 0. The penalty wins over the reward.

Why have the KL penalty at all? Why not just maximise the reward model’s score?

The score is a proxy, learned from a limited set of comparisons, and it is wrong in places nobody checked. Without the penalty the model moves all its probability to whichever reply the proxy rates highest, wrong ones included. The penalty keeps it where the reward model is still trustworthy.

The reward model scores reply A at 2.0 and reply B at 0.5. How often does it say people prefer A?

The sigmoid of the gap, 1.5: 1 / (1 + e−1.5) = 0.82, so 82% of the time.

Used in

Go deeper