ConceptsINTERNALS.md

Reinforcement learning

Learning by trial and error from a reward signal, with no labelled answers. Used for game players, robots, and the tuning step of chat models.

  • An agent sees a state, picks an action, and gets a reward.
  • Here the only rewards are +1 at the goal and −1 in the pit.
  • It keeps a table of how much reward it expects from each move, and nudges the table after every step.
  • Credit for the goal flows backwards, one cell per update, to the moves that led there.

Try this: press Run 50 episodes, then drag γ down and run again.

Try it

This page needs JavaScript.

What just happened

  1. Every cell starts at 0, so the first episodes are random walks. Nothing is learned until the agent stumbles onto the goal or the pit, because those are the only places with a reward.
  2. After each move the agent nudges its estimate for that move toward the reward it just got, plus γ times the best value of the cell it landed on. The nudge size is α = . This is Q-learning.
  3. The goal cell is worth +1. The cell next to it learns that, then the cell next to that learns it from the neighbour, each time discounted by γ = . The start is moves from the goal, so its best possible value is γ to the power 5, .
  4. With the settings above, the first 10 episodes reach the goal times and episodes 41 to 50 reach it times, in about steps each. The start cell’s learned value after episodes is , against the best possible .
  5. ε = means it ignores its table and moves at random of the time. That is how it keeps finding out whether a different route is better.
The update rule

Q(s, a) ← Q(s, a) + α · (r + γ · max Q(s′, a′) − Q(s, a)). Q(s, a) is the expected total discounted reward for taking action a in state s and playing well afterwards. The bracket is the gap between what it expected and what it saw one step later. When the episode ends at s′ the max term is 0. This is the one-step rule of Watkins (1989), as given in chapter 6 of Sutton and Barto.

The grid is deterministic and tiny, so a table is enough. Real problems have too many states for a table, and the table is replaced by a neural network, which makes learning far less stable.

Where you meet it

Reinforcement learning fits problems where you can score the outcome of a whole sequence of decisions but cannot write down the right decision at each step.

Where it breaks

The agent maximises the reward you wrote, not the one you meant. Here a coin sits in the bottom row and pays a small bonus every time the agent steps onto it. Nobody meant the coin to matter more than the goal. Drag the bonus up.

At γ = the agent switches from finishing the task to farming the coin at a bonus of about . This is called reward hacking, or specification gaming. This page solves the grid exactly, so the switch is exact and not a lucky training run.

Check yourself

Why does the value spread outward from the goal one cell at a time, instead of reaching the start at once?

Each update looks only one step ahead. A cell learns about the goal only after its neighbour has already learned it. The start cell, six moves away, needs the chain to build up first.

You drag γ down to 0.5. What happens to the value of the start cell, and why does that matter?

It falls to 0.5⁵, about 0.03, because the +1 is halved for every step on the way. The agent still prefers the goal, but far from it the signal is tiny and learning is slow. A small γ makes an agent short-sighted.

Why can a +0.2 coin beat a +1 goal?

The goal pays once and the episode ends. The coin pays again every second step: 0.2 + 0.2·γ² + 0.2·γ⁴ and so on, which is 0.2 divided by (1 − 0.81), about 1.05. That is more than 1.

Used in

Go deeper