ConceptsINTERNALS.md

LoRA

Low-rank adaptation. A way to fine-tune a large model by training a tiny add-on instead of the whole thing.

  • Fine-tuning changes every weight, so you train and store as many numbers as the model has.
  • LoRA freezes the weights and learns the change as the product of two thin matrices.
  • The rank r is how thin: a sum of r simple patterns, each one a column times a row.
  • If the change really is simple, a small r captures almost all of it.

Try this: drag r from 1 to 3 and watch “Missed” fade.

Try it

This page needs JavaScript.

What just happened

  1. The left grid is the update that full fine-tuning would make to one weight matrix, here 8 by 8. It has 64 numbers.
  2. LoRA writes the update as B times A, where B is 8 by r and A is r by 8. Rank means 2 × 8 × numbers to learn, not 64.
  3. The middle grid is the best that rank can do, and “Missed” is what is left over. For this update, rank 1 keeps of its strength, rank keeps and rank 3 keeps . Strength here is the sum of squares of the numbers.
  4. The saving is large because it grows with the matrix. For a real by layer, full fine-tuning trains numbers. Rank trains , which is of that.
The formula, the start, and what LoRA does not do

The layer computes x · (W + (α/r) · B A). W is frozen. B starts at zero and A starts random, so the update is zero at the start and training begins from the pretrained model. The α/r factor is a scale so that changing r does not change the size of the update much. After training, B A is added into W, so the model runs at the same speed as before, with no extra layers.

The grids use a fixed update built so its best low-rank approximation is known exactly. The best rank-r approximation keeps the r largest singular values (the Eckart–Young theorem). LoRA never computes that. It learns B and A by gradient descent, and the best rank is the most it could reach.

For a real layer the update is not a fixed grid. Whether it is low rank is an empirical question that depends on the task. The next section tries a spread-out update.

Where you meet it

Fine-tuning a large model normally needs memory for a gradient and optimiser state for every weight. LoRA needs them only for B and A, and a LoRA add-on is small enough to store and swap per task.

Where it breaks

LoRA only helps if the change you need is close to low rank. Here is the same slider on a different update, one whose strength is spread over many patterns. Switch the kind of update.

At rank the spread-out update keeps only of its strength, against for the low-rank one, and it needs about rank 6 to reach . The grids are made up to show the idea. Real updates sit somewhere between these two. Biderman and others (2024) compared LoRA with full fine-tuning on code and maths and found LoRA learned less on those harder targets, and also forgot less of the original model.

Check yourself

A weight matrix is 1,000 by 1,000. How many numbers does LoRA train at rank 4?

4 × (1,000 + 1,000) = 8,000, against 1,000,000 for the full matrix: 0.8%.

Why start B at zero?

Then B A is zero, so the model begins exactly as the pretrained model. Training moves it away only as far as the data pushes it. If both started random, the first step would already be a random change to every layer.

Doubling r doubles the trainable numbers. Does it always double what you can express?

No. In the low-rank example rank 1 already keeps 75% and rank 2 keeps 94%. Later ranks add very little because the update has little left in those patterns. In the spread-out example each rank adds a similar share.

Used in

Go deeper