Bi-encoder vs cross-encoder
Two ways to use a neural model to match a query to documents. Bi-encoders find candidates fast, and cross-encoders re-rank a short list.
- A bi-encoder reads the query and each document separately.
- So it can read every document once, ahead of time, and save the result.
- A cross-encoder reads the query and one document together.
- So it must run once for every document, on every search.
Try this: drag N up to a million, then press Run a search a few times.
Try it
What just happened
No model ran here. The page only counts how often each kind of model would have to run. That count is the whole difference.
Where you meet it
A search app has a million documents. It cannot read them all with a slow model on every search. So it uses a fast first step to find about a hundred candidates, then a slow, careful model to put those hundred in order. This is called retrieve and rerank.
- The first step is often a bi-encoder, or BM25, or both. The Sentence Transformers documentation describes this setup, with a bi-encoder retrieving and a cross-encoder reranking.
- The same documentation says cross-encoders achieve higher performance than bi-encoders, but do not scale well to large collections.
- The Sentence-BERT paper gives a real anchor: finding the most similar pair among 10,000 sentences takes about 50 million BERT computations, or about 65 hours, when every pair is read together.
Where it breaks
The cross-encoder cannot search a big collection. Its cost grows with every document, and nothing can be saved. The bi-encoder is quick, but its document vector is made before the query exists. The text “The hotel has no pool” becomes one vector that has to serve a search about pools, a search about prices and a search about noise. Pick how many top results the cross-encoder rereads.
This page counts work. It does not say which model is more accurate on your data. For that you need graded searches and a measure like nDCG.
Check yourself
A new document is added to the collection. What must each system do before the next search?
The bi-encoder must read the new document once and save its vector. The cross-encoder needs nothing in advance, because it never saved anything. It just reads the new document along with the query when the search runs.
You will run one search over 1,000 documents, once. Which system does fewer model passes?
The cross-encoder: 1,000 passes against 1,001 for the bi-encoder, which spends 1,000 saving the vectors and 1 on the query. Saving only pays back from the second search. That is the “Start over” then “Run a search” experiment above.
Why can't a cross-encoder save a vector for each document?
From the first layer on, each document word looks at the query words. Nothing the model computes for the document is independent of the query, so there is nothing to compute early and reuse.
Used in
- Loading…
Go deeper
- Reimers and Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks” (2019). Shows why reading every pair together is too slow for search, and how separate vectors fix it.
- Nogueira and Cho, “Passage Re-ranking with BERT” (2019). A cross-encoder used to rerank candidates from a first step.
- Karpukhin and others, “Dense Passage Retrieval for Open-Domain Question Answering” (2020). Two separate encoders, with document vectors made ahead of time.