Cross-encoder vs bi-encoder: the two retrieval architectures that divide search work

A bi-encoder turns a query and each document into separate vectors. Comparing them is cheap, so it owns first-stage search over large corpora. A cross-encoder reads the query and one document together and scores the pair in one pass. It is more precise and far slower. Real search systems use both, in turn.

Bi-encoder, the glossary entry

A bi-encoder is a model with two twin halves that run one at a time. One half turns the query into a vector. The other half turns a document into a vector of the same shape. The system then compares the two vectors, a step most systems run with cosine similarity. The model never sees the query and the document together. That one restraint is the whole design.

The method rewards prep work. You encode the whole corpus once, ahead of time. The vectors then sit in a vector index. At query time you encode only the query. The index then hands back the nearest vectors. The Sentence-BERT paper (Reimers and Gurevych, 2019) introduced this pattern and sized the wall it removes. Scoring every pair among 10,000 sentences costs about 50 million joint passes, about 65 hours. Twin embeddings answer the same job in about 5 seconds. The sentence-transformers docs walk the same retrieve and re-rank setup with runnable code.

Where it wins: first-stage search over thousands to millions of documents. It also wins any job where the texts can be embedded ahead of time. Clustering, dedup, and similarity search at scale live here. Where it loses: fine grain. Each text collapses into one pooled vector, so word level signals are lost. Negations and small condition words tend to score the same as their opposites. The final order of a shortlist is where a bi-encoder is weakest.

Cross-encoder, the glossary entry

A cross-encoder is a model that reads the query and one document as one sequence. It outputs one relevance score. It makes no embeddings at all. The first demonstration is passage reranking with BERT (Nogueira and Cho, 2019). Each query and passage pair went through the model as one input. The score then ranked the passages. That system topped the MS MARCO passage leaderboard of its day. It scored 27 percent better in MRR@10 than the prior best.

The engine is joint attention. The two texts are joined with a separator token. Self attention then runs across both texts in every layer. Each token can see every token on the other side. A small scoring head maps the pooled output to one score. On the served row here, bge-reranker-v2-m3, raw logits pass through a sigmoid, so scores land between 0 and 1.

Where it wins: final order. On a shortlist of tens to a couple hundred texts, a cross-encoder reads each pair in full. It splits texts that a bi-encoder scores almost the same. RAG context choice and legal or medical search belong to it. So does any task where the top five results carry the value. Where it loses: corpus scale. Every text must pass through the full model with the query, at request time, every query. None of it can be done ahead. A cross-encoder fetches nothing by itself. It only scores what a retriever hands it.

Head to head: the four dimensions that decide a stage

The two rows complete each other; they do not compete. The table is the whole design rule for a two stage pipeline. Size the first stage wide enough that the right texts survive it. Size the second stage small enough that pair scoring stays cheap.

Late interaction, the third architecture (ColBERT)

Late interaction sits between the two head terms. The ColBERT paper (SIGIR 2020) names the pattern. Encode the query and the document apart, as a bi-encoder does. But keep one vector per token, not one pooled vector per text. Then run one cheap step after encoding. Every query token takes its best match against all document tokens. The paper calls this step MaxSim. The per token best matches sum to the score.

Because encoding stays apart, document token vectors are still computed offline and stored. Vector indexes still serve them. Because any interaction happens at all, word level matching survives. The paper reports near cross-encoder quality at two orders of magnitude lower cost per query. ColBERTv2 (NAACL 2022) cut the storage bill by another 6 to 10 times with residual compression.

Where it wins: exact term and phrase matching inside semantic search. It also wins fields where one word decides. Where it loses: storage and plumbing. A late interaction index holds many vectors per text, not one. Fewer hosted platforms carry it. As of 2026-10-28, no hosted late interaction rerank row appears on the shelves we checked. The open bge-m3 model behind this platform's embeddings lane ships all three modes in its weights. They are dense, sparse, and multi-vector. The last is ColBERT style.

Worked example: one query, two stages, real numbers

Take a corpus of 500,000 document chunks, a mid size product knowledge base. A user asks: how do the two encoder designs differ. Here is the full path.

Stage one, bi-encoder search. You embed all 500,000 chunks once, ahead of time, with the embeddings lane row bge-m3. The query arrives. You embed the query, one pass. The vector index returns the 100 nearest chunks. The cross-encoder path over the whole corpus would mean 500,000 passes for this one query, one per chunk. Every query would pay the same again. The Sentence-BERT paper frames the same wall with small numbers. Even 10,000 sentences cost about 50 million pair passes when every pair needs the joint model.

Stage two, cross-encoder rerank. The 100 options go to the rerank lane in one request. The served row is bge-reranker-v2-m3, a 568M parameter cross-encoder on an XLM-RoBERTa base. It scores all 100 pairs and returns them sorted, with a relevance score between 0 and 1 for each. You keep the top 5 for the model that answers the user.

What it costs. Metering counts one search per request, not one per document. So 1,000 rerank requests bill as 1,000 searches. Monthly cost = (Q / 1,000) x R. Q is your request count. R is the rate the fence above renders from the platform's model data. One sensitivity note: the total scales linearly with Q. The embeddings side is a one-time corpus cost, not a per query cost. The full cost picture for rerank units and latency bands lives in the rerank pricing guide.

Latency. Our records from 2026-09-09 put p95 at about 0.9 s for a single rerank request. That number covers the rerank stage only. Your vector search adds its own latency, set by your index.

What each architecture is called on the shelf today

The hosted rerank field is young and it moves. As of 2026-10-28, here is what each provider's own model card or docs say. The cells cover the design behind each rerank or embedding product. Where a vendor does not say, the cell says so.

The pattern in the table is the point. Every disclosed design on the rerank shelf is a cross-encoder, in sizes from 568M to 8B parameters. The bi-encoder side of the shelf is the embeddings catalog. Late interaction is the open weights path, not the hosted API path. That holds as of the read date above.

Which one should you use

Most readers land in the first two rows. Embed the corpus once. Rerank the shortlist on demand. The two stage decision guide walks the build or buy version of that call.

Make the first rerank call

The rerank lane is live on this platform. The request takes a closed set of four fields: model, query, documents, and top_n. The first three are required. Unknown fields are rejected. Send the same Idempotency-Key header on a retry. The platform replays the first result instead of billing twice.

curl https://api.ironstratum.com/v1/rerank \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bge-reranker-v2-m3",
    "query": "how do the two encoder designs differ",
    "documents": [
      "A bi-encoder embeds query and document apart and compares vectors.",
      "A cross-encoder reads query and document together and scores the pair.",
      "Chunk three: an unrelated document about vector database tuning."
    ],
    "top_n": 2
  }'

The response is a results array. Each entry carries the index of one document from your list. It also carries its relevance_score between 0 and 1, best first. A key from the console and wallet credit is all it takes. The rerank lane page lists the full field reference, and pricing shows the current rate card.

Frequently asked questions

Which is better, a bi-encoder or a cross-encoder?

Neither. They own different stages. A bi-encoder fetches a shortlist from a large corpus in milliseconds. A cross-encoder orders that shortlist with far better judgment. A search stack that needs both speed and precision runs them in turn. The second stage can only be as good as the first. A reranker cannot rescue a text the retriever never surfaced.

What is a cross-encoder used for?

Reranking. A cross-encoder scores one query and document pair at a time, fully. It orders the top 50 to 200 options a faster retriever returns. Common uses are RAG context choice, search result order, and dedup of close matches. Legal, medical, and financial search join them, since the top few results carry the value.

What are the four types of encoders?

The sentence-transformers library, the reference tooling here, ships four model types. Sentence Transformer is the bi-encoder embeddings type. Cross Encoder is the pair scoring type. Sparse Encoder makes learned sparse vectors, SPLADE style. Multi-Vector Encoder is the late interaction type, ColBERT style. Most comparison pages cover the first two. The third and fourth matter when term matching or index size drives your design.

Is ColBERT a cross-encoder?

No. ColBERT is the late interaction design. It encodes query and document apart, which is the bi-encoder side. But it keeps one vector per token and runs one cheap MaxSim step over them. That step is where its edge comes from. It lands near cross-encoder quality at a fraction of the per query cost. The price is larger indexes.

Last verified: 2026-10-28. Design claims follow the papers and library docs linked above (arXiv 1908.10084, arXiv 1901.04085, arXiv 2004.12832, arXiv 2112.01488, sbert.net docs). Model facts follow the BAAI, Qwen, NVIDIA, Voyage, and Cohere pages linked in the shelf table. Platform facts follow the platform's model data and measurement records named in the text.