Reranker vs embeddings retrieval: when the second stage pays for itself

Embeddings search a whole corpus fast, and their cost is mostly paid once at indexing. A reranker reads each candidate next to the query and orders the shortlist far better. It adds a serial hop to every request. Neither one replaces the other. The real decision is whether your pipeline needs the second stage. That turns on corpus size, precision need, latency budget, and one break-even sum.

Two designs, one decision

A one-stage pipeline embeds every chunk once, stores the vectors, and answers each query with a vector search. The index hands back the nearest few hundred chunks in milliseconds. That stage is built for recall: gather everything that might matter, fast.

A two-stage pipeline keeps that first stage and adds a second one. The top candidates from the vector search go to a reranker. It reads the query and each candidate together, as one input, and scores the pair. The shortlist comes back in a better order. That stage is built for precision: the few chunks you actually use rank first.

The two stages run different model shapes. The embedding side is a bi-encoder. It encodes the query and the documents apart, so the document work happens once, ahead of time. The rerank side is a cross-encoder. It reads the query and the document as one sequence, so the work repeats for every pair, on every query. The cross-encoder vs bi-encoder guide covers those shapes in depth. This page stays on the pipeline call.

That table is the whole trade. You are not picking a winner between two products. You are deciding whether your queries need the second row's ordering badly enough to pay for its serial hop.

What the recall stage gives up

An embedding compresses each text into a single vector. That compression is where the losses live. Three failure shapes show up again and again in production.

First, fine grain. Two chunks that differ by one decisive word can land almost on top of each other in vector space. A renewal deadline and a termination deadline are near neighbors. The order inside the top ten can come down to noise. Second, conditions. Some queries hinge on one small word, such as "not", or on a numeric condition, such as "after 90 days". Those queries rely on a close read of the text. A single pooled vector smooths over that reading. Third, scale. The vector search returns a fixed window of size k. In a corpus of five million chunks, thousands may be plausible. Only k of them fit in the window. More near-duplicates compete for the same slots. The order inside the window gets noisier as the corpus grows. Recall can stay high while the top-order quality falls.

The reference tooling states the gap plainly. The sentence-transformers docs say it in one line: the first stage "might retrieve documents that are not that relevant". Their fix is a second step that scores every candidate with a cross-encoder. BAAI, the publisher of the open bge family, writes the same verdict on its own model card. As cross-encoders, re-rankers demonstrate higher accuracy than bi-encoder embedding models. The card and the library docs agree. That is the sign of settled ground.

The scale wall behind the two shapes is old and well measured. The Sentence-BERT paper (Reimers and Gurevych, 2019) put a number on it. Scoring all pairs among 10,000 sentences with a joint model costs about 50 million inference runs. That is about 65 hours. Twin embeddings do the same job in about 5 seconds. That is why the cross-encoder never runs the whole corpus, and why the pipeline splits at all.

What the second stage costs

The rerank stage buys its ordering with two resources: time on every request, and a growing amount of work per candidate.

Latency first. The second stage is serial. The reranker scores only after the first stage returns the shortlist. Its latency lands on top of everything else. On this platform's two lanes, our records from the 9 September 2026 window give two numbers. The embeddings lane: about 1.24 seconds p95 for a first response, single flight. The rerank lane: about 0.9 seconds p95, same shape. Read both as planning inputs from one measurement window, not as service promises. A two-stage request on these lanes pays the query embedding, the vector search, and the rerank hop in sequence. The rerank hop is not the smallest one.

Throughput next. A cross-encoder scores every query-document pair in full. Work scales with the shortlist size and with document length. One vendor publishes its own benchmark for scale. Cohere reports that Rerank 5 Fast reranks about 187 documents of 1,000 tokens each per second on a single A100 GPU. The shape of the test: 96 documents per request, 8 requests in flight. That is about 2.9 times the throughput of its Pro tier, per Cohere's Rerank 5 post, read 28 October 2026. A vendor measuring its own model on its own hardware is one source, and only one. The lesson is the shape: pairs cost, and pairs scale.

Shortlist size is the dial that sets both bills. The reference pattern from the sentence-transformers docs retrieves the top 100 and reranks those. BAAI's own evaluation of the bge reranker reranks the top 100 candidates from a retriever, per its model card. Vendors cap the request instead of the pattern. Voyage's reranker docs cap one rerank-3 request at 600,000 total tokens, read 28 October 2026. On this platform's rerank lane, one request is one metered search at any document count. A longer shortlist moves your latency, never your bill. The unit rules across vendors differ far more than that, and the rerank pricing guide holds that census.

The break-even math

The pages ranking today stop at "add a reranker for quality". None of them prints the arithmetic, as of our reads on 28 October 2026. This section does. All rates stay symbolic and render from the platform's model data, so the math never goes stale.

Stage one, the indexing side. The corpus is embedded once, and again whenever chunks change:

indexing cost  =  ( C x D / 1,000,000 ) x R_e

C:  chunks embedded in the month, re-embeds included
D:  average tokens per chunk
R_e:  per-million-token rate for the embeddings row, from the fenced table

Stage one, the query side, plus stage two:

query-side cost  =  ( Q x q / 1,000,000 ) x R_e
rerank cost      =  ( Q / 1,000 ) x R_r

Q:  queries in the month
q:  average tokens per query
R_r:  per-1k-search rate for the rerank row, from the fenced table

A worked shape, on named counts. Take 100,000 chunks at 400 tokens each. Indexing moves 40 million tokens. The one-time stage-one bill is 40 x R_e. Take 1,000 queries at 20 tokens each. The query-side embedding cost is 0.02 x R_e for the month. The rerank stage over those 1,000 queries costs R_r. On this platform's flat rule, shortlist length changes nothing in that line. Fifty candidates and 500 candidates both bill as one search.

Now the value side, which is the part every incumbent skips. The second stage pays when the loss it prevents exceeds its bill:

second stage pays  when  E x L  >  R_r / 1,000   per query

E:  share of queries whose stage-one top order is wrong enough to change the result
L:  loss on one wrong order: wasted generation tokens, a retry, a human review

Both sides scale with query volume. Volume alone never flips the call. A million queries a month and a hundred face the same per-query comparison. What flips it is precision need, not traffic. High-stakes ordering crosses the line at once. Browsing-style retrieval never crosses it.

Two cells in this math stay dashed, honestly. The recall at k=100 for a specific corpus of yours: dash. It depends on your chunks and your queries, and BAAI publishes its retrieval results as charts, not tables. A p95 for a 1,000-document rerank request on this lane: dash. The 0.9 second figure comes from the single-flight measurement window, and stretching the shortlist tenfold was not part of it. Measure once on your own shortlist before you commit. The local-model forums say the same thing when they rank rerankers: test on your own data, because no global winner exists.

The decision table

Read the three keys in order: corpus size, precision need, latency budget. The table gives the call, not a verdict about products.

One regime deserves its own sentence. The smallest corpora hold the surprise. Run the reranker alone, with no vector index at all. One request can carry the whole corpus. The metering rule prices that request as one search. You get full cross-encoder judgment over every chunk, and no index to build or refresh.

Corpus regimes, and the freshness edge

The corpus-size rows above follow one logic. A vector search is a fixed-size window over a growing pile. The bigger the pile, the more the order inside the window depends on luck. A rerank stage reads the actual text of each candidate. Its judgment does not degrade with corpus size. It degrades with shortlist size, and you control that.

Freshness is the quieter axis, and it points the same way for churn-heavy systems. An embedding index is a snapshot. A chunk that changed yesterday still answers with its old vector until someone re-embeds it. A reranker holds no index. It reads whatever text you send it now. Rewritten chunks, late pages, and hourly feeds are scored at their current state, with no index work. For a corpus that turns over fast, that property can outweigh a tight latency budget. How you split documents into chunks is its own subject. The document parsing guide covers it.

Scale has one more cost line worth naming. At millions of chunks, re-embedding turns into a project. A model change or a chunking fix means the same 40 x R_e shape from the math above, with a bigger C and a schedule. The rerank side never pays that line. It holds no index to rebuild.

Two lanes, one training family

This platform serves both stages, and the two rows were built for each other.

The embeddings lane serves bge-m3. Its model card lists 1,024 dimensions, an 8,192-token window, and more than 100 working languages. One pass yields three retrieval outputs: dense, sparse, and multi-vector. The rerank lane serves bge-reranker-v2-m3. It is a 568M-parameter cross-encoder on a bge-m3 base, per the family table on its model card. Scores arrive between 0 and 1 through a sigmoid. The embeddings lane also carries a multimodal row, wemm-embed, for corpora that mix text with images and video.

BAAI's own advice for retrieval pipelines is the two-stage shape this page computes. Its bge-m3 card recommends hybrid retrieval plus re-ranking, with the reranker filtering the retriever's text. On the hosted side, the rerank class is young, and its billing is split: some vendors bill per search, some per token. The rerank pricing guide carries the dated census. That same open reranker appears on other hosts' rerank menus too, per Pinecone's rerank docs, read 28 October 2026. The weights are open, so the self-host exit stays open.

A first rerank call, for the stage your math says you need:

curl https://api.ironstratum.com/v1/rerank \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "bge-reranker-v2-m3",
    "query": "when does the second retrieval stage pay for itself",
    "documents": [
      "Rerank the top 100 candidates when top-order precision changes the answer.",
      "One-stage vector search keeps latency flat for browsing-style queries.",
      "Chunk three: an unrelated note about vector index tuning."
    ],
    "top_n": 2
  }'

The response is an ordered list with one relevance score per document, best first. The request takes four fields, model, query, documents, and top_n, and rejects anything else. Send an Idempotency-Key header on retries. A replayed request bills once, per the idempotency docs. The wallet is prepaid and a refused request bills nothing. Create a key in the console, then price your real shortlist on the pricing page rather than on any headline.

Frequently asked questions

What is the difference between an embedding model and a reranker?

An embedding model turns one text into one vector. The texts are encoded apart, so the corpus work happens once, and a vector index serves it. A reranker takes a query and one candidate together as a single input and returns a relevance score for that pair. The first is built for fast recall over a whole corpus. The second is built for accurate ordering of a shortlist. BAAI's model cards state the accuracy relation directly: cross-encoder rerankers score higher than bi-encoder embedding models.

Is a reranker better than vector search?

Better is the wrong axis; they solve different halves. Vector search fetches a plausible window from millions of chunks in milliseconds. A reranker then reads the query against each fetched chunk and fixes the order of the window. Where top order matters, the lift is large. The BERT reranking system of Nogueira and Cho (2019) beat the prior best on the MS MARCO passage task by 27 percent relative in MRR@10 (Nogueira and Cho, 2019). Where any plausible result satisfies the reader, the lift buys nothing.

How many documents should I rerank?

Start at 100. That is the reference pattern in the sentence-transformers docs and the shape BAAI uses in its own reranker evaluations. Go wider, to the low hundreds or a thousand, only when your first stage's recall at 100 is the measured bottleneck. Every extra candidate costs serial time in the second stage. On this platform's lane the bill stays flat at one search per request. The dial trades latency, and nothing else.

How much latency does a reranker add?

One serial hop, sized by your shortlist. On this platform's rerank lane, the 9 September 2026 window put first-response p95 at about 0.9 seconds single flight. For scale on vendor hardware, Cohere reports about 187 documents of 1,000 tokens per second on one A100 for its Fast tier. The shape there: 96 documents per request. Both numbers are dated observations at named shapes, not promises. Budget the hop as stage-one p95 plus rerank p95, then measure once on your own shortlist.

Do I need a reranker for a small knowledge base?

Often the opposite. Under about 1,000 chunks, skip the vector index and send the whole corpus in each rerank request. You get cross-encoder judgment over every chunk, no index to build, and no re-embedding when content changes. The cost is one metered search per query. Below a few hundred chunks, even one-stage embedding search may be more machinery than the problem needs.

Last verified: 2026-10-28. Design claims follow the papers and library docs linked above (arXiv 1908.10084, arXiv 1901.04085, sentence-transformers docs). Model facts follow the BAAI model cards for bge-m3 and bge-reranker-v2-m3. Vendor figures follow Cohere's Rerank 5 post, Voyage's reranker docs, and Pinecone's rerank docs, each read 28 October 2026. Platform facts follow the platform's model data and the measurement records named in the text.