Reranking: Cross-Encoders and Cascades

A retriever may return the relevant passage at rank 7 when the prompt includes only the first five results. Bi-encoder retrieval is optimized for fast recall over a large corpus, not precise ordering within the shortlist. A reranker scores each query-candidate pair jointly and reorders that shortlist before prompt assembly.

Reranking

A reranker is a function score(query, candidate) → relevance applied per-candidate at query time. It does not build an index; it does not enumerate the corpus. It receives a small list of candidates (already shortlisted by retrieval) and produces a per-pair relevance score that you sort on.

The defining property is the joint input: the model attends over query and candidate tokens together in the same forward pass. That is the entire architectural difference from the embedding-based retriever you put in front of it.

Bi-encoders and cross-encoders

The retrieval models discussed in text embeddings are bi-encoders: two parallel encoders, one for queries, one for documents, with cosine similarity glued on top. Documents are embedded ahead of time. The query is embedded once at runtime. Comparison is a dot product. This factorization is what lets ANN over vector databases scale to billions of vectors: the document side is fully precomputable.

A cross-encoder breaks the factorization on purpose. The query and document are concatenated into a single input, fed through one transformer, and a regression head produces a single scalar relevance score. Every query token can attend to every document token. The model can see “the device fails at 03:14 UTC” in the document and notice that the question asked specifically about 3:14 UTC, not just device failure. That cross-attention is the entire reason cross-encoders outperform bi-encoders on fine-grained ranking; and it is also the entire reason they cannot be indexed.

This is not a strict dichotomy in practice. Late-interaction models like ColBERT and ColPali split the difference by keeping per-token embeddings on both sides and computing a MaxSim score at query time; more expressive than a single cosine, far cheaper than a full cross-encoder. They show up in the same pipeline slot a reranker would, with different cost characteristics.

Reranker models

A few options dominate, split between hosted APIs and open-weights models you can self-host.

Hosted, closed weights:

• Cohere Rerank v3.5; 4096-token context, multilingual (100+ languages), strong on BEIR and enterprise domains (finance, e-commerce). Priced at $2.00 per 1,000 searches at the time of writing. The default “just works” choice if you don’t want to think about it.
• Voyage rerank-2.5 and rerank-2.5-lite; 32K-token context (8× Cohere’s), instruction-following (you can steer ranking with natural-language criteria), and competitive accuracy on Voyage’s internal 93-dataset benchmark. The long context is the differentiator if your candidates are full pages rather than chunks.

Open weights, run yourself:

• BAAI bge-reranker-v2-m3; 0.6B-parameter multilingual cross-encoder built on bge-m3. The standard self-hosted baseline. Easy to deploy, fast inference, well-understood.
• BAAI bge-reranker-v2.5-gemma2-lightweight; heavier 9B model on gemma2-9b with token-compression and layerwise pruning. Better quality, much higher cost.
• mixedbread mxbai-rerank-large-v2; 1.5B-parameter model trained with a three-stage RL recipe (GRPO + contrastive + preference). Open weights, multilingual, 8K context.
• Jina reranker v2 base multilingual; 278M parameters, 1024-token context. Smaller and faster than the open-weights alternatives; good when latency is tight and corpora are short.

Model choice depends on domain data, latency budget, and whether the team operates GPU inference. A held-out retrieval set should decide between hosted and self-hosted options.

Latency budget

Per-pair cross-encoder cost depends on model size, sequence length, and hardware. Rough orders of magnitude on a modern GPU:

• Small (~22M MiniLM-class) cross-encoder: a few ms per pair; sub-second for 100 pairs even on CPU.
• Mid-size (300M–1.5B) cross-encoder: 5–20 ms per pair on GPU; 50–200 ms total for a top-50 rerank.
• Large (7B+ LLM-as-reranker): 50–200 ms per pair; reranking >20 candidates breaks an interactive budget.

A useful reference benchmark: published numbers for ms-marco-MiniLM-L6-v2 show roughly 12 ms for 1 document up to ~740 ms for 100 documents on a CPU, with GPU bringing 30–50 ms for ~30 candidates and 100–200 ms still typical on CPU at that size. The shape is linear in candidate count, which is why every cascade caps candidate count explicitly; the reranker has no “approximate” mode.

For an interactive RAG endpoint with a ~1.5 s total budget, you can typically afford to rerank 50–100 candidates with a mid-size open-weights model on GPU, or 100 candidates with a hosted API on the order of 100–300 ms. Beyond that, latency dominates and you need a smaller reranker, a smaller candidate pool, or batch processing.

Hosted reranking with Cohere

The hosted path is the lowest-friction. The Cohere Python SDK v2 exposes a single rerank call. Install: pip install cohere.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import os
import cohere

co = cohere.ClientV2(api_key=os.environ["COHERE_API_KEY"])

def cohere_rerank(query: str, candidates: list[str], top_n: int = 5):
    response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=candidates,
        top_n=top_n,
    )
    # response.results is sorted by relevance_score descending
    return [(r.index, r.relevance_score) for r in response.results]

# Usage: candidates are the strings returned by your hybrid retrieval stage.
ranked = cohere_rerank(
    query="how do I rotate a Postgres replication slot's primary?",
    candidates=hybrid_top_100,  # from the previous article's hybrid_search()
    top_n=10,
)

The TypeScript shape is symmetric via the Cohere TypeScript SDK. Install: npm install cohere-ai.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

import { CohereClientV2 } from "cohere-ai";

const co = new CohereClientV2({ token: process.env.COHERE_API_KEY! });

async function cohereRerank(
  query: string,
  candidates: string[],
  topN = 5,
): Promise<Array<{ index: number; score: number }>> {
  const response = await co.rerank({
    model: "rerank-v3.5",
    query,
    documents: candidates,
    topN,
  });
  return response.results.map((r) => ({
    index: r.index,
    score: r.relevanceScore,
  }));
}

relevance_score is a value in [0, 1] that Cohere calibrates as a probability. You can threshold on it; drop candidates below 0.3; though calibration in your domain may differ from the model’s training distribution and a thresholding cut typically wants its own held-out tuning.

Retrieval cascade

The whole pipeline ties together as:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

def rag_pipeline(query: str, *, retrieve_k: int = 200, rerank_k: int = 25,
                 final_k: int = 8):
    # Stage 1: hybrid retrieval (BM25 + dense, fused by RRF)
    retrieved = hybrid_search(query, top_k=retrieve_k, candidate_k=retrieve_k * 2)
    docs = [load_chunk(doc_id) for doc_id, _ in retrieved]

    # Stage 2: cross-encoder rerank
    ranked = local_rerank(query, docs, top_n=rerank_k)
    reranked = [docs[i] for i, _ in ranked]

    # Stage 3 (optional): LLM judge over rubric — skipped by default
    final_docs = reranked[:final_k]

    return final_docs

The numbers 200 → 25 → 8 are not magic; they are the answer to “how much can I afford at each stage?” given your reranker latency and your end-to-end budget. Two principles set them:

1. Stage 1 over-fetches by 5–10×. Hybrid retrieval is cheap; cross-encoder reranking is not. You want stage 1’s recall@retrieve_k to be very high (say 0.95) on your eval set, even if precision@k is poor. The reranker will sort it out.
2. Stage 2 cuts to what the generator can plausibly use. If your generator packs 5–10 chunks into the prompt, the reranker output of 25 is a small over-fetch that lets a final lightweight dedup or diversity step have room to work.

Failure modes

A reranker cannot fix recall. If the gold passage is not in the top-200 from retrieval, the reranker never sees it. Reranking amplifies retrieval quality; it does not substitute for it. Always measure recall@retrieve_k first.

Token truncation is silent and lossy. Most hosted rerankers have 4–32K-token windows; most open-weights models default to 512 tokens. A 1,500-token document being reranked by a 512-token model is silently truncated to the first 512 tokens. If the relevant span is at the end, the reranker will score it as if the relevant span did not exist. The same chunking discipline from the chunking strategies article applies: keep candidates inside the reranker’s window or use a model with a longer one.

Scores are not always probabilities. Cohere calibrates [0, 1]. Open-weights cross-encoders trained on MS MARCO output raw logits; applying a sigmoid produces only pseudo-probabilities. Tune thresholds per model and domain. Sorting by score is stable across queries; absolute cutoffs are not.

Domain shift hits rerankers hard. A reranker trained primarily on web search and news will underperform on legal contracts, biomedical text, or your internal log lines. Voyage’s instruction-following models partially address this by letting you steer with natural-language criteria; for the rest, fine-tune on a domain-labeled set or accept the gap. The standard signal: rerank improves nDCG by 10+ points on MS MARCO and 0–2 points on your domain. That’s the gap.

LLM-as-reranker is a real option, with real cost. A frontier-model judge with a custom rubric is the strongest reranker available; it follows arbitrary criteria, can do multi-hop reasoning over candidates, and adapts instantly to new domains. It is also 100×–1000× the cost of a dedicated reranker per call. The pragmatic pattern is to use it in stage 3 over a small set (5–10 candidates) after a cheap cross-encoder has done stage 2’s 200 → 25 cut.

Top-K matters more than you think. Reranking the top 10 and reranking the top 100 are two different operations. The first is “polish what’s already good”; the second is “find the gem you missed.” If your retriever’s recall@10 is already 0.9, reranking top 10 barely moves the needle. If recall@100 is 0.95 but recall@10 is 0.6, reranking top 100 → 10 is the entire game. Pick the rerank pool size against retrieval’s recall curve, not by feel.

Cost dominates at scale. A hosted rerank at $2/1k searches is cheap per query and meaningful at 100M queries. A self-hosted GPU reranker has fixed amortized cost; at sufficient throughput, it is cheaper; below it, the GPU is idle and expensive. Run the math at your projected QPS before committing to either side.

Don’t double-rerank by accident. It’s easy to add a Cohere rerank and forget that LlamaIndex or LangChain wrapped the same call inside a higher-level abstraction. Two cross-encoder passes in a single query are silently expensive and almost never improve quality.

References

• Anthropic; Contextual Retrieval; the paper that quantified reranking’s contribution: 49% retrieval-failure reduction without it, 67% with a Cohere reranker added. The clearest production case study you can read in 15 minutes.
• Jo Bergum (Vespa); Minimizing LLM Distraction with Cross-Encoder Re-Ranking; Vespa’s chief scientist on why cross-encoders sit between retrieval and the LLM, and the declarative multi-phase ranking pattern Vespa exposes. Worth reading anything Bergum has written about ranking.
• Pinecone; Rerankers and Two-Stage Retrieval; a thorough walkthrough of cross-encoder reranking, including a worked example with sentence-transformers and an honest discussion of when reranking adds noise.
• Cohere; An Overview of Rerank; the canonical product reference; reads as documentation but contains useful evaluation patterns and integration sketches that generalize beyond Cohere’s specific model.