A Guide to Search Relevance Metrics and Evaluation

A team ships a new reranker, and the top results look different. Without a measurement, nobody can say whether the change helped, hurt, or only reshuffled the list. Search relevance metrics turn that question into a number you can compare across versions.

This guide computes five core metrics on one shared example and shows how relevance labels get produced. It then compares offline and online testing and explains where the metrics mislead in RAG and AI agent pipelines.

What Are Search Relevance Metrics?

Search relevance metrics are numbers that score how well a ranked list of results matches what a query needs. They grade each result with a relevance label and look only at the top k positions.

Every metric depends on three inputs:

• A relevance label is a grade for one query-document pair. Labels are binary (relevant or not) or graded, such as 0 (not relevant) to 3 (perfect).
• The cutoff k is the number of top results the metric reads. P@5 looks at the first five results and ignores the rest.
• A query set is the fixed list of test queries you score. Each metric is computed per query, then averaged across the set.

These ranking metrics grade the ranker's output, whatever the retrieval method. BM25 (a keyword-scoring formula), embedding search, hybrid retrieval, and web search APIs all end in an ordered list. For background on how those lists get built, see how semantic search works and how web search APIs rank results.

Set-Based vs Rank-Aware Metrics

Set-based metrics treat the top k as an unordered group and ignore order inside it. Rank-aware metrics give more credit when relevant results appear higher.

Swapping the results at ranks 1 and 5 leaves precision@5 unchanged. The same swap changes MRR, MAP, and nDCG@5.

The Running Example Used in This Guide

Every metric below uses one query and the same five returned results. Each result has a grade from 0 (not relevant) to 3 (perfect match).

The corpus holds four relevant documents for this query, graded 3, 2, 1, and 1. Three of them appear in the top five. The fourth, Doc F with grade 1, was never retrieved.

Grades like these are called relevance judgments. In this example, "relevant" means any grade above 0.

The Core Search Relevance Metrics

Five metrics cover most retrieval evaluation work. Each one answers a different question about the same ranked list, so two metrics can disagree about the same change. Only nDCG uses the full 0 to 3 grades.

Precision@k

Precision@k is the share of the top k results that are relevant.

P@k = (relevant results in the top k) / k

In the running example, ranks 2, 3, and 5 are relevant, so P@5 = 3/5 = 0.6. The Elasticsearch ranking evaluation API documentation reads the metric the same way: "A precision at 10 (P@10) value of 0.6 then means 6 out of the 10 top hits are relevant."

Precision@k ignores position inside the cutoff, so relevant results at ranks 1 to 3 score the same as relevant results at ranks 3 to 5.

Duplicates inflate precision. If one page appears twice under different URLs, both copies count as hits, yet the second adds no new information. For agents, the copy also takes up context window space, so deduplicating search results for agents before scoring gives a more honest number.

Recall@k

Recall@k is the share of all relevant documents that appear in the top k.

R@k = (relevant results in the top k) / (total relevant documents)

The example has four relevant documents and three of them in the top five, so R@5 = 3/4 = 0.75. The missing grade-1 document accounts for the gap.

Recall needs the total count of relevant documents, which is hard to know on the open web. Teams usually pick one of two workarounds:

• Build a gold set of known-good sources for each query and measure recall against that list.
• Use hit rate, which scores 1 if at least one correct source appears in the top k and 0 otherwise.

Recall@k matters most when an LLM reads every result in the window. A source that does not reach the model cannot be used. Order inside the window matters less.

Query rewriting is a common change tested this way, and the guide to query expansion for AI agents walks through a recall@k comparison.

Mean Reciprocal Rank (MRR)

Mean reciprocal rank is the average, across queries, of 1 divided by the rank of the first relevant result. For a single query, that value is the reciprocal rank (RR).

RR = 1 / (rank of the first relevant result), and MRR = average RR across all queries

In the example, the first relevant result sits at rank 2, so RR = 1/2 = 0.5. The same Elasticsearch documentation notes that "finding the first relevant result in position 3 means the reciprocal rank is 1/3."

MRR fits single-answer and navigational queries, such as finding a login page or one product spec. It ignores everything after the first hit, so a list with one good result scores the same as a list with five. A query with no relevant result scores 0.

Mean Average Precision (MAP)

Average precision (AP) adds up the precision at each rank where a relevant result appears, then divides by the total number of relevant documents. Mean average precision (MAP) is the average of AP across all queries.

In the example, relevant results appear at ranks 2, 3, and 5. Precision at those ranks is 1/2, 2/3, and 3/5. That gives AP = (1/2 + 2/3 + 3/5) / 4 = 0.44.

The divisor is 4 because the unretrieved document still counts. AP therefore rewards ranking every relevant result high and penalizes missed documents.

MAP works with binary labels only, so a grade-3 result and a grade-1 result count the same. That is the core of nDCG vs MAP: nDCG keeps the grades.

Normalized Discounted Cumulative Gain (nDCG)

nDCG scores a ranking with graded labels and gives less credit to results lower in the list. It divides the list's score by the best possible score, so the result falls between 0 and 1.

Discounted cumulative gain (DCG) adds up each result's grade divided by log2(rank + 1). Ideal DCG (IDCG) is the DCG of the best possible order of all relevant documents. The scikit-learn ndcg_score docs describe the same steps: sum "the true scores ranked in the order induced by the predicted scores, after applying a logarithmic discount," then divide "by the best possible score (Ideal DCG, obtained for a perfect ranking)."

For the running example:

• DCG@5 = 3/log2(3) + 2/log2(4) + 1/log2(6) = 1.893 + 1.0 + 0.387 = 3.28
• The ideal order [3, 2, 1, 1, 0] gives IDCG@5 = 3 + 1.262 + 0.5 + 0.431 = 5.19
• nDCG@5 = 3.28 / 5.19 = 0.63

Tools can compute nDCG differently, so compare nDCG numbers only within one implementation and one set of labels. A query with no relevant documents has an ideal DCG of 0, which makes nDCG a division by zero. Decide up front whether those queries score 0, score 1, or get dropped, and apply that rule to every run.

Computing All Five Metrics in Python

This code uses only Python's built-in math module and computes every value from the running example.

import math

retrieved = [0, 3, 2, 0, 1]  # grades at ranks 1-5
all_relevant = [3, 2, 1, 1]  # grades of every relevant doc for the query
k = 5


def precision_at_k(grades, k):
    return sum(1 for g in grades[:k] if g > 0) / k


def recall_at_k(grades, total_relevant, k):
    return sum(1 for g in grades[:k] if g > 0) / total_relevant


def reciprocal_rank(grades):
    for rank, g in enumerate(grades, start=1):
        if g > 0:
            return 1 / rank
    return 0.0  # no relevant result


def average_precision(grades, total_relevant):
    hits, total = 0, 0.0
    for rank, g in enumerate(grades, start=1):
        if g > 0:
            hits += 1
            total += hits / rank  # precision at this rank
    return total / total_relevant if total_relevant else 0.0


def dcg_at_k(grades, k):
    return sum(g / math.log2(rank + 1)
               for rank, g in enumerate(grades[:k], start=1))


def ndcg_at_k(grades, relevant_grades, k):
    ideal = dcg_at_k(sorted(relevant_grades, reverse=True), k)
    if ideal == 0:
        return 0.0  # no relevant docs: pick a rule and apply it everywhere
    return dcg_at_k(grades, k) / ideal


print(f"P@5: {precision_at_k(retrieved, k):.2f}")
print(f"R@5: {recall_at_k(retrieved, len(all_relevant), k):.2f}")
print(f"RR: {reciprocal_rank(retrieved):.2f}")
print(f"AP: {average_precision(retrieved, len(all_relevant)):.2f}")
print(f"nDCG@5: {ndcg_at_k(retrieved, all_relevant, k):.2f}")

Expected output:

P@5: 0.60
R@5: 0.75
RR: 0.50
AP: 0.44
nDCG@5: 0.63

How to Choose the Right Metric

The right metric depends on who consumes the results. A person scanning ten links reads from the top and stops early, while an LLM in a RAG pipeline reads every chunk it receives.

Track one primary metric plus one guardrail metric. For example, a RAG team might optimize recall@10 while checking that precision@10 holds steady.

Where Relevance Labels Come From

Every metric depends on labels, and the label source often shapes the score more than the formula does. The same ranking can score differently under strict human raters, a lenient LLM judge, or click data. Each source trades cost against a different kind of bias.

Human Judgment Lists

A judgment list is a set of (query, document, grade) rows used as ground truth. TREC-style evaluations call these files qrels.

Building a judgment list takes four steps:

1. Sample real queries from logs across head (frequent) and tail (rare) traffic.
2. Write a short grading guide with at least one example for each grade.
3. Have two raters label the same sample and measure how often they agree.
4. Refresh labels when the content behind the results changes.

Low rater agreement usually means the grading guide is unclear, so fix the guide before labeling more data.

Unlabeled results also need a rule. When a new ranker returns a document nobody graded, you can count it as irrelevant or skip it. Elasticsearch exposes this choice as the ignore_unlabeled option.

Graders should judge what the system actually returned, not a page title or a memory of the site. Store each page's content alongside its rank. The Olostep Searches endpoint returns deduplicated links with titles and descriptions for a query; pair it with the Scrapes or Batches endpoints to fetch the page bodies you grade, so each grade sits beside the returned text.

Click and Behavior Signals

Clicks, query reformulations, abandoned searches, and dwell time (how long a user stays on a result) arrive for free at scale. For search evaluation, the catch is that clicks reflect position as well as relevance. The OpenSearch search quality guide notes that "Behavioral data often contains biases like position and presentation bias."

The AWR 2026 CTR study estimated for Google US desktop in July 2026 that "the average organic result earned a 20.02% CTR at position 1, 10.36% at position 2, and 3.89% at position 3." Those figures mix two effects: stronger results tend to rank higher, and users click higher positions more often. Raw click rates cannot separate the two.

Two methods compare rankers on behavior while controlling for position. An A/B test splits users between rankers and compares outcomes.

Interleaving merges results from two rankers into one list and credits whichever ranker's results get clicked, so both face the same position effects. The Airbnb interleaving paper, from KDD '25, states the credit rule: "Preferences are determined by counting which 'team' has gathered more clicks."

LLM-as-a-Judge

LLM-as-a-judge means an LLM grades (query, document) pairs using a prompt that contains the grading scale. It labels far more pairs per hour than a human rater.

The TREC 2024 RAG judging study compared an LLM judge (GPT-4o with the UMBRELA prompt) with NIST assessors across 77 runs and 27 fully manual topics. Kendall's τ measures how closely two rankings agree, from −1 to 1. In that TREC 2024 RAG study, τ between UMBRELA and fully manual judgments was 0.890 for run-level nDCG@20, but only 0.553 per topic.

Those τ values measure agreement on how systems rank, not agreement on individual labels. The authors also wrote that "Overall, human assessors appear to be stricter than UMBRELA in applying relevance criteria."

The Clarke and Dietz critique, presented at EVIA 2025, argues that the average hides weakness at the top. In their analysis, agreement "drops to 0.51 (24% swaps) among the top 20 systems, and to 0.56 (21% swaps) among the top 15." A swap is a pair of systems placed in the opposite order, and the top systems are where teams pick a winner.

Use LLM judges with guardrails:

• Calibrate the judge against a human-labeled sample before trusting its grades.
• Keep human raters on the top candidates, where judge agreement is weakest.
• Judge with a different LLM than the one that ranks results, so the judge does not grade its own preferences.

Offline vs Online Evaluation

Offline evaluation scores a ranker on a fixed query set with stored labels before launch. It is fast and repeatable, so the same test can run on every change.

Online evaluation runs A/B tests or interleaving on live traffic. It measures real user behavior, but it takes longer and needs enough users to reach a result.

Offline wins do not always carry over to production. Common reasons include:

• Labels go stale as the underlying content changes.
• The query set does not match the real traffic mix.
• The metric does not match the outcome you care about, such as task success.

Check whether a difference is real by running a paired significance test over the same queries. The ranx statistical tests docs document Fisher's randomization test and the two-sided paired Student's t-test for comparing runs.

Then look at per-query wins and losses, because a small average gain can hide large losses on important queries.

Evaluating Retrieval in RAG and AI Agent Pipelines

In RAG, the reader of the results is an LLM, so the goal shifts from good order to getting the right content to the model. A person can skip a weak result at rank 3, but the model reads it and may use it.

Retrieval quality and answer quality are separate layers, as retrieval-augmented generation explained shows in more depth.

Why Retrieval Scores Don't Guarantee Good Answers

Higher retrieval scores track answer quality only loosely. The retrieval-to-RAG correlation study, from ICTIR '26, compared 15 retrieval stacks and 4 RAG pipelines on TREC NeuCLIR 2024 and RAG 2024 at cutoff 20. For retrieval metrics against RAG coverage in that study, "topic-level correlations in Table 3 are moderate in absolute terms (r between 0.26–0.56)."

The UDCG paper on RAG metrics is an October 2025 preprint that tested 6 LLMs on 5 QA datasets. In that preprint, the authors write: "All traditional IR metrics would score these two scenarios equally, although their end-to-end accuracy changes significantly, exhibiting differences of up to 9 accuracy points." The difference came from distracting passages, which look relevant but pull the model toward a wrong answer.

Measure the answer as well as the retrieval. Score answer correctness and check that each citation supports its claim. Track distractors in the context window, not only relevant hits.

Extra Checks for Live Web Retrieval

For agents on the live web, a relevant link fails if the page cannot be fetched, renders only after JavaScript runs, or returns truncated text. The label says relevant, but the model receives nothing useful.

Add these checks next to relevance metrics:

• Fetch success rate, the share of top-k URLs that return usable content
• Extraction completeness, whether the main text arrives without truncation or page clutter
• Freshness, since labels for time-sensitive queries expire and need re-judging on a schedule
• Latency, measured per query at the k your agent reads

The guide to evaluating AI web search APIs groups these checks into four layers. The layers are retrieval quality, content and extraction quality, answer and citation accuracy, and freshness and operations.

A Simple Search Relevance Evaluation Workflow

1. Define who consumes the results, then pick one primary metric and one guardrail.
2. Sample real queries across head and tail traffic, aiming for a broad set that represents your users.
3. Collect the ranked results and the page content each system returned.
4. Label a sample with human raters, then scale with an LLM judge calibrated against them.
5. Compute metrics at the k your consumer actually reads.
6. Compare runs with a paired significance test and per-query wins and losses.
7. Confirm the winner with an A/B test or interleaving where traffic allows.
8. Refresh labels on a schedule, and sooner for time-sensitive queries.

FAQ

What is a good nDCG score?

No universal threshold exists, because nDCG depends on the dataset, labels, cutoff k, and implementation. Compare your score against a baseline ranker on the same query set and labels.

What is the difference between precision@k and recall@k?

Precision@k is the share of the top k results that are relevant. Recall@k is the share of all relevant documents that appear in the top k.

When should I use MRR instead of nDCG?

Use MRR when one correct result is enough, as with navigational or single-answer queries. Use nDCG when several results matter and their quality grades differ.

What is the difference between nDCG and MAP?

MAP uses binary labels and treats every relevant result the same. nDCG uses graded labels with a log discount, so a perfect match at rank 1 earns more than a partial match.

How many queries do I need for search evaluation?

No universal number exists, so sample real user queries; OpenSearch's search quality guide recommends "a broad and representative set of queries." Add queries if a significance test cannot separate two rankers.

Can I use clicks as relevance labels?

Yes, with care, because clicks are biased by position and presentation. Use interleaving or click-debiasing methods before treating clicks as labels.

Is LLM-as-a-judge reliable for search relevance?

LLM judges help at scale and agreed with human assessors on average in TREC 2024 RAG, but were weaker at separating top systems. Calibrate the judge on a human-labeled sample and keep people on close decisions.

How do I evaluate retrieval in a RAG pipeline?

Measure recall@k against gold sources and precision@k to limit distractors. Then score the final answer for correctness and check that each citation supports its claim.