How to Evaluate AI Web Search APIs: A Practical Framework

To evaluate web search for AI, test whether a search tool helps your model give correct, cited, and timely answers to your own queries. Then measure the latency and cost of each correct answer.

A web search API is a service that takes a query and returns results as structured data, such as JSON, for software to read. A human search engine returns a page of links for a person to scan and judge.

That difference changes the test. A person skips a bad link and tries the next one, while a model reads whatever the API returns and builds an answer from it. LLM web search (a large language model calling a search tool) works only if the right text reaches the model in a usable form.

Treat web search for AI as a pipeline with five steps: find, fetch, extract, answer, and cite. A failure at any step breaks the final answer, even when the earlier steps worked. When you test an AI search API, such as the Olostep web search API or any other provider, score each step on its own.

Why Ranked Links Alone Are the Wrong Test

Giving a model search access does not guarantee good answers. In OpenAI's BrowseComp results, "Enabling browsing for GPT‑4o led to a modest improvement in accuracy (from 0.6% to 1.9%), but performance remained low."

BrowseComp is a deliberately hard set of multi-hop problems. A multi-hop question needs facts from several pages combined. Those numbers describe GPT-4o with browsing on that set only, and they show that access alone does little when the search strategy is weak.

Retrieval design, by contrast, can move accuracy a lot. In the FRAMES RAG benchmark paper, the paper's authors reported "…achieving 0.40 accuracy with no retrieval… an accuracy of 0.66 (>50% improvement)." RAG, or retrieval-augmented generation, means the model answers using documents fetched at query time.

That gain came from a multi-step retrieval pipeline the authors built for the paper. It is evidence that retrieval strategy matters. It is not a score for generic web search.

A provider can also return relevant links and still fail the task. The page may block the fetch, load its text with JavaScript, or come back cut off, so the model never sees the answer. The risk grows in agentic search, where an agent plans, searches, reads, and searches again in a loop, as covered in agentic search explained.

The Five Layers to Score

Score web search for AI agents across five layers: retrieval, content, answer and citation, freshness and operations, and cost per resolved task. Each layer catches a failure the others miss.

Layer 1: Retrieval Quality

Retrieval quality asks one question: did the results include a page that holds the answer? Grade it against a gold set, which is a list of known-good source URLs you record for each test query.

The core metric is recall@k. It is the share of queries where at least one correct source appears in the top k results, such as the top 5. Also track relevance rate, the share of returned results a grader marks as on topic.

def recall_at_k(results, gold_urls, k=5):
    """results: {query_id: [url, ...]}, gold_urls: {query_id: {url, ...}}"""
    hits = 0
    for query_id, urls in results.items():
        if any(url in gold_urls[query_id] for url in urls[:k]):
            hits += 1
    return hits / len(results)

Normalize URLs before you compare them by removing tracking parameters, trailing slashes, and "www." Accept more than one gold source per query, because the same fact often appears on several pages.

A SERP API returns what a search engine results page shows, usually titles, URLs, and short snippets. An AI search API is built for models and may return longer page text with each result. Both can be scored on recall@k, which makes it a fair first comparison.

Layer 2: Content and Extraction Quality

Content quality asks whether the model received the full, clean text of the right page. A perfect URL is useless if the fetch fails or the text arrives cut off.

Three page types cause frequent trouble. JavaScript-heavy pages build their text in the browser, PDFs need a separate parser, and snippets (short excerpts) often stop before the answer. Score this layer with four metrics:

• Fetch success rate: The share of result URLs that return usable content instead of an error, a block, or an empty page.
• Extraction completeness: The share of a page's main text that reaches the model. Check it against hand-copied versions of 20-30 pages.
• Token efficiency: The share of tokens that are main content instead of menus, ads, and footers. A token is the unit of text a model reads and bills by.
• Field-level accuracy: For structured output, the share of JSON fields that match the gold values.

Test snippets and full pages as separate settings. Simple facts often fit in a snippet, but multi-hop and numeric questions usually need the whole page. If a provider returns only snippets, add a fetch step that can scrape full page content as Markdown, and score it too.

Layer 3: Answer and Citation Accuracy

Answer accuracy asks whether the final answer matches the gold answer. Citation accuracy asks a separate question: does each cited page actually support the claim attached to it?

These two scores can split apart. In a Tow Center study of eight AI search tools, "Collectively, they provided incorrect answers to more than 60 percent of queries." The task in those February 2025 tests was narrow: given an excerpt from a news article, identify the source article and its URL.

That study measured source attribution, so it shows why citations need their own test. Use this method to check them:

1. Split each answer into claims, and pair each claim with the URL cited for it.
2. Fetch every cited page and extract its main text.
3. Ask a grader whether the page text supports the claim.
4. Compute citation precision: supported citations divided by total citations.

Also score abstention, which is when the system says it cannot find an answer. A "not found" is safer than a confident wrong answer, so give correct answers +1, abstentions 0, and wrong answers -1.

Some APIs expose these signals directly. Olostep's answers endpoint with citations returns a sources array of the URLs used. When a field can't be verified from live web sources, it returns "NOT_FOUND" instead of a guess.

Layer 4: Freshness and Operations

Freshness asks how quickly new information shows up in search results. Measure it with a dated test instead of trusting a provider's claim:

1. Pick 20-30 recent events with a known publish time, such as a product launch or an earnings report.
2. Write one query per event that only a new page can answer.
3. Run the queries daily, or hourly for news, and log when a correct source first appears.
4. Report index lag, the median time from publication to first appearance.

Our glossary entry on how fresh search data is covers this dimension in more depth. Rotate in new events each time you re-run, because last quarter's fresh questions are now old.

Operations asks whether the service stays fast and stable under real load. Log p50 latency, the median response time, and p95 latency, the time within which 95% of requests finish. The p95 number matters for agents, because one slow call stalls every step after it.

Also track error rate, timeout rate, and rate limits, which are caps on requests per second or minute. Run the load test at the concurrency you expect in production. A provider that is quick at one request at a time may slow down at 50.

Layer 5: Cost per Resolved Task

Cost per resolved task is total spend divided by the number of queries answered correctly. Price per call misleads, because a cheap call that needs retries and extra fetches can cost more per correct answer.

cost per resolved task =
    (search cost + page fetch cost + model token cost + retry cost)
    / number of correctly answered queries

Model tokens belong in the formula. A provider that returns long, noisy pages raises token spend, which links this layer back to token efficiency in Layer 2.

Check each provider's docs for how it bills failed or empty requests. If failures are free, failed attempts add only model and retry costs. If they are billed, they inflate cost per resolved task.

How to Run Your Own Evaluation, Step by Step

This vendor-neutral process takes a small team a week or less. It has four steps: shortlist providers, build a test set, run and grade, then weight the scores.

Step 1: Shortlist Providers and Lock the Harness

Pick three or four providers that cover different types. Common types are SERP APIs, AI-native search APIs, and the built-in search some model providers offer. Our post comparing web search APIs can help you build the list.

A harness is the fixed test setup that runs every query the same way. Keep the model, prompt, number of results, and fetch settings identical across providers. Configure each provider as its docs recommend, and write down every setting.

Fair setup matters most in head-to-head tests such as Exa vs Tavily, where default settings can favor one tool. If you publish results, state any ties you have to the vendors you tested.

Step 2: Build a Test Set From Real Queries

Build a test set of 100-200 queries pulled from your logs or expected user tasks. For each query, record a gold answer and at least one gold source URL. Mix four kinds of queries:

• Simple facts: Single-answer questions, such as a company's founding year. These test basic retrieval.
• Multi-hop: Questions that need facts from two or more pages combined.
• Fresh: Questions about events after a known date. These feed the Layer 4 test.
• Hard pages: Answers that live on JavaScript-heavy pages, PDFs, or long documents. These test Layer 2.

Run the model once with search turned off. Queries it already answers from memory tell you little about search, so cut them or give them less weight.

Public sets can add a hard baseline. According to OpenAI's SimpleQA benchmark announcement, "SimpleQA was created to be a greater challenge for frontier models (e.g., GPT‑4o scores less than 40%)." That score is GPT-4o on a question set built to be hard on purpose, so it is not a general hallucination rate.

A hallucination is a confident answer that is false. Use SimpleQA-style questions to check difficulty, and keep most of your set grounded in real queries.

Step 3: Run, Grade, and Calibrate the Judge

Run every query three times per provider, because search results and model outputs change between runs. Report the mean score and the spread across runs.

At 150 queries, three runs, and four providers, you have 1,800 pipeline runs to manage. Automate them with a job queue or an API that can batch-run evaluation queries, so every run uses the same settings.

Grade answers with an LLM judge, which is a model that scores each answer against a written rubric. Using an LLM as a judge is fast, but its grades need checking. Have a person review about 10% of graded items, chosen at random, and rewrite the rubric if they often disagree with the judge.

Store each graded item as a record you can audit later. A sample record looks like this:

{
  "query_id": "q_042",
  "provider": "provider_b",
  "run": 2,
  "gold_answer": "2019",
  "model_answer": "2019",
  "answer_correct": true,
  "cited_urls": ["https://example.com/about"],
  "citations_supported": 1,
  "citations_total": 1,
  "failure_class": null,
  "latency_ms": 2140
}

Label every failure with one class, so you know which layer to fix:

• No results means the search returned nothing useful for the query.
• Fetch or extraction failure means the right URL appeared, but its text never reached the model.
• Wrong answer means the model had the right text but answered incorrectly.
• Bad citation means the answer is right, but a cited page does not support it.

Treat small gaps with care. At 150 queries and 70% accuracy, the 95% margin of error is about ±7 percentage points. A 3-point lead between providers may be noise.

Step 4: Weight the Scorecard by Use Case

Weight each layer by what your product needs, then compute one weighted score per provider. The weights below are starting points to adjust for your own product.

Chatbot grounding gives weight to latency, because users wait on each reply. Deep research leans on retrieval and citations across many pages. Data enrichment depends on clean field extraction and unit cost at volume, while monitoring depends on how fast changes appear.

Re-run the evaluation every quarter. Also re-run it when a provider changes its index, model, or pricing, and version your test set so scores stay comparable over time.

Frequently Asked Questions

What is a good web search tool for an AI agent?

A good web search tool for an AI agent returns the answer page, delivers its full clean text, and stays fast under load. The right choice depends on your use case, so test a shortlist on your own queries.

How many test queries do I need to compare search APIs?

Use 100-200 real queries for a first comparison, and run each one three times per provider. Smaller sets work for a quick screen, but gaps of a few points will fall inside the margin of error.

Should I evaluate search results or the agent's final answer?

Evaluate both, because good search results can still lead to a bad final answer. Score retrieval, extraction, and answers separately so you can tell where each failure happened.

Are SimpleQA, FRAMES, and BrowseComp enough to choose a provider?

No, they are sanity checks that test models and research systems on fixed question sets. Your own queries, page types, and latency limits decide which provider fits your product.

How do I measure the freshness of a search API?

Query recent events with known publish times each day, and log when a correct source first appears. The median gap between publication and first appearance is the provider's index lag.

Can I trust an LLM judge for search evaluations?

Yes, once you calibrate it against human review of about 10% of graded items. If people and the judge often disagree, fix the rubric and re-grade before you trust the scores.

How often should I re-run the evaluation?

Re-run it every quarter and whenever a provider changes its index, model, or pricing. Refresh the fresh-query slice each time, since old "fresh" questions go stale.