How to Build a Web Search Chatbot With Reliable Citations

A chatbot that searches the web is easy to demo. A chatbot whose citations hold up when a user clicks them takes more work. This guide walks through the full pipeline in Python, including a step that is easy to skip: checking each citation before the user sees it.

What a Web Search Chatbot That Cites Its Sources Does

To build a web search chatbot that cites its sources, you search the web for each question, read the top pages as clean text, and give the model numbered passages. The model must cite a passage number after every claim. Your code then checks each citation against the passage before the answer is shown.

A citation is a pointer from one claim to the exact passage that supports it. A list of links at the bottom of an answer is not the same thing. A source list tells users where the model looked, while a citation tells them where a specific sentence came from.

That difference matters because citation errors are common in real products. In a Tow Center AI search study of eight AI search tools asked to identify news articles from excerpts in February 2025, the researchers reported: "Collectively, they provided incorrect answers to more than 60 percent of queries." The task was narrow, but it shows that adding web search does not make citations reliable on its own.

Why Chatbot Citations Break

Chatbot citations break in three ways: the URL is invented or broken, the claim is credited to the wrong source, or a real source sits next to a claim it does not support. Each failure needs a different fix, so it helps to name them clearly.

• Fabricated or broken URLs: The model writes a link from memory that leads to an error page or never existed.
• Wrong-source attribution: The fact may be correct, but it came from a different page than the one cited.
• Unsupported claims: The cited page is real and relevant, but it does not say what the sentence claims.

Models without retrieval are weak at naming sources. On the CiteME citation attribution benchmark, which asks models to identify the paper behind a claim in machine learning research, the authors reported "LMs achieving only 4.2-18.5% accuracy and humans 69.7%." Giving the model search and reading tools helped, but it still trailed humans.

Retrieval reduces the problem without removing it. A RAG faithfulness detection paper accepted at ICLR 2026 states that "in practice, grounding does not eliminate unfaithfulness." Models can still contradict the text they were given or add details it does not contain.

The BBC/EBU news integrity toolkit, an audit of AI assistants published in October 2025, reports "45% of responses having a significant issue – something which could materially mislead the user – of some form." That figure covers all kinds of issues in news answers, not only citation errors. A link that opens is not proof that a citation is correct.

The Architecture: Five Stages From Question to Cited Answer

A web search chatbot that cites its sources runs five stages for every user message: rewrite the query, search, read, generate, and verify. Each stage has one job and one common failure.

You often do not need a vector database for live web chat. Each question fetches fresh pages, so you can rank passages from those pages in memory and throw them away after the answer. A vector database makes more sense when you index a fixed set of documents, such as your own help center.

Step-by-Step: Build the Chatbot in Python

The code below builds the five stages with Olostep for search and reading, plus any chat model that returns JSON. Each step is short so you can swap pieces later.

Prerequisites:

• Python 3.10 or newer, plus the requests and openai packages
• An Olostep API key, set as OLOSTEP_API_KEY
• An LLM API key, set as OPENAI_API_KEY (any provider with JSON output works)

import os, json, re, requests
from openai import OpenAI

OLOSTEP = "https://api.olostep.com/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['OLOSTEP_API_KEY']}"}
llm = OpenAI()
MODEL = "gpt-4.1-mini"

def ask_json(system: str, user: str) -> dict:
    resp = llm.chat.completions.create(
        model=MODEL,
        response_format={"type": "json_object"},
        messages=[{"role": "system", "content": system},
                  {"role": "user", "content": user}],
    )
    return json.loads(resp.choices[0].message.content)

Step 1: Rewrite the User's Question Into a Search Query

The first step turns the latest message into a search query that makes sense on its own. A follow-up like "what about their pricing?" means nothing to a search engine without the earlier messages.

def rewrite_query(history: list[dict], message: str) -> str:
    convo = "\n".join(f"{m['role']}: {m['content']}" for m in history[-6:])
    out = ask_json(
        "Rewrite the user's last message as one standalone web search query. "
        'Return JSON: {"query": "..."}',
        f"Conversation:\n{convo}\n\nLast message: {message}",
    )
    return out["query"]

Keep only the last few turns. Longer histories usually add tokens without improving the query.

Step 2: Search the Web for Candidate Sources

The search step returns a ranked list of URLs that might answer the query. The Olostep web search API takes a natural language query and returns deduplicated links with titles and descriptions.

def search(query: str, limit: int = 6) -> list[dict]:
    body = {
        "query": query,
        "limit": limit,
        "exclude_domains": ["pinterest.com"],
        "scrape_options": {"formats": ["markdown"]},
    }
    r = requests.post(f"{OLOSTEP}/searches", headers=HEADERS, json=body, timeout=60)
    r.raise_for_status()
    return r.json()["result"]["links"]

Use include_domains and exclude_domains to control source quality. A medical or legal chatbot, for example, can restrict results to official and academic domains.

Step 3: Read Each Page as Clean Markdown

The reading step gets the full text of each page so the model cites what it actually read. Search snippets are often one or two sentences, which is too little to support most claims and too short to quote back to users.

Passing scrape_options in Step 2 tells Olostep to embed markdown_content for each link in the same response. If you already have URLs, the scrape endpoint for clean Markdown does the same for one page at a time.

def read_pages(links: list[dict]) -> list[dict]:
    pages = []
    for link in links:
        text = link.get("markdown_content")
        if not text:  # scrape failed, timed out, or page was empty
            continue
        pages.append({"url": link["url"], "title": link.get("title") or link["url"], "text": text})
    return pages

Markdown matters because LLM-ready web data strips the navigation, scripts, and footers that fill raw HTML. In one example from Cloudflare Markdown for Agents, a single blog post took "16,180 tokens in HTML and 3,150 tokens when converted to markdown." Savings vary by page.

Some pages load their content with JavaScript, so they must be rendered in a browser before the text exists. Olostep's scrape endpoint is built to handle JavaScript-heavy sites. When a page still returns no content, skip it rather than citing a page your system never read.

Step 4: Split Pages Into Numbered Passages

Passages are short, numbered sections of text that the model can cite by ID. Numbering makes each citation point to a specific quote instead of a whole page.

def make_passages(pages: list[dict], query: str, top_n: int = 12) -> list[dict]:
    terms = set(re.findall(r"\w+", query.lower()))
    chunks = []
    for page in pages:
        for block in re.split(r"\n(?=#)|\n\n", page["text"]):
            block = block.strip()
            if 200 <= len(block) <= 1500:
                score = len(terms & set(re.findall(r"\w+", block.lower())))
                chunks.append({**page, "text": block, "score": score})
    chunks.sort(key=lambda c: c["score"], reverse=True)
    return [{"id": i + 1, **c} for i, c in enumerate(chunks[:top_n])]

This version ranks passages by keyword overlap, which is fast and needs no extra service. You can replace it with an embedding or reranker model later without changing the other steps.

Passage size is a trade-off. Large passages give the model more context, while small passages give users more exact quotes. Splitting on headings and paragraphs keeps each passage close to one idea.

Step 5: Generate an Answer That Cites Passage IDs

The generation step asks the model to answer only from the passages and to cite a passage ID for every claim. The prompt also gives the model permission to say it does not know.

SYSTEM = """Answer using ONLY the numbered passages.
Split the answer into short claims. Each claim must cite one or more passage ids.
Copy a short exact quote (under 25 words) from each cited passage that supports the claim.
If the passages do not answer the question, return an empty claims list.
Return JSON: {"claims": [{"text": "...", "passage_ids": [1], "quote": "..."}]}"""

def generate(question: str, passages: list[dict]) -> list[dict]:
    context = "\n\n".join(f"[{p['id']}] ({p['url']})\n{p['text']}" for p in passages)
    return ask_json(SYSTEM, f"Question: {question}\n\nPassages:\n{context}")["claims"]

Asking for a quote with each claim gives the next step something it can check. It also gives users the exact words behind each sentence.

Some model providers offer built-in citation features. The Claude Citations API docs state that "citations are guaranteed to contain valid pointers to the provided documents." A valid pointer shows where a passage is, but it does not prove the passage supports the claim, so the next step still applies.

Step 6: Verify Every Citation Before Showing It

Verification checks each claim against its cited passage and removes claims that fail. This step separates checked citations from a plain list of links.

The checks run from cheapest to most expensive:

1. ID check: Every cited passage ID must exist in the passages you sent.
2. Quote check: The quote must appear in the cited passage, allowing for small whitespace differences.
3. Support check: A second model call judges whether the passage supports the claim.

def norm(s: str) -> str:
    return re.sub(r"\s+", " ", s).strip().lower()

JUDGE = """Does the passage fully support the claim? Be strict.
Return JSON: {"supported": true|false, "reason": "..."}"""

def verify(claims: list[dict], passages: list[dict]) -> list[dict]:
    by_id = {p["id"]: p for p in passages}
    approved = []
    for c in claims:
        cited = [by_id[i] for i in c.get("passage_ids", []) if i in by_id]
        if not cited:
            continue  # ID check failed
        if not any(norm(c.get("quote", "")) in norm(p["text"]) for p in cited):
            continue  # quote check failed
        source = "\n\n".join(p["text"] for p in cited)
        verdict = ask_json(JUDGE, f"Claim: {c['text']}\n\nPassage:\n{source}")
        if verdict.get("supported"):
            approved.append({**c, "sources": [{"id": p["id"], "url": p["url"], "title": p["title"]} for p in cited]})
    return approved

Dropping a failed claim is the safest default. In some products you may prefer to show the claim with a warning label, so users can see what the system could not confirm. The judge model can also make mistakes, which is why the quote check runs first: it catches the clearest failures without any model call.

The full pipeline fits in one function:

def answer(history: list[dict], message: str) -> dict:
    query = rewrite_query(history, message)
    pages = read_pages(search(query))
    passages = make_passages(pages, query)
    if not passages:
        return {"answer": "I couldn't find reliable sources for that.", "claims": []}
    claims = verify(generate(message, passages), passages)
    if not claims:
        return {"answer": "The sources I found don't clearly answer that.", "claims": []}
    return {"answer": " ".join(f"{c['text']} [{','.join(map(str, c['passage_ids']))}]" for c in claims),
            "claims": claims}

Step 7: Render Citations in the Chat Interface

The interface shows each claim with a small numbered marker that opens its source. A good source card shows the page title, the domain, and the exact quote, so users can check the claim without leaving the chat.

A verified response looks like this:

{
  "answer": "Olostep's search endpoint can return page Markdown inline [2].",
  "claims": [
    {
      "text": "Olostep's search endpoint can return page Markdown inline",
      "passage_ids": [2],
      "quote": "pass scrape_options to also scrape every returned URL",
      "sources": [{"id": 2, "url": "https://docs.olostep.com/api-reference/searches/create", "title": "Create Search"}]
    }
  ]
}

Showing the quote on hover lets users spot a wrong source quickly, without opening each link. For a deeper look at the response shape, see how to build agents that return structured citations.

The Faster Path: One Call to an Answers Endpoint

If you do not need control over each stage, one API call can search, read, and answer for you. The Olostep Answers endpoint takes a task and an optional JSON shape, then returns structured data along with the source URLs it used.

r = requests.post(f"{OLOSTEP}/answers", headers=HEADERS, json={
    "task": "What is the latest stable Python release and when was it published?",
    "json_format": {"version": "", "release_date": ""},
}, timeout=60)
print(r.json()["result"])

When a field cannot be found on the web, the endpoint returns NOT_FOUND instead of a guess. Olostep's docs list execution time as 3 to 30 seconds, depending on how many pages the task needs.

You can start with the Answers endpoint to test demand, then move to the DIY pipeline when users need quote-level citations.

Failure Modes and How to Fix Them

Common citation bugs fall into a few patterns, and each has a specific fix. Use this table as a checklist when testing.

Respect each site's terms, robots.txt rules, and rate limits when you fetch pages. A citation pipeline should only read content it is allowed to access.

How to Measure Citation Quality

You measure citation quality by running a fixed set of test questions and scoring each verified answer. Re-run the same set after every prompt, model, or retrieval change so you can see what got better or worse.

Start with 30 to 50 questions that match real user traffic, including a few that have no good answer. Track these metrics:

• Citation precision: The share of shown claims whose cited passage actually supports them. Check a sample by hand.
• Unsupported-claim rate: The share of generated claims that verification removed. A rising rate points to a retrieval or prompt problem.
• Correct refusals: How often the bot says "I don't know" on questions with no good sources.
• Latency: Median and 95th-percentile time per answer, split by stage.
• Cost per answer: Search and scrape calls plus model tokens for generation and verification.

Splitting latency by stage shows where to spend effort. Fetching pages in parallel and streaming the answer while verification runs are two fixes to try first.

Connect the Chatbot to Agents and Frameworks

The same search and read steps can run as tools inside an agent instead of a fixed pipeline. The agent decides when to search, and your code still verifies the citations at the end.

The Olostep MCP server exposes search and scraping to clients that support the Model Context Protocol, so you don't have to write HTTP calls by hand. If your stack already uses LangChain, the Olostep LangChain integration gives an agent the same tools. In both cases, keep Step 6 outside the agent so every answer passes the same checks.

Frequently Asked Questions

Do I Need a Vector Database to Build a Web Search Chatbot?

No, because each question fetches fresh pages that you can rank in memory and discard after answering. A vector database helps when you index a fixed document set, such as your own docs.

Why Does My Chatbot Cite Sources That Don't Support Its Answer?

The model often blends what it already knows with the retrieved text, then attaches the nearest source. Requiring an exact quote per claim and checking it against the passage catches many of these cases.

Can a Chatbot Guarantee Accurate Citations?

No system can guarantee it, because both the generator and the judge model can make mistakes. Verification, exact quotes, and regular testing lower the error rate and make the remaining errors easy for users to spot.

How Do I Handle Follow-Up Questions?

Rewrite each follow-up into a standalone search query using the last few chat turns. Then run the full search, read, and verify pipeline again.

Should I Cite Search Snippets or Full Pages?

Cite full pages, because snippets are too short to support most claims. Read each page as Markdown and cite the exact passage you used.

How Do I Handle JavaScript-Heavy Pages and PDFs?

Use a scraper that renders JavaScript and supports PDFs, and skip any page that returns no usable text. Never cite a page your system did not actually read.