You give an OpenAI agent live web data by adding a tool that searches, reads, and returns current web content while the agent runs. That tool is either OpenAI's built-in web search tool or a dedicated web data API connected as a function tool or an Olostep MCP server.
"Live web data" means information pulled from the internet at request time, not from the model's memory. An "OpenAI agent" here means an agent built with the OpenAI Agents SDK or the Responses API, which can call tools to do work.
This matters because the web is now largely accessed by machines. According to the Thales 2026 Bad Bot Report, bots made up more than 53% of all web traffic in 2025, up from 51% the previous year, while human activity fell to 47%. Agents are becoming a normal client of the live web, and they need a reliable way to reach it.
The full job has four parts: find the right pages, read them, turn them into clean data, and keep that data fresh. This guide covers each part and shows how to wire it into an OpenAI agent.
Why an LLM's Training Data Goes Stale
An LLM's knowledge is frozen at its training cutoff, so it cannot answer questions about events after that date. Ask about this week's news and the model either declines or guesses.
Grounding the model in retrieved web data is the standard fix. In one peer-reviewed study, a retrieval-augmented approach called Hyper-RAG improved accuracy by an average of 12.3% over direct LLM use, measured on a medical dataset across six models. The size of the gain depends on the domain, but the direction is consistent: fresh, retrieved evidence helps.
Live web data closes this gap directly. The agent fetches current facts at runtime instead of relying on what the model already learned.
Search, Scrape, and Structured Data: Three Kinds of Web Data
Web data for agents comes in three forms, and each answers a different need. Knowing which one you need decides how you set the agent up.
- Search: finds relevant pages or returns a synthesized answer to a question.
- Scrape: reads one full page and returns its content.
- Structured extraction: returns specific fields as typed JSON, ready to store or pass downstream.
Built-in search tools usually return short snippets and citations, not full pages. When an agent needs to compare sources, quote precisely, or fill a database, it often needs the full page or a JSON schema instead. Converting messy HTML into token-dense LLM-ready data is what makes that content usable inside a model's limited context window.
The Two Main Ways to Give an OpenAI Agent Live Web Data
There are two practical ways to connect an OpenAI agent to the live web. You can use OpenAI's built-in web search tool, or you can bind a dedicated web data API as a custom function tool or MCP server.
The tradeoff is simple to state. The built-in tool is the fastest way to start, while a dedicated API gives you more control over content, structure, cost, and portability across frameworks.
Option 1 — OpenAI's Built-In Web Search Tool
OpenAI's built-in web search tool is native to the Agents SDK and Responses API, so you enable it by adding it to the agent's tool list. It supports three modes: non-reasoning search for quick lookups, agentic search that plans multi-step queries, and deep research for longer investigations.
It works well for general questions and quick starts. As OpenAI's web search guide notes, the search context window is limited to 128k, even when the model context window is larger, so it is tuned for focused lookups rather than large-scale reading.
Option 2 — A Dedicated Web Data API or MCP Server
A dedicated web data API returns full page content as clean Markdown or JSON, crawls whole sites, and can watch pages for changes over time. You connect it either as a custom function tool inside the Agents SDK or through an Olostep MCP server that any MCP client can call.
This path suits production agents that need full content, structured output, or data that stays current. It also keeps your integration portable across frameworks.
How to Add OpenAI's Built-In Web Search Tool
Adding the built-in tool takes one change to your agent definition. The steps below use the Python Agents SDK and the Responses API.
Enable Web Search in the Agents SDK or Responses API
In the Agents SDK, import WebSearchTool and add it to the agent's tools list. The SDK generates the search queries and runs them for you, then returns an answer with cited sources.
from agents import Agent, Runner, WebSearchTool
agent = Agent(
name="Research assistant",
instructions="Answer using current web sources and cite them.",
tools=[WebSearchTool()],
)
result = Runner.run_sync(agent, "What did OpenAI announce this week?")
print(result.final_output)With the Responses API, you add a web_search object to the tools array in the request body.
{
"model": "gpt-5",
"tools": [{ "type": "web_search" }],
"input": "Summarize today's top AI infrastructure news, with sources."
}Configure Domain Filters, Context Size, and Location
The tool exposes a few controls you can set directly on it. You can allow or block specific domains, set how much search content is passed back, and pass an approximate location for local results.
{
"type": "web_search",
"search_context_size": "medium",
"filters": { "allowed_domains": ["openai.com", "arxiv.org"] },
"user_location": { "type": "approximate", "country": "US" }
}Per OpenAI's web search guide, filters accepts up to 100 allowed or blocked domains, and search_context_size takes low, medium, or high. These knobs help, but the underlying index and ranking stay inside OpenAI's implementation.
Know the Limits: Context Cap, Cost, and No Crawl or Extraction
The built-in tool has four documented limits worth planning around. None of them make it a bad choice; they just mark where a dedicated API takes over.
- Context cap: search content is capped at a 128K window even on models with larger context.
- Cost at volume: OpenAI's published pricing lists web search (all models) at $10.00 / 1k calls, plus search content tokens billed at model rates (as of September 2026).
- Snippets, not pages: it returns citations and short extracts, not full page content or schema-based JSON.
- Portability: results are tied to OpenAI endpoints, so the same setup does not move to other models or frameworks.
How to Connect a Dedicated Web Data API for Fuller Control
A dedicated web data API fills the gaps above. It returns full content, structured JSON, whole-site crawls, and scheduled monitoring, and it works across frameworks.
The sections below map each need to a specific endpoint and show how to bind them to an OpenAI agent.
Get Grounded, Cited Answers from the Live Web
A web search and answers endpoint takes a natural-language question and returns a synthesized answer with the sources it used. Olostep's web search API for grounded answers also returns NOT_FOUND for low-confidence fields instead of a fabricated guess.
This grounds an agent in one request, without wiring together a search API, a scraper, and a separate LLM call. You can also request the answer in a JSON schema you define.
Scrape Full Pages into Clean, LLM-Ready Data
When the agent needs a full page rather than a snippet, a scrape endpoint reads the page and returns it as clean Markdown or JSON. Olostep's scrape endpoint renders JavaScript and strips page clutter, so token-dense content fits inside the model's context budget.
This is what makes the earlier 128K limit manageable: you decide what enters the window. Clean Markdown carries more meaning per token than raw HTML.
Crawl Whole Sites and Monitor Pages for Changes
Some tasks need more than one page or need data that changes over time. To read many pages, you can crawl an entire site and collect its content at scale.
For data that updates, you can monitor pages for changes on a schedule and alert the agent when something is new. Recurring monitoring is the piece most agent tutorials skip, yet it is what keeps a pricing, news, or compliance agent accurate between runs.
Connect via MCP or a Custom Function Tool
You have two ways to bind a web data API to an OpenAI agent. The first is a custom function tool that calls the API from inside the Agents SDK.
import os, requests
from agents import Agent, function_tool
@function_tool
def search_web(question: str) -> str:
"""Search the live web and return a grounded, cited answer."""
resp = requests.post(
"https://api.olostep.com/v1/answers",
headers={"Authorization": f"Bearer {os.environ['OLOSTEP_API_KEY']}"},
json={"task": question},
)
return resp.text
agent = Agent(
name="Grounded agent",
instructions="Use search_web for anything time-sensitive. Always cite sources.",
tools=[search_web],
)The second is the Model Context Protocol. Register an Olostep MCP server once, and any MCP client, such as the Agents SDK, LlamaIndex, LangChain, Claude Code, or Cursor, can call the same web tools.
{
"mcpServers": {
"olostep-web": {
"command": "npx",
"args": ["-y", "olostep-mcp"],
"env": { "OLOSTEP_API_KEY": "YOUR_API_KEY" }
}
}
}Built-In Tool vs Dedicated Web Data API: How to Choose
Both approaches give an agent live web data, so the choice comes down to how much control and structure you need. The table compares them across the same criteria.
| Criterion | Built-In Web Search Tool | Dedicated Web Data API |
|---|---|---|
| Setup effort | Lowest — one line in tools | Low — function tool or MCP server |
| Output | Snippets and citations | Full Markdown, HTML, or schema JSON |
| Crawl and extraction | Not available | Crawl, batch, and structured extraction |
| Context control | Fixed 128K search window | You choose what enters the window |
| Transparency | Index and ranking are internal | Inspect queries, results, and sources |
| Ongoing data | Per-request only | Scheduled monitors for changes |
| Portability | OpenAI endpoints only | Any tool-calling model or framework |
Use the built-in tool when you want quick, general lookups with minimal code. Add a dedicated API when you need full page content, structured JSON, whole-site crawls, ongoing monitoring, tighter cost control, or portability across frameworks. Many teams use both and route each query to the right one.
A Practical Setup: Search, Read, Structure, and Monitor
A production agent usually needs the full lifecycle, not a single call. Take an agent that tracks a competitor's pricing and announcements as an example.
- Search: find the competitor's current pricing and news pages.
- Read: scrape those pages into clean Markdown so the model can reason over the full text.
- Structure: extract the fields you care about, such as plan names and prices, into JSON.
- Monitor: schedule a monitor so the agent is alerted when a price or announcement changes.
This same pattern powers recurring deep research workflows, where an agent gathers, cleans, and updates data across many sources on a schedule. Building the lifecycle once, behind one API, keeps the agent's data both complete and current.
Cost and Reliability Considerations
Estimate cost by call volume before you ship. OpenAI's published pricing lists web search at $10.00 / 1k calls plus content tokens, so an agent running 10,000 searches a day carries a meaningful daily fee before token costs. A dedicated API can shift that math when you need full content or high volume.
Reliability is the other factor. Dynamic, JavaScript-heavy, and anti-bot-protected sites are hard to fetch consistently, which is why managed web-data infrastructure exists. Interest in this space is climbing: by one market forecast, the AI agents market is projected to grow to USD 182.9 billion by 2033, at a CAGR of 49.6% from 2026 to 2033.
Whichever path you choose, keep requests within each site's terms and rate limits. Responsible access keeps your agent's data pipeline durable.
Frequently Asked Questions
Can AI agents browse the web?
Yes. When you give an agent a web search or scraping tool, it can query the live web, read pages, and use the results during a run.
Where do OpenAI agents get their live data?
They get it from a tool you add, either OpenAI's built-in web search tool or a dedicated web data API connected as a function tool or MCP server.
Does OpenAI have a browser agent?
OpenAI offers built-in web search and deep research modes in the Agents SDK and Responses API, which let an agent search and read pages during a task.
How much does OpenAI's web search tool cost?
OpenAI's published pricing lists web search at $10.00 per 1,000 calls plus search content tokens billed at model rates, as of September 2026; always check the live pricing page.
Can I use both the built-in tool and a custom web search tool?
Yes. You can register both on one agent and use instructions to route each query to the tool that fits it best.
Do built-in search tools return full page content?
Usually not; they return short snippets and citations, so use a scrape or extraction endpoint when the agent needs the full page or structured JSON.