Information retrieval (IR) is the process of finding the documents or passages in a large collection that best satisfy an information need. It then ranks them by relevance. You type a query, the system matches it against an index, and it returns a ranked list of the most useful results.
The standard academic definition comes from the Stanford IR textbook by Manning, Raghavan and Schütze: "Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers)." In everyday use, "search" and "information retrieval" mean almost the same thing.
A few terms come up again and again:
- Information need: the thing you actually want to know, such as how to reset a router.
- Query: the words you give the system to express that need.
- Document: any unit the system can return, such as a web page, a PDF or a text chunk.
- Collection (corpus): the full set of documents the system can search.
- Relevance: how well a document satisfies the information need, not just whether it contains the query words.
How Does Information Retrieval Work?
An IR system works in five stages: acquire documents, represent them, index them, match and rank them against a query, and evaluate the results. Each stage depends on the one before it.
Many guides start at indexing. For systems built on web data, retrieval quality starts earlier, with how the collection is gathered and cleaned.
Step 1: Acquire the Document Collection
A system can only retrieve what it has collected. For a library, that means cataloged books. For a search engine or AI application, it means pages fetched from the web.
Web acquisition has two parts. Discovery finds the URLs, using sitemaps and links, and tools such as URL discovery with Maps return the URLs on a site. Fetching then downloads each page, and crawling a whole site collects many pages in one job. Responsible acquisition respects robots directives, site terms and rate limits. JavaScript-heavy pages may also need browser rendering before their final content can be extracted.
Step 2: Clean and Represent Each Document
Raw HTML is noisy. Menus, cookie banners, ads and scripts sit next to the content you actually want to retrieve. If that noise gets indexed, it can match queries it has nothing to do with.
Extracting the main content into LLM-ready web data formats such as Markdown or JSON gives the indexer cleaner text. It also gives a language model fewer wasted tokens. The system then prepares the text:
- Tokenization: splits text into words or sub-word units.
- Normalization: lowercases text and removes punctuation.
- Stemming or lemmatization: reduces words to a base form, so "routers" matches "router".
- Chunking: splits long documents into passages, which RAG systems need.
Step 3: Build an Index
An index is a data structure that makes search fast. The classic version is the inverted index. It maps each term to a posting list, which records every document containing that term. An inverted index works like the index at the back of a book.
Modern systems often add a vector index, which stores numeric embeddings and finds the nearest ones to a query. Approximate nearest neighbor (ANN) methods such as HNSW keep vector search fast at scale. Indexes can get very large: Google's Search index "covers hundreds of billions of web pages and is well over 100,000,000 gigabytes in size," according to Google.
Step 4: Process the Query and Rank Results
The system processes the query the same way it processed the documents, using the same tokens and stems. It pulls candidate documents from the index, and a ranking function then scores each candidate and returns the top results (the "top-k").
Simple systems rank by text relevance alone. Web search layers many signals, including BM25-style term scoring, link analysis such as PageRank, and neural language matching. Olostep's glossary covers these ranking algorithms in web search in more detail.
Step 5: Evaluate and Improve
Evaluation compares the system's results against relevance judgments, which are human or model labels saying which documents are actually relevant. The scores show where the system misses.
Common improvements include relevance feedback, where clicks or labels re-weight the query, and query expansion, which adds synonyms or related terms. Large language models (LLMs) now often rewrite queries too.
What Are the Main Information Retrieval Models?
The three classic types of information retrieval models are Boolean, vector space and probabilistic. Modern systems add dense (neural) retrieval and hybrid retrieval, which combines keyword and neural methods.
| Model | How It Matches | Main Strength | Main Weakness | Typical Use |
|---|---|---|---|---|
| Boolean | Exact rules with AND, OR, NOT | Precise, predictable | No ranking; strict syntax | Legal and patent search, filters |
| Vector space (TF-IDF) | Weighted term vectors, cosine similarity | Ranks by partial match | Misses synonyms | Classic text search |
| Probabilistic (BM25) | Estimated probability of relevance | Strong, fast baseline | Misses paraphrases | Default in many search engines |
| Dense retrieval | Neural embeddings, nearest neighbors | Matches meaning | Can miss exact IDs and rare terms | Semantic search, RAG |
| Hybrid | Fuses keyword and dense results | Covers both failure modes | More components to tune | Production RAG and search |
Boolean Retrieval Model
The Boolean model treats a query as a logical rule. A query such as router AND password NOT email returns every document that satisfies the rule and nothing else.
Results form a set, not a ranked list, so a document either matches or it does not. Boolean logic still powers legal and patent search, and it powers the metadata filters in many search tools.
Vector Space Model and TF-IDF
The vector space model represents each document and query as a vector of term weights. It then ranks documents by how closely their vectors point in the same direction, using cosine similarity.
TF-IDF is the usual weighting scheme. Term frequency (TF) rewards words that appear often in a document. Inverse document frequency (IDF) rewards words that are rare across the collection. TF-IDF vectors are sparse word counts, not learned embeddings. The two are often confused.
Probabilistic Model and BM25
The probabilistic model ranks documents by their estimated probability of being relevant to the query. BM25 (Best Matching 25) is its best-known scoring function and the default ranking function in Lucene-based engines such as Elasticsearch.
BM25 improves on raw TF-IDF with two controls:
- k1 (term saturation): limits how much repeated words add, so the tenth mention of "router" counts far less than the first.
- b (length normalization): stops long documents from winning just because they contain more words.
Dense, Sparse and Hybrid Retrieval
Dense retrieval converts queries and documents into neural embeddings and matches them by meaning. This approach is often called semantic search or vector search. Sparse retrieval, such as BM25, matches exact terms. Hybrid search runs both and merges the ranked lists, often with Reciprocal Rank Fusion (RRF).
Dense retrieval does not automatically win. On financial text-and-table documents, a 2026 retrieval benchmark (an arXiv preprint) reported Recall@5 of 0.644 for BM25 and 0.587 for dense retrieval. Hybrid RRF scored 0.695, and hybrid plus a reranker reached 0.816. The authors note that results may not carry over beyond finance. Treat BM25 as a strong baseline, and measure every upgrade against it.
An Information Retrieval Example, Step by Step
Here is a small example that shows ranking and evaluation together. The collection has four documents, and the query is "reset router password".
| Doc | Text | Query Terms Matched (After Stemming) | Relevant? |
|---|---|---|---|
| D1 | How to reset your router password | reset, router, password | Yes |
| D2 | Factory reset steps for Wi-Fi routers | reset, router | Yes |
| D3 | Reset a forgotten email password | reset, password | No |
| D4 | Best password managers in 2026 | password | No |
D1 matches all three terms, so it ranks first. D2 and D3 each match two terms, but "router" appears in only 2 of 4 documents while "password" appears in 3. The rarer term carries more weight, so D2 ranks above D3. The final order is D1, D2, D3, D4.
Now score the top three results. Two of the three are relevant, so precision@3 is 2/3, or 0.67. Both relevant documents appear in the top three, so recall@3 is 2/2, or 1.0.
Stemming changes the outcome. Without it, "routers" in D2 does not match "router", so D2 drops below D3. Precision@2 then falls from 1.0 to 0.5. This is a small example of why representation choices in Step 2 shape ranking in Step 4.
The same pattern applies to the live web. This request sends a query to Olostep's Searches endpoint and prints the ranked links it returns:
import requests
resp = requests.post(
"https://api.olostep.com/v1/searches",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={"query": "reset router password", "limit": 5},
)
for link in resp.json()["result"]["links"]:
print(link["url"], "-", link["title"])Information Retrieval vs. Data Retrieval
Information retrieval finds relevant unstructured content and ranks it. Data retrieval fetches exact records from structured data. A SQL query that returns every order over $100 is data retrieval. A search for "laptop that stays cool" is information retrieval.
| Dimension | Information Retrieval | Data Retrieval |
|---|---|---|
| Data type | Unstructured text, pages, media | Structured rows and fields |
| Query | Natural language or keywords | Formal language such as SQL |
| Matching | Partial, by relevance | Exact, by rule |
| Result | Ranked list | Complete set |
| Error tolerance | Some irrelevant results expected | One wrong record is a bug |
| Example | Web search, RAG | Database lookups |
In practice, the line is blurry. Production search often combines ranked text retrieval with exact metadata filters such as date, language or domain.
How Is Retrieval Quality Measured?
Retrieval quality is measured by comparing results to relevance judgments. Precision is the share of retrieved results that are relevant. Recall is the share of all relevant documents that were retrieved.
| Metric | What It Measures | Use It When |
|---|---|---|
| Precision | Share of retrieved results that are relevant | Wrong results are costly |
| Recall | Share of relevant documents that were found | Missing documents is costly |
| Precision@k | Precision within the top k results | Users or LLMs read only the top results |
| Recall@k | Recall within the top k results | Checking RAG context windows |
| MRR | How high the first relevant result ranks | One right answer is enough |
| nDCG | Ranking quality with graded relevance | Order and degree of relevance both matter |
Public test collections such as TREC, MS MARCO and BEIR let teams compare methods. For a real application, build a small golden set. Collect real queries from your users, label the relevant documents, and re-run the set after every change to chunking, embeddings or ranking.
Information Retrieval for RAG and AI Agents
An LLM's built-in knowledge stops at its training data. Information retrieval supplies the current, specific context they lack, which makes IR the foundation of retrieval-augmented generation and of AI agents that use the web.
How Retrieval-Augmented Generation Uses IR
Retrieval-augmented generation (RAG) retrieves relevant passages, places them in the model's prompt, and generates an answer grounded in those passages. Systems such as Olostep's source-grounded answers endpoint combine web retrieval with generation and return answers with their sources.
The answer is usually only as good as the retrieved context. Better retrieval does not remove every error, though. In a peer-reviewed RAG study, hybrid retrieval raised accuracy for LLaMA3.1-8B from 37.92% to 46.29%. The authors also found that "hybrid retrieval introduces a trade-off between accuracy and faithfulness." Evaluate both whether answers are correct and whether they stay faithful to the sources.
Why AI Agents Are the Web's Second User
AI crawlers and agents are a growing class of web user, alongside people. According to Cloudflare's 2025 Radar review, "traffic from AI bots accounted for an average of 4.2% of HTML requests" on sites behind Cloudflare in 2025. That figure covers AI crawlers and agents combined.
Agents retrieve differently from people. They issue many short, machine-written queries in loops, refine them based on what they find, and need structured, citable output within latency and token budgets. Olostep's guide explains how agentic search works in that loop.
Static Index vs. Live Web Retrieval
A static index is a collection you crawl and store ahead of time. Live retrieval searches and fetches the web at query time. Many production systems use both.
| Factor | Static Index | Live Web Retrieval |
|---|---|---|
| Freshness | As fresh as the last crawl | Current at query time |
| Latency | Low; lookup is local | Higher; pages are fetched on demand |
| Cost profile | Upfront crawl and storage | Per-query search and fetch |
| Control | Full control over chunking and ranking | Depends on search and fetch quality |
| Best fit | Stable docs, internal knowledge | News, prices, fast-changing pages |
Freshness is a retrieval problem, not just a storage problem. A perfectly ranked page with outdated information can still produce a wrong answer. Store timestamps with every document, re-crawl on a schedule that matches how often sources change, and use monitoring pages for changes to update the index when content moves.
Why Information Retrieval Fails in Practice
Many retrieval failures start upstream, in the data rather than the ranking model. Use this table to diagnose common problems.
| Symptom | Likely Cause | Fix |
|---|---|---|
| Relevant docs use different words than the query | Vocabulary mismatch | Add dense retrieval or query expansion |
| Exact product codes or names are missed | Embeddings blur rare tokens | Add BM25 to make retrieval hybrid |
| Results match menus or footers | Noisy HTML extraction | Extract main content to Markdown |
| Answers miss key context | Chunks split ideas apart | Chunk by headings; add overlap |
| Answers cite outdated facts | Stale index | Track timestamps; monitor and re-crawl |
| Results from the wrong region or date | No metadata filters | Store and filter on metadata |
Adding hybrid search can barely change results on some datasets. The gain depends on the query mix and the chunk quality. Test every change against your golden set before you ship it.
Where Information Retrieval Is Used
Information retrieval powers products that need to find relevant content in a large collection. Web search is the largest example: Google reports more than 5 trillion searches per year. In Google's words, "We already see more than 5 trillion searches on Google annually."
Other common applications include:
- Enterprise search: finding internal documents, tickets and wikis.
- E-commerce search: matching shopper queries to products.
- Legal and patent search: high-recall searches across case law and filings.
- Digital libraries: searching academic papers and archives.
- RAG assistants and AI agents: retrieving live context before generating answers.
A Short History of Information Retrieval
Information retrieval grew from library science into one of computing's core fields.
- 1945: Vannevar Bush describes the memex, an imagined machine for linking and retrieving records.
- Around 1950: Calvin Mooers coins the term "information retrieval".
- 1960s: Gerard Salton's SMART system introduces the vector space model.
- 1970s: Karen Spärck Jones and Stephen Robertson develop inverse document frequency and probabilistic retrieval.
- 1992: The first Text REtrieval Conference (TREC) creates shared test collections.
- 1990s: BM25 emerges from the Okapi system and becomes a standard baseline.
- 1998: PageRank brings link analysis to web search.
- Around 2019–2020: Neural models such as BERT bring dense retrieval into mainstream use.
- 2020: Retrieval-augmented generation connects retrieval directly to language models.
FAQ
What Are the Three Types of Information Retrieval?
The three classic information retrieval models are Boolean, vector space and probabilistic. Modern systems add dense neural retrieval and hybrid retrieval.
Is Information Retrieval the Same as Search?
Mostly, yes. "Search" is the everyday word for information retrieval, while IR is the field that studies how search systems find and rank content.
Is Information Retrieval Part of AI or NLP?
Information retrieval is its own field of computer science that overlaps heavily with natural language processing and machine learning. Modern IR often uses neural language models for ranking and retrieval.
What Is an Example of an Information Retrieval System?
Web search engines, library catalogs, e-commerce site search and RAG chatbots are all information retrieval systems. Each one matches queries against an index and returns ranked results.
What Is the Difference Between BM25 and Vector Search?
BM25 ranks documents by exact term matches weighted by rarity and frequency. Vector search ranks documents by the similarity of their neural embeddings, so it can match meaning even when the words differ.
Do Long Context Windows Replace Retrieval?
Usually not. Even large context windows have cost, latency and size limits, so a system still needs retrieval to choose which current, relevant content goes into the prompt.
How Do You Keep a Retrieval Index Fresh?
Store a timestamp with every document, re-crawl sources on a schedule that matches how often they change, and use change monitoring to update pages after they change.
to update pages after they change.