What Is Text Mining? How It Works, Techniques, and Examples

Text mining is the automated process of turning unstructured text, like documents, web pages, reviews, and emails, into structured data you can count and analyze. Unstructured text has no fixed rows or columns, so software must find patterns across a corpus, the collection of documents being studied.

The NNLM text mining definition puts it in one line: "Text mining is the process of extracting meaning from unstructured text data." Our guide to structured vs. unstructured data explains how the two data types differ.

Why Text Mining Matters

Text mining matters because organizations and the public web produce more text than people can read. A support team can't review 10,000 tickets by hand, but a script can count issues per product across all of them.

According to the IBM unstructured data overview, "analysts estimate that up to 90% of enterprise-generated data falls into this unstructured category." That figure is an estimate, and it covers images, audio, and logs as well as text.

The public web is another large text source. The Common Crawl September 2026 crawl "contains 2.17 billion web pages (or 361.4 TiB of uncompressed content)" from 33.2 million registered domains. That is one two-week crawl sample, not the size of the whole web.

Text Mining vs. Text Analytics, NLP, and Data Mining

Text mining, text analytics, natural language processing (NLP), and data mining overlap, and published sources use the terms differently. Some treat text mining and text analytics as synonyms; others separate them. The table uses our working definitions, with the same criteria for each term.

In these working definitions, NLP supplies the methods and text mining applies them to a corpus. Text analytics is often the reporting layer on top. Data mining is the parent discipline for structured data, and text mining borrows its methods once text becomes numbers.

How Does Text Mining Work?

Text mining works by moving text through five stages, and each stage depends on the one before it. An error at collection, such as keeping menu text from web pages, carries through to every result.

1. Collect the text that forms your corpus.
2. Clean and preprocess the text.
3. Convert the words into numbers.
4. Analyze the numbers for patterns.
5. Evaluate the results against human judgment.

Step 1: Collect the Text

Collection decides input quality, because analysis only finds patterns in the text you gathered. Internal sources include support tickets, emails, chat logs, contracts, and PDFs. External sources include web pages, product reviews, forum posts, and public filings.

Collecting text from the web is part of web mining, and it uses two methods. Scraping fetches the content of one known URL. Crawling starts from a page, follows its links, and gathers many pages across a site.

Olostep's Crawls endpoint can crawl entire websites up to the page and depth limits you set, and its Scrapes endpoint fetches single URLs. Both can return Markdown when you request it, which gives later steps cleaner input than raw HTML.

PDFs need parsing, and scanned PDFs need optical character recognition (OCR) to turn page images into text. Olostep's Scrapes endpoint also accepts PDF URLs, which saves you from running a separate PDF parser.

Step 2: Clean and Preprocess the Text

Cleaning removes text that doesn't belong to the content you want to study. Preprocessing then splits the rest into units a program can count.

Web pages need cleanup first. Menus, cookie banners, and footers repeat on every page, so words like "login" and "subscribe" can top your counts. Main-content extractors use density signals: blocks with high text density and low link density usually hold content, and blocks packed with links usually hold navigation.

Our guide to extracting only main content covers these heuristics. JavaScript-heavy pages may load text after the first HTML response, so a plain HTTP request can return a nearly empty page. Those pages may need browser rendering before extraction.

Duplicate pages, such as one article at two URLs, inflate counts. Hashing each page's cleaned text finds exact duplicates to drop. For broader dataset fixes, see our data cleaning guide.

After web cleanup, linguistic preprocessing prepares the words themselves:

• Tokenization: Splits text into tokens, the small units a program counts, usually words or punctuation marks.
• Lowercasing: Converts letters to lowercase so "Refund" and "refund" count as one word.
• Stop-word removal: Drops very common words, such as "the" and "and," that carry little meaning alone.
• Stemming: Cuts words to a rough root by rule, so "running" becomes "run" and "studies" becomes "studi."
• Lemmatization: Maps each word to its dictionary form, so "studies" becomes "study" and "better" becomes "good."

Stemming is faster and cruder. Lemmatization needs a dictionary but produces real words.

Step 3: Turn Words Into Numbers

Models work with numbers, so text mining converts each document into a vector, which is a list of numbers. Three common methods capture progressively more meaning:

• Bag-of-words counts how often each word appears in a document and ignores word order.
• TF-IDF (term frequency-inverse document frequency) gives a word more weight when it is frequent in one document but rare across the corpus.
• Embeddings are vectors produced by a trained model, so texts with similar meaning get similar numbers.

In a corpus of hotel reviews, "room" appears in almost every review, so TF-IDF gives it a low weight. "Mold" appears in only a few reviews, so it gets a high weight in those reviews.

Step 4: Analyze for Patterns

Analysis runs a text mining technique, such as classification or clustering, on the vectors from Step 3. Sorting tickets into known categories calls for classification, and discovering unknown themes calls for clustering or topic modeling.

Step 5: Evaluate the Results

Evaluation checks whether the output is correct before anyone acts on it. The two standard measures are precision and recall:

• Precision is the share of flagged items that are actually correct. If a model tags 100 tickets as "billing" and 90 really are, precision is 90%.
• Recall is the share of all true items the model found. If 120 tickets are truly about billing and the model caught 90, recall is 75%.

To get those numbers, have a person label a random sample of documents, then compare those labels with the model's.

Repeat the check on a schedule. New products, slang, or page layouts can shift the input over time, a problem called drift, and lower accuracy without any error message.

Common Text Mining Techniques

Text mining techniques are the methods that find patterns in prepared text, and one pipeline can combine several. This table lists eight common types of text mining and their output.

Paired with topic modeling, sentiment analysis shows which themes draw negative reviews. Information extraction can also run on web pages directly. With schema-based extraction, you define fields such as product name and price in a JSON schema, and the extractor returns those fields.

Text Mining Examples by Use Case

Each of these text mining examples answers a question from more text than a team can read:

• A product team classifies app-store reviews by topic and sentiment to find which feature draws the most complaints.
• An e-commerce team scrapes competitor product pages each week and extracts prices and stock status into JSON.
• A research group clusters thousands of paper abstracts by method to find topics with few published studies.
• A legal team uses NER and keyword search to flag contracts that mention auto-renewal or a specific governing law.
• An AI team cleans and deduplicates crawled web pages before using them for retrieval-augmented generation (RAG), where a model answers from retrieved documents.

Text Mining in Python: A Minimal Example

This minimal illustrative example fetches one web page as Markdown and prints its 10 most frequent terms. It uses the Olostep Scrapes endpoint for collection and Python's standard library for counting.

You need Python 3, the requests library (pip install requests), and an Olostep API key. Replace the example URL with a page you are allowed to collect.

# Minimal illustrative example: fetch a page as Markdown, then count top terms.
import re
from collections import Counter

import requests

url = "https://example.com"  # Replace with a page you are allowed to collect.

response = requests.post(
    "https://api.olostep.com/v1/scrapes",
    headers={"Authorization": "Bearer <YOUR_API_KEY>"},
    json={"url_to_scrape": url, "formats": ["markdown"]},
    timeout=60,
)
response.raise_for_status()
result = response.json()

# Confirm response field names in the Olostep API docs before relying on them.
markdown = result["result"]["markdown_content"]

# Drop link targets like (https://...) so URLs don't become top terms.
text = re.sub(r"\(https?://[^)]*\)", " ", markdown)

# Tokenize: lowercase the text and keep runs of letters.
tokens = re.findall(r"[a-z]+", text.lower())

# Remove a small stop-word set.
stop_words = {"the", "and", "for", "are", "with", "that", "this", "from", "you", "your", "was", "not"}
terms = [token for token in tokens if token not in stop_words and len(token) > 2]

print(Counter(terms).most_common(10))

The script prints 10 (term, count) pairs, most frequent first. If words like "menu" or "cookie" appear, boilerplate is still in the input.

This example skips lemmatization and uses a short stop-word list. Real projects add those steps with a library such as spaCy or NLTK.

How LLMs Are Changing Text Mining

A large language model (LLM) is an AI model trained on very large amounts of text to predict and generate language. LLMs can classify, extract fields, and summarize from a written prompt, without a labeled training set.

That flexibility has trade-offs. Every document an LLM reads is billed or computed per token, so labeling millions of documents with an LLM can cost more than running a TF-IDF model on the same machine. LLM outputs can also vary between runs unless you fix settings and check results. An LLM can also hallucinate, meaning it states facts that are not in the source text.

Published results are mixed. A clinical NER study on rare-disease text found that "without proper fine-tuning, local LLMs generally struggle with token-level NER, despite prior training on clinical texts." After fine-tuning, which means further training on labeled examples, one LLM in the study approached BioClinicalBERT, a smaller model built for clinical text.

That study tested 2023-era models on one English clinical dataset, so it doesn't settle how LLMs compare with classic models in general.

A practical split is classic methods such as TF-IDF and clustering for counting at scale, and LLMs for extraction against a fixed schema. Evaluate both with the same precision and recall checks.

Is Text Mining Legal?

Whether text mining is legal depends on the jurisdiction, the source, and how you use the results. This section describes EU law in general terms and is not legal advice.

Under the EU Copyright Directive text, text and data mining "means any automated analytical technique aimed at analysing text and data in digital form in order to generate information which includes but is not limited to patterns, trends and correlations". The Directive, numbered (EU) 2019/790, sets two exceptions that each member state implements through national law.

Article 3 covers research organizations and cultural heritage institutions that mine works they have lawful access to for scientific research. Article 4 is a broader exception for lawfully accessible works. Article 4 doesn't apply when rightholders have expressly reserved their rights, and for online content that reservation can be made by machine-readable means.

The EU AI Act adds a related duty for providers of general-purpose AI models only. AI Act Article 53 requires those providers to have a copyright policy and to "identify and comply with, including through state-of-the-art technologies, a reservation of rights expressed pursuant to Article 4(3) of Directive (EU) 2019/790". Article 53 applies to providers of general-purpose AI models, not to every team that mines text.

Before collecting web text, run through this checklist:

• Read the site's robots.txt file, which the robots.txt protocol uses to tell crawlers which paths they may fetch.
• Read the site's terms of service for rules on automated access and reuse.
• Remove or protect personal data, such as names and email addresses, unless you have a clear legal basis to process it.
• Limit your request rate so collection doesn't overload the site.

Olostep's Terms of Service require customers to respect applicable website terms, robots.txt directives, and rate limits.

Text Mining Challenges and Limitations

The challenges of text mining come from language itself and from the input data, and each can lower accuracy without a visible error:

• Ambiguity and sarcasm confuse models, so a review saying "great, another delay" can score as positive.
• Domain language, such as medical or legal terms, can be misread by general models not trained on it.
• Multilingual text needs language-specific tokenizers and stop-word lists, and Chinese text doesn't separate words with spaces.
• Noisy input, such as boilerplate, broken characters, and duplicate pages, skews counts and topics.
• Bias in source data carries into results, so reviews from one platform may not reflect all customers.
• Privacy limits what you can store, because tickets, emails, and forum posts can contain personal data.

Frequently Asked Questions

What is an example of text mining?

A common example is counting complaint themes across thousands of support tickets. Another is extracting prices from competitor product pages into a table.

What are some tools used for text mining?

Python libraries such as NLTK, spaCy, and scikit-learn handle preprocessing and modeling, and R has its own text mining packages. Browser-based tools also let you explore a corpus without writing code.

Can you give me some examples of text data?

Emails, support tickets, chat logs, survey answers, product reviews, social media posts, contracts, research papers, and web pages are all text data. Semi-structured formats like HTML and JSON also count when they contain text.

Is text mining the same as data mining?

No, though they are closely related. Data mining finds patterns mostly in structured tables, and text mining first converts unstructured text into numbers so similar methods can run on it.

Do I need to know Python for text mining?

No, because some browser-based tools need no code. Python or R gives you control over each pipeline step and handles larger corpora.

Is text mining part of AI?

Text mining often uses AI methods such as machine learning and NLP, so it is commonly grouped with AI. Simple word counting, like the Python example above, uses no AI at all.