Data cleaning is the process of finding and fixing errors, duplicates, and inconsistencies in a dataset so the data is accurate and ready to use. It is also called data cleansing or data scrubbing. All three terms describe the same work.
For example, a customer list shows "Jane Smith" twice, once as "jane smith" and once with a missing email. Cleaning merges the two rows into one correct record.
Clean data is commonly described by five qualities. It is accurate, complete, consistent, unique, and valid. These qualities apply whether the data comes from a spreadsheet, a database, or pages collected from the web.
Why Data Cleaning Matters
Data cleaning matters because every report, dashboard, and AI model is only as good as its data. If the input is wrong, the output is usually wrong too. People call this "garbage in, garbage out."
Data teams still worry about this problem. In the dbt Labs 2026 report, "A significant majority of respondents (71%) are concerned about hallucinated or incorrect data reaching stakeholders."
Unreliable data also holds back AI projects as they move into production. In the Informatica CDO Insights 2026 survey, Informatica reports: "Even as companies race to production, 57% admit data reliability is a top barrier."
You may have heard that data scientists spend 80% of their time cleaning data. That number is repeated often, but we could not find a reliable, recent primary source for it. The real amount depends on how messy your data is when you collect it.
Common Types of Dirty Data
Dirty data is any data with errors that make it wrong, confusing, or hard to use. Most problems fall into a few common types.
- Missing values: Blank cells, such as a customer with no email or a product with no price.
- Duplicates: The same record appears more than once, sometimes with small differences.
- Inconsistent formats: The same thing is written in different ways, like "10/17/2026" and "2026-10-17."
- Structural errors: Typos, odd capitalization, or extra spaces, such as "N/A," "n/a," and "NA."
- Outliers: Values far outside the normal range, like an age of 430.
- Invalid values: Data that breaks a rule, such as an email with no "@" sign.
- Irrelevant data: Rows or columns that do not help your goal.
- Leftover markup and encoding errors: Common in web data. HTML tags remain in the text, or characters turn into gibberish like "’" instead of an apostrophe.
Web pages are a special case. A page arrives as unstructured HTML, so it must be parsed before it can be cleaned. Learn more about structured vs. unstructured data.
The table below shows what cleaning looks like in practice.
| Problem | Before Cleaning | After Cleaning |
|---|---|---|
| Inconsistent format | 10/17/2026, Oct 17 2026 | 2026-10-17 |
| Structural error | " jane SMITH " | Jane Smith |
| Duplicate | Two rows for the same customer | One merged row |
| Invalid value | jane.smith.com | Flagged for review |
| Encoding error | It’s ready | It's ready |
How the Data Cleaning Process Works
The data cleaning process is a loop, not a one-time task. You check the data, fix it, and check it again.
Most teams follow six steps. The order can change, but each step has a clear job.
Step 1: Profile the Data
Profiling means looking at your data before you change anything. Count the rows, count the blanks in each column, and list the unique values. Check the lowest and highest numbers to spot odd values.
Keep a raw copy of the original data before you start. If a fix goes wrong, you can start over.
Step 2: Remove Duplicates and Irrelevant Data
Duplicates come in two kinds. Exact duplicates match in every field. Near-duplicates describe the same thing with small differences, like "Acme Inc." and "ACME, Inc."
Web data has its own version of this problem. The same article can appear at several URLs, such as a print version or a link with tracking codes. Remove columns and rows that do not serve your goal at this step too.
Step 3: Fix Structural Errors and Standardize Formats
Standardizing means making the same kind of value look the same everywhere. Pick one date format, one unit, and one spelling for each category. Trim extra spaces and fix capitalization.
Also check text encoding. Encoding is how a computer stores letters as numbers. A mismatch can turn an apostrophe into "’."
Step 4: Handle Missing Values and Outliers
You have three choices for missing values. You can drop the row, fill in a value (called imputation), or flag the gap so others know it exists. The right choice depends on how much data is missing and why.
Check outliers before you delete them. Some are typos, like an age of 430. Others are real and important, like a sudden spike in sales.
Step 5: Validate the Results
Validation means testing the cleaned data against clear rules. For example, ages must be between 0 and 120, and every email must contain "@." Run your profile again and compare the row counts to the raw copy.
If a rule fails, go back to the step that caused it. That return trip is why cleaning works as a loop.
Step 6: Document and Automate the Fixes
Write your cleaning steps as a script instead of fixing cells by hand. A script runs the same way every time new data arrives. Hand edits are hard to repeat and easy to forget.
Keep a short log of what changed and why. A change log helps teammates trust the data.
A Simple Data Cleaning Example in Python
Python is widely used for data cleaning. The pandas library handles tables of data, called DataFrames. This short script covers the main steps.
import pandas as pd
# Load the raw data and keep an untouched copy
raw = pd.read_csv("customers.csv")
df = raw.copy()
# Standardize text: trim spaces and fix capitalization
df["name"] = df["name"].str.strip().str.title()
df["email"] = df["email"].str.strip().str.lower()
# Standardize dates into one format (bad dates become NaT)
df["signup_date"] = pd.to_datetime(df["signup_date"], errors="coerce")
# Remove exact duplicates, then duplicates by email
df = df.drop_duplicates()
df = df.drop_duplicates(subset="email", keep="first")
# Flag missing values instead of silently dropping them
df["missing_email"] = df["email"].isna()
# Validate: every email that exists must contain "@"
invalid = df[df["email"].notna() & ~df["email"].str.contains("@", na=False)]
print(f"Rows before: {len(raw)}, after: {len(df)}, invalid emails: {len(invalid)}")The final line prints a quick check. It shows how many rows you started with, how many remain, and how many emails still need review.
Data Cleaning vs. Data Transformation vs. Data Validation
Data cleaning, transformation, and validation are related, but each has a different job. Cleaning fixes errors. Transformation changes the shape or format of data, and validation checks that data follows the rules.
| Data Cleaning | Data Transformation | Data Validation | |
|---|---|---|---|
| Goal | Fix or remove wrong data | Change data's structure or format | Confirm data meets rules |
| Example | Merge duplicate customers | Convert HTML to JSON, or join two tables | Check every email has "@" |
| When it happens | After collection, before analysis | Before loading or analysis | At entry and after cleaning |
You may also see the terms data wrangling and data preprocessing. Both are umbrella terms. They include cleaning plus other steps that get data ready for analysis or machine learning.
Why Web Data Needs Extra Cleaning
Web pages are built for people, not for programs. Besides the main content, a page holds menus, ads, footers, scripts, and styling code.
When you collect data from publicly accessible websites, those extra parts come along too. Cleaning web data means separating the content you want from everything else.
Boilerplate, Markup, and Encoding Problems
Boilerplate is the repeated material found on many pages of a site, such as navigation bars, cookie banners, and footers. Markup is the HTML code that tells a browser how to display the page. Neither belongs in your dataset.
An HTML parser reads the page and builds a tree of its parts, called the Document Object Model (DOM). Code can then keep the main content and drop the rest. Read more about how an HTML parser works.
Encoding problems are common too. Pages use different character sets, so text can arrive garbled. Fix these early, before the errors spread into other steps.
How Dirty Web Data Affects AI Models
Dirty web data hurts AI models in two ways: it adds noise and it adds cost. Large language models (LLMs) read text in units called tokens, and many AI APIs charge per token processed. Every menu link and HTML tag uses tokens without adding meaning.
Olostep's glossary gives an example: a webpage's HTML might use 50,000 tokens, while the same content in Markdown uses about 5,000. Markdown is a simple text format that keeps headings and lists. See this overview of formats for feeding web data to AI.
One study points the same way. In the FinerWeb line-level filtering study, researchers removed low-quality lines from web text used to train small GPT-2 models: "Remarkably, both models reach the original dataset's highest score approximately 6k steps earlier, a 32% reduction in training time." That result comes from small models on one benchmark, so treat it as a signal, not a rule.
Data leaders feel this pressure. According to Hex State of Data Teams, "31% of data leaders cite trust and data accuracy as their #1 AI concern — nearly 2× the next most common barrier."
Clean at the Source: Collecting Data That Starts Clean
The cheapest place to clean data is usually where you collect it. Every error you stop at the start is one you do not have to find and fix later.
For web data, that means asking for clean output when you collect it. Instead of saving raw HTML, you can request Markdown or plain text with boilerplate removed.
You can also request JSON shaped by a schema, which is a list of the field names and data types you want. Read more on converting HTML into structured JSON.
The Olostep scrape endpoint works this way. It returns a page as Markdown, HTML, text, or structured JSON, so your pipeline starts with content instead of markup.
Cleaning at the source does not remove every step. You still need to validate the results, remove duplicates across pages, and handle missing values.
Source cleaning also cannot fix information that was wrong on the original page. For a deeper look at preparing data for AI, read our guide to LLM-ready data.
Frequently Asked Questions About Data Cleaning
Is Data Cleaning the Same as Data Cleansing?
Yes, data cleaning, data cleansing, and data scrubbing all mean the same thing: finding and fixing errors in data.
Can I Use Python for Data Cleaning?
Yes, Python is widely used for data cleaning, and the pandas library handles most common tasks like removing duplicates and fixing formats.
Can You Clean Data in Excel?
Yes, Excel can work for small datasets with tools like Remove Duplicates, TRIM, and filters. For large or repeating jobs, a script is easier to repeat and check.
How Long Does Data Cleaning Take?
The time depends on how messy the data is and how it was collected. Collecting cleaner data at the source and automating your fixes can shorten the work.
Can AI Clean Data Automatically?
AI tools can spot errors and suggest fixes, but people still need to set the rules and check the results.