What Is Data Profiling? A Practical Guide With Examples

Every dataset has two versions: the one its documentation describes and the one that actually exists. Data profiling tells you which one you have. This guide explains what data profiling is, what a profile contains, how the process works, and how to profile semi-structured web data before it reaches your analytics, AI agents or RAG pipeline.

What Is Data Profiling?

Data profiling is the process of examining a dataset to produce statistics and metadata that describe its structure, content and quality before you use it. The result, called a data profile, is a factual summary of what is really in the data.

In simple terms, profiling answers basic questions with numbers. How many records are there? How many values are missing? Which formats appear in each field? Are there duplicates?

A profile measures the shape of data, not its truth. It can show that 17% of prices are missing, but it cannot show that a present price is correct. Profiling is a diagnostic step, and it does not verify accuracy.

What a Data Profile Contains

A data profile contains a set of per-field (column) statistics plus checks across fields. Column profiling is the most common starting point.

Why Data Profiling Matters

Data profiling matters because you cannot fix a data problem you have not measured. A profile replaces assumptions about a dataset with facts, before those assumptions break a migration, a dashboard or a model.

Poor data quality is expensive. According to IBM's 2026 analysis, "over a quarter of organizations estimate they lose more than USD 5 million annually due to poor data quality, with 7% reporting losses of USD 25 million or more." These figures are organizations' own estimates.

Data teams report the same problem. In the dbt Labs 2025 survey, "Poor data quality continues to be the challenge most frequently reported by data teams, cited by over 56% of respondents."

Profiling is the measurement step inside broader data quality management practices. It shows where quality work should start.

Types of Data Profiling

Data profiling is commonly grouped into three types. Each type answers a different question about the dataset.

Structure Discovery

Structure discovery checks whether data is formatted consistently. It looks at data types, field lengths and value patterns, such as whether every date uses the same format.

For JSON records, structure discovery also measures key presence rate. Key presence rate is the share of records that contain a given field at all. It also catches type drift, where a field is a number in some records and a string in others.

Content Discovery

Content discovery inspects the individual values inside each field. It finds nulls, outliers, invalid values and unusual frequency distributions.

For example, content discovery might show that one "country" value appears in 40% of rows. That often means a default value filled the field when the real value was unknown.

Relationship Discovery

Relationship discovery finds connections between fields and between tables. It identifies candidate keys, overlapping values between tables and duplicates across sources.

It also detects functional dependencies. A functional dependency means one field determines another, such as a postal code determining a city. When the same postal code maps to two cities, the profile flags a conflict.

How Data Profiling Works

Data profiling works by computing statistics over a dataset and comparing the results with what you expect. Most profiling follows the same ordered steps.

1. Choose the dataset and scope. Decide which tables, files or record sets to profile, and which fields matter most.
2. Decide sample or full scan. Sampling is faster on large data, but a sample can miss rare values and rare errors.
3. Compute column statistics. Calculate null rates, distinct counts, types, ranges, lengths and patterns for each field.
4. Run cross-field and cross-table checks. Test keys, dependencies and duplicates.
5. Review findings against expectations. Compare the profile with the schema, documentation or business rules.
6. Turn findings into rules or fixes. A finding such as "price must be numeric" becomes a validation rule or a cleaning step.
7. Re-profile after changes. Run the profile again after cleaning, and again when the source changes.

The expected result is a short list of known issues, each tied to a field and a count. That list tells you what to fix first.

A Data Profiling Example in Python and SQL

The example below profiles six product records scraped from three online stores. The records are JSON objects, which is a common output format for web data.

[
  {"url": "https://shop-a.example/p/1", "name": "Trail Shoe", "price": 89.0, "currency": "USD", "in_stock": true},
  {"url": "https://shop-a.example/p/2", "name": "Rain Jacket", "price": "120.00", "currency": "USD", "in_stock": true},
  {"url": "https://shop-b.example/item/7", "name": "Trail Shoe", "price": null, "currency": "usd", "in_stock": false},
  {"url": "https://shop-b.example/item/8", "name": "Wool Socks", "price": 14.5, "currency": "USD"},
  {"url": "https://shop-a.example/p/1", "name": "Trail Shoe", "price": 89.0, "currency": "USD", "in_stock": true},
  {"url": "https://shop-c.example/x/3", "name": "", "price": 0.0, "currency": "EUR", "in_stock": true}
]

This short pandas script computes a basic column profile:

import pandas as pd

df = pd.DataFrame(records)  # records = the JSON list above

profile = pd.DataFrame({
    "null_rate": df.isna().mean().round(2),
    "distinct": df.nunique(),
    "python_types": df.apply(lambda c: sorted({type(v).__name__ for v in c.dropna()})),
})
print(profile)
print("duplicate urls:", df["url"].duplicated().sum())

prices = pd.to_numeric(df["price"], errors="coerce")
print("price min/max:", prices.min(), prices.max())

Sample output:

null_rate  distinct  python_types
url            0.00         5         [str]
name           0.00         4         [str]
price          0.17         4  [float, str]
currency       0.00         3         [str]
in_stock       0.17         2        [bool]
duplicate urls: 1
price min/max: 0.0 120.0

Each line of the profile points to a specific issue:

• Null rate on price (0.17): One record has no price, so extraction missed it or the page did not show it.
• Mixed types on price: One price arrived as the string "120.00" instead of a number.
• Three currency values: "USD" and "usd" are the same currency written two ways, plus one "EUR" record that needs conversion.
• Null rate on in_stock (0.17): One record is missing the field entirely, which is a key presence problem.
• One duplicate URL: The same product page was collected twice.
• Minimum price of 0.0: A zero price is likely a placeholder, not a real price.
• An empty name: The name field has no nulls, but one value is an empty string, which a null check alone misses.

The same checks work in SQL once the records land in a warehouse table:

SELECT
  COUNT(*)                                         AS row_count,
  AVG(CASE WHEN price IS NULL THEN 1.0 ELSE 0 END) AS price_null_rate,
  COUNT(DISTINCT url)                              AS distinct_urls,
  COUNT(*) - COUNT(DISTINCT url)                   AS duplicate_urls,
  COUNT(DISTINCT UPPER(currency))                  AS currencies_normalized,
  MIN(price)                                       AS min_price,
  MAX(price)                                       AS max_price
FROM scraped_products;

For a full automated report, open-source tools such as ydata-profiling (formerly pandas-profiling) generate these statistics for every field at once.

Profiling Web and External Data

Profiling web and external data requires extra checks, because you do not control the source. A website can rename a field, move a price or change a layout without notice, and your dataset changes with it.

External data also arrives without owner documentation. The profile becomes the documentation. For semi-structured JSON, that profile should cover key presence rates, type consistency per field, and the length of nested arrays such as product variants.

A declared target schema gives the profile something to check against. With schema-based extraction, you define field names and types before extraction begins. The profile then measures how closely real records match that contract.

Profiling also needs enough records to be meaningful. Six records illustrate the method, but a real profile usually needs hundreds or thousands of pages. The Olostep Batches endpoint processes large URL lists in one asynchronous job, which produces a dataset large enough to profile.

Profiling Data for AI and RAG

Profiling data for AI and RAG means checking text documents, not just table columns. RAG (retrieval-augmented generation) systems retrieve documents to ground model answers, so weak documents can lead to weak answers.

Useful checks for a document corpus include:

• Empty or near-empty documents: Pages that returned only navigation or an error message.
• Length distribution: Very short documents and cut-off documents both stand out as outliers.
• Duplicates and near-duplicates: The same page collected under different URLs.
• Boilerplate share: How much of each document is menus, footers or cookie notices.
• Language mix: Documents in languages your pipeline does not expect.
• Freshness: Publish or crawl dates that show how old each document is.

These checks describe what LLM-ready data should look like before it is chunked and embedded.

Continuous Profiling and Data Drift

Continuous profiling means running the same profile on a schedule and comparing each result with a baseline. A one-time profile goes stale as soon as the source changes.

Data issues keep appearing after the first cleanup. Monte Carlo's 2026 telemetry reports, "Today, it's an average of 1 data quality issue for every 10 tables in your environment per year." This figure comes from Monte Carlo's own customer base.

The signals to compare between profile runs are simple:

• Null-rate jumps: A field that was 2% empty is suddenly 60% empty.
• Row-count drops: A source that returned 5,000 records now returns 3,000.
• New categories: A status field gains values nobody expected.
• Type changes: A numeric field starts returning text. This is called schema drift.

For web sources, a sudden profile change is often the first sign that a site changed its layout. The Olostep Monitors API re-checks pages on a schedule, which gives a re-profiling job fresh data to compare.

Data Profiling vs. Related Practices

Data profiling describes data, while related practices enforce, fix or analyze it. The table compares each practice on the same three criteria.

Profiling often feeds validation: a profile finding becomes a rule. Learn more about how data validation works and about data cleaning techniques that act on profile findings.

Data Profiling Tools

Data profiling tools range from one-line Python libraries to enterprise suites. The right choice depends on data size, where the data lives, and whether you need a one-off report or recurring checks.

Start with the simplest tool that covers your data size. Move to a validation framework once the same checks need to run on every load.

Frequently Asked Questions

What Is Data Profiling in Simple Terms?

Data profiling is a quick factual summary of what is inside a dataset. It counts missing values, formats, duplicates and ranges so you know what you are working with.

What Is an Example of Data Profiling?

Profiling scraped product records and finding that 17% have no price is an example of data profiling. The same profile might also find prices stored as text and duplicate product URLs.

What Are the Best Tools for Data Profiling?

The best tool depends on scale: ydata-profiling suits notebooks, Great Expectations and Soda suit recurring pipeline checks, and Deequ suits Spark-scale data.

How Is Data Profiling Different From Data Mining?

Data profiling describes the structure and quality of data. Data mining searches trusted data for business patterns and insights.

When Should You Profile Data?

Profile data before first use, after every cleaning step, and on a schedule for sources that change, such as websites.

Can Data Profiling Fix Data Quality Problems?

No, data profiling surfaces problems but does not fix them. Data cleansing corrects the issues, and data validation prevents them from returning.