Data transformation is the process of converting data from one format, structure or set of values into another so a target system can use it. That target can be a database, a dashboard, a machine learning model or an AI agent.
A simple example is a day-first date such as "03/04/2026" becoming "2026-04-03," so every system reads it as April 3. The term also has a statistics meaning (log scaling) and a machine learning meaning (feature preprocessing), but this guide covers the data engineering meaning.
Structured, Semi-Structured and Unstructured Data
Data comes in three shapes, and the shape decides how much transformation it needs. Structured data sits in tables with fixed columns, like a spreadsheet or a SQL table. Semi-structured data uses tags or keys without fixed columns, like JSON files and HTML.
Unstructured data has no predictable layout, like free text, PDFs and raw web pages. A raw web page behaves like unstructured data in practice, because the facts you want are buried inside layout markup.
Transformation usually moves data toward more structure, such as turning unstructured data into structured data that code can query. A price mentioned in a paragraph becomes a number in a price column. Olostep's explainer on structured vs. unstructured web data covers how these shapes apply to scraped pages.
Why Data Transformation Matters
Data transformation matters because each system expects data in a specific shape, and mismatched data breaks the work downstream. A join fails when one table stores customer IDs as text and another stores them as integers. A report double-counts revenue when duplicate orders slip through.
Poor-quality data also has a cost that organizations try to put a number on. According to IBM's data quality research, published in January 2026, "over a quarter of organizations estimate they lose more than USD 5 million annually due to poor data quality, with 7% reporting losses of USD 25 million or more." Each figure is an organization's own estimate.
Keeping data usable also fills much of the workday for data practitioners. In dbt Labs' 2025 survey, "57% of respondents spend most of their workdays maintaining or organizing data sets, nearly identical to last year's report." The 57% counts survey respondents, and the survey ran from October to December 2024.
Common Types of Data Transformation
Most transformations fall into a small set of repeatable operations. Real pipelines often chain several of them together.
- Cleaning: Fix errors and invalid values, such as replacing "N/A" in a revenue column with a true null.
- Standardizing formats: Convert values to one convention, such as storing every weight in kilograms.
- Deduplication: Remove repeated records, such as two identical order rows created by a retried job.
- Mapping: Match source fields to target fields, such as sending
cust_nminto acustomer_namecolumn. - Aggregation: Summarize many rows into fewer, such as counting orders per customer.
- Enrichment: Add data from another source, such as attaching an industry label to a sales lead.
- Filtering: Keep only the rows or fields a consumer needs, such as dropping test accounts.
- Restructuring: Change the data's shape, such as flattening nested JSON like
address.cityinto acitycolumn.
Data normalization is a related term with its own meaning. It usually refers to organizing database tables or scaling numbers into a common range.
Is Data Transformation the Same as Data Cleaning?
No. Data cleaning, also called data cleansing, is one type of data transformation that fixes errors and removes bad records.
Transformation also changes structure and format, such as reshaping an HTML page into JSON. Data wrangling and data preparation are broader terms that describe the whole workflow of getting data ready, with transformation as one part.
How the Data Transformation Process Works
The data transformation process has six steps that run in order. Defining the target shape early gives every later step something to test against.
- Profile the source to learn its fields, types, null rates and odd values.
- Define the target shape, or schema, which lists the exact fields and types the consumer expects.
- Map each source field to a target field, and flag fields with no match.
- Apply the transformation rules in code, SQL or a dedicated tool.
- Validate the output against the schema with row counts, type checks and spot checks.
- Load or deliver the output to its destination, such as a warehouse table, an API or a search index.
Validation is where pipelines tend to fail quietly. A wrong value can look exactly like a correct one, so checks need to test values as well as job status.
A Simple Before-and-After Example
This order export has four issues: day-first dates, full names in one column, prices stored as text and a repeated row.
order_id,customer,order_date,price
1001,Maria Lopez,15/03/2026,"$1,299.00"
1002,Sam Lee,16/03/2026,$45.50
1002,Sam Lee,16/03/2026,$45.50Four rules produce the clean version. Dates become year-month-day, names split on the first space, prices become plain numbers and the duplicate order 1002 is removed.
order_id,first_name,last_name,order_date,price_usd
1001,Maria,Lopez,2026-03-15,1299.00
1002,Sam,Lee,2026-03-16,45.50The name rule is a simplification. A name like "Mary Ann Smith" needs a smarter rule or separate name fields in the source.
ETL vs. ELT: Where Transformation Happens
ETL stands for extract, transform, load, so data is transformed before it reaches storage. ELT stands for extract, load, transform, so raw data is loaded first and transformed inside the warehouse.
AWS's ETL vs. ELT comparison defines them this way: "The ETL approach uses a set of business rules to process data from several sources before centralized integration. The ELT approach loads data as it is and transforms it at a later stage, depending on the use case and analytics requirements."
| Criterion | ETL | ELT |
|---|---|---|
| Order of steps | Extract, transform, load | Extract, load, transform |
| Where transformation runs | A processing layer before the warehouse | Inside the warehouse, usually in SQL |
| Typical tools | Integration tools such as Informatica or Talend, or custom scripts | dbt running SQL models in warehouses such as Snowflake or BigQuery |
| When it fits | Data must be reshaped, filtered or masked before storage | The warehouse can hold raw data and teams want to remodel it later |
dbt is built for SQL modeling inside the warehouse. Raw HTML is a harder case, because SQL is a poor tool for pulling clean fields out of page markup.
For web data, the first transformation usually happens before loading, and dbt models can still run on the clean records afterward. Olostep works at that upstream step and does not replace dbt.
Transforming Web Data for LLMs and AI Agents
When an LLM (large language model) or AI agent reads the data, the most important transformation turns unstructured web pages into Markdown or structured JSON. Raw HTML carries navigation menus, scripts, ads and styling code that a model has to read past.
Clean input lets the model spend its context window, the limit on how much text it can take in at once, on actual content. Olostep's guide to what makes data LLM-ready covers the requirements in more depth.
HTML to Markdown for LLM Context
HTML to Markdown conversion keeps a page's headings, lists and links and drops the clutter, so the same content uses fewer tokens. A token is a small chunk of text, often part of a word, that models count toward both input limits and pricing.
This HTML snippet includes a menu and a tracking script around a short policy section:
<nav><a href="/">Home</a> | <a href="/pricing">Pricing</a></nav>
<div class="post-body">
<h2 class="post-title">Return Policy</h2>
<p>Items can be returned within <strong>30 days</strong>.</p>
<ul><li>Keep the receipt</li><li>Use the original box</li></ul>
</div>
<script src="/analytics.js"></script>The Markdown version keeps only the content and its structure:
## Return Policy
Items can be returned within **30 days**.
- Keep the receipt
- Use the original boxOlostep's glossary entry on HTML-to-Markdown conversion walks through the process for scraped pages. In a February 2026 post on Cloudflare's Markdown for Agents, Cloudflare reported: "This blog post you're reading takes 16,180 tokens in HTML and 3,150 tokens when converted to markdown. That's a 80% reduction in token usage." That figure describes one page, and savings on other pages depend on how much markup each one carries.
HTML to Structured JSON for Apps and Agents
Structured data extraction turns a page into JSON with named fields that code can read directly. A schema defines those fields and their types, such as name as a string and price as a number. The extractor then fills them in from the page.
For a sample product page, the output looks like this:
{
"name": "Trail Runner 2",
"price": 89.99,
"currency": "USD",
"in_stock": true
}Olostep's glossary page on converting HTML into structured JSON explains schema-based extraction step by step.
Valid JSON is not the same as correct JSON, because a schema fixes the shape of the output and cannot confirm its values. If a page shows a list price of $119.99 and a sale price of $89.99, either number passes a price: number check. Test extraction on pages where a field is missing, and confirm the output returns null for that field instead of an invented value.
Matching the Output Format to the Consumer
The right output format depends on what reads the data next.
| Consumer | Format | Why it fits |
|---|---|---|
| BI dashboard | Table or CSV | Dashboards query rows and columns, and joins need consistent types |
| RAG pipeline | Markdown chunks | Headings mark natural split points, and clean text keeps token counts down |
| AI agent or app | JSON that matches a schema | Code reads fields by name, so names and types must stay stable |
| Fine-tuning job | JSONL | Each line holds one training example as a JSON object |
RAG (retrieval-augmented generation) is a setup that finds relevant text and adds it to a model's prompt. One data transformation pipeline can feed several consumers, so the same page may need more than one format.
Rule-Based Parsers vs. LLM Extraction
Web pages become JSON through one of two methods: a rule-based parser or LLM extraction. A parser uses fixed rules, such as CSS selectors (patterns that point to elements on a page), to find each field. LLM extraction sends the page and a schema or prompt to a language model, which returns the fields.
| Criterion | Rule-based parser | LLM extraction |
|---|---|---|
| Repeatability | Same page in, same output out | Output can vary slightly between runs |
| New or changed layouts | Breaks or returns empty fields until the rules are updated | Can handle layouts it has not seen before |
| Cost | Cheaper than LLM extraction, per Olostep's docs | 20 credits per llm_extract call vs. 1 credit for a standard scrape, per Olostep's pricing |
| Speed | Faster than LLM extraction, per Olostep's docs | Slower, because each page adds a model call |
| Best use | Recurring jobs with stable fields like price, title and SKU | One-off schemas, unfamiliar layouts and fuzzy fields |
A hybrid setup uses each method where it fits. Parsers extract stable fields such as price and SKU, and LLM extraction handles fuzzy fields such as a summary or a product category. Olostep's comparison of parsers versus LLM extraction covers when to choose each.
Parsers also have a silent failure mode when a site changes its layout. Olostep's February 2026 post on parsers describes it: "The job succeeds, status is green, and the HTML still arrives… Price becomes null." Monitor each field's null rate and value range, and alert when either shifts after a site change (often called drift).
Transforming Web Data at Fetch Time With Olostep
Olostep's scrape endpoint can return Markdown, HTML, text or structured JSON for a URL in one request. The transformation happens as the page is retrieved, before the data reaches your pipeline.
This minimal request asks for Markdown plus structured JSON that follows a schema:
curl -X POST https://api.olostep.com/v1/scrapes \
-H "Authorization: Bearer $OLOSTEP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url_to_scrape": "https://example.com/products/trail-runner-2",
"formats": ["markdown", "json"],
"llm_extract": {
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"price": {"type": "number"}
}
}
}
}'The formats list sets the outputs, and llm_extract with a schema defines the structured fields. The response includes the Markdown and the structured JSON for the page.
For large jobs, batch processing for large URL lists applies the same extraction across many pages. Per Olostep's docs, a single batch can hold up to 10k URLs.
Store provenance with every record: the source URL, the retrieval timestamp and the parser ID or schema version. Provenance is a record of where data came from, and it lets you trace a bad value back to its page and rule.
To see the difference on your own data, scrape one page you already use and compare its Markdown with the raw HTML. Olostep offers 500 free credits to start.
Frequently Asked Questions
The answers that follow cover examples, synonyms, tools, when to skip transformation and formats for LLMs.
What Are Examples of Data Transformation?
Common data transformation examples include converting date formats, splitting a full name into first and last name columns and totaling daily sales into monthly sales. Turning a web page into Markdown for an LLM, or into JSON for an app, is also data transformation.
What Is Another Word for Data Transformation?
Data conversion is the closest synonym, though it often means a one-time format change such as XML to JSON. Data wrangling (or munging) stresses hands-on, exploratory reshaping, while data preparation covers every step before analysis.
What Is a Data Transformation Tool?
A data transformation tool is software that applies transformation rules to data, such as dbt for SQL in the warehouse or pandas for Python. Web data APIs such as Olostep apply the same idea to web pages, turning them into Markdown or JSON.
Does Every Dataset Need to Be Transformed?
No. Data that already matches the target system's shape can be loaded as is, and every transformation adds cost, so apply only what the consumer needs.
What Format Is Best for Feeding Web Data to an LLM?
Markdown works best when a model needs to read the page, such as for RAG context or summaries. Schema-based JSON works best when code or an agent needs specific fields, such as a price or a date.