Data integrity means data stays accurate, complete, and consistent from the moment it is created or collected until the moment it is used. Data with integrity has not been changed, lost, or corrupted along the way, whether by accident or on purpose.
NIST's glossary defines data integrity as "a property whereby data has not been altered in an unauthorized manner since it was created, transmitted, or stored." In short, the data you read is the same data that was written.
For example, a customer places an order for $120. If the database later shows $12, or the invoice shows a different customer, the order record has lost its integrity.
Why Data Integrity Matters
Data integrity matters because software, reports, and AI systems act on data without a person checking each value. One bad record can flow into dashboards, billing, model training, and agent decisions before anyone notices.
Trust is already low. A Precisely and Drexel survey of data and analytics professionals reported "67% of respondents saying they don't have complete trust in their organizations' data for decision-making – up from 55% last year." That survey was published in November 2024 by Precisely, a vendor of data integrity software.
Bad data tends to become harder to fix the further it travels. Catching a wrong value at the source is cheap. Finding it after it has shaped a forecast or an AI answer is not.
Data Integrity vs. Data Quality vs. Data Security
Data integrity, data quality, and data security are related, but each one answers a different question. Integrity asks whether data is intact and valid, and quality asks whether data is fit for its purpose. Security asks whether data is protected from people who shouldn't access it.
| Concept | Question It Answers | Example Failure | Typical Controls |
|---|---|---|---|
| Data integrity | Is the data unaltered, complete, and structurally valid? | A file is corrupted during transfer | Checksums, constraints, validation, audit trails |
| Data quality | Is the data accurate and useful for this task? | A customer list is full of outdated emails | Profiling, cleansing, enrichment |
| Data security | Is the data protected from unauthorized access? | An attacker steals a database | Encryption, access control, monitoring |
These terms overlap, and teams draw the lines between them differently. The UK regulator's MHRA data integrity guidance notes that "controls required for integrity do not necessarily guarantee the quality of the data generated." A practical model is to treat them as three overlapping checks: data can pass one and fail another.
A valid record is not necessarily a true record. A record can match every rule in your schema and still be wrong, because the source itself was wrong or out of date.
Security and integrity also meet in the CIA triad, the infosec model of confidentiality, integrity, and availability. The NIST information security definition describes protecting information "in order to provide confidentiality, integrity, and availability." A breach can damage all three.
IBM's 2026 breach report puts the global average cost of a data breach at USD 4.99 million, "a 12% increase over last year and a record high." That figure measures security incidents as a whole, not integrity failures alone.
Types of Data Integrity
Data integrity comes in two main types: physical and logical. Physical integrity protects the data where it is stored and moved. Logical integrity makes sure the data makes sense inside a system.
Physical Integrity
Physical integrity is the protection of data against damage to the hardware, storage, or network that holds it. Threats include disk failure, power loss, bit flips in memory, and errors during file transfers.
Controls for physical integrity fall into three jobs:
- Detect: checksums and hashes (such as SHA-256) produce a short fingerprint of a file, and if the fingerprint changes, the file changed. The NIST checksum definition describes a checksum as "A value computed on data to detect error or manipulation."
- Prevent: error-correcting memory and redundant storage (RAID) reduce the chance that hardware faults corrupt data.
- Recover: backups and replicas let you restore a correct copy after damage.
Logical Integrity
Logical integrity is the set of rules that keeps data correct and consistent inside a database or application. Database engines can enforce most of these rules automatically.
| Type | What It Ensures | Example Rule | Enforced By |
|---|---|---|---|
| Entity integrity | Every record is unique and identifiable | Each customer has one non-empty ID | Primary key |
| Referential integrity | Links between tables stay valid | Every order points to a real customer | Foreign key |
| Domain integrity | Values have the right type and range | Order totals are numbers above zero | Data types, check constraints |
| User-defined integrity | Business rules are followed | A discount can't exceed 50% | Triggers, application logic |
In the PostgreSQL constraints documentation, a primary key "requires that the values be both unique and not null." The same page says a foreign key "maintains the referential integrity between two related tables."
Many databases also group changes into ACID transactions. ACID stands for atomicity, consistency, isolation, and durability. Atomicity is the all-or-nothing part: the PostgreSQL transactions tutorial says a transaction "bundles multiple steps into a single, all-or-nothing operation."
What Breaks Data Integrity
Data integrity often breaks through ordinary events as well as attacks. Common causes include:
- Human error: typos, duplicate entries, or a wrong value pasted into a form.
- Transfer errors: files truncated or altered as they move between systems.
- Software bugs: a failed migration or a faulty transformation script rewrites good data.
- Malicious changes: an attacker or insider edits or deletes records.
- Hardware failure: disks, memory, or servers fail mid-write.
Data collected from outside your systems adds a quieter set of problems. A website redesign can rename or move fields, which is called schema drift.
A page can load only partly before it is captured. A price can arrive as text with a currency symbol instead of a number.
Silent failures are the hardest of these to detect. Many silent scraping failures happen when a site redesign breaks extraction and the scraper keeps running while returning empty or null values. Row-count and null-rate checks can catch these problems before bad records reach downstream systems.
How to Ensure Data Integrity
Ensuring data integrity means building controls that prevent bad data, detect changes, and recover from damage. Grouping practices by these three jobs makes gaps easier to spot.
Prevent Bad Writes With Validation and Constraints
Prevention stops bad data before it is saved. These are the main controls:
- Input validation: reject values that have the wrong type, format, or range.
- Database constraints: use primary keys, foreign keys, and check constraints.
- Access control: give each user and service only the permissions it needs.
- Transactions: wrap related changes so they succeed or fail together.
Detect Changes With Checks, Hashes, and Audit Trails
Detection catches problems that get past prevention. Compare checksums before and after every transfer. Reconcile row counts between source and destination.
For pipelines, set thresholds on null rates and value ranges, and alert when a run returns far fewer rows than usual. Keep audit trails that record who changed what and when.
Recover With Backups and Versioning
Recovery restores a known-good copy when detection finds damage. Take regular backups, keep point-in-time restore enabled, and version important datasets.
Test your restores. A backup that has never been restored has not been proven to work.
Data Integrity for Web and External Data
Data integrity for external data means checking data at the moment you collect it, because you don't control the source. Database-focused integrity controls assume data starts inside your own database. Many AI products, enrichment tools, and monitoring systems depend on data that starts on the web.
Four checks address the main risks for web data:
- Source fidelity: capture what the page actually showed, including content rendered by JavaScript. The Olostep Scrapes endpoint returns rendered pages as Markdown, HTML, or structured JSON, so the extraction step starts from the rendered page.
- Structural integrity: validate every extracted record against a declared schema. With schema-based extraction, you define field names and types up front, so a price arrives as a number, not a string.
- Completeness: confirm you collected every page or listing you expected. Start by crawling an entire site, then compare record counts to past runs to see when pages go missing.
- Provenance and freshness: store where and when each record came from, and watch for changes. Scheduled web monitors re-check sources on a schedule, and the guide on how change tracking works explains the baseline, re-scrape, and diff cycle.
A JSON Schema is a simple way to write the structural rules down. This example requires a product name, a positive numeric price, a source URL, and a fetch time:
{
"type": "object",
"required": ["name", "price", "source_url", "fetched_at"],
"properties": {
"name": { "type": "string", "minLength": 1 },
"price": { "type": "number", "exclusiveMinimum": 0 },
"currency": { "type": "string", "enum": ["USD", "EUR", "GBP"] },
"source_url": { "type": "string", "format": "uri" },
"fetched_at": { "type": "string", "format": "date-time" }
}
}A record that passes the schema also carries its own provenance. Sample record:
{
"name": "Trail Running Shoe",
"price": 129.0,
"currency": "USD",
"source_url": "https://example.com/products/trail-shoe",
"fetched_at": "2026-10-17T09:30:00Z",
"content_hash": "sha256:9f2c..."
}The content_hash field lets you check later whether the stored content changed. The fetched_at field records when the price was collected, so an old price can't pass as a current one.
Data Integrity in AI and RAG Pipelines
Data integrity in AI means the data a model reads is complete, current, and traceable to its source. Retrieval-augmented generation (RAG) is a method where an AI system retrieves stored documents and uses them to ground its answer. A RAG answer can only be as reliable as the documents it retrieves.
Gartner's AI-ready data prediction states: "Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data."
Use this checklist for RAG ingestion:
- Keep chunks complete: split documents without cutting tables or sentences in half, and preserve text encoding.
- Carry metadata: attach the source URL, fetch time, and content hash to every chunk.
- Re-index on change: when a source changes, update or remove the old chunks.
- Cite sources: return source links with answers so users can verify claims.
- Validate extracted fields: check that values pulled by an LLM appear in the source text, because a model may fill an empty field with a value that isn't on the page.
Clean inputs make each step easier. Starting from LLM-ready web data in Markdown or JSON means navigation, scripts, and other page noise are stripped out before chunking.
ALCOA: Data Integrity in Regulated Industries
ALCOA is a set of five data integrity principles used in regulated industries such as pharmaceuticals. FDA's ALCOA framework states: "Data integrity refers to the completeness, consistency, and accuracy of data. Complete, consistent, and accurate data should be attributable, legible, contemporaneously recorded, original or a true copy, and accurate (ALCOA)."
| Letter | Principle | Plain Meaning |
|---|---|---|
| A | Attributable | You know who or what created the data |
| L | Legible | The data can be read and understood |
| C | Contemporaneous | The data was recorded when the event happened |
| O | Original | It is the first record or a true copy |
| A | Accurate | The data is correct |
The WHO data integrity guideline (2021) extends these principles to data that is also "complete, consistent, enduring, and available; commonly referred to as “ALCOA+”." The core ideas carry over well beyond pharma: "attributable" and "contemporaneous" map directly to storing a source URL and fetch time for web data.
Frequently Asked Questions
What Is Data Integrity in Simple Terms?
Data integrity means your data stays correct, complete, and unchanged from when it is created or collected to when it is used.
What Are the Main Types of Data Integrity?
The two main types are physical integrity, which protects stored and moving data from damage, and logical integrity, which includes entity, referential, domain, and user-defined rules.
What Is an Example of a Data Integrity Violation?
An order that points to a customer ID that no longer exists is a referential integrity violation.
Is Data Integrity the Same as Data Quality?
No. Integrity means data is intact and valid, while quality means data is accurate and useful for a specific purpose.
How Do Checksums Protect Data Integrity?
A checksum creates a short fingerprint of data, so if the fingerprint changes after a transfer or over time, you know the data changed.
Is Stale Data a Data Integrity Problem?
Stale data is usually a quality issue, but it becomes an integrity issue when an old value is stored or shown as current without a timestamp.
How Do You Check the Integrity of Scraped Data?
Validate each record against a schema, track row counts and null rates across runs, and store the source URL and fetch time with every record.
Does a Database Guarantee Data Integrity End to End?
No. Database constraints protect data inside the database, but they can't catch errors that happened before the data arrived, such as a page that was captured only partly.