Ground truth in machine learning is the verified reference data that a model learns from and is scored against. It is the set of labels, values, or answers that a team treats as correct for each example, such as "cat" for a photo of a cat or the actual sale price of a house.
Ground truth is a decision, not a discovery. People choose what counts as correct, how to label it, and which examples to include. An ICML 2026 position paper notes that the term "ground truth" commonly refers to datasets viewed as containing the true values of a given concept, and its authors argue that those values are built through human choices.
That distinction matters in practice. If the reference is wrong, a model can be scored as accurate while giving wrong answers, or scored as wrong while giving correct ones.
Why Is It Called Ground Truth?
The term has been used in remote sensing since the 1960s, as the ICML 2026 paper explains. There, it meant information gathered by measuring directly at ground level, instead of by interpreting remotely collected data.
"Ground truthing" means checking data against direct observation. The hyphenated form, "ground-truth," is used as an adjective or verb, as in "ground-truth labels" or "to ground-truth a map."
How Ground Truth Is Used in Training, Validation and Testing
Ground truth is used at every stage of a model's life, from learning to production. Each stage uses a separate slice of data so the model is not graded on examples it has already seen.
| Stage | What ground truth does | Example |
|---|---|---|
| Training | Gives the model labeled examples to learn patterns from | 50,000 emails marked spam or not spam |
| Validation | Helps tune settings and choose between model versions | A held-out set used to compare two classifiers |
| Testing | Produces the final score on data the model never saw | A locked test set scored once before release |
| Production | Checks live predictions against real outcomes that arrive later | Whether a flagged transaction was later confirmed as fraud |
Real outcomes can arrive days or months after a prediction. Until they arrive, teams cannot measure live accuracy directly.
Ground Truth Examples by Task
Ground truth takes a different shape for each type of machine learning task:
- Classification: a category for each item, such as "spam" or "not spam."
- Regression: a measured number, such as a home's final sale price.
- Object detection: boxes drawn around each car or person in an image.
- Image segmentation: a label for every pixel, such as "road" or "sidewalk."
- LLM question answering: a reference answer plus the facts any correct answer must include.
How Ground Truth Data Is Created
Ground truth data is created in three main ways: people label it, real-world measurements record it, or models draft labels that people then review. A single project can combine these methods.
- Human annotation: crowd workers or domain experts label each example. Experts cost more but are needed for tasks like medical or legal review.
- Measurement and outcomes: sensors, transactions, and logged events record what actually happened.
- Model-assisted labeling: a model proposes labels and humans accept, fix, or reject them.
Teams also choose between building ground truth in-house and buying it. A list of AI training data providers can help compare outside options.
Tool names can cause confusion here. Amazon SageMaker Ground Truth, for example, is a service for training data labeling using humans, as described in AWS's SageMaker Ground Truth docs. AWS notes that the service is no longer open to new customers, though existing customers can keep using it. It is a product, not the concept itself.
Measuring Label Quality With Inter-Annotator Agreement
Inter-annotator agreement measures how often different people give the same label to the same item, beyond what chance would produce. A common score is Krippendorff's alpha. In a 2025 label-variation case study, the authors use this scale, where 1 means perfect agreement and 0 means chance-level agreement.
Agreement drops as tasks become more subjective. In the same study, Krippendorff's alpha ranged from 0.81 to 0.38 across questions on one dataset, compared with a 0.667 reliability threshold. The easy question ("Is this post political?") scored high, while a judgment-heavy question scored low.
Low agreement carries information about the item being labeled. Majority voting can hide real disagreement and produce a single label that does not reflect how people actually judged the item.
Ground Truth vs. Labels, Benchmarks and Reference Answers
Ground truth is the overall reference standard, while the related terms describe specific parts or uses of it.
| Term | What it means | Typical use |
|---|---|---|
| Ground truth | The full set of data treated as correct | Training and scoring models |
| Label | One annotation on one example | A single "spam" tag |
| Gold standard | A small, expert-verified subset trusted most | Auditing labelers and other labels |
| Benchmark | A public, fixed test set | Comparing models across teams |
| Reference answer | Ground truth for generative tasks, often with several acceptable forms | Scoring LLM and RAG outputs |
When Ground Truth Is Wrong
Ground truth can be wrong. It inherits human mistakes, design choices, and data leaks, and a model's score can only be as accurate as the reference it is measured against.
Label Errors in Popular Benchmarks
Even well-known test sets contain wrong answers. In the MMLU-Redux study, researchers wrote: "We estimate that 6.49% of MMLU questions contain errors."
The MMLU-Redux authors say these errors obscure the true capabilities of LLMs. When answer keys are wrong, scores reflect both model skill and mistakes in the key.
Label Design Changes the Score
How you define the labels can change accuracy without changing the model. The ICML 2026 paper reports a Mars terrain case: a classifier scored 74% accuracy with 14 terrain classes, and a version trained after regrouping the same pixels into 5 broader classes scored 92%.
The model did not get better at seeing terrain. The definition of "correct" became easier. Document your label scheme so readers can tell real gains from changed rules.
Contamination: When Test Answers Leak Into Training Data
Contamination happens when test questions and their answers end up in a model's training data. A high score may then reflect memorization rather than skill.
An EMNLP 2024 contamination report found contamination levels ranging from 1% to 45% across benchmarks. Public test sets can spread online and end up in training data, so keep a private, held-out test set as well.
Ground Truth Has an Expiration Date
Many reference answers are only true at a point in time. Prices, officeholders, software versions, and company policies all change, so a reference answer written last year can mark today's correct answer as wrong.
This problem affects LLM evaluation directly. In the FreshQA benchmark (Aug 2024), researchers observed that "all LLMs struggle to answer questions that require fast-changing world knowledge." Its authors also committed to updating FreshQA's ground-truth answers on a regular schedule.
It helps to separate two kinds of change. Sometimes the world changes, so the true answer is different. Other times the labeling rules or labelers change, so the same item gets a different label.
How to Keep Web-Derived Ground Truth Current
Keeping web-derived ground truth current means treating each reference answer as dated data with a source, not as a permanent fact. Use this checklist:
- Store provenance: save the source URL and retrieval date with every reference answer.
- Tag time sensitivity: mark each question as stable or time-sensitive, and re-check time-sensitive ones more often.
- Watch sources: use monitoring source pages for changes to flag answers whose sources were edited.
- Refresh in bulk: re-collect sources on a schedule with batch re-collection of URLs instead of rebuilding the full set by hand.
- Keep inputs clean: convert collected pages into LLM-ready data such as Markdown, so labelers and models read the same text.
- Review and version: send changed answers to human review, then publish a new version of the eval set with a changelog.
For time-sensitive questions, the Olostep Answers endpoint searches the live web and returns an answer with a sources array listing the URLs it used. Reviewers can check those sources before accepting a new reference answer.
Olostep handles source collection and change detection. It does not decide what is correct, so humans still approve every change. This review-and-refresh loop matches the approach described in agent-led dataset maintenance.
FAQ
What Is Ground Truthing?
Ground truthing is the process of checking data or model outputs against direct, verified observation. The term has long been used in remote sensing, where it meant measuring directly at ground level instead of interpreting remote data.
Is Ground Truth Always Correct?
No. Ground truth reflects human labeling choices and can contain errors, disagreement, or outdated answers.
What Is the Difference Between Ground Truth and Training Data?
Training data is the input a model learns from, and ground truth is the correct label or answer attached to that data. Ground truth is also used for validation and testing, not just training.
Can an LLM Create Ground Truth?
An LLM can draft labels or reference answers, but people should review them before they are treated as ground truth. Unreviewed model output can copy the same mistakes you are trying to measure.
How Much Ground Truth Do I Need?
It depends on the task, the number of classes, and how much the data varies. Start with a small, high-quality test set and grow it where the model makes the most errors.
How Often Should I Refresh Ground Truth?
Refresh time-sensitive answers whenever their sources change, and review stable answers on a regular schedule. Version each update so scores stay comparable over time.
What Is Amazon SageMaker Ground Truth?
Amazon SageMaker Ground Truth is an AWS service for labeling training data with human workers, and AWS has closed it to new customers. It is a tool for creating ground truth, not a different meaning of the term.