Human-in-the-loop (HITL) is a system design where a person reviews, corrects, or approves an AI system's work at defined points before that work counts. The AI does most of the work, and a human makes the calls that need judgment.
The "loop" is the cycle of input, model output, and feedback. A human is "in" the loop when the cycle cannot finish without their decision. An older form of the term, "man in the loop," describes the same idea.
Today, HITL covers two jobs. People help train machine learning models, and people supervise AI agents that take real actions. The second job is growing fast: McKinsey's 2026 State of AI survey reports that forty percent of respondents from large organizations (those with annual revenues of more than $1 billion) report scaling AI agents, up from 27 percent last year.
How Does a Human-in-the-Loop System Work?
A HITL system has five parts: a trigger, a reviewer, the evidence the reviewer sees, a decision, and feedback that changes what happens next. The trigger decides when the system stops and asks a person.
These parts show up in two places. One is model training. The other is an agent's runtime, while it is doing live work.
Human-in-the-Loop in Machine Learning Training
In training, humans supply the judgments a model learns from. This is the oldest form of HITL in machine learning.
- Labeling: People tag examples, such as "spam" or "not spam," so a model can learn from them (supervised learning).
- Active learning: The model picks the examples it is least sure about and asks a person to label only those.
- RLHF: In reinforcement learning from human feedback, people rank model outputs, and those rankings train a reward model that steers the main model.
Human-in-the-Loop in AI Agents at Runtime
At runtime, an agent pauses before a consequential action, such as sending an email, issuing a refund, or deleting a record. It saves its state in a checkpoint, waits for approval, and then resumes from that point.
The mechanics matter in production. Approvals need a timeout or escalation path, or the agent can wait indefinitely.
Steps should be idempotent, meaning safe to run twice, because some frameworks re-run part of a step when a paused agent resumes. Teams usually place these checkpoints using AI agent orchestration patterns that separate planning, tool calls, and approvals.
Human-in-the-Loop vs. Human-on-the-Loop vs. Human-out-of-the-Loop
The three terms describe how close the human sits to each decision. In the loop, a person approves before the action runs.
On the loop, the system acts and a person monitors and steps in when needed. Out of the loop, the system acts with no human involved.
| Model | Who acts | When the human steps in | Added latency | Best for |
|---|---|---|---|---|
| Human-in-the-loop | AI proposes, human approves | Before every gated action | High | High-risk, irreversible actions |
| Human-on-the-loop | AI acts, human supervises | On alerts, exceptions, or audits | Low | High-volume work with clear rules |
| Human-out-of-the-loop | AI acts alone | Never during operation | None | Low-risk, easily reversed actions |
Many production systems mix all three. A refund agent might approve small refunds on its own, flag mid-sized ones for later review, and require approval above a set amount.
Why Teams Add Humans to AI Workflows
Teams add humans where an AI mistake is costly, hard to undo, or legally sensitive. People also bring context a model lacks, such as a customer's history or a policy exception.
Regulation is making oversight a design requirement. Article 14 of the EU AI Act states that high-risk AI systems shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use. The same article names "automation bias," the tendency to over-trust a system's output, as a risk overseers must stay aware of.
The deadlines are set. According to the European Commission's enforcement timeline, the rules for high-risk systems listed in Annex III apply from 2 December 2027, while the rules for high-risk AI systems embedded into regulated products apply from 2 August 2028.
Where Human Review Breaks Down
Human review fails when people are asked to approve too much. This is called approval fatigue: reviewers stop reading and start clicking "approve."
Coding agents show the pattern clearly. In Anthropic's engineering write-up on Claude Code, published in March 2026, the company reported that users accept 93% of manual permission prompts. Anthropic's August 2026 post puts the figure at 97% of permission prompts.
The cost shows up in catch rates. In a controlled Anthropic study with 1,053 paid professional testers, human review caught just 13.6% of dangerous commands, while auto mode caught 89%. Humans also did worse as sessions got longer: they blocked about 17% of dangerous commands early on, dropping to about 5% after 50 or more prior prompts.
This is vendor research on one coding tool, so treat it as a signal, not a universal rate. The same trade-off can apply to other HITL designs: the more decisions you send to people, the less each review is worth. Large queues also stall agents while work waits for a free reviewer.
How to Decide What Goes to a Human
Send a decision to a human when it is risky, hard to reverse, or uncertain. Let the system handle everything else, and audit it afterward.
This matches how we approach agent-led dataset maintenance: agents detect, refresh, and validate changes, while humans review ambiguous diffs, schema remaps, and policy-sensitive content. Humans review only the cases that need their judgment.
Route by Risk and Reversibility
Risk and reversibility together tell you how much review an action needs. Plot each agent action on this grid.
| Easy to reverse | Hard to reverse | |
|---|---|---|
| Low risk | Automate fully | Light approval or batch sign-off |
| High risk | Automate, then review samples after | Required approval before action |
A wrong tag on an internal record is low risk and easy to fix. A wire transfer or a mass email is high risk and hard to undo.
Route by Confidence Thresholds
Confidence thresholds route each output based on how sure the system is. A common setup uses three bands.
- High confidence: Accept automatically.
- Medium confidence: Send to a review queue.
- Low confidence: Reject, retry, or re-fetch the source.
For extracted data, confidence signals include schema validation failures, missing required fields, and values outside an expected range. Reviewer corrections then become labeled examples for tuning the thresholds. The extraction method also changes how much reaches a reviewer, which is the core trade-off in parsers versus LLM extraction: deterministic parsers produce repeatable fields, while free-form extraction yields more borderline records.
Human-in-the-Loop for Agents That Use Live Web Data
Agents that search, scrape, and monitor the web face a specific problem: their source data changes without notice. Review triggers should come from changes in the data itself.
Show Reviewers the Source, Not Just the Answer
A reviewer can only judge an agent's output if they can see what it was based on. Show the source URL, the fetch timestamp, the page content as Markdown, and the schema used for extraction.
Show changes at the field level against the last known value, so a reviewer checks one diff instead of a full record. Tie each approval to the exact payload that was approved, so nothing different runs afterward. Our guide to data provenance for AI agents covers how to carry this evidence through a pipeline.
Trigger Reviews on Meaningful Changes
Instead of a person re-checking pages on a schedule, a monitor can watch them and open a review only when something meaningful changes. That moves the human from in the loop to on the loop.
For example, an agent tracking competitor pricing pages can use the Olostep Monitors API to detect a change, then route only that change to a person before updating a pricing model.
Sample Review at Batch Scale
At the scale of thousands of URLs, review failures, anomalies, and a random sample instead of every record. Reviewing everything at this volume brings back approval fatigue.
Running batch jobs with tracked outputs makes this practical, because missing or failed items are visible at the job level and can be routed to review on their own.
How to Measure Whether Human-in-the-Loop Works
You measure HITL by tracking whether reviews change outcomes, not just whether they happen. These metrics give a working baseline.
| Metric | What it tells you | Warning sign |
|---|---|---|
| Review rate | Share of decisions sent to humans | Rising with no drop in errors |
| Override rate | Share of reviews where the human changes the outcome | Near zero, which can mean rubber-stamping |
| Approval latency | Time from request to decision | Agents stalled for hours |
| Queue depth and timeouts | Work waiting for a reviewer | Growing backlog or expired requests |
| Post-approval error rate | Errors found after a human approved | Approved items failing later |
| Time per review | How long reviewers spend on each item | A few seconds on complex items |
A very low override rate is not proof that the AI is right. It can also mean reviewers have stopped looking, so pair it with spot checks of approved items.
Human-in-the-Loop Examples
HITL is common where an AI output carries real consequences. Common examples include:
- Content moderation queues where people review posts a classifier flagged as uncertain.
- Loan and insurance claim approvals where a model recommends and an officer decides.
- Coding agents that ask permission before running shell commands or pushing code.
- E-commerce repricing agents that need approval before a large price change goes live.
- Lead-enrichment pipelines that send low-confidence company records to a researcher.
Frequently Asked Questions
What does human-in-the-loop mean?
Human-in-the-loop means an AI system includes a person who reviews, corrects, or approves its work at set points before the result is used.
Is ChatGPT human-in-the-loop?
ChatGPT was trained with human feedback (RLHF), which is a form of HITL, but a normal chat is not HITL at runtime. Agent features that ask you to confirm an action before it runs are HITL.
What does "man in the loop" mean?
"Man in the loop" is an older form of the same term: a person must take part in the decision cycle.
Is human-in-the-loop the same as RLHF?
No. RLHF is one HITL technique used to train models, while HITL also covers runtime review and approval of AI actions.
When should you not use human-in-the-loop?
Skip per-decision review for high-volume, low-risk actions that are easy to reverse. Review there adds delay without catching many errors, so monitoring and audits work better.
Does the EU AI Act require human-in-the-loop?
The EU AI Act requires effective human oversight for high-risk AI systems, which does not always mean approving every decision. Oversight can also mean monitoring the system and being able to intervene or stop it.