What are HTTP Status Codes?

HTTP looks simple from the outside: send a request, get a response. But for API teams, web scrapers, crawlers, and AI agents, the difference between a reliable workflow and a broken one often comes down to how well you handle http status codes.

In this guide, we’ll break down the standard status codes, how they work, which common http status codes matter most, and how Olostep uses clear HTTP behavior to make web data extraction easier to automate.

What is an HTTP status code?

HTTP status codes are three-digit response codes returned by a web server to indicate whether a client's request succeeded, failed, or needs more action. They are part of the hypertext transfer protocol and help browsers, API clients, scrapers, and AI agents decide what to do next.

Every response from Olostep’s Web Data API includes an http status code that drives client behavior, retry logic, and error handling. API functionality relies on status codes to inform the client about successful or failed requests.

There are five classes of HTTP status codes:

1xx informational, such as 100 Continue
2xx success, such as 200 OK
3xx redirection, such as 301 moved permanently
4xx client error, such as 400 bad request
5xx server error, such as 500 internal server error

HTTP status codes indicate whether a request succeeded or failed. Understanding appropriate status codes is critical for stable REST APIs, web scraping pipelines, and automated client requests.

For example, imagine an AI agent calling Olostep to scrape a product page. A 200 OK means the agent can parse the requested data, a 404 Not Found means the product page probably does not exist, a 429 Too Many Requests means the agent should slow down, and a 503 service unavailable means it should retry later instead of assuming the data is gone.

How HTTP status codes work in client–server communication

An http request starts when a client sends an http method, URL, request headers, and sometimes a request body. The server responds with a status line, response headers, and an optional message body.

A typical http status line looks like this:

HTTP/1.1 200 OK

It contains the http version, numeric response code, and reason phrase. The reason phrase is helpful for humans, but automated systems should rely on the numeric http status.

HTTP codes are categorized by their first digit indicating the nature of the response:

1 means informational
2 means success
3 means redirection
4 means client error
5 means server error

A simple Olostep timeline looks like this:

An AI data pipeline sends a post request to Olostep with a target URL.
Olostep validates the request headers, request body, API key, and requested resource.
Olostep fetches the target resource, renders JavaScript if needed, and structures the output.
The server responds with a final response: 200 OK with data, or a 4xx/5xx code with a JSON error payload.

Example error payload:

{
  "status": 400,
  "error_code": "invalid_parameter",
  "message": "Parameter 'url' must be a valid URL.",
  "details": {
    "url": "Missing protocol."
  },
  "request_id": "req_abc123"
}

Many observability tools, server logs, and customer logging stacks key off status code ranges. A spike in 4xx client error responses often points to client-side bugs, while a spike in 5xx server error responses usually signals platform, dependency, or upstream server problems.

The five classes of HTTP status codes

The HTTP specification RFC 9110 defines five standard classes of status codes. There are five classes of HTTP status codes, and each class tells the client how to interpret the response at a high level.

1xx codes indicate informational responses from the server.
2xx codes confirm successful processing of a client's request.
3xx codes require additional action, usually URL redirection.
4xx codes indicate client errors, requiring user action to fix.
5xx codes signal server errors, indicating issues on the server side.

Most production APIs, including web data platforms like Olostep, lean heavily on 2xx, 4xx, and 5xx codes in daily traffic. 3xx is more common across front-end sites, CDNs, documentation pages, and search engines. Later sections cover common http status codes like 200 OK, 400 bad request, 404 Not Found, 500 internal server error, and 503 service unavailable.

1xx informational responses

1xx status codes are interim responses. They mean the server has received the request header fields and is still working toward a final response.

The most familiar example is 100 Continue. A client can send the expect request header field before uploading a large request entity. If the origin server accepts the request headers, it returns 100 Continue, so the client can complete request transmission without unnecessary risk processing.

This is useful for large uploads, batch scraping jobs, or any case where sending the full message body is expensive.

Other examples include:

101 Switching Protocols, often used when upgrading HTTP to WebSocket.
102 Processing, used for a webdav request when work is still happening.
103 Early Hints, sometimes used by CDNs to preload assets.

Most developers do not handle 1xx directly because client libraries often hide the interim response and surface only the final response.

2xx success responses

2xx success codes indicate successful request processing. 2xx status codes mean the request succeeded, with different codes explaining the type of success.

The 200 OK status code indicates a successful request. 200 OK means the request was successful and data returned. A 200 OK status code is ideal for SEO because it tells browsers and search engines that the page is available.

In Olostep, a successful get request to retrieve scraped HTML, markdown, or structured JSON product data would normally return 200 OK.

201 Created indicates a new resource was successfully created. Use it when a post request creates a crawl job, stored query, crawl configuration, or other new resource.

202 Accepted means the request is accepted but not yet completed. This is useful for asynchronous crawling, batch URL processing, or long-running extraction jobs.

204 No Content means the request was successful with no data returned. It is a good fit for canceling a scheduled crawl, deleting a stored configuration, or updating something where no response body is needed.

3xx redirection responses

3xx codes tell the client to look elsewhere or take an extra step before receiving the requested data. 3xx codes require additional action, usually URL redirection.

A 301 Moved Permanently status code indicates a permanent URL change. A 301 status code tells search engines to update their index to the new URL, and a 301 redirect preserves link equity for SEO.

308 Permanent Redirect is also permanent, but it preserves the request method and request body. This matters for APIs, where changing POST to GET during a redirected request can break behavior.

302 Found and 307 Temporary Redirect are for temporary moves. 307 is safer for APIs because it preserves the original request method.

A scraper or Olostep’s internal fetcher should usually follow redirects, but only up to a maximum depth. Redirect loops slow down crawlers, bots, and human users.

304 Not Modified is different. It supports caching: if a previous request already fetched a page and the content has not changed, the server can return 304 so the client uses its cached copy.

Search engines use status codes to determine how to index a website. A 404 status code tells search engines to remove a page from their index. A 410 Gone status code indicates permanent removal from search index.

4xx client error responses

4xx status codes indicate client-side errors. 4xx codes indicate client errors that require user action.

400 Bad Request means the server couldn't understand the request. Use 400 bad request for malformed request syntax, invalid request message framing, invalid query parameters, or a broken JSON request body.

401 Unauthorized means the client did not provide valid authentication credentials. In some cases, the origin server requires valid authentication before it will return the target resource.

403 Forbidden means the client lacks permission to access the resource. The 403 Forbidden status code means access is denied to the resource, even if the client has valid authentication.

404 Not Found indicates the requested resource doesn't exist. The 404 Not Found status code means the resource doesn't exist, such as an invalid job ID, broken product URL, or unknown endpoint.

410 Gone indicates the resource was intentionally deleted and won't return.

409 Conflict is useful when request conflicts occur, such as duplicate resource creation or version mismatch.

422 Unprocessable Entity is common when syntax is valid but the request body fails business validation.

Lesser-known client-side codes also matter. 407 can require a client to authenticate with a specified proxy before using network access through a proxy server. 421 can signal deceptive request routing. 414 can apply when a request uri is too long. 413 can apply when a request entity is too large. 426 may require an upgrade. 451 can be used for legal restrictions.

5xx server error responses

5xx status codes indicate server-side errors. 5xx codes signal server errors that need immediate attention.

A 5xx server error means the server understood the client's request but could not complete it because the server encountered an internal problem, dependency issue, or upstream failure.

500 Internal Server Error indicates an unknown server issue. The 500 Internal Server Error indicates a server-side issue and often maps to unhandled exceptions, bugs, or an internal configuration error. A production internal server error response should never leak stack traces.

502 Bad Gateway occurs when an invalid response is received from an upstream server. This can happen when a gateway, proxy, or scraping backend receives broken data from the target site.

503 Service Unavailable means the server is temporarily down. A 503 Service Unavailable status can affect user experience and SEO, especially if it persists.

504 Gateway Timeout indicates a timeout from an upstream server. It means a proxy or gateway did not get a timely response.

Other server-side codes include 501 Not Implemented and 505 http version not supported. In all cases, clients should treat many 5xx responses as retryable, but only with exponential backoff to avoid turning one server failure into a retry storm.

Most common HTTP status codes developers must know

Although the spec defines dozens of standard status codes, a small subset appears in most API dashboards.

Code	Meaning	When to use it
200	OK	Successful get request or synchronous API call with data returned.
201	Created	A new resource was created successfully.
204	No Content	The request succeeded, but no message body is returned.
301	Moved Permanently	A permanent URL migration.
304	Not Modified	Cached content is still valid after a conditional head request or GET.
400	Bad Request	Malformed JSON, invalid parameters, or malformed request syntax.
401	Unauthorized	Missing or invalid authentication credentials.
403	Forbidden	Access denied despite valid authentication.
404	Not Found	The requested resource does not exist.
409	Conflict	The request conflicts with current resource state.
422	Unprocessable Entity	The syntax is valid, but validation fails.
429	Too Many Requests	The client sent too many requests and should slow down.
500	Internal Server Error	Unknown server-side failure.
502	Bad Gateway	Gateway received an invalid response from an upstream service.
503	Service Unavailable	Temporary overload or maintenance.
504	Gateway Timeout	Upstream service failed to return in time.
A few comparisons matter:

301 vs 302 vs 307: permanent vs temporary, and whether the request method is preserved.
401 vs 403: authenticate first vs authenticated but not allowed.
400 vs 422: malformed request vs semantically invalid request.
404 vs 410: unknown resource vs intentionally removed resource.

Olostep’s Web Data API is designed around consistent JSON responses so AI agents can reliably branch logic based on the code instead of parsing free-form error pages.

Success & redirection codes in practice (2xx, 3xx)

For CRUD-style APIs, the mapping should be simple:

Use 200 OK when returning a representation.
Use 201 Created when a post request creates a crawl config, stored query, or job.
Use 202 Accepted when work starts but continues asynchronously.
Use 204 No Content when an update or delete succeeds without returning data.

For redirection, 301 and 308 are useful when migrating documentation URLs, dashboards, or API-facing frontends. 301 moved permanently is especially important for SEO because it tells search engines that the forwarding address is now canonical.

CDNs and load balancers may inject 302 or 307 redirects, often from HTTP to HTTPS. API clients should handle them gracefully, but avoid long chains such as 301 → 302 → 301.

When sending redirect codes, always include a valid Location header so future requests and subsequent requests can follow the destination deterministically.

Client error codes in practice (4xx)

You can group 4xx codes by cause:

Syntax and validation: 400, 422
Authentication and authorization: 401, 403
Resource issues: 404, 409, 410
Rate and size limits: 413, 414, 429

Example 400 bad request payload:

{
  "status": 400,
  "error_code": "malformed_json",
  "message": "The request body is not valid JSON.",
  "details": {
    "line": 1,
    "field": "url"
  },
  "request_id": "req_400"
}

Example 422 payload:

{
  "status": 422,
  "error_code": "validation_failed",
  "message": "The URL must use http or https.",
  "details": {
    "url": "Unsupported protocol."
  },
  "request_id": "req_422"
}

Olostep-style integrations should return 401 Unauthorized for invalid API keys and 403 Forbidden when a plan lacks access to an endpoint, domain, or feature.

For rate limits, 429 Too Many Requests should include Retry-After or rate-limit counters when possible. According to http.dev’s 429 reference, 429 exists specifically to signal that the user has sent too many requests in a given time.

4xx client error codes should not be auto-retried by agents without changing the request. Retrying a bad request only creates noise and can make a request failure harder to debug.

Server error codes in practice (5xx)

Server-side failures usually fall into a few groups:

Application exceptions: 500
Unsupported behavior: 501
Upstream connectivity: 502, 504
Controlled overload or maintenance: 503

Olostep may surface 5xx codes when target sites throttle, block, fail, or time out. Its infrastructure can retry upstream requests internally while still returning precise codes and structured error bodies to the user when the request failed.

Good client strategies include:

Retry selected 5xx responses with exponential backoff.
Use circuit breakers when 500, 502, 503, or 504 rates spike.
Fall back to cached data when 503 service unavailable appears frequently.
Avoid leaking stack traces for 500 internal server error.

Set alerts on spikes in 5xx server error rates. They usually indicate infrastructure, deployment, dependency, or target-site instability.

Choosing appropriate status codes for REST & data APIs

Picking the right http status code is part of API contract design. It is not just an implementation detail.

The principle is straightforward:

2xx for success
3xx for redirects
4xx client error when the request is wrong
5xx server error when the server failed

For REST operations:

POST should return 201 Created or 202 Accepted. Use 400/422 for validation failure and 409 for conflicts.
GET should return 200 OK, 304 Not Modified, or 404 Not Found.
PUT/PATCH should return 200 or 204 on success, with 400/422, 404, or 409 on failure.
DELETE should usually return 204 No Content or 404 Not Found, depending on your idempotency policy.

Idempotency matters. Repeating a PUT or DELETE should still yield a correct and consistent status code. For example, deleting a stored crawl might return 204 every time, or it might return 404 after the first deletion. Either policy can work if it is documented.

Olostep endpoints are designed to use predictable, standards-aligned codes so AI workflows can be authored once and reused across projects.

HTTP status codes in web scraping, crawling, and AI workflows

Status codes become even more important in web data pipelines because you are dealing with both your API and the target website.

A crawler should interpret:

3xx redirects as “follow carefully”
403 Forbidden as “blocked, restricted, or permission denied”
404 Not Found as “resource does not exist”
429 Too Many Requests as “slow down”
5xx as “server or upstream instability”

Robots, WAFs, bot detection systems, and anti-scraping controls may return 403, 429, or custom 4xx codes. In digital payment systems, healthcare portals, and other sensitive sites, a client certificate may also be required before the client can gain network access to protected resources.

Olostep helps normalize this complexity. A user’s own 400 bad request to Olostep is different from a 503 service unavailable coming from the target domain. Clean error_code values and response metadata help separate API misuse from target-site instability.

A reliable AI agent should follow this order:

Inspect the status code.
Parse the structured response body.
Decide whether to retry, re-parameterize, switch endpoint, or stop.

Proper handling of status codes enhances user experience by showing meaningful error pages. Proper error handling improves API reliability and user experience. Returning meaningful error messages aids in debugging API issues.

Status code strategies for resilient crawlers

A resilient crawler should treat status codes as routing signals.

Useful defaults:

Do not retry most 4xx client error responses.
Retry 408, 429, 502, 503, and 504 only with delay and backoff.
Follow 3xx redirects up to a maximum hop count.
Treat 404 and 410 as hard failures.
Treat 5xx, 429, and temporary 503 responses as soft or transient failures.

For reporting, label hard failures separately from soft failures. A high 404 rate may mean stale URLs. A high 503 rate may mean target instability. A high 400 rate may mean your integration is sending invalid payloads.

Olostep’s Web Data API can act as a single layer that handles JavaScript rendering, redirects, anti-bot complexity, and many upstream 5xx issues while returning clean JSON and coherent status codes.

Log additional metadata with each status code:

target domain
endpoint
crawl profile
job ID
region
user agent
latency
request_id

This makes large-scale scraping clusters much easier to debug.

Status-based routing also helps AI agents. If a direct scrape returns 404 Not Found, try a search endpoint. If 503 service unavailable appears repeatedly, switch region, use cached data, or try a mirror source.

Rate limits, throttling, and automation-friendly codes

429 Too Many Requests protects APIs and target sites from overload. It tells bots and agents to pause instead of sending more traffic.

Retry-After and custom headers can tell clients when to resume future requests. Rate-limit headers can also show remaining quota, reset time, or concurrency limits.

Automation-safe APIs should use:

deterministic codes
structured JSON for all non-2xx responses
no HTML-only error pages
stable error_code values
clear distinction between user-level throttling and global service protection

Use 429 for per-user or per-key throttling. Use 503 service unavailable for major incidents, maintenance, or platform-wide overload.

Olostep uses clear 4xx/5xx error patterns and structured bodies so teams can adapt agent behavior rather than hard-failing an entire workflow.

Best practices for implementing and documenting HTTP error codes

Status codes plus good error bodies are the primary debugging interface for API consumers.

Use the most specific standard status codes available. Avoid returning 200 for errors, and avoid using generic 500 for every failure. Multiple status codes exist because clients need precise signals.

A good 4xx or 5xx response should include:

status
error_code
message
details
correlation_id or request_id
optional retry_after
optional documentation link

Security matters. A 500 internal server error should not reveal stack traces, SQL fragments, hostnames, secrets, or deployment details. Log sensitive context internally, not in the public response.

Accurate API documentation should list expected http status codes per endpoint. This is especially important for teams integrating with Olostep or similar B2B SaaS APIs.

Designing error payloads and developer experience

A concise error payload template looks like this:

{
  "status": 403,
  "error_code": "domain_not_allowed",
  "message": "Your plan does not allow access to this domain.",
  "details": {
    "domain": "example.org"
  },
  "request_id": "req_403",
  "docs_url": "https://docs.example.com/errors/domain-not-allowed"
}

Messages should be stable enough for automation but clear enough for humans during debugging.

Clients should never need to scrape free-form text from error pages. They should rely on the numeric response code and machine-readable fields.

When an endpoint changes behavior or returned error codes, maintain a changelog. Status-code drift can break clients that made reasonable assumptions.

Testing status code behavior

Use curl, Postman, and automated test suites to verify that every endpoint returns the intended http status in both success and failure paths.

Test edge cases such as:

invalid auth
missing fields
conflicting updates
oversized payloads
unsupported http method
upstream timeout
proxy failure
rate-limit behavior

Olostep’s internal testing approach validates mappings from upstream website errors, such as 404, 500, or a 5xx server error, into user-facing API responses.

Simulate high-load scenarios to confirm correct 429 and 503 service unavailable behavior, including Retry-After headers and retry hints.

Coherent test coverage reduces surprises in production and strengthens the API contract.

Monitoring, analytics, and observability by HTTP status

Monitoring http status distributions is one of the fastest ways to detect regressions in APIs, crawlers, and web data pipelines.

Track:

share of 2xx vs 4xx vs 5xx
top individual error codes
latency by status code
response code by endpoint
request failure rate by target domain
server failure rate by region

Spikes in 4xx client error codes, such as many 400 bad request or 401 responses, often indicate client-side bugs, rollout issues, expired credentials, or bad configuration.

Sustained increases in 5xx server error rates, 500 internal server error, or 503 service unavailable values usually indicate infrastructure, dependency, or upstream server problems.

Olostep customers often filter logs and dashboards on http status code to optimize AI data pipelines and quickly spot scraping issues.

Alerting and SLOs based on status codes

Set alerts around error budgets or SLOs. For example, you might require that 99.5% of requests return 2xx or 3xx over a rolling window.

Separate 4xx and 5xx alerts because they usually point to different owners. Client teams often own 4xx spikes. Platform or SRE teams usually own 5xx spikes.

Critical codes worth alerting on include:

500
502
503
504
bursts of 429 Too Many Requests

Tag every error with dimensions such as domain, region, endpoint, job ID, plan, and crawl mode. This reduces triage time when a large crawl starts failing.

Feed status-code insights back into product decisions. If users frequently trigger 400 or 422 errors, your docs, SDK validation, or UI may need improvement.

Summary and next steps

HTTP status codes are the compact language by which clients, APIs, proxies, and web servers coordinate. 2xx means success, 3xx means redirection, 4xx means client error, and 5xx means server error.

Choosing appropriate status codes, especially for common http status codes like 200 OK, 400 bad request, 404 Not Found, 500 internal server error, and 503 service unavailable, directly improves the reliability of AI workflows and web data pipelines.

Olostep’s Web Data API surfaces clear, standards-based status codes and structured error bodies to simplify debugging, retry logic, scraping, crawling, search, and agentic research.

If your team depends on live web data, audit your current APIs and scraping clients for status-code misuse. Then consider Olostep if you want to offload crawling, scraping, redirects, rate limits, JavaScript rendering, and 4xx/5xx normalization into one well-instrumented API layer.