A good API rate limits strategy has two halves. On the server side, you set limits from tested capacity, enforce them with an algorithm that fits your traffic, and tell clients clearly when they hit a limit. On the client

side, you honor those signals with backoff, jitter and queues instead of hammering the endpoint again. Skip either half and the first traffic spike (or the first runaway AI agent) finds the gap.

This guide covers both halves, with the numbers and sources behind each decision.

TL;DR

  • There is no universally good limit. Load-test to find sustainable capacity, then set limits below it with headroom for failure recovery.
  • Pick the algorithm for the behavior you need: token bucket for controlled bursts, leaky bucket for smooth output, sliding window where precision matters.
  • Return HTTP 429 with a Retry-After header, and make clients wait, back off exponentially with jitter, and cap retries.
  • For LLM and AI integrations, limit requests and tokens, and ramp traffic gradually.
  • Roll out in dark mode first, watch what would be rejected, and keep a kill switch.

Table of Contents

Rate limiting, throttling and quotas: what is the difference?

People use the terms interchangeably, and in casual conversation that is fine. In architecture practice they are slightly different things. Rate limiting is the policy: how much work a caller may submit in a period. Throttling is the enforcement behavior: slowing, queuing or rejecting work when the policy or the system’s capacity is exceeded. The Azure Architecture Center’s rate limiting pattern frames it from the caller’s side: control how fast your application sends requests so you stay inside the service’s throttling limits.

That distinction leads to the most useful idea in this whole topic. Admission control protects the service. Client-side back-pressure protects your workflow and your users. Microsoft’s guidance is explicit that limiting reduces throttling errors but does not remove the need to handle throttling responses. If you run an API, you need the first. If you consume one, you need the second. Most teams do both, because most products are both a provider and a consumer.

A quota is a third concept: a longer-horizon allowance, such as a monthly cap or a credit balance. It behaves differently when exhausted, which matters later when we get to retries.

Start with capacity, not a magic number

“What is a good rate limit for an API?” is the question we hear most, and the honest answer is: the one your system can sustain. AWS’s Well-Architected guidance on throttling recommends throttling unexpected spikes and using load testing to establish known capacity. Requests under the rate get processed; requests over it get rejected with a message saying so.

Two mistakes show up again and again, and both are named as anti-patterns in that same AWS guidance:

  • Testing only one dimension. A benchmark of many small requests says little about a handful of huge ones. Test request rate and request size together.
  • Leaving defaults in place or ignoring scale-out. Default throttles are rarely tuned to your workload, and horizontally scaled consumers can multiply load unless you cap them.

Choose what you are counting

A limit needs a unit. Microsoft’s examples include 20 requests per second, 2 GiB per minute, or 20,000 resource units per second. Which one fits depends on where your real cost lives:

Unit Best when Watch out for
Requests per time window Requests cost roughly the same Cheap and expensive endpoints treated alike
Data volume Payload size drives load Harder for clients to predict
Weighted cost units Endpoints vary widely in cost More design work up front

Expensive endpoints, like search, exports or report generation, should generally consume more quota than a health check. A flat “100 per minute” that lets a client spend all 100 on your heaviest query is a limit in name only.

Choose who is being limited

Limits can be scoped per API key, user, tenant, IP address, endpoint, project, or a shared global pool. Per-key or per-tenant limits give the fairest results for authenticated APIs. IP limits are a reasonable first line for anonymous traffic, with the usual caveat that many real users can share one address. A global pool protects the backend but lets one noisy neighbor starve everyone, so pair it with a per-client limit rather than relying on it alone.

Pick the algorithm that matches your traffic

The algorithm is not an implementation detail. It decides how your API feels under pressure. Redis’s overview of throttling algorithms is a solid reference, and the trade-offs boil down to this:

Algorithm Behavior Use it when
Fixed window counter Counts requests per calendar window You want the simplest option and can tolerate boundary bursts
Sliding window log Stores each request timestamp Exact rolling enforcement matters, such as sensitive authentication flows
Sliding window counter Approximates a rolling window High traffic where approximation is acceptable
Token bucket Refills at a steady rate, allows bursts up to bucket size Clients legitimately send short bursts
Leaky bucket Drains at a constant rate The downstream system needs smooth, steady input

A few specifics worth knowing. Fixed-window counting is O(1) but vulnerable to boundary bursts: a client can send a full allowance at the end of one window and another at the start of the next. Sliding-window logs fix that precision problem but use O(N) space per identity. Token buckets are the default choice for most public APIs because they define two clean knobs: a refill rate for steady-state traffic and a bucket capacity for the largest burst you will accept.

Distributed enforcement

Once you run more than one API node, counters have to be shared or reconciled. Distributed implementations need atomic read-modify-write behavior, and Redis Lua scripts or an equivalent atomic mechanism are the common way to get it for token and leaky bucket decisions. If centralized coordination adds too much latency, some teams move decisions toward the edge with local counters per node and accept a looser global limit. That is a legitimate trade, as long as you choose it on purpose.

Shared and local counters across API nodes

Tell clients what happened

A limit that fails silently is a support ticket generator. When you reject a request, return HTTP 429 and say what the client should do next. The Microsoft Dev Proxy documentation on 429 describes it well: the request itself was fine, it was just sent too often, so waiting and retrying usually succeeds.

Include a Retry-After header, which can carry seconds or an HTTP date. Add headers that expose the limit, remaining allowance and reset time so well-behaved clients can pace themselves before they ever see a rejection.

Keep the status semantics clean. A 429 means this caller exceeded its allowance. A 503 means the service itself is overloaded. Mixing them up sends clients the wrong instruction, and it makes your own dashboards harder to read.

Expect inconsistency when you are the consumer. GitHub’s REST API distinguishes primary and secondary rate limits, and in some cases signals a limit with a 403 and a reset header rather than a 429. Every provider has its own dialect, so read the docs and test the failure path instead of assuming.

Handle rate limits as a client

Here is the order of operations we recommend when your code hits a limit:

  1. Check what kind of failure it is. A 429 for too many requests is recoverable. Quota or credit exhaustion may not be, and retrying it just burns time.
  2. Honor Retry-After if it is present. The server knows better than your guess.
  3. If there is no server delay, use exponential backoff with random jitter. Jitter prevents every client from retrying at the same instant.
  4. Cap retries. Infinite retries turn a short outage into a long one.
  5. Reduce concurrency or pace requests before the next attempt, rather than only waiting.
  6. Make writes idempotent so a retry cannot charge a card or create a record twice.

Microsoft specifically warns that blind retries create retry storms: a throttled service gets more load exactly when it can least handle it. Atlassian’s guidance, as summarized in our research, adds a refinement: back off only the quota scope that was affected, not your whole integration.

The Sentry engineering blog on dealing with API rate limits covers the same ground from a practical angle, including the question of whether you should always retry at all. Sometimes the right answer is to fail fast and tell the user.

Use a durable queue when you cannot drop work

If your app can produce work faster than a throttled downstream service can ingest it, buffering in memory is a trap. If the process stops, the data goes with it. Azure’s guidance points to durable messaging such as Service Bus, Queue Storage or Event Hubs, with controlled workers pulling from the queue at the permitted rate. That pattern absorbs bursts without losing data and keeps your user-facing path fast.

AI and LLM integrations need their own rules

Rate limits for AI APIs are different in one important way: requests are not the only currency. Large prompts and long completions cost tokens, and providers limit both. OpenAI’s rate limit documentation shows headers for request limits, token limits, remaining capacity, reset times and Retry-After, so your client should track requests and tokens separately.

There is also a trap that surprises teams. OpenAI notes that traffic can get a slow_down 429 even when requests-per-minute and tokens-per-minute quotas are not exhausted, if traffic ramps too quickly. Its documentation currently gives a rule of thumb for traffic at 1 million input tokens per minute: increase by no more than 50% every 15 minutes, with the exact threshold varying by model and conditions. So staying under the numbers is not enough. Warm up gradually after a launch, a batch job or a failover, and treat ramp rate as part of capacity planning.

A few practical consequences for AI features:

  • Estimate tokens before sending, and reject or trim oversized prompts yourself.
  • Put batch and background work in a queue with its own concurrency cap, separate from interactive traffic.
  • Cache repeated prompts and results where the output is deterministic enough.
  • Cap agent loops. An autonomous agent that retries a failing tool call can burn a day’s allowance in minutes.

If latency is your other pain point, see our guide on how to reduce LLM latency in production apps, and for the control side of agent tool use, hardening LLM function calling. The same thinking about gradual traffic growth applies when you wire AI into publishing or SEO pipelines, and API-led integration layers show how structured API connections speed that work up.

A rollout plan you can actually follow

Rate limiting is an operational control, not a one-time middleware feature. Stripe’s write-up on scaling its API with rate limiters makes that case well, and its advice maps onto a sequence like this:

  1. Inventory callers and endpoint cost. Know who calls what, how often, and what each call costs you.
  2. Define quotas and burst policy. Decide the unit, the scope and how much burst you will tolerate.
  3. Choose the algorithm and enforcement layer. Gateway, middleware or service, token bucket or otherwise.
  4. Load-test normal and worst-case traffic. Include large and expensive requests.
  5. Emit a 429 with actionable headers. Clients should never have to guess.
  6. Ship client backoff and idempotency. Including your own SDKs and internal callers.
  7. Observe. Track rejects, latency, queue depth and cost.
  8. Dark-launch, tune, and keep a kill switch. Measure what the limiter would reject before it rejects anything, then adjust thresholds against real client behavior.

One more point from Stripe that saves outages: hook the limiter into your middleware so that if the limiter has a bug, or Redis goes down, requests are not all failed. Failing open, where appropriate, keeps a limiter problem from becoming an API outage. For sensitive endpoints such as login, you may decide the opposite, and that is fine if it is deliberate.

Our take: limits are a product decision

We see rate limiting go wrong in two opposite ways. The first is no limits at all, usually in a fast-built app where a prototype became production. It works until a customer script or a looping AI agent finds the endpoint. The second is a limit copied from a blog post, say 100 requests per minute, that nobody tested and nobody monitors. It either blocks legitimate users or protects nothing.

The pattern we see most with AI-generated and vibe-coded apps is the missing client half. The server may have a limit, but the code calling third-party APIs retries immediately, in a tight loop, with no cap and no jitter. One rate-limited provider and the app creates its own outage. These are the last-mile gaps that separate a working demo from something you can trust under real traffic.

Our advice is simple. Treat the limit as a promise you make to customers, tested against real capacity. Return signals clients can act on. Make the failure path as carefully built as the happy path. And measure, because the right number changes as your product and your traffic change. If your integration surface is growing, our API integration strategy playbook goes deeper on hardening it in stages.

— Chad

Want a second pair of eyes on your limits?

If you are not sure how your app behaves when a provider throttles it, or when your own traffic spikes, we can look. A Vibe Check is $449 one-off and reviews AI-generated or vibe-coded apps for exactly this kind of production-readiness gap. A Senior Developer Review starts from $449, and our pricing page lists the other assessments, including infrastructure assessment and automated monitoring, logging and alerts. For larger builds, see our services.

FAQ

What is a good rate limit for an API?

There is no universal number. Load-test to find the sustainable capacity of your system, using realistic request sizes as well as rates, then set limits below that with headroom for failure recovery. Weight expensive endpoints more heavily than cheap ones, and revisit the numbers as traffic grows.

What is the difference between API throttling and rate limiting?

Rate limiting is the policy that defines how much a caller may send in a period. Throttling is the enforcement behavior that slows, queues or rejects work when that policy or system capacity is exceeded. In practice the terms overlap, and many tools use both.

How do you handle rate limiting in an API client?

Honor the Retry-After header when it is present. If it is not, use exponential backoff with random jitter, cap the number of retries, and reduce concurrency before trying again. Do not retry quota or credit exhaustion blindly, because waiting will not fix it.

Which algorithm should I use?

Use a token bucket if clients legitimately send bursts, since it has a refill rate and a bucket capacity you can tune. Use a leaky bucket when the backend needs smooth, constant input, and a sliding window log where exact enforcement matters. Fixed windows are simplest but allow bursts at window boundaries.

Sources