A customer's integration went into a retry loop at 2am. No backoff, no cap — just a failed request, immediately retried, forever. By morning it had made about 1.4 million requests, saturated the connection pool, and taken the API down for everyone else.
They were not attacking us. Their code had a bug. But the effect was identical to an attack, and the reason it worked was that we had no rate limiting at all.
Rate limiting is usually filed under security. In my experience it is much more often about protecting yourself from a customer's mistake — and from your own.
Four Algorithms, and When Each Is Wrong
Fixed window — count requests per calendar minute, reset on the boundary. Trivially simple, one counter per key, and it has a real flaw: a client can send the full limit at 10:00:59 and the full limit again at 10:01:00. You allow double your intended rate across that boundary. Fine for rough protection, bad if the limit is load-bearing.
Sliding window log — store a timestamp per request, count the ones inside the window. Exactly accurate, no boundary problem, and it stores every request. At high volume that memory cost is real.
Sliding window counter — keep the current and previous window counts, and weight the previous one by how far into the current window you are. Approximate, cheap, and close enough that this is what I use most of the time.
Token bucket — tokens refill at a steady rate up to a maximum; each request spends one. This is the one I reach for when the API is user-facing, because it does something the others do not: it allows bursts. A user who has been idle can make ten requests at once, then settles to the sustained rate. That matches how people actually use software.
Token bucket in Redis, done atomically so concurrent requests cannot both pass:
-- rate_limit.lua — KEYS[1]=bucket ARGV: capacity, refill/sec, now, cost
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])
local b = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(b[1]) or capacity
local ts = tonumber(b[2]) or now
tokens = math.min(capacity, tokens + (now - ts) * rate) -- refill
if tokens < cost then
redis.call("HMSET", KEYS[1], "tokens", tokens, "ts", now)
redis.call("EXPIRE", KEYS[1], math.ceil(capacity / rate) * 2)
return {0, tokens} -- denied
end
tokens = tokens - cost
redis.call("HMSET", KEYS[1], "tokens", tokens, "ts", now)
redis.call("EXPIRE", KEYS[1], math.ceil(capacity / rate) * 2)
return {1, tokens} -- allowed
Doing this in a Lua script matters. Read-then-write from application code is a race, and under exactly the load where the limit matters, several requests will read the same count and all pass.
What You Limit By Is More Important Than How
Most rate limiting is broken not because of the algorithm but because of the key.
Per IP is the common default and it is wrong for authenticated APIs. An office, a university or a mobile carrier NATs hundreds of people behind one address, so you throttle innocent users while a distributed client sails through. Use it only for unauthenticated endpoints.
Per API key or user is right for authenticated traffic. It is who you are actually billing and who you actually want to control.
Per tenant matters in multi-tenant SaaS. One customer's runaway script should not consume the shared budget — that is the noisy neighbour problem, and it is exactly what happened to us.
Per endpoint, with different costs. A login attempt, a search and a report generation are not equivalent. Give expensive endpoints a higher cost against the same bucket rather than maintaining separate limits.
In practice I layer them: a global per-IP limit at the edge as a blunt shield, plus a per-tenant limit in the application where the business rules live.
Tell the Client What Is Happening
A limit with no feedback produces clients that hammer you harder. Return the standard headers and a proper status:
HTTP/1.1 429 Too Many Requests
RateLimit-Limit: 1000
RateLimit-Remaining: 0
RateLimit-Reset: 43
Retry-After: 43
{"error":{"code":"rate_limited","message":"Rate limit exceeded. Retry in 43s."}}
Send RateLimit-Remaining on successful responses too, not just failures. A well-behaved client that can see its budget will pace itself, which generates less load than one that discovers the limit by hitting it.
And use 429, not 403. Clients have retry logic keyed to status codes, and 403 tells them to give up permanently.
The Retry Side of the Contract
Half of this problem lives in the client, and if you write API clients you own that half.
Exponential backoff with jitter, and respect Retry-After:
async function call(fn: () => Promise<Response>, max = 5) {
for (let attempt = 0; ; attempt++) {
const res = await fn();
if (res.status !== 429 && res.status < 500) return res;
if (attempt >= max) return res;
const server = Number(res.headers.get("retry-after")) * 1000;
const backoff = Math.min(30_000, 2 ** attempt * 500);
// Jitter matters: without it every client retries in the same instant.
const wait = server || backoff * (0.5 + Math.random() / 2);
await new Promise(r => setTimeout(r, wait));
}
}
The jitter is not a detail. When a service recovers from an outage, every client that failed retries simultaneously and knocks it over again. Randomising the delay spreads the reconnection out, and it is two lines.
Related Protections Worth Having
Rate limiting is one of several bounds, and the others are often missing alongside it:
Request body size limits. Otherwise one client uploads 500MB of JSON.
Query result caps. Maximum page size, mandatory pagination, a maximum date range on reports.
Statement timeouts in the database, so a single query cannot run for five minutes.
Concurrency limits per tenant, separate from rate. Ten requests a second is fine; ten simultaneous ten-second reports is not.
A spend cap on anything with per-use cost, especially AI features. A loop that burns tokens is a bill, not just load.
Rolling It Out Without Breaking Customers
Turning limits on suddenly will break someone who was quietly exceeding them.
What works: log first, enforce later. Run the limiter in shadow mode for a couple of weeks, recording who would have been blocked and by how much. Then set the actual limits above the legitimate observed usage, tell affected customers directly, and enforce.
You will find that one or two customers are wildly above everyone else, and usually for a reason worth a conversation rather than a 429.
We shipped a per-tenant token bucket the week after that 2am incident. The same customer's integration broke again a month later — the same bug, never fixed on their side. That time it got 429s for an hour and nobody else noticed.
