Distributed Systems & Caching

Design a Distributed Rate Limiter & Abuse Prevention Gateway (Cloudflare / Stripe)

Staff / Principal

Architect an ultra-low latency distributed rate limiter deployed across global edge points of presence (PoPs) to throttle DDoS attacks and enforce per-API-key quotas.

Production Scale: 500 Billion Requests/Day • 1,000,000 Peak QPS • Sub-2ms Evaluation

Functional Requirements

  • •Enforce rate limits based on Client IP, API Key, User ID, and HTTP endpoint
  • •Support configurable algorithm strategies: Token Bucket, Leaky Bucket, and Sliding Window Counter
  • •Return standard HTTP 429 (Too Many Requests) headers: X-RateLimit-Limit, X-RateLimit-Remaining, and Retry-After

Non-Functional Requirements

  • •Ultra-low latency overhead (<2ms p99 latency per request)
  • •High fault tolerance (Fail-Open policy: if rate limiter fails, allow user traffic through)
  • •Global consistency across multi-region edge clusters

Capacity & Scale Estimation

Peak Throughput1,000,000 QPS globally
Active Clients Tracked500 Million unique keys / day
Memory Footprint~32GB RAM in Redis Cluster (64 bytes per counter)
Evaluation Latency Target<1.5ms via local memory L1 + Redis L2

Core Architectural Components

1Local In-Memory L1 Cache (Go / Envoy Filter)

Evaluates 90% of non-violating traffic locally using in-memory sliding windows to avoid remote network calls.

2Distributed Redis Cluster L2 (Lua Scripts)

Atomic Redis Lua scripts executing Token Bucket increment and TTL expiration without race conditions across nodes.

3Asynchronous Rule Synchronizer (gRPC / Kafka)

Broadcasts dynamic quota updates and blacklisted IP ranges from management dashboards to all edge nodes in <500ms.

4Telemetry & Anomaly Detection Pipeline (ClickHouse)

Ingests rate limiting events asynchronously to identify coordinated botnet attacks and adjust thresholds dynamically.

Architectural FAQs & Interview Deep Dives

Why is the Token Bucket algorithm preferred over Fixed Window Counter for rate limiting?

Fixed Window Counters suffer from boundary burst vulnerabilities (e.g. allowing 2x the limit at the turn of a minute). Token Bucket allows smooth burst handling while enforcing strict sustained average rate limits.

How do you eliminate race conditions when multiple workers update a shared counter in Redis?

By executing the rate check and token decrement inside an atomic Redis Lua script (`EVAL`), ensuring the read-modify-write cycle executes uninterrupted on a single thread in memory.

Should a rate limiter fail-open or fail-closed if Redis becomes unreachable?

For public consumer APIs, fail-open is the industry standard: availability takes priority over strict rate limit enforcement. For critical internal financial endpoints, fail-closed with alert paging is preferred.

How do you handle clock skew across distributed servers in sliding window log algorithms?

By using synchronized NTP servers and monotonic timestamp offsets, or relying on server-side Redis cluster time rather than untrusted client timestamps.

What headers must a rate limiter return in HTTP 429 responses?

`Retry-After` (seconds until reset), `X-RateLimit-Limit` (max quota), `X-RateLimit-Remaining` (remaining tokens), and `X-RateLimit-Reset` (epoch timestamp).

How does local L1 memory caching reduce load on the central Redis cluster?

By syncing local token counts in batches every 100ms, edge proxies can absorb 95% of high-volume traffic locally, reducing Redis cluster network operations by 20x.