Back to problems

Rate Limiter — Inbound + Outbound

System Design · xAI · Medium

Requirements Functional: Limit incoming traffic by API route or by user identity (representative figures: 100 requests/second arriving and 10 requests/second sent to a downstream worker). If downstream processing cannot keep pace with arrivals, place requests into a capacity-limited queue; once that queue reaches capacity, reject or discard incoming work. Send callers an appropriate throttling response: 429 together with a retry-after header. Preserve fairness across users:…

Checking your access…