System Design · xAI · Medium
Requirements Functional: Limit incoming traffic by API route or by user identity (representative figures: 100 requests/second arriving and 10 requests/second sent to a downstream worker). If downstream processing cannot keep pace with arrivals, place requests into a capacity-limited queue; once that queue reaches capacity, reject or discard incoming work. Send callers an appropriate throttling response: 429 together with a retry-after header. Preserve fairness across users:…
Checking your access…