AI Chatbot Application

What the product does

An AI chatbot app is a web product that enables users to converse with an LLM in real time. After a user submits a message, the app forwards it to the model and progressively renders the reply as tokens arrive, so the response appears to be typed out live. ChatGPT, Claude, and Gemini are common examples.

System scope

We are designing backend and streaming infrastructure for an ephemeral AI chatbot. A person signs in, sends a text prompt, and receives model output streamed token-by-token over Server-Sent Events. The server never persists messages: the browser holds the entire conversation, and a refresh clears it. The central challenge is providing a fast, progressive token stream through stateless infrastructure while handling stream failures, unpredictable upstream model behavior, and long-lived connections protected by authentication.

High-level architecture of the stateless AI chatbot streaming backend.

Core capabilities

• A signed-in user can submit a text prompt and get an AI answer streamed token-by-token in real time.
• The server passes the prompt to an upstream model and streams the answer back through SSE, with a typing indicator shown while output is being generated.
• Earlier turns are kept in the frontend and included with each new prompt, allowing the model to use multi-turn context.
• The server does not keep any chat state. Messages and conversation data are not stored on the backend.
• Users authenticate by logging in and getting a JSON Web Token (JWT), which is required for all requests, including streaming connections.

Excluded from this design

• Saving messages or conversations in a database
• Resuming a conversation across sessions or syncing across devices
• Images, audio, video, file uploads, or other rich media
• Switching between models, model selection, or fine-tuning options
• Admin tooling, analytics, plugins, or function calling
• Shared or collaborative conversations
• Offline use

Quality requirements

• The time to first token should be below 500 ms at p95. After the stream begins, token-to-render latency should be imperceptible.
• The streaming route should aim for 99.9% uptime, with graceful fallbacks when the external LLM is slow or unavailable.
• The platform should handle 50K concurrent streams at peak, including bursts when many users send prompts at once.
• All endpoints require JWT authentication, including long-running SSE connections.
• Each user must have rate limits on prompt count and total token usage to keep upstream API spending under control.
• Reduce unnecessary calls to the AI provider. If a user stops watching a stream mid-response, the stream should be stopped quickly to prevent extra charges.

Working assumptions

• The LLM is an external provider, such as OpenAI, with an endpoint that supports streaming.
• The backend is only a stateless streaming proxy. It keeps no message or conversation state.
• To preserve multi-turn context, the client includes the full conversation history with each request.
• Only text chat is considered. There is one model provider and no model routing.

[!TIP] During an interview, an early clarification should be whether messages are persisted or conversations are ephemeral. That decision affects backend complexity, the data model, and how failures are handled.

Back-of-the-envelope estimate

The number of simultaneous streams shows that one server cannot keep every SSE connection open, so the system must run as a horizontally scaled set of instances behind a load balancer. Since streams average 15-30 seconds, a connection can outlive a short-lived JWT, but the duration is still easier to handle than persistent WebSocket sessions. Token volume is the most important factor for cost: moving roughly 1.5M tokens per minute through an external API makes per-user budgets and rate limits essential.