Streaming AI Responses for Chat UIs

What is it?

Without streaming, the server waits until the model finishes, then returns one JSON blob. With streaming, the model emits What are Tokens in AI? gradually; the HTTP response stays open while chunks arrive. Common transports are Server-Sent Events (SSE) or raw chunked text. Chat UIs feel faster because users see the first words sooner.

Why it matters

Long answers can take many seconds. Streaming improves perceived latency and lets users cancel mid-flight. It also changes error handling: failures can occur after partial text was already shown.

How to do it

Client responsibilities:

1. fetch your backend with Accept headers your API documents (often SSE).
2. Obtain a ReadableStream from response.body.
3. Decode bytes to text; split on event boundaries.
4. Parse each event's data field; append token text to the assistant message in state (Chatbot State and Conversation UI in React).
5. Stop on a documented done event or when the stream closes.
6. Handle abort (user clicks stop) via AbortController.

Server-side Node details live on Streaming AI Responses with Node.js. Keep vendor protocols behind your backend so the browser speaks one stable format.

Example

Illustrative browser loop over SSE-like lines:

async function readChatStream(res, onToken) {
  const reader = res.body.getReader();
  const dec = new TextDecoder();
  let buf = "";
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buf += dec.decode(value, { stream: true });
    const parts = buf.split("\n");
    buf = parts.pop() || "";
    for (const line of parts) {
      if (!line.startsWith("data:")) continue;
      const payload = line.slice(5).trim();
      if (payload === "[DONE]") return;
      const json = JSON.parse(payload);
      if (json.token) onToken(json.token);
    }
  }
}

Exact event shapes vary — adapt to your backend contract.

When to use it

Stream when answers are long or latency-sensitive. Skip streaming for short classifications, embeddings, or batch jobs where one JSON result is simpler.

Practical notes

UX details matter as much as the protocol. Show a cursor or shimmer while the first token has not arrived; switch to appending text once streaming starts. If the user sends a second message while streaming, either queue or finish the first stream — define the rule and stick to it. Scroll behaviour should follow new tokens unless the user scrolled up to read earlier context. On mobile networks, expect chunk delays and timeouts; surface a reconnect or retry without duplicating the assistant message. Keep your public event schema small ('token', 'done', 'error') even if the vendor sends richer events, so Chatbot State and Conversation UI in React stays simple.

Common mistakes

• Parsing incomplete JSON across chunk boundaries without a buffer.
• Updating React state on every token without batching, causing jank.
• Forgetting to cancel the upstream vendor request when the user aborts.
• Exposing the vendor stream format directly to the browser (tight coupling).

Related concepts