Streaming AI Responses with Node.js

What is it?

Vendors often return streamed completions. Your Node process is the middle layer: it authenticates the user, opens the vendor stream with the secret key, and writes events to the browser. This page focuses on the server side; client parsing is in Streaming AI Responses for Chat UIs.

Why it matters

Without a server proxy, the browser would need the vendor key. With a proxy, you can enforce auth, hide provider formats, and cancel upstream work when the client disconnects — essential for cost control.

How to do it

1. Client opens GET or POST to your /api/chat/stream.
2. Node validates the user and body.
3. Node fetches the vendor with stream: true (field names vary).
4. Set response headers for SSE (Content-Type: text/event-stream, Cache-Control: no-cache).
5. For each upstream chunk, parse provider events, extract text deltas, write data: …\n\n to the client.
6. On client close (req.on('close')), abort the upstream fetch.
7. On upstream error after headers were sent, write an error event and end.

Example

export async function streamChat(req, res, messages) {
  const key = process.env.MODEL_API_KEY;
  const upstream = await fetch("https://api.example.com/v1/chat/completions", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${key}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "your-model-id", messages, stream: true }),
  });
  if (!upstream.ok || !upstream.body) {
    res.writeHead(502, { "Content-Type": "application/json" });
    res.end(JSON.stringify({ error: "upstream" }));
    return;
  }
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    Connection: "keep-alive",
  });
  const reader = upstream.body.getReader();
  const dec = new TextDecoder();
  req.on("close", () => reader.cancel().catch(() => {}));
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    // simplify: forward decoded text; real code parses provider SSE
    const chunk = dec.decode(value, { stream: true });
    res.write(`data: ${JSON.stringify({ raw: chunk })}\n\n`);
  }
  res.write("data: [DONE]\n\n");
  res.end();
}

Production code should parse provider events and emit a stable { token } schema for React.

When to use it

Stream from Node when the UI needs token-by-token updates. For batch jobs and evals, non-streaming JSON is simpler.

Practical notes

Watch for proxy buffering: some reverse proxies buffer SSE until a flush threshold — configure them to disable buffering on the stream route. Send a comment heartbeat (': ping') every 15–30 seconds if connections traverse idle timeouts. Normalize vendor quirks (different delta field names) before writing to the client. If the vendor supports cancellation via DELETE or by closing the body, wire that to the Node 'close' event. Backpressure: if 'res.write' returns false, pause the upstream reader until 'drain'. Record metrics for time-to-first-token and bytes streamed. Keep non-stream JSON available on a sibling route for clients and tests that do not need SSE.

Common mistakes

• Forgetting to abort upstream on client disconnect.
• Buffering the entire vendor stream before writing (defeats streaming).
• Mixing compressed responses without handling byte boundaries.
• Leaking vendor error bodies that contain internal details.

Related concepts