Streaming Responses & Real-Time Tokens

How Streaming Works

When you set stream: true in your request payload, InfinityRouter opens an HTTP chunked stream with content type text/event-stream.

Each event contains an incremental delta of the completion text, tool call arguments, or usage metrics.

textSSE Wire Format

data: {"id":"req_8a12f","model":"claude-sonnet-5","choices":[{"delta":{"role":"assistant"}}]}

data: {"id":"req_8a12f","model":"claude-sonnet-5","choices":[{"delta":{"content":"Distributed"}}]}

data: {"id":"req_8a12f","model":"claude-sonnet-5","choices":[{"delta":{"content":" systems"}}]}

data: [DONE]

Real-Time Metering & Connection Abort

One of the primary benefits of streaming through InfinityRouter is real-time token accounting:

Pre-request Credit Hold: We hold credits for the prompt tokens plus the declared max_tokens ceiling.
Stream Metering: As tokens stream to your client, token counters update incrementally.
Instant Disconnect Refund: If your client closes the TCP connection or cancels an AbortController midway, downstream generation stops and unused hold credits are immediately released back to your balance.

Browser Fetch API Example

If you are building custom frontends without SDK wrappers, you can read SSE streams using the native Web Streams API:

typescriptbrowser_stream.ts

async function streamChat(prompt: string) {
  const response = await fetch("https://infinityrouter.qd.je/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": "Bearer " + process.env.NEXT_PUBLIC_INFINITY_KEY,
    },
    body: JSON.stringify({
      model: "claude-sonnet-5",
      messages: [{ role: "user", content: prompt }],
      stream: true,
    }),
  });

  if (!response.body) throw new Error("ReadableStream not supported.");

  const reader = response.body.getReader();
  const decoder = new TextDecoder("utf-8");
  let buffer = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n");
    buffer = lines.pop() || "";

    for (const line of lines) {
      const trimmed = line.trim();
      if (!trimmed || !trimmed.startsWith("data: ")) continue;
      const dataStr = trimmed.replace("data: ", "");
      if (dataStr === "[DONE]") return;

      try {
        const parsed = JSON.parse(dataStr);
        const token = parsed.choices[0]?.delta?.content || "";
        process.stdout.write(token);
      } catch (err) {
        console.error("Failed parsing stream chunk:", err);
      }
    }
  }
}