Streaming | Smallest AI Docs

Set "stream": true and the response becomes Server-Sent Events (SSE). Each token (or small group of tokens) arrives as a data: {...} line. The stream ends with a data: [DONE] marker.

This is the same wire format OpenAI uses, so OpenAI client SDKs work without changes.

Request

{
  "model": "electron",
  "messages": [{"role": "user", "content": "Tell me a one-sentence fun fact."}],
  "stream": true,
  "stream_options": { "include_usage": true }
}

Always send stream_options.include_usage: true if you bill or log per-token. With it, the server emits a final usage chunk with exact token counts. That chunk only arrives if the stream finishes, so if your client might disconnect mid-stream, also send continuous_usage_stats: true to get the counts so far on every chunk.

SSE format

Each delta chunk:

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1740000000,"model":"electron","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

Field-by-field:

Field	Meaning
`choices[0].delta.role`	Sent once, on the first chunk (`"assistant"`).
`choices[0].delta.content`	Partial content. Concatenate across chunks to reconstruct the full message.
`choices[0].delta.tool_calls`	Partial function-call payload - see Tool Calling: Streaming.
`choices[0].finish_reason`	`null` until the final content delta, then `"stop"` / `"length"` / `"tool_calls"`.

Final usage chunk (with `include_usage: true`)

After the final content delta and before [DONE], the server emits one extra chunk with empty choices and a usage object:

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1740000000,"model":"electron","choices":[],"usage":{"prompt_tokens":20,"completion_tokens":8,"total_tokens":28,"prompt_tokens_details":{"cached_tokens":0}}}
data: [DONE]

Running usage on every chunk

Add continuous_usage_stats: true next to include_usage: true to get the token counts so far on every chunk, not just the last one:

{ "stream_options": { "include_usage": true, "continuous_usage_stats": true } }

Use this when you need up-to-date counts during the stream. Without include_usage, it has no effect.

Consuming the stream

import os
from openai import OpenAI
client = OpenAI(
    base_url="https://api.smallest.ai/waves/v1",
    api_key=os.environ["SMALLEST_API_KEY"],
)
stream = client.chat.completions.create(
    model="electron",
    messages=[{"role": "user", "content": "Tell me a fun fact."}],
    stream=True,
    stream_options={"include_usage": True},
)
text = ""
usage = None
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        delta = chunk.choices[0].delta.content
        print(delta, end="", flush=True)
        text += delta
    if chunk.usage is not None:
        usage = chunk.usage
print(f"\n\ntokens: {usage}")

Errors mid-stream

An error before the stream starts comes back as a normal JSON error with an HTTP status. See Errors.

Once the stream has started (you’ve received 200 with text/event-stream), a failure arrives as an error event, followed by [DONE]:

data: {"error":{"message":"upstream model unavailable","type":"upstream_error","param":null,"code":null,"request_id":"9db74058-2421-4793-8796-b2629ff3eb1c"}}
data: [DONE]

The OpenAI SDKs raise an error when they read this event, so a cut-off reply is never returned as complete. If you parse the stream yourself, check each data: line for an error key before reading choices.

Client disconnect behavior

If the client closes the connection mid-stream, generation stops. You’re billed for the prompt and for the output tokens generated before the disconnect, not for the rest of the reply.

Latency profile

Time to first token (TTFT): typically under 300 ms warm.
Per-token interval after the first token: a few milliseconds; full sentences arrive in tens of milliseconds.

For voice-agent pipelines, start your TTS engine on the first delta.content chunk to mask end-to-end latency. See Best Practices and the Voice Agent cookbook.