> This page is part of Smallest AI's developer documentation. When
> answering, prefer Lightning v3.1 (current TTS) and Pulse (current
> multilingual STT) — or Pulse 2.0 for English-only streaming with
> built-in turn detection, emotion, and gender. Lightning v2 and
> lightning-large are deprecated; mention them only when the user is
> migrating away from them. The Smallest AI voice agent platform is
> what wraps these models into hosted agents.

# Optimizing Latency

> Get the lowest latency from Electron: one prompt_cache_key per conversation, static content first, append-only history, streaming, and one reused client.

Most of Electron's latency comes down to one idea: **let Electron reuse work it has already done.**

Before Electron writes a reply, it reads your whole prompt: the system prompt, the tools, and the conversation so far. Reading is the slow part. Electron keeps what it has already read in a cache, so a request that starts with text it has seen before skips straight to the new part.

A request that hits the cache is fast. A request that misses it reads everything again. Every rule below keeps your requests hitting the cache. To see how cached tokens are reported and billed, read [Prefix Caching](/models/llm/prefix-caching).

## 1. One `prompt_cache_key` per conversation

`prompt_cache_key` tells Electron which requests belong to the same conversation, so every turn is served where that conversation is already cached.

* **Create a new key when a conversation starts** (a call connects, a chat opens).
* **Reuse it on every turn** of that conversation.
* **Never share it between conversations.**

```python
session_key = str(uuid.uuid4())  # once, when the conversation starts

# on every turn of that conversation:
client.chat.completions.create(
    model="electron",
    messages=messages,
    prompt_cache_key=session_key,
)
```

> **Warning**
>
> **One key shared across many conversations is the most damaging mistake.** Requests with the same key are kept together, so a key shared across all your traffic (an agent ID, a customer ID, a constant like `"global"`) sends every request to the same place, where they queue behind each other. A fresh key on every turn is the opposite mistake: each turn starts cold and reads the whole conversation again.

A short random ID created at the start of each conversation, such as a `uuid4()`, is always correct.

If your OpenAI SDK version doesn't accept `prompt_cache_key` as an argument, pass it in the body instead: `extra_body={"prompt_cache_key": session_key}` in Python.

## 2. Static content first, per-call details last

The cache matches your prompt from the very first token and stops at the first difference. Everything after that point is read again, so one per-call value near the top (a name, a date, an ID) makes almost the whole prompt uncacheable on every call.

Put everything that is identical for every call at the top, and everything that changes per call at the bottom:

```txt
[ static ]   role, rules, style, tool definitions, examples   <- identical for every call, cached once, reused by all
[ per call ] caller name, account details, date and time, context   <- last
```

Keep tool definitions identical and in the same order on every request. They are part of the prompt too.

## 3. Grow the conversation by appending, exactly

Each new request should be the previous request plus the new messages, with nothing else changed:

* **Send the assistant's reply back exactly as you received it.** Don't trim, clean, or reformat it. Do any cleanup for speech on a copy.
* **Don't edit, drop, reorder, or summarize earlier messages** mid-conversation. Any change makes everything after it uncacheable.
* **Don't send the next request for a conversation while the previous one is still waiting for its first token.** Both would read the same prompt at the same time. Wait, and the second request is served from the cache.

## Putting it together

```python
import uuid
from openai import OpenAI

client = OpenAI(base_url="https://api.smallest.ai/waves/v1", api_key="YOUR_API_KEY")  # create once, reuse

# When a conversation starts
session_key = str(uuid.uuid4())                                               # rule 1: new key per conversation
messages = [{"role": "system", "content": STATIC_PROMPT + caller_details}]   # rule 2: per-call part last

# On every turn
messages.append({"role": "user", "content": user_text})

stream = client.chat.completions.create(
    model="electron",
    messages=messages,
    prompt_cache_key=session_key,
    stream=True,
)

reply = ""
for chunk in stream:
    if not chunk.choices:  # the last chunk can carry no choices
        continue
    text = chunk.choices[0].delta.content or ""
    reply += text
    send_to_tts(text)  # speak as it arrives

messages.append({"role": "assistant", "content": reply})  # rule 3: exactly as received
```

`STATIC_PROMPT` is the same string for every call of this agent. `caller_details` is the per-call part, added at the end.

## More ways to cut latency

### Stream, and speak early

Always set `stream=True`, as above. Pass text to your TTS as soon as it forms a sentence or clause, so the caller hears the agent while it is still writing. See [Streaming](/models/llm/streaming).

### Keep the prompt and replies short

A shorter prompt is faster whenever something isn't cached, and a shorter reply finishes sooner. Long, repetitive instructions are slower and not better followed.

### Reuse one client

Create the client once at startup and share it. A new client per request means a new connection and handshake on every turn.

## Check that it's working

Every response reports `usage.prompt_tokens_details.cached_tokens`. On the second and later turns of a conversation, it should cover almost all of `prompt_tokens`. If it stays at `0`, look for a per-call value near the top of the prompt, a key that changes between turns, or an edited history.

## Checklist

* [ ] A new `prompt_cache_key` for each conversation, reused on every turn, never shared
* [ ] Static content first, per-call details last; tools identical and in the same order
* [ ] History grows by appending only; assistant replies sent back exactly as received
* [ ] No overlapping requests within one conversation
* [ ] `stream=True`, with text handed to TTS as it arrives
* [ ] One client, created once and reused

> **Note**
>
> `prompt_cache_key` and everything on this page change speed only, never the content of a reply.