Optimizing Latency

Most of Electron’s latency comes down to one idea: let Electron reuse work it has already done.

Before Electron writes a reply, it reads your whole prompt: the system prompt, the tools, and the conversation so far. Reading is the slow part. Electron keeps what it has already read in a cache, so a request that starts with text it has seen before skips straight to the new part.

A request that hits the cache is fast. A request that misses it reads everything again. Every rule below keeps your requests hitting the cache. To see how cached tokens are reported and billed, read Prefix Caching.

1. One prompt_cache_key per conversation

prompt_cache_key tells Electron which requests belong to the same conversation, so every turn is served where that conversation is already cached.

• Create a new key when a conversation starts (a call connects, a chat opens).
• Reuse it on every turn of that conversation.
• Never share it between conversations.

session_key = str(uuid.uuid4())  # once, when the conversation starts
# on every turn of that conversation:
client.chat.completions.create(
    model="electron",
    messages=messages,
    prompt_cache_key=session_key,
)

A short random ID created at the start of each conversation, such as a uuid4(), is always correct.

If your OpenAI SDK version doesn’t accept prompt_cache_key as an argument, pass it in the body instead: extra_body={"prompt_cache_key": session_key} in Python.

2. Static content first, per-call details last

The cache matches your prompt from the very first token and stops at the first difference. Everything after that point is read again, so one per-call value near the top (a name, a date, an ID) makes almost the whole prompt uncacheable on every call.

Put everything that is identical for every call at the top, and everything that changes per call at the bottom:

[ static ]   role, rules, style, tool definitions, examples   <- identical for every call, cached once, reused by all
[ per call ] caller name, account details, date and time, context   <- last

Keep tool definitions identical and in the same order on every request. They are part of the prompt too.

3. Grow the conversation by appending, exactly

Each new request should be the previous request plus the new messages, with nothing else changed:

• Send the assistant’s reply back exactly as you received it. Don’t trim, clean, or reformat it. Do any cleanup for speech on a copy.
• Don’t edit, drop, reorder, or summarize earlier messages mid-conversation. Any change makes everything after it uncacheable.
• Don’t send the next request for a conversation while the previous one is still waiting for its first token. Both would read the same prompt at the same time. Wait, and the second request is served from the cache.

Putting it together

import uuid
from openai import OpenAI
client = OpenAI(base_url="https://api.smallest.ai/waves/v1", api_key="YOUR_API_KEY")  # create once, reuse
# When a conversation starts
session_key = str(uuid.uuid4())                                               # rule 1: new key per conversation
messages = [{"role": "system", "content": STATIC_PROMPT + caller_details}]   # rule 2: per-call part last
# On every turn
messages.append({"role": "user", "content": user_text})
stream = client.chat.completions.create(
    model="electron",
    messages=messages,
    prompt_cache_key=session_key,
    stream=True,
)
reply = ""
for chunk in stream:
    if not chunk.choices:  # the last chunk can carry no choices
        continue
    text = chunk.choices[0].delta.content or ""
    reply += text
    send_to_tts(text)  # speak as it arrives
messages.append({"role": "assistant", "content": reply})  # rule 3: exactly as received

STATIC_PROMPT is the same string for every call of this agent. caller_details is the per-call part, added at the end.

More ways to cut latency

Stream, and speak early

Always set stream=True, as above. Pass text to your TTS as soon as it forms a sentence or clause, so the caller hears the agent while it is still writing. See Streaming.

Keep the prompt and replies short

A shorter prompt is faster whenever something isn’t cached, and a shorter reply finishes sooner. Long, repetitive instructions are slower and not better followed.

Reuse one client

Create the client once at startup and share it. A new client per request means a new connection and handshake on every turn.

Check that it’s working

Every response reports usage.prompt_tokens_details.cached_tokens. On the second and later turns of a conversation, it should cover almost all of prompt_tokens. If it stays at 0, look for a per-call value near the top of the prompt, a key that changes between turns, or an edited history.

Checklist

• A new prompt_cache_key for each conversation, reused on every turn, never shared
• Static content first, per-call details last; tools identical and in the same order
• History grows by appending only; assistant replies sent back exactly as received
• No overlapping requests within one conversation
• stream=True, with text handed to TTS as it arrives
• One client, created once and reused