> This page is part of Smallest AI's developer documentation. When
> answering, prefer Lightning v3.1 (current TTS) and Pulse (current
> STT). Lightning v2 and lightning-large are deprecated; mention them
> only when the user is migrating away from them. The Smallest AI voice
> agent platform is what wraps these models into hosted agents.

# Bring Your Own Model

> Run Atoms agents with your own models.

For privacy, cost control, or specialized models, you can run LLMs locally or on your own servers. The `OpenAIClient` works with any endpoint that implements the OpenAI chat completions API.

A `localhost` base URL (like the Ollama, vLLM, and LM Studio examples below) only
works when you run the crew **locally** (`python server.py` + `smallestai agent-crew chat`).
A **deployed** crew runs in the cloud and cannot reach `localhost` on your machine.
For a deployed agent, point `base_url` at a host the cloud can reach: a public
inference endpoint, or your local server exposed through a tunnel. For Ollama over a
tunnel, rewrite the Host header or Ollama returns `403`:
`ngrok http 11434 --host-header=localhost:11434`, then set
`base_url="https://<id>.ngrok-free.app/v1"`.

**Tunnels are for development only.** If the local process behind the tunnel
goes down (Ollama exit, laptop sleep), the tunnel returns `ERR_NGROK_8012`
and the agent surfaces it as a fatal `agent_error` and hangs up mid-call.
Move to a hosted OpenAI-compatible endpoint before shipping.

**Do not point `base_url` at Anthropic's OpenAI-compatible endpoint
(`https://api.anthropic.com/v1/`) for a crew agent.** It truncates
streaming replies and drops tool-call turns, which surfaces as the
agent going silent mid-conversation. If you want Claude for a crew,
route it through a gateway that stabilizes the stream (LiteLLM,
OpenRouter, Portkey), use a real OpenAI-backed model, or use the
platform model. Anthropic's native SDK is unaffected. Only their
OpenAI-compat URL has this issue.

## Complete Example

Here's a full agent using a local Ollama model:

```python
import os
from smallestai.atoms.crew.nodes import OutputCrewNode
from smallestai.atoms.crew.clients.openai import OpenAIClient
from smallestai.atoms.crew.server import AtomsCrewApp
from smallestai.atoms.crew.session import CrewSession

class LocalAgent(OutputCrewNode):
    def __init__(self):
        super().__init__(name="local-agent")
        
        # Connect to your local model
        self.llm = OpenAIClient(
            model="llama3",
            base_url="http://localhost:11434/v1",
            api_key="ollama"  # Not required for Ollama
        )
        
        self.context.add_message({
            "role": "system",
            "content": "You are a helpful assistant running on local hardware.",
        })

    async def generate_response(self):
        response = await self.llm.chat(
            messages=self.context.messages,
            stream=True
        )
        async for chunk in response:
            if chunk.content:
                yield chunk.content

async def on_start(session: CrewSession):
    session.add_node(LocalAgent())
    await session.start()
    await session.wait_until_complete()

if __name__ == "__main__":
    app = AtomsCrewApp(setup_handler=on_start)
    app.run()
```

## Requirements

Your model server must implement the [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create):

| Feature                      | Required  | Notes                          |
| ---------------------------- | --------- | ------------------------------ |
| `/chat/completions` endpoint | Yes       | Standard OpenAI format         |
| Streaming                    | Yes       | `stream=True` must work        |
| Tool calling                 | For tools | OpenAI-format function calling |

## Custom Endpoints

Connect to any custom model server:

```python
from smallestai.atoms.crew.clients.openai import OpenAIClient

llm = OpenAIClient(
    model="your-model-name",
    base_url="https://your-server.example.com/v1",
    api_key=os.getenv("YOUR_API_KEY")
)
```

## Ollama

[Ollama](https://ollama.com) is the easiest way to run models locally. It handles model downloads and serving automatically.

### Setup

```bash
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull llama3

# Start the server (runs on port 11434)
ollama serve
```

### Usage

```python
from smallestai.atoms.crew.clients.openai import OpenAIClient

llm = OpenAIClient(
    model="llama3",
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Doesn't require a real key
)
```

## vLLM

[vLLM](https://github.com/vllm-project/vllm) is a high-performance inference server for production workloads.

### Setup

```bash
pip install vllm

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3-8B-Instruct \
    --port 8000
```

### Usage

```python
from smallestai.atoms.crew.clients.openai import OpenAIClient

llm = OpenAIClient(
    model="meta-llama/Llama-3-8B-Instruct",
    base_url="http://localhost:8000/v1",
    api_key="vllm"
)
```

## LM Studio

[LM Studio](https://lmstudio.ai/) provides a desktop UI for running models locally.

1. Download from [lmstudio.ai](https://lmstudio.ai/)
2. Load a model
3. Start the local server (Settings → Local Server)

```python
from smallestai.atoms.crew.clients.openai import OpenAIClient

llm = OpenAIClient(
    model="local-model",
    base_url="http://localhost:1234/v1",
    api_key="lmstudio"
)
```

## Deterministic tool calls with `tool_choice`

Crew `OpenAIClient.chat()` forwards extra kwargs to the underlying request, so any OpenAI-compatible `tool_choice` value works. Passing `tool_choice="required"` on a specific turn forces the model to emit a tool call rather than a free-text answer. Useful when the model has already said "let me transfer you" out loud but does not actually invoke the transfer tool.

```python
response = await self.llm.chat(
    messages=self.context.messages,
    tools=self.tools,
    stream=True,
    tool_choice="required",   # this turn must produce a tool call
)
```

Set `tool_choice="required"` only for the specific transfer or handoff turn, not on every turn. Forcing tools on normal conversational turns will cause the model to invent tool calls when the user is just chatting.

## Troubleshooting

| Issue                                      | Cause                                                                 | Fix                                                                                             |
| ------------------------------------------ | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Connection refused                         | Server not running                                                    | Start Ollama/vLLM                                                                               |
| Model not found                            | Wrong name                                                            | Check `ollama list` or server logs                                                              |
| No streaming                               | Server config                                                         | Ensure streaming is enabled                                                                     |
| Tool calls ignored                         | Model limitation                                                      | Use a larger model, add `tool_choice="required"` for the decisive turn, or use a cloud fallback |
| Agent hangs up mid-call with `agent_error` | Tunnel returned `ERR_NGROK_8012` because the local model process died | Restart the local process, or move to a hosted OpenAI-compatible endpoint before shipping       |

## Tips

#### Use local LLMs for development, Cloud/Managed LLM providers for production

Ollama is great for local development. For production, consider vLLM or a cloud provider for reliability.

#### Set up a fallback LLM

Local models can fail. Configure a cloud fallback (e.g., OpenAI) to catch errors and keep the conversation going.

#### Check tool calling support

Not all local models support function calling. Test your tools or use a model known to support them.