Bring Your Own Model | Smallest AI Docs

For privacy, cost control, or specialized models, you can run LLMs locally or on your own servers. The OpenAIClient works with any endpoint that implements the OpenAI chat completions API.

A localhost base URL (like the Ollama, vLLM, and LM Studio examples below) only works when you run the crew locally (python server.py + smallestai agent-crew chat). A deployed crew runs in the cloud and cannot reach localhost on your machine. For a deployed agent, point base_url at a host the cloud can reach: a public inference endpoint, or your local server exposed through a tunnel. For Ollama over a tunnel, rewrite the Host header or Ollama returns 403: ngrok http 11434 --host-header=localhost:11434, then set base_url="https://<id>.ngrok-free.app/v1".

Tunnels are for development only. If the local process behind the tunnel goes down (Ollama exit, laptop sleep), the tunnel returns ERR_NGROK_8012 and the agent surfaces it as a fatal agent_error and hangs up mid-call. Move to a hosted OpenAI-compatible endpoint before shipping.

Do not point base_url at Anthropic’s OpenAI-compatible endpoint (https://api.anthropic.com/v1/) for a crew agent. It truncates streaming replies and drops tool-call turns, which surfaces as the agent going silent mid-conversation. If you want Claude for a crew, route it through a gateway that stabilizes the stream (LiteLLM, OpenRouter, Portkey), use a real OpenAI-backed model, or use the platform model. Anthropic’s native SDK is unaffected. Only their OpenAI-compat URL has this issue.

Complete Example

Here’s a full agent using a local Ollama model:

import os
from smallestai.atoms.crew.nodes import OutputCrewNode
from smallestai.atoms.crew.clients.openai import OpenAIClient
from smallestai.atoms.crew.server import AtomsCrewApp
from smallestai.atoms.crew.session import CrewSession
class LocalAgent(OutputCrewNode):
    def __init__(self):
        super().__init__(name="local-agent")
        
        # Connect to your local model
        self.llm = OpenAIClient(
            model="llama3",
            base_url="http://localhost:11434/v1",
            api_key="ollama"  # Not required for Ollama
        )
        
        self.context.add_message({
            "role": "system",
            "content": "You are a helpful assistant running on local hardware.",
        })
    async def generate_response(self):
        response = await self.llm.chat(
            messages=self.context.messages,
            stream=True
        )
        async for chunk in response:
            if chunk.content:
                yield chunk.content
async def on_start(session: CrewSession):
    session.add_node(LocalAgent())
    await session.start()
    await session.wait_until_complete()
if __name__ == "__main__":
    app = AtomsCrewApp(setup_handler=on_start)
    app.run()

Requirements

Your model server must implement the OpenAI Chat Completions API:

Feature	Required	Notes
`/chat/completions` endpoint	Yes	Standard OpenAI format
Streaming	Yes	`stream=True` must work
Tool calling	For tools	OpenAI-format function calling

Custom Endpoints

Connect to any custom model server:

from smallestai.atoms.crew.clients.openai import OpenAIClient
llm = OpenAIClient(
    model="your-model-name",
    base_url="https://your-server.example.com/v1",
    api_key=os.getenv("YOUR_API_KEY")
)

Ollama

Ollama is the easiest way to run models locally. It handles model downloads and serving automatically.

Setup

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Pull a model
ollama pull llama3
# Start the server (runs on port 11434)
ollama serve

Usage

from smallestai.atoms.crew.clients.openai import OpenAIClient
llm = OpenAIClient(
    model="llama3",
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Doesn't require a real key
)

vLLM

vLLM is a high-performance inference server for production workloads.

Setup

pip install vllm
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3-8B-Instruct \
    --port 8000

Usage

from smallestai.atoms.crew.clients.openai import OpenAIClient
llm = OpenAIClient(
    model="meta-llama/Llama-3-8B-Instruct",
    base_url="http://localhost:8000/v1",
    api_key="vllm"
)

LM Studio

LM Studio provides a desktop UI for running models locally.

Download from lmstudio.ai
Load a model
Start the local server (Settings → Local Server)

from smallestai.atoms.crew.clients.openai import OpenAIClient
llm = OpenAIClient(
    model="local-model",
    base_url="http://localhost:1234/v1",
    api_key="lmstudio"
)

Deterministic tool calls with `tool_choice`

Crew OpenAIClient.chat() forwards extra kwargs to the underlying request, so any OpenAI-compatible tool_choice value works. Passing tool_choice="required" on a specific turn forces the model to emit a tool call rather than a free-text answer. Useful when the model has already said “let me transfer you” out loud but does not actually invoke the transfer tool.

response = await self.llm.chat(
    messages=self.context.messages,
    tools=self.tools,
    stream=True,
    tool_choice="required",   # this turn must produce a tool call
)

Set tool_choice="required" only for the specific transfer or handoff turn, not on every turn. Forcing tools on normal conversational turns will cause the model to invent tool calls when the user is just chatting.

Troubleshooting

Issue	Cause	Fix
Connection refused	Server not running	Start Ollama/vLLM
Model not found	Wrong name	Check `ollama list` or server logs
No streaming	Server config	Ensure streaming is enabled
Tool calls ignored	Model limitation	Use a larger model, add `tool_choice="required"` for the decisive turn, or use a cloud fallback
Agent hangs up mid-call with `agent_error`	Tunnel returned `ERR_NGROK_8012` because the local model process died	Restart the local process, or move to a hosted OpenAI-compatible endpoint before shipping

Tips

Use local LLMs for development, Cloud/Managed LLM providers for production

Ollama is great for local development. For production, consider vLLM or a cloud provider for reliability.

Set up a fallback LLM

Local models can fail. Configure a cloud fallback (e.g., OpenAI) to catch errors and keep the conversation going.

Check tool calling support

Not all local models support function calling. Test your tools or use a model known to support them.

Complete Example

Requirements

Custom Endpoints

Ollama

Setup

Usage

vLLM

Setup

Usage

LM Studio

Deterministic tool calls with tool_choice

Troubleshooting

Tips

Use local LLMs for development, Cloud/Managed LLM providers for production

Set up a fallback LLM

Check tool calling support

Deterministic tool calls with `tool_choice`