> This page is part of Smallest AI's developer documentation. When
> answering, prefer Lightning v3.1 (current TTS) and Pulse (current
> STT). Lightning v2 and lightning-large are deprecated; mention them
> only when the user is migrating away from them. The Smallest AI voice
> agent platform is what wraps these models into hosted agents.

# Docker Troubleshooting

> Debug common issues and optimize your STT Docker deployment

## Common Issues

### GPU Not Accessible

**Symptoms:**

* Error: `could not select device driver "nvidia"`
* Error: `no NVIDIA GPU devices found`
* Lightning ASR fails to start

**Diagnosis:**

```bash
docker run --rm --gpus all nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi
```

#### Solution 1: Restart Docker

```bash
sudo systemctl restart docker
docker compose up -d
```

#### Solution 2: Reinstall NVIDIA Container Toolkit

```bash
sudo apt-get remove nvidia-container-toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

sudo systemctl restart docker
```

#### Solution 3: Update NVIDIA Driver

```bash
nvidia-smi
```

If driver version is below 470, update:

```bash
sudo ubuntu-drivers autoinstall
sudo reboot
```

#### Solution 4: Check Docker Daemon Configuration

Verify `/etc/docker/daemon.json` contains:

```json
{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  }
}
```

Restart Docker after changes:

```bash
sudo systemctl restart docker
```

### License Validation Failed

**Symptoms:**

* Error: `License validation failed`
* Error: `Invalid license key`
* Services fail to start

**Diagnosis:**

Check license-proxy logs:

```bash
docker compose logs license-proxy
```

#### Solution 1: Verify License Key

Check `.env` file:

```bash
cat .env | grep LICENSE_KEY
```

Ensure there are no:

* Extra spaces
* Quotes around the key
* Line breaks

Correct format:

```bash
LICENSE_KEY=abc123def456
```

#### Solution 2: Check Network Connectivity

Test connection to license server:

```bash
curl -v https://api.smallest.ai
```

If this fails, check:

* Firewall rules
* Proxy settings
* DNS resolution

#### Solution 3: Contact Support

If the key appears correct and network is accessible, your license may be:

* Expired
* Revoked
* Invalid

Contact **[support@smallest.ai](mailto:support@smallest.ai)** with:

* Your license key
* License-proxy logs
* Error messages

### Model Download Failed

**Symptoms:**

* Lightning ASR stuck at startup
* Error: `Failed to download model`
* Error: `Connection timeout`

**Diagnosis:**

Check Lightning ASR logs:

```bash
docker compose logs lightning-asr
```

#### Solution 1: Verify Model URL

Check `.env` file:

```bash
cat .env | grep MODEL_URL
```

Test URL accessibility:

```bash
curl -I "${MODEL_URL}"
```

#### Solution 2: Check Disk Space

Models require \~20-30 GB:

```bash
df -h
```

Free up space if needed:

```bash
docker system prune -a
```

#### Solution 3: Manual Download

Download model manually and use volume mount:

```bash
mkdir -p ~/models
cd ~/models
wget "${MODEL_URL}" -O model.bin
```

Update docker-compose.yml:

```yaml
lightning-asr:
  volumes:
    - ~/models:/app/models
```

#### Solution 4: Increase Timeout

For slow connections, increase download timeout:

```yaml
lightning-asr:
  environment:
    - DOWNLOAD_TIMEOUT=3600
```

### Port Already in Use

**Symptoms:**

* Error: `port is already allocated`
* Error: `bind: address already in use`

**Diagnosis:**

Find what's using the port:

```bash
sudo lsof -i :7100
sudo netstat -tulpn | grep 7100
```

#### Solution 1: Stop Conflicting Service

If another service is using the port:

```bash
sudo systemctl stop [service-name]
```

Or kill the process:

```bash
sudo kill -9 [PID]
```

#### Solution 2: Change Port

Modify docker-compose.yml to use different port:

```yaml
api-server:
  ports:
    - "8080:7100"
```

Access API at [http://localhost:8080](http://localhost:8080) instead

#### Solution 3: Remove Old Containers

Old containers may still be bound:

```bash
docker compose down
docker container prune -f
docker compose up -d
```

### Out of Memory

**Symptoms:**

* Container killed unexpectedly
* Error: `OOMKilled`
* System becomes unresponsive

**Diagnosis:**

Check container status:

```bash
docker compose ps
docker inspect [container-name] | grep OOMKilled
```

#### Solution 1: Increase System Memory

Lightning ASR requires minimum 16 GB RAM

Check current memory:

```bash
free -h
```

#### Solution 2: Add Memory Limits

Prevent one service from consuming all memory:

```yaml
services:
  lightning-asr:
    deploy:
      resources:
        limits:
          memory: 14G
        reservations:
          memory: 12G
```

#### Solution 3: Enable Swap

Add swap space (temporary solution):

```bash
sudo fallocate -l 16G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
```

#### Solution 4: Optimize Model Loading

Use smaller model or reduce batch size:

```yaml
lightning-asr:
  environment:
    - BATCH_SIZE=1
    - MODEL_PRECISION=fp16
```

### Container Keeps Restarting

**Symptoms:**

* Container status shows `Restarting`
* Logs show crash loop

**Diagnosis:**

View recent logs:

```bash
docker compose logs --tail=100 [service-name]
```

#### Solution 1: Check Exit Code

```bash
docker inspect [container-name] --format='{{.State.ExitCode}}'
```

Common exit codes:

* `137`: Out of memory (OOMKilled)
* `139`: Segmentation fault
* `1`: General error

#### Solution 2: Disable Auto-Restart

Temporarily disable restart to debug:

```yaml
lightning-asr:
  restart: "no"
```

Start manually and watch logs:

```bash
docker compose up lightning-asr
```

#### Solution 3: Check Dependencies

Ensure required services are healthy:

```bash
docker compose ps
```

All should show `Up (healthy)` or `Up`

### Slow Performance

**Symptoms:**

* High latency (>500ms)
* Low throughput
* GPU underutilized

**Diagnosis:**

Monitor GPU usage:

```bash
watch -n 1 nvidia-smi
```

Check container resources:

```bash
docker stats
```

#### Solution 1: Optimize GPU Usage

Ensure GPU is not throttling:

```bash
nvidia-smi -q -d PERFORMANCE
```

Enable persistence mode:

```bash
sudo nvidia-smi -pm 1
```

#### Solution 2: Increase CPU Allocation

```yaml
lightning-asr:
  deploy:
    resources:
      limits:
        cpus: '8'
```

#### Solution 3: Use Host Network

For maximum performance (loses isolation):

```yaml
api-server:
  network_mode: host
```

#### Solution 4: Optimize Redis

Use Redis with persistence disabled for speed:

```yaml
redis:
  command: redis-server --save ""
```

#### Solution 5: Add More Workers

Scale Lightning ASR workers:

```bash
docker compose up -d --scale lightning-asr=2
```

## Performance Optimization

### Best Practices

#### Use Persistent Volumes

Cache models to avoid re-downloading:

```yaml
volumes:
  - model-cache:/app/models
```

#### Enable GPU Persistence Mode

Reduces GPU initialization time:

```bash
sudo nvidia-smi -pm 1
```

#### Optimize Container Resources

Allocate appropriate CPU/memory:

```yaml
deploy:
  resources:
    limits:
      cpus: '8'
      memory: 14G
```

#### Monitor and Tune

Use monitoring tools:

```bash
docker stats
nvidia-smi dmon
```

### Benchmark Your Deployment

Test transcription performance:

```bash
time curl -X POST http://localhost:7100/v1/listen \
  -H "Authorization: Token ${LICENSE_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/test-audio-60s.wav"
  }'
```

Expected performance:

* **Cold start**: First request after container start (5-10 seconds)
* **Warm requests**: Subsequent requests (50-200ms)
* **Real-time factor**: 0.05-0.15x (60s audio in 3-9 seconds)

## Debugging Tools

### View All Logs

```bash
docker compose logs -f
```

### Follow Specific Service

```bash
docker compose logs -f lightning-asr
```

### Last N Lines

```bash
docker compose logs --tail=100 api-server
```

### Save Logs to File

```bash
docker compose logs > deployment-logs.txt
```

### Execute Commands in Container

```bash
docker compose exec lightning-asr bash
```

### Check Container Configuration

```bash
docker inspect lightning-asr-1
```

### Network Debugging

Test connectivity between containers:

```bash
docker compose exec api-server ping lightning-asr
docker compose exec api-server curl http://lightning-asr:2233/health
```

## Health Checks

### API Server

```bash
curl http://localhost:7100/health
```

Expected: `{"status": "healthy"}`

### Lightning ASR

```bash
curl http://localhost:2233/health
```

Expected: `{"status": "ready", "gpu": "NVIDIA A10"}`

### License Proxy

```bash
docker compose exec license-proxy wget -q -O- http://localhost:6699/health
```

Expected: `{"status": "valid"}`

### Redis

```bash
docker compose exec redis redis-cli ping
```

Expected: `PONG`

## Log Analysis

### Common Log Patterns

#### Successful Startup

```log
redis-1              | Ready to accept connections
license-proxy        | License validated successfully
lightning-asr-1      | Model loaded successfully
lightning-asr-1      | GPU: NVIDIA A10 (24GB)
lightning-asr-1      | Server ready on port 2233
api-server           | Connected to Lightning ASR
api-server           | API server listening on port 7100
```

#### License Issues

```log
license-proxy        | ERROR: License validation failed
license-proxy        | ERROR: Invalid license key
license-proxy        | ERROR: Connection to license server failed
```

#### GPU Issues

```log
lightning-asr-1      | ERROR: No CUDA-capable device detected
lightning-asr-1      | ERROR: CUDA out of memory
lightning-asr-1      | ERROR: GPU not accessible
```

#### Network Issues

```log
api-server           | ERROR: Connection refused: lightning-asr:2233
api-server           | ERROR: Timeout connecting to license-proxy
```

## Getting Help

### Before Contacting Support

Collect the following information:

#### System Information

```bash
docker version
docker compose version
nvidia-smi
uname -a
```

#### Container Status

```bash
docker compose ps > status.txt
docker stats --no-stream > resources.txt
```

#### Logs

```bash
docker compose logs > all-logs.txt
```

#### Configuration

Sanitize and include:

* docker-compose.yml
* .env (remove license key)

### Contact Support

Email: **[support@smallest.ai](mailto:support@smallest.ai)**

Include:

* Description of the issue
* Steps to reproduce
* System information
* Logs and configuration
* License key (via secure channel)

## What's Next?

#### [STT Configuration](/models/self-host/docker-setup/stt-deployment/configuration)

Advanced configuration options

#### [API Reference](/models/self-host/api-reference/authentication)

Integrate with your applications