Common Issues

Overview

This guide provides quick solutions to the most common issues encountered with Smallest Self-Host across Docker and Kubernetes deployments.

Installation Issues

License Key Invalid

Symptoms:

• License validation failed
• Invalid license key
• Services fail to start

Quick Fix:

$
echo $LICENSE_KEY | wc -c

$
curl -H "Authorization: Bearer ${LICENSE_KEY}" https://console-api.smallest.ai/validate

Image Pull Failed

Symptoms:

• ImagePullBackOff
• unauthorized: authentication required
• manifest unknown

Quick Fix:

$
docker login quay.io
$
Username: your-username
$
Password: your-password
$
$
docker pull quay.io/smallestinc/lightning-asr:latest

Model Download Failed

Symptoms:

• Lightning ASR stuck at startup
• Failed to download model
• Connection timeout

Quick Fix:

1. Verify URL:

   $
   curl -I $MODEL_URL

2. Check disk space:

   $
   df -h

   Need at least 30 GB free

3. Test network:

   $
   wget --spider $MODEL_URL

4. Increase timeout (Kubernetes):

   1
   lightningAsr:
   2
     env:
   3
       - name: DOWNLOAD_TIMEOUT
   4
         value: "3600"

Runtime Issues

High Latency

Symptoms:

• Requests taking >1 second
• Slow transcription
• Timeouts

Quick Fix:

$
nvidia-smi
$
kubectl exec -it <lightning-asr-pod> -- nvidia-smi

$
kubectl describe pod <pod-name> | grep -A5 "Limits"

1
lightningAsr:
2
  resources:
3
    limits:
4
      memory: 16Gi
5
      cpu: 8

$
sudo nvidia-smi -pm 1

1
gpu-operator:
2
  driver:
3
    env:
4
      - name: NVIDIA_DRIVER_CAPABILITIES
5
        value: "compute,utility"

Out of Memory

Symptoms:

• Pod killed (exit code 137)
• OOMKilled status
• Memory errors in logs

Quick Fix:

1. Increase memory limit:

   1
   lightningAsr:
   2
     resources:
   3
       limits:
   4
         memory: 20Gi
   5
       requests:
   6
         memory: 16Gi

2. Check memory leaks:

   $
   kubectl top pod <pod-name>

3. Restart pod:

   $
   kubectl delete pod <pod-name>

Connection Refused

Symptoms:

• Cannot connect to API
• Connection refused
• Service unavailable

Quick Fix:

$
kubectl get pods -n smallest
$
docker compose ps

Performance Issues

Slow Autoscaling

Symptoms:

• HPA not scaling fast enough
• Pods stuck in Pending
• Cluster Autoscaler delayed

Quick Fix:

1. Reduce HPA stabilization:

   1
   scaling:
   2
     auto:
   3
       lightningAsr:
   4
         hpa:
   5
           scaleUpStabilizationWindowSeconds: 0

2. Check metrics available:

   $
   kubectl get hpa
   $
   kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"

3. Verify node capacity:

   $
   kubectl describe nodes | grep -A5 "Allocated resources"

Request Queue Building Up

Symptoms:

• Increasing active requests
• Users experiencing delays
• HPA shows high metrics

Quick Fix:

1. Manual scale up:

   $
   kubectl scale deployment lightning-asr --replicas=10

2. Check autoscaling limits:

   1
   scaling:
   2
     auto:
   3
       lightningAsr:
   4
         hpa:
   5
           maxReplicas: 20

3. Add cluster capacity:

   $
   eksctl scale nodegroup --cluster=smallest-cluster --name=gpu-nodes --nodes=5

Data Issues

Transcription Quality Poor

Symptoms:

• Low confidence scores
• Incorrect transcriptions
• Missing words

Quick Fix:

1. Check audio quality:

   • Sample rate: 16 kHz minimum (44.1 kHz recommended)
   • Format: WAV or FLAC preferred
   • Channels: Mono for best results

2. Enable punctuation:

   1
   {
   2
     "url": "...",
   3
     "punctuate": true,
   4
     "language": "en"
   5
   }

3. Verify correct language:

   1
   {
   2
     "url": "...",
   3
     "language": "es"
   4
   }

Missing Timestamps

Symptoms:

• No word-level timing data
• Unable to sync with video

Quick Fix:

Enable timestamps in request:

1
{
2
  "url": "...",
3
  "timestamps": true
4
}

Response will include:

1
{
2
  "words": [
3
    {"word": "Hello", "start": 0.0, "end": 0.5}
4
  ]
5
}

Network Issues

Cannot Reach License Server

Symptoms:

• Grace period activated
• Connection to license server failed
• Services still working but warnings

Quick Fix:

1. Test connectivity:

   $
   curl -v https://console-api.smallest.ai

2. Check firewall rules:

   • Allow outbound HTTPS (port 443)
   • Whitelist console-api.smallest.ai

3. Review network policies (Kubernetes):

   $
   kubectl get networkpolicy -n smallest

4. Monitor grace period:

   $
   kubectl logs -l app=license-proxy | grep -i "grace"

Slow Downloads

Symptoms:

• Model download taking >30 minutes
• Audio file upload slow

Quick Fix:

1. Use faster network:

   • AWS S3 in same region
   • CloudFront CDN

2. Enable parallel downloads:

   1
   lightningAsr:
   2
     env:
   3
       - name: DOWNLOAD_WORKERS
   4
         value: "4"

3. Use shared storage (Kubernetes):

   1
   models:
   2
     volumes:
   3
       aws:
   4
         efs:
   5
           enabled: true

Quick Diagnostics

One-Command Health Check

$
curl http://localhost:7100/health && \
>
  kubectl get pods -n smallest && \
>
  kubectl top nodes && \
>
  kubectl top pods -n smallest

Collect All Logs

$
kubectl logs -l app=lightning-asr -n smallest --tail=100 > asr-logs.txt
$
kubectl logs -l app=api-server -n smallest --tail=100 > api-logs.txt
$
kubectl logs -l app=license-proxy -n smallest --tail=100 > license-logs.txt

Test Transcription

$
curl -X POST http://localhost:7100/v1/listen \
>
  -H "Authorization: Token ${LICENSE_KEY}" \
>
  -H "Content-Type: application/json" \
>
  -d '{"url": "https://www2.cs.uic.edu/~i101/SoundFiles/StarWars60.wav"}'

Getting Help

If issues persist: