Grafana Dashboards

Overview

Grafana provides powerful visualization of Lightning ASR metrics, autoscaling behavior, and system performance. This guide covers accessing Grafana, importing dashboards, and creating custom visualizations.

Access Grafana

Enable Grafana

Ensure Grafana is enabled in your Helm values:

1
scaling:
2
  auto:
3
    enabled: true
4
5
kube-prometheus-stack:
6
  grafana:
7
    enabled: true
8
    adminPassword: "admin-password"

Port Forward

Access Grafana locally:

$
kubectl port-forward -n default svc/smallest-prometheus-stack-grafana 3000:80

Open http://localhost:3000 in your browser.

Default Credentials

• Username: admin
• Password: prom-operator (or custom password from adminPassword)

1
grafana:
2
  adminPassword: "your-secure-password"

Expose Externally

For permanent access, expose via LoadBalancer or Ingress:

1
kube-prometheus-stack:
2
  grafana:
3
    service:
4
      type: LoadBalancer

Import ASR Dashboard

The Smallest Self-Host repository includes a pre-built ASR dashboard.

Import from File

Import via ConfigMap

Automatically load dashboard on Grafana startup:

1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: asr-dashboard
5
  namespace: default
6
  labels:
7
    grafana_dashboard: "1"
8
data:
9
  asr-dashboard.json: |
10
    {
11
      "dashboard": ...,
12
      "overwrite": true
13
    }

Or enable via Helm:

1
kube-prometheus-stack:
2
  grafana:
3
    dashboardProviders:
4
      dashboardproviders.yaml:
5
        apiVersion: 1
6
        providers:
7
          - name: 'default'
8
            folder: 'Smallest'
9
            type: file
10
            options:
11
              path: /var/lib/grafana/dashboards/default
12
    
13
    dashboards:
14
      default:
15
        asr-dashboard:
16
          file: dashboards/asr-dashboard.json

ASR Dashboard Overview

The pre-built dashboard includes the following panels:

Active Requests

Shows current requests being processed:

• Metric: asr_active_requests
• Visualization: Stat panel with thresholds
• Colors:
  • Green: 0-5 requests
  • Yellow: 5-10 requests
  • Orange: 10-20 requests
  • Red: 20+ requests

Request Rate

Requests per second over time:

• Metric: rate(asr_total_requests[5m])
• Visualization: Time series graph
• Use: Track traffic patterns

Error Rate

Failed requests percentage:

• Metric: rate(asr_failed_requests[5m]) / rate(asr_total_requests[5m]) * 100
• Visualization: Stat panel + time series
• Alert: Warning if > 5%

Response Time

Request duration percentiles:

• Metrics:
  • P50: histogram_quantile(0.50, asr_request_duration_seconds_bucket)
  • P95: histogram_quantile(0.95, asr_request_duration_seconds_bucket)
  • P99: histogram_quantile(0.99, asr_request_duration_seconds_bucket)
• Visualization: Time series graph

Pod Count

Number of Lightning ASR replicas:

• Metric: count(asr_active_requests)
• Visualization: Stat panel
• Use: Monitor autoscaling

GPU Utilization

GPU usage per pod:

• Metric: asr_gpu_utilization
• Visualization: Time series graph
• Use: Ensure GPUs are utilized

GPU Memory

GPU memory usage:

• Metric: asr_gpu_memory_used_bytes / 1024 / 1024 / 1024
• Visualization: Gauge + time series
• Use: Monitor memory leaks

Create Custom Dashboards

Add New Dashboard

Useful Queries

Average Active Requests

1
avg(asr_active_requests)

Total Throughput (requests/hour)

1
sum(rate(asr_total_requests[1h])) * 3600

Pod Resource Usage

1
sum(container_memory_usage_bytes{pod=~"lightning-asr.*"}) by (pod) / 1024 / 1024 / 1024

Autoscaling Events

1
kube_deployment_status_replicas{deployment="lightning-asr"}

GPU Temperature

1
asr_gpu_temperature_celsius

Dashboard Variables

Add variables for dynamic filtering:

Namespace Variable

1
asr_active_requests{namespace="$namespace"}

Pod Variable

label_values(asr_active_requests{namespace="$namespace"}, pod)

Time Range Variable

$__interval

Use in queries for dynamic aggregation.

Alerting

Configure Alert Rules

WHEN avg() OF query(A, 5m, now) IS ABOVE 20

Alert Notification Channels

Configure notifications:

Pre-Built Dashboard Examples

System Overview Dashboard

1
{
2
  "title": "Smallest Self-Host Overview",
3
  "panels": [
4
    {
5
      "title": "Active Requests",
6
      "targets": [{"expr": "sum(asr_active_requests)"}]
7
    },
8
    {
9
      "title": "Request Rate",
10
      "targets": [{"expr": "sum(rate(asr_total_requests[5m]))"}]
11
    },
12
    {
13
      "title": "Pod Count",
14
      "targets": [{"expr": "count(asr_active_requests)"}]
15
    },
16
    {
17
      "title": "Error Rate %",
18
      "targets": [{"expr": "sum(rate(asr_failed_requests[5m])) / sum(rate(asr_total_requests[5m])) * 100"}]
19
    }
20
  ]
21
}

Autoscaling Dashboard

Track HPA behavior:

1
kube_deployment_status_replicas{deployment="lightning-asr"}
2
kube_deployment_status_replicas_available{deployment="lightning-asr"}
3
kube_horizontalpodautoscaler_status_desired_replicas{horizontalpodautoscaler="lightning-asr"}
4
kube_horizontalpodautoscaler_status_current_replicas{horizontalpodautoscaler="lightning-asr"}

Cost Dashboard

Monitor resource costs:

1
sum(kube_pod_container_resource_requests{pod=~"lightning-asr.*"}) by (resource)
2
count(kube_node_info{node=~".*gpu.*"}) * 1.00

Best Practices

$
kubectl get configmap asr-dashboard -o jsonpath='{.data.asr-dashboard\.json}' > asr-dashboard.json
$
git add asr-dashboard.json
$
git commit -m "Update ASR dashboard"

Troubleshooting

Grafana Not Showing Data

Check Prometheus data source:

Grafana → Configuration → Data Sources → Prometheus

• URL: http://smallest-prometheus-stack-prometheus:9090
• Access: Server (default)

Test connection with “Save & Test” button.

Check Prometheus is running:

$
kubectl get pods -l app.kubernetes.io/name=prometheus

Queries Returning No Data

Verify metric exists in Prometheus:

$
kubectl port-forward svc/smallest-prometheus-stack-prometheus 9090:9090

Open http://localhost:9090 and query the metric.

Check time range: Ensure time range includes data.

Dashboard Not Loading

Check Grafana logs:

$
kubectl logs -l app.kubernetes.io/name=grafana

Increase memory if needed:

1
kube-prometheus-stack:
2
  grafana:
3
    resources:
4
      limits:
5
        memory: 512Mi

What’s Next?