Kubernetes SetupStorage & PVC

Redis Persistence

View as MarkdownOpen in Claude

Overview

Redis provides caching and state management for Smallest Self-Host. This guide covers configuring Redis persistence, high availability, and performance optimization.

Redis Deployment Options

Option 1: Embedded Redis (Default)

Smallest Self-Host includes Redis as a subchart.

Advantages:

  • Simple setup
  • Automatic configuration
  • Included in Helm chart

Disadvantages:

  • Single point of failure
  • No data persistence by default
  • Limited to cluster resources

Configuration:

values.yaml
1redis:
2  enabled: true
3  auth:
4    enabled: true
5    password: "your-secure-password"
6  master:
7    persistence:
8      enabled: false
9  replica:
10    replicaCount: 1
11    persistence:
12      enabled: false

Option 2: External Redis

Use Amazon ElastiCache or self-managed Redis.

Advantages:

  • Managed service (ElastiCache)
  • High availability
  • Better performance
  • Independent scaling

Disadvantages:

  • Additional cost
  • More complex setup

Configuration:

values.yaml
1redis:
2  enabled: false
3  externalHost: "my-redis.abc123.0001.use1.cache.amazonaws.com"
4  port: 6379
5  ssl: false
6  auth:
7    enabled: true
8    password: "redis-password"

Enable Redis Persistence

With Embedded Redis

Enable AOF (Append-Only File) persistence:

values.yaml
1redis:
2  enabled: true
3  auth:
4    enabled: true
5    password: "your-secure-password"
6  master:
7    persistence:
8      enabled: true
9      storageClass: "gp3"
10      size: 8Gi
11      accessModes:
12        - ReadWriteOnce
13  replica:
14    replicaCount: 2
15    persistence:
16      enabled: true
17      storageClass: "gp3"
18      size: 8Gi

This creates:

  • 1 master pod with persistent volume
  • 2 replica pods with persistent volumes
  • Automatic failover

Verify Persistence

Check PVCs created:

$kubectl get pvc -n smallest | grep redis

Expected output:

redis-data-smallest-redis-master-0    Bound    8Gi
redis-data-smallest-redis-replicas-0  Bound    8Gi
redis-data-smallest-redis-replicas-1  Bound    8Gi

High Availability

Sentinel Mode

Redis Sentinel provides automatic failover:

values.yaml
1redis:
2  enabled: true
3  sentinel:
4    enabled: true
5    quorum: 2
6  master:
7    persistence:
8      enabled: true
9      size: 8Gi
10  replica:
11    replicaCount: 2
12    persistence:
13      enabled: true
14      size: 8Gi

Cluster Mode

For very high throughput:

values.yaml
1redis:
2  enabled: true
3  architecture: replication
4  master:
5    count: 3
6  replica:
7    replicaCount: 2

AWS ElastiCache Integration

Create ElastiCache Cluster

Using AWS Console:

2

Cluster Settings

  • Cluster mode: Disabled (for simplicity)
  • Name: smallest-redis
  • Engine version: 7.0+
  • Node type: cache.r6g.large (or larger)
3

Subnet Group

Select subnet group in same VPC as EKS cluster

4

Security

  • Security group: Allow port 6379 from EKS cluster
  • Encryption in transit: Enabled
  • Encryption at rest: Enabled
5

Backup

  • Automatic backups: Enabled
  • Retention: 7 days
6

Create

Review and create (takes 10-15 minutes)

Note the Primary endpoint

Configure Helm Chart

values.yaml
1redis:
2  enabled: false
3  externalHost: "smallest-redis.abc123.0001.use1.cache.amazonaws.com"
4  port: 6379
5  ssl: true
6  auth:
7    enabled: false
8
9lightningAsr:
10  env:
11    - name: REDIS_URL
12      value: "rediss://smallest-redis.abc123.0001.use1.cache.amazonaws.com:6379"
13    - name: REDIS_TLS
14      value: "true"

Performance Tuning

Memory Configuration

Set memory limits for embedded Redis:

values.yaml
1redis:
2  master:
3    resources:
4      limits:
5        memory: 2Gi
6      requests:
7        memory: 1Gi
8  replica:
9    resources:
10      limits:
11        memory: 2Gi
12      requests:
13        memory: 1Gi

Eviction Policy

Configure memory eviction:

values.yaml
1redis:
2  master:
3    configuration: |
4      maxmemory-policy allkeys-lru
5      maxmemory 1gb

Disable Persistence for Performance

For non-critical data (faster performance):

values.yaml
1redis:
2  master:
3    configuration: |
4      save ""
5      appendonly no
6    persistence:
7      enabled: false

Without persistence, all data is lost if Redis restarts. Only use for truly ephemeral data.

Monitoring Redis

Check Redis Status

$kubectl get pods -l app.kubernetes.io/name=redis -n smallest

Connect to Redis CLI

$kubectl exec -it <redis-pod> -n smallest -- redis-cli

Inside redis-cli:

1AUTH your-password
2INFO
3DBSIZE
4KEYS *

Monitor Memory Usage

$kubectl exec -it <redis-pod> -n smallest -- redis-cli INFO memory

Monitor Performance

$kubectl exec -it <redis-pod> -n smallest -- redis-cli INFO stats

Backup and Recovery

Manual Backup

Create snapshot:

$kubectl exec -it <redis-master-pod> -n smallest -- redis-cli BGSAVE

Copy RDB file:

$kubectl cp <redis-master-pod>:/data/dump.rdb ./redis-backup.rdb -n smallest

Scheduled Backups

Create CronJob for automatic backups:

redis-backup-cronjob.yaml
1apiVersion: batch/v1
2kind: CronJob
3metadata:
4  name: redis-backup
5  namespace: smallest
6spec:
7  schedule: "0 2 * * *"
8  jobTemplate:
9    spec:
10      template:
11        spec:
12          containers:
13          - name: backup
14            image: redis:7-alpine
15            command:
16              - sh
17              - -c
18              - |
19                redis-cli -h smallest-redis-master BGSAVE
20                sleep 60
21                kubectl cp smallest-redis-master-0:/data/dump.rdb /backup/redis-$(date +%Y%m%d).rdb
22            volumeMounts:
23              - name: backup
24                mountPath: /backup
25          volumes:
26            - name: backup
27              persistentVolumeClaim:
28                claimName: redis-backup-pvc
29          restartPolicy: OnFailure

Restore from Backup

$kubectl cp ./redis-backup.rdb <redis-master-pod>:/data/dump.rdb -n smallest
$
$kubectl exec -it <redis-master-pod> -n smallest -- redis-cli SHUTDOWN NOSAVE
$
$kubectl delete pod <redis-master-pod> -n smallest

Pod will restart and load from backup.

Security

Enable Authentication

Always use password authentication:

values.yaml
1redis:
2  auth:
3    enabled: true
4    password: "strong-random-password"

Or use existing secret:

values.yaml
1redis:
2  auth:
3    enabled: true
4    existingSecret: "redis-secret"
5    existingSecretPasswordKey: "redis-password"

Enable TLS

For embedded Redis:

values.yaml
1redis:
2  tls:
3    enabled: true
4    authClients: true
5    certFilename: "tls.crt"
6    certKeyFilename: "tls.key"
7    certCAFilename: "ca.crt"

Network Policies

Restrict access to Redis:

redis-network-policy.yaml
1apiVersion: networking.k8s.io/v1
2kind: NetworkPolicy
3metadata:
4  name: redis-policy
5  namespace: smallest
6spec:
7  podSelector:
8    matchLabels:
9      app.kubernetes.io/name: redis
10  policyTypes:
11    - Ingress
12  ingress:
13    - from:
14        - podSelector:
15            matchLabels:
16              app: lightning-asr
17        - podSelector:
18            matchLabels:
19              app: api-server
20      ports:
21        - protocol: TCP
22          port: 6379

Scaling Redis

Vertical Scaling

Increase resources:

values.yaml
1redis:
2  master:
3    resources:
4      limits:
5        memory: 4Gi
6        cpu: 2

Restart pods:

$kubectl rollout restart statefulset smallest-redis-master -n smallest

Horizontal Scaling

Add more replicas:

values.yaml
1redis:
2  replica:
3    replicaCount: 3

Troubleshooting

Connection Refused

Check Redis pod is running:

$kubectl get pods -l app.kubernetes.io/name=redis -n smallest
$kubectl logs -l app.kubernetes.io/name=redis -n smallest

Test connection:

$kubectl run redis-test --rm -it --restart=Never \
>  --image=redis:7-alpine \
>  --command -- redis-cli -h smallest-redis-master -a your-password ping

Out of Memory

Check memory usage:

$kubectl exec -it <redis-pod> -n smallest -- redis-cli INFO memory

Increase memory limit or enable eviction:

1redis:
2  master:
3    resources:
4      limits:
5        memory: 4Gi
6    configuration: |
7      maxmemory-policy allkeys-lru

Slow Performance

Check latency:

$kubectl exec -it <redis-pod> -n smallest -- redis-cli --latency

Check slow queries:

$kubectl exec -it <redis-pod> -n smallest -- redis-cli SLOWLOG GET 10

Data Loss

Check if persistence is enabled:

$kubectl exec -it <redis-pod> -n smallest -- redis-cli CONFIG GET save
$kubectl exec -it <redis-pod> -n smallest -- redis-cli CONFIG GET appendonly

Best Practices

Enable password authentication even for internal Redis:

1redis:
2  auth:
3    enabled: true
4    password: "strong-password"

Use AOF for maximum durability:

1redis:
2  master:
3    persistence:
4      enabled: true
5    configuration: |
6      appendonly yes
7      appendfsync everysec

At least 2 replicas for high availability:

1redis:
2  replica:
3    replicaCount: 2

Use Redis exporter for Prometheus:

$helm install redis-exporter prometheus-community/prometheus-redis-exporter \
>  --set redisAddress=redis://smallest-redis-master:6379

Schedule automatic backups:

  • ElastiCache: Enable automatic backups
  • Self-managed: Use CronJob for BGSAVE

What’s Next?

HPA Configuration

Configure autoscaling for Lightning ASR

Metrics Setup

Set up Prometheus metrics collection