Analyse Redis

README.md

Redis Performance Analysis and Monitoring Guide

Overview

This guide outlines steps to analyze Redis performance issues, particularly focusing on latency problems and ensuring data safety before restart procedures.

Analyzing Redis State

1. Memory Analysis

redis-cli INFO memory

Key metrics to watch:

• used_memory_human: Total memory used by Redis
• used_memory_rss_human: Total memory used from system perspective
• used_memory_peak_human: Peak memory consumed
• mem_fragmentation_ratio: > 1.5 indicates significant fragmentation
• mem_allocator: Memory allocator in use

2. Persistence Status

redis-cli INFO persistence

Critical metrics:

• rdb_last_save_time: Timestamp of last successful save
• rdb_changes_since_last_save: Number of changes since last dump
• rdb_last_bgsave_status: Must be "ok" for safe restart
• rdb_last_save_time_sec: How long last save took
• rdb_current_bgsave_time_sec: Current save progress if running

Example healthy output:

rdb_last_save_time:1612137171
rdb_changes_since_last_save:0
rdb_last_bgsave_status:ok

3. CPU and System Load

redis-cli INFO cpu

Monitor:

• used_cpu_sys: System CPU consumed by Redis
• used_cpu_user: User CPU consumed by Redis
• used_cpu_sys_children: System CPU consumed by background processes
• used_cpu_user_children: User CPU consumed by background processes

4. Keyspace Statistics

redis-cli INFO keyspace

Analyze:

• Number of keys per database
• Expiration settings
• Key distribution

5. Error Statistics

redis-cli INFO stats

Important metrics:

• total_connections_received
• rejected_connections
• keyspace_hits
• keyspace_misses
• latest_fork_usec: Time taken for last fork operation

Verifying Persistence Configuration

1. Check RDB Configuration

Verify in redis.conf:

# Snapshot settings
save 900 1        # After 900 sec if at least 1 change
save 300 10       # After 300 sec if at least 10 changes
save 60 10000     # After 60 sec if at least 10000 changes

# Directory settings
dir /var/lib/redis
dbfilename dump.rdb

2. Verify File Permissions

# Check Redis working directory
ls -la /var/lib/redis/

# Verify dump file permissions
ls -la /var/lib/redis/dump.rdb

3. Test Save Capability

# Manual save test
redis-cli SAVE

# Verify save success
redis-cli INFO persistence | grep rdb_last_save

Latency Investigation

1. Real-time Latency Monitoring

# Monitor command latency
redis-cli --latency

# Detailed latency analysis
redis-cli LATENCY DOCTOR

2. Memory-Related Latency

# Check if swapping is occurring
redis-cli INFO stats | grep process_id
ps -o majflt,minflt -p <pid>

3. Command Analysis

# Monitor slow commands
redis-cli SLOWLOG GET 10

# Reset slow log if needed
redis-cli SLOWLOG RESET

Pre-Restart Checklist

1. Verify persistence configuration is active:

redis-cli CONFIG GET save

2. Check last successful save:

redis-cli INFO persistence | grep rdb_last_save

3. Trigger manual save:

redis-cli SAVE

4. Verify save file:

ls -la /var/lib/redis/dump.rdb

5. Document current metrics:

redis-cli INFO memory
redis-cli DBSIZE

Post-Restart Verification

1. Check data persistence:

redis-cli DBSIZE  # Compare with pre-restart value
redis-cli INFO keyspace

2. Verify performance improvement:

redis-cli --latency
redis-cli INFO memory  # Check fragmentation ratio

Additional Considerations

1. Monitor file descriptors:

redis-cli INFO clients | grep connected_clients

2. Check client connections:

redis-cli CLIENT LIST | wc -l

3. AOF status (if enabled):

redis-cli INFO persistence | grep aof

Memory, keys, and last used

# Check total memory usage
redis-cli INFO memory | grep used_memory_human

# Count total keys
redis-cli DBSIZE

# Count keys by pattern (be careful on production)
redis-cli --scan --pattern '*' | wc -l

# Get key counts by database
redis-cli INFO keyspace

# Analyze key distribution and sizes
redis-cli --bigkeys

# Check if approaching system memory
redis-cli INFO memory | grep maxmemory_human

# Sample key patterns to identify potential issues
redis-cli --scan --pattern '*' | head -n 10

# Check key age and expiration
# Get keys with TTL
redis-cli --scan | xargs -L 100 redis-cli TTL

# Find keys without expiration (TTL = -1)
redis-cli --scan | xargs -L 100 redis-cli TTL | grep -c -- -1

# Check for keys with specific patterns and their TTLs
redis-cli --scan --pattern 'pattern:*' | xargs -L 100 redis-cli TTL

# Get key idle time (how long since last access)
redis-cli OBJECT IDLETIME keyname

# Sample idle time for random keys
redis-cli --scan --pattern '*' | head -n 50 | xargs -L 1 redis-cli OBJECT IDLETIME

# Get creation time of keys (if available)
redis-cli OBJECT FREQ keyname

Large memory and old keys

# Check total memory usage
redis-cli INFO memory | grep used_memory_human

# Count total keys
redis-cli DBSIZE

# Count keys by pattern (be careful on production)
redis-cli --scan --pattern '*' | wc -l

# Get key counts by database
redis-cli INFO keyspace

# Analyze key distribution and sizes
redis-cli --bigkeys

# Check if approaching system memory
redis-cli INFO memory | grep maxmemory_human

# Sample key patterns to identify potential issues
redis-cli --scan --pattern '*' | head -n 10

# Check key age and expiration
# Get keys with TTL
redis-cli --scan | xargs -L 100 redis-cli TTL

# Find keys without expiration (TTL = -1)
redis-cli --scan | xargs -L 100 redis-cli TTL | grep -c -- -1

# Check for keys with specific patterns and their TTLs
redis-cli --scan --pattern 'pattern:*' | xargs -L 100 redis-cli TTL

# Get key idle time (how long since last access)
redis-cli OBJECT IDLETIME keyname

# Sample idle time for random keys
redis-cli --scan --pattern '*' | head -n 50 | xargs -L 1 redis-cli OBJECT IDLETIME

# Get creation time of keys (if available)
redis-cli OBJECT FREQ keyname

Regular monitoring using these commands helps identify issues before they become critical. Always ensure proper persistence configuration before any restart procedure.

Sentinel.md

Redis StatefulSet Management Guide

Overview

This document provides instructions and best practices for managing Redis StatefulSets in Kubernetes, including rolling restarts, monitoring, and recovery procedures.

Basic Operations

Rolling Restart Procedure

To perform a rolling restart of the Redis cluster:

# Rolling restart the Redis pods (includes master and replicas)
kubectl rollout restart statefulset redis

# Rolling restart Sentinel pods
kubectl rollout restart statefulset redis-sentinel

StatefulSet Benefits

The StatefulSet deployment provides several key advantages:

1. Maintains stable network identities
2. Handles orderly termination
3. Maintains replica count
4. Handles persistent volumes correctly
5. Respects PodDisruptionBudgets

Monitoring Status

Check the rollout status:

# Check Redis StatefulSet status
kubectl rollout status statefulset redis

# Check Sentinel StatefulSet status
kubectl rollout status statefulset redis-sentinel

# Watch pods during rollout
kubectl get pods -l app=redis -w

# Check Redis cluster health
redis-cli cluster info

# View detailed events
kubectl describe statefulset redis

Rollback Procedure

If issues arise during the rollout:

# Rollback Redis StatefulSet
kubectl rollout undo statefulset redis

# Rollback Sentinel StatefulSet
kubectl rollout undo statefulset redis-sentinel

Advanced Configuration

Health Check Implementation

Add readiness probes to ensure proper pod health:

kubectl patch statefulset redis --patch '
spec:
  template:
    spec:
      containers:
      - name: redis
        readinessProbe:
          exec:
            command: ["redis-cli", "ping"]
          initialDelaySeconds: 15
          periodSeconds: 5'

Version Control

Enable version tracking for rollouts:

kubectl rollout restart statefulset redis --record=true

Rate Limiting Configuration

Configure update strategy with rate limiting:

kubectl patch statefulset redis --patch '
spec:
  updateStrategy:
    rollingUpdate:
      partition: 0
      maxUnavailable: 1'

Pre-rollout Checklist

1. Verify cluster state:

   redis-cli cluster nodes
   redis-cli sentinel master mymaster

2. Check PDB status:

   kubectl get pdb redis-pdb

3. Verify backup status

4. Document current replica count

5. Check resource quotas

Post-rollout Validation

1. Check replication status:

   kubectl exec redis-0 -- redis-cli info replication

2. Verify Sentinel configuration:

   kubectl exec redis-sentinel-0 -- redis-cli -p 26379 sentinel master mymaster

3. Monitor replication lag

4. Verify client connectivity

5. Check error logs

Best Practices

Resource Management

• Set appropriate resource requests and limits
• Configure horizontal pod autoscaling
• Implement proper storage class configuration

Monitoring

• Set up Prometheus metrics collection
• Configure alerts for replication lag
• Monitor memory usage and connections

Backup and Recovery

• Implement regular backup schedule
• Test recovery procedures
• Document backup retention policy

Security

• Enable Redis authentication
• Configure network policies
• Implement proper RBAC rules

Troubleshooting

Common Issues

1. Pod scheduling failures
2. Persistent volume claims issues
3. Network connectivity problems
4. Memory pressure

Resolution Steps

1. Check pod events:

   kubectl describe pod redis-0

2. Review logs:

   kubectl logs redis-0
   kubectl logs redis-sentinel-0

3. Verify network connectivity:

   kubectl exec redis-0 -- redis-cli ping

Key Considerations

1. High Availability

   • Maintain sufficient replica count
   • Configure appropriate PodDisruptionBudgets
   • Implement proper failover settings

2. Performance

   • Monitor memory usage
   • Track latency metrics
   • Configure appropriate persistence settings

3. Maintenance

   • Schedule regular updates
   • Plan maintenance windows
   • Document procedures

References

• Kubernetes StatefulSet documentation
• Redis Sentinel documentation
• Redis Cluster specification