Redis Basics: In-Memory Data Store

Redis Basics: In-Memory Data Store

Redis (REmote DIctionary Server) is an open-source, in-memory data structure store used as a database, cache, and message broker. It is known for its blazing-fast performance, supporting sub-millisecond response times by keeping data in RAM. Unlike traditional databases that store data on disk, Redis stores everything in memory, making it ideal for use cases that require real-time performance, such as caching, session management, real-time analytics, and message queuing.

Redis supports a rich set of data structures including strings, hashes, lists, sets, sorted sets, bitmaps, hyperloglogs, and geospatial indexes. It also provides persistence options (RDB snapshots and AOF logs), replication, Lua scripting, transactions, and pub/sub messaging. To understand Redis properly, it is helpful to be familiar with NoSQL databases, database caching, and database replication.

┌─────────────────────────────────────────────────────────────┐
│                         Redis                                │
├─────────────────────────────────────────────────────────────┤
│  In-Memory        → Ultra-fast (microsecond latency)        │
│  Persistent       → Optional disk persistence (RDB/AOF)     │
│  Data Structures  → Strings, Hashes, Lists, Sets, Sorted Sets│
│  Replication      → Master-replica for high availability    │
│  Cluster          → Horizontal scaling with sharding        │
│  Atomic Operations→ All operations are atomic               │
└─────────────────────────────────────────────────────────────┘

What Is Redis

Redis is an in-memory key-value store that is often described as a data structure server. Unlike simple key-value stores that only support strings, Redis supports complex data types and operations. It is written in C and is extremely fast, capable of handling hundreds of thousands of operations per second on modest hardware.

• In-Memory: All data is stored in RAM for maximum speed.
• Persistent: Data can be saved to disk for durability.
• Atomic: All Redis operations are atomic, ensuring data consistency.
• Data Structures: Supports strings, hashes, lists, sets, sorted sets, bitmaps, hyperloglogs, streams, and geospatial indexes.
• Replication: Master-replica replication for read scaling and high availability.
• High Availability: Redis Sentinel provides automatic failover.
• Clustering: Redis Cluster provides automatic sharding across multiple nodes.

Why Redis Matters

Redis is one of the most popular databases in the world, used by companies like Twitter, GitHub, Stack Overflow, and Snapchat. Its performance and versatility make it essential for modern applications.

• Ultra-Fast Performance: Sub-millisecond latency for millions of operations per second.
• Versatile Data Structures: One tool for caching, queuing, real-time analytics, and more.
• Simple API: Easy to learn and use, with clients available in all major programming languages.
• Reduced Database Load: Cache frequently accessed data to reduce load on primary databases.
• Real-Time Processing: Handle streaming data, leaderboards, and real-time counters.
• High Availability: Built-in replication and automatic failover with Sentinel.

Installing Redis

# Ubuntu / Debian
sudo apt update
sudo apt install redis-server

# macOS (Homebrew)
brew install redis

# Docker
docker run --name redis -d -p 6379:6379 redis

# Start Redis server
redis-server

# Connect to Redis CLI
redis-cli

# Test connection
ping
# Response: PONG

Redis Data Types

1. Strings

Strings are the simplest Redis data type, storing binary-safe strings up to 512 MB. They are used for caching, session storage, counters, and more.

# Set and get string values
SET user:1000:name "John Doe"
GET user:1000:name

# Set with expiration (seconds)
SET session:abc123 "user_data" EX 3600

# Set if not exists (for locks)
SETNX lock:resource "locked"

# Increment/decrement counters
INCR page_views
INCRBY page_views 10
DECR stock_count
DECRBY stock_count 5

# Get and set atomically
GETSET counter 0

# Multiple operations
MSET key1 "value1" key2 "value2"
MGET key1 key2

# Append to string
APPEND log "new entry"

# String length
STRLEN user:1000:name

2. Hashes

Hashes store field-value pairs, similar to objects or dictionaries. They are ideal for storing objects with multiple fields.

# Set hash fields
HSET user:1001 name "John Doe"
HSET user:1001 email "john@example.com" age 30

# Set multiple fields at once
HMSET user:1002 name "Jane Smith" email "jane@example.com" age 28

# Get fields
HGET user:1001 name
HMGET user:1001 name email
HGETALL user:1001

# Get all field names or values
HKEYS user:1001
HVALS user:1001

# Increment numeric field
HINCRBY user:1001 age 1

# Check if field exists
HEXISTS user:1001 email

# Delete fields
HDEL user:1001 age

# Get number of fields
HLEN user:1001

3. Lists

Lists are ordered collections of strings, implemented as linked lists. They are ideal for queues, stacks, and message buffers.

# Push to left (head) and right (tail)
LPUSH queue "task1"
RPUSH queue "task2"
RPUSH queue "task3"

# Pop from left and right (FIFO queue)
LPOP queue   # Removes from head
RPOP queue   # Removes from tail

# Blocking pop (for message queues)
BLPOP queue 5  # Wait 5 seconds for element

# Get range of elements
LRANGE queue 0 -1   # All elements
LRANGE queue 0 9    # First 10 elements

# Get length
LLEN queue

# Trim list (keep only range)
LTRIM queue 0 99   # Keep only first 100 items

# Insert at position
LINSERT queue BEFORE "task2" "new_task"

4. Sets

Sets are unordered collections of unique strings. They are ideal for tags, unique visitors, and relationships.

# Add members
SADD tags "redis" "database" "nosql"

# Remove members
SREM tags "database"

# Check if member exists
SISMEMBER tags "redis"

# Get all members
SMEMBERS tags

# Get number of members
SCARD tags

# Random member
SRANDMEMBER tags
SPOP tags  # Remove and return random member

# Set operations
SADD set1 1 2 3
SADD set2 2 3 4

SUNION set1 set2      # Union: 1,2,3,4
SINTER set1 set2      # Intersection: 2,3
SDIFF set1 set2       # Difference: 1

# Store results
SUNIONSTORE union_set set1 set2

5. Sorted Sets

Sorted sets are similar to sets but with an associated score, allowing ordered retrieval. They are ideal for leaderboards, rate limiting, and priority queues.

# Add members with scores
ZADD leaderboard 100 "player1"
ZADD leaderboard 95 "player2" 85 "player3"

# Increment score
ZINCRBY leaderboard 5 "player2"

# Get rank (0 = highest)
ZRANK leaderboard "player2"
ZREVRANK leaderboard "player2"  # Reverse rank

# Get by rank range
ZRANGE leaderboard 0 2 WITHSCORES   # Top 3 ascending
ZREVRANGE leaderboard 0 2 WITHSCORES # Top 3 descending

# Get by score range
ZRANGEBYSCORE leaderboard 90 100 WITHSCORES

# Get count in score range
ZCOUNT leaderboard 80 100

# Remove members
ZREM leaderboard "player3"

# Get score
ZSCORE leaderboard "player1"

6. Bitmaps

Bitmaps are compact arrays of bits, ideal for tracking boolean states like user check-ins or feature flags.

# Set bit at offset
SETBIT user:1000:active 7 1   # Set 8th bit to 1

# Get bit
GETBIT user:1000:active 7

# Count set bits
BITCOUNT user:1000:active

# Bitwise operations
BITOP AND result key1 key2
BITOP OR result key1 key2

7. HyperLogLogs

HyperLogLogs provide approximate cardinality counting with very low memory usage (12KB per key). They are ideal for counting unique visitors.

# Add elements
PFADD unique_visitors "user1" "user2" "user3"

# Get approximate count
PFCOUNT unique_visitors

# Merge multiple HyperLogLogs
PFMERGE daily_unique visitors_day1 visitors_day2

8. Geospatial Indexes

Geospatial indexes store longitude and latitude coordinates for location-based queries.

# Add locations
GEOADD locations 13.361389 38.115556 "Palermo"
GEOADD locations 15.087269 37.502669 "Catania"

# Calculate distance (km/mi)
GEODIST locations Palermo Catania km

# Get coordinates
GEOPOS locations Palermo

# Find nearby locations
GEORADIUS locations 15 37 100 km
GEORADIUSBYMEMBER locations Palermo 100 km

9. Streams

Streams are append-only log structures for event sourcing and message queuing (similar to Kafka).

# Add to stream
XADD mystream * sensor-id 1234 temperature 19.8

# Read from stream
XRANGE mystream - +

# Create consumer group
XGROUP CREATE mystream mygroup 0

# Read as consumer
XREADGROUP GROUP mygroup consumer1 COUNT 1 STREAMS mystream >

Redis Expiration (TTL)

Redis allows you to set expiration times on keys, automatically deleting them after the specified time. This is essential for caching, session management, and temporary data.

# Set expiration when creating key
SET session:abc123 "data" EX 3600

# Set expiration on existing key
EXPIRE session:abc123 3600

# Set expiration at specific Unix timestamp
EXPIREAT session:abc123 1735689600

# Check remaining time (seconds)
TTL session:abc123

# Remove expiration
PERSIST session:abc123

# Set expiration in milliseconds
PEXPIRE key 5000
PTTL key  # Time in milliseconds

Redis Persistence

Redis offers two persistence options to protect against data loss on restart.

# Save snapshot manually
SAVE   # Blocking (not recommended)
BGSAVE # Non-blocking (fork)

# Configure in redis.conf
save 900 1   # Save after 900 seconds if at least 1 key changed
save 300 10  # Save after 300 seconds if at least 10 keys changed
save 60 10000

# RDB files are compact and good for backups

# Enable AOF (redis.conf)
appendonly yes
appendfilename "appendonly.aof"

# Sync policies
appendfsync always    # Every write (slowest, safest)
appendfsync everysec  # Every second (default, balanced)
appendfsync no        # Let OS decide (fastest, risk of loss)

# AOF is more durable than RDB but larger files

# Enable both RDB and AOF
# AOF includes RDB preamble for faster recovery
aof-use-rdb-preamble yes

Redis Replication

Redis replication allows you to create read replicas, improving read scalability and providing high availability.

# On replica (redis.conf)
replicaof 192.168.1.100 6379

# Or via command
SLAVEOF 192.168.1.100 6379

# Check replication status
INFO replication

# Make replica a master (failover)
SLAVEOF NO ONE

Redis Sentinel (High Availability)

Redis Sentinel provides automatic failover, monitoring, and configuration management for Redis deployments.

sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

Redis Transactions

Redis transactions group multiple commands and execute them atomically using the MULTI/EXEC commands.

# Basic transaction
MULTI
SET user:1000:name "John"
SET user:1000:email "john@example.com"
INCR user:1000:login_count
EXEC

# Discard transaction
MULTI
SET key1 value1
SET key2 value2
DISCARD

# Optimistic locking with WATCH
WATCH user:1000:balance
balance = GET user:1000:balance
MULTI
SET user:1000:balance balance - 100
EXEC  # Fails if balance changed during WATCH

Redis Lua Scripting

Lua scripts allow you to execute complex logic atomically on the Redis server, reducing round trips and ensuring atomicity.

# Simple Lua script
EVAL "return redis.call('GET', KEYS[1])" 1 user:1000:name

# Atomic increment with limit
EVAL "
    local current = redis.call('GET', KEYS[1]) or 0
    if tonumber(current) < tonumber(ARGV[1]) then
        redis.call('INCR', KEYS[1])
        return 1
    end
    return 0
" 1 rate_limit:user:1000 100

# Load script for reuse
SCRIPT LOAD "return redis.call('GET', KEYS[1])"
# Returns SHA: abc123...
EVALSHA abc123... 1 user:1000:name

Redis Pub/Sub (Publish/Subscribe)

Redis Pub/Sub allows messages to be sent to multiple subscribers without storing them. It is ideal for real-time notifications and messaging.

# Subscriber 1
SUBSCRIBE notifications

# Subscriber 2
SUBSCRIBE notifications

# Publisher
PUBLISH notifications "Hello subscribers"

# Pattern subscription
PSUBSCRIBE news.*

# Unsubscribe
UNSUBSCRIBE notifications
PUNSUBSCRIBE news.*

Common Redis Use Cases

Common Redis Mistakes to Avoid

• Using Redis as Primary Database: Redis is in-memory; data loss is possible unless persistence is configured. Use Redis as a cache or secondary store.
• Large Keys and Values: Large keys and values consume memory and slow operations. Keep keys short and values small.
• KEYS Command in Production: KEYS scans the entire keyspace and blocks Redis. Use SCAN instead.
• No Expiration on Cached Data: Cached data without TTL causes memory exhaustion.
• Single Point of Failure: Run Redis with replicas and Sentinel for high availability.
• Ignoring Memory Limits: Set maxmemory and eviction policies to prevent out-of-memory crashes.
• Slow Lua Scripts: Lua scripts block Redis during execution. Keep scripts fast.

Redis Best Practices

• Use Connection Pooling: Reuse connections to reduce overhead.
• Set Appropriate TTL: Always set expiration for cached data.
• Use SCAN Instead of KEYS: SCAN iterates without blocking.
• Batch Commands with Pipelines: Reduce round-trip time for multiple commands.
• Monitor Memory Usage: Track memory and set maxmemory-policy (allkeys-lru for cache).
• Use Redis Cluster for Large Datasets: Distribute data across multiple nodes.
• Enable Persistence for Critical Data: Use AOF with everysec for balance of performance and durability.

import redis
r = redis.Redis()

# Without pipeline (multiple round trips)
for i in range(1000):
    r.incr('counter')

# With pipeline (single round trip)
pipe = r.pipeline()
for i in range(1000):
    pipe.incr('counter')
pipe.execute()

Frequently Asked Questions

1. What is the difference between Redis and Memcached?
   Redis supports rich data structures (strings, hashes, lists, sets), persistence, replication, and transactions. Memcached is a simple key-value cache. Redis is more versatile; Memcached is simpler for basic caching.
2. Is Redis persistent?
   Redis can be persistent using RDB snapshots or AOF logs. By default, Redis is not fully persistent; configure persistence for durability.
3. What is the maximum key size in Redis?
   Keys can be up to 512 MB. Values can also be up to 512 MB, but large values are not recommended.
4. Does Redis support clustering?
   Yes, Redis Cluster provides automatic sharding across multiple nodes, supporting up to 1000 nodes.
5. What is the difference between Redis and a relational database?
   Redis is in-memory and schema-less, optimized for speed. Relational databases are disk-based with fixed schemas, optimized for complex queries and ACID transactions.
6. What should I learn next after Redis basics?
   After mastering Redis basics, explore advanced Redis features, Redis Cluster, Redis Sentinel for HA, and Redis Streams for event processing.