Redis Pub/Sub vs Redis Streams: Choosing the Right Messaging Pattern for Your Application

Redis Pub/Sub vs Redis Streams: Choosing the Right Messaging Pattern

Table of Contents

• Introduction
• Core Concepts
• Architecture Overview
• Step-by-Step Guide
• Real-World Examples
• Production Code Examples
• Comparison Table
• Best Practices
• Common Mistakes
• Performance Tips
• Security Considerations
• Deployment Notes
• Debugging Tips
• FAQ
• Conclusion

Introduction

When building event-driven or real-time systems with Redis, developers often reach a crossroads: should they use Redis Pub/Sub or Redis Streams? Both are built into Redis and both handle message passing, but their design philosophies, reliability guarantees, and use cases diverge significantly. Choosing the wrong pattern leads to data loss, duplicated processing, unnecessary infrastructure complexity, or poor scalability.

Redis Pub/Sub is a fire-and-forget broadcast mechanism optimized for low-latency fan-out messaging to many subscribers simultaneously. Redis Streams, introduced in Redis 5.0, is a log-based data structure that provides persistence, consumer groups, acknowledgments, and replay capability. This guide walks through both mechanisms in depth, compares them across dimensions that matter in production, and gives you a clear decision framework backed by code examples you can use today.

Core Concepts

Redis Pub/Sub Fundamentals

Redis Pub/Sub has three actor types: publishers, subscribers, and channels. A publisher sends a message to a named channel. Every client currently subscribed to that channel receives the message in real time. Messages are not stored. If no subscriber is listening when a message arrives, the message is dropped permanently.

The publish operation is O(N) where N is the number of subscribers on that channel, because Redis delivers the message to each subscriber synchronously within the single command execution. This makes Pub/Sub fast for small to moderate subscriber counts but potentially blocking under high fan-out scenarios.

Redis Streams Fundamentals

Redis Streams is an append-only log data structure, conceptually similar to Apache Kafka but running inside Redis itself. Each entry in a stream has a unique, monotonically increasing ID composed of a millisecond timestamp and a sequence number. Entries can carry any number of field-value pairs.

Streams support consumer groups, which allow multiple consumers to cooperatively process entries from a stream. Each consumer in a group tracks its own delivery counter, enabling at-least-once processing semantics. Consumers acknowledge processed entries, and unacknowledged entries can be reclaimed by the group after a timeout. This gives Streams the reliability of a proper message queue while keeping Redis's in-memory performance characteristics.

Architecture Overview

Pub/Sub Architecture

In a Pub/Sub architecture, publishers connect to Redis as regular clients, issue PUBLISH commands, and move on. Subscribers use the SUBSCRIBE or PSUBSCRIBE commands to register interest in channels or patterns. Redis internally maintains a mapping from channels to client connection lists and fans out each message to every listed subscriber over their respective TCP connections.

This architecture is simple and requires no additional infrastructure. There are no consumer groups, no offsets to manage, and no persisted backlog. The trade-off is strict at-most-once delivery with no durability guarantee. If a subscriber disconnects, it misses all messages sent while disconnected.

Streams Architecture

A Streams architecture has three key components: producers, consumers in consumer groups, and the stream itself. Producers use XADD to append entries. Consumer groups use XREADGROUP to read entries, and XACK to mark them as processed. Redis stores pending entries in the PEL (Pending Entries List) for each consumer group member, enabling dead-letter handling and reliable redelivery.

The stream is a persistent data structure that survives consumer disconnections and Redis restarts (with AOF or RDB persistence enabled). This fundamental difference from Pub/Sub makes Streams suitable for any workflow where message loss is unacceptable or where consumers need to catch up on missed messages after reconnecting.

Step-by-Step Guide

Setting Up Pub/Sub

Start by launching Redis and connecting with two separate client instances — one for publishing, one for subscribing. In redis-cli, open a terminal and subscribe to a test channel:

SUBSCRIBE news.alerts

In a second terminal, publish a message:

PUBLISH news.alerts "Breaking: Redis 8 released"

The subscriber terminal receives the message as a three-element array: the message type ("message"), the channel name ("news.alerts"), and the payload ("Breaking: Redis 8 released"). To subscribe to pattern-matched channels, use PSUBSCRIBE with a glob-style pattern like "news.*".

Setting Up Streams

Create a stream and write an entry using XADD:

XADD mystream * event user_login user_id 42 ip 203.0.113.5

The asterisk tells Redis to auto-generate the entry ID. Next, create a consumer group with XGROUP:

XGROUP CREATE mystream mygroup 0 MKSTREAM

Read entries from the group using XREADGROUP:

XREADGROUP GROUP mygroup worker-1 COUNT 1 STREAMS mystream >

The > symbol tells Redis to deliver only new entries not yet delivered to this consumer group member. Acknowledge processing with XACK:

XACK mystream mygroup 

To claim pending entries that have not been acknowledged within a timeout, use XCLAIM:

XCLAIM mystream mygroup worker-2 5000 

This reassigns a stuck or failed consumer's pending entries to another worker, enabling basic failure recovery without external tooling.

Real-World Examples

Use Case 1: Live Chat with Pub/Sub

A chat application needs to broadcast new messages to all participants in a room. Pub/Sub is ideal here because the messages are ephemeral, delivery guarantees beyond at-most-once are unnecessary, and latency matters more than durability. When a user sends a message, the server publishes it to a channel named "chat:room:{roomId}". All connected clients subscribed to that channel receive the message instantly and render it in the UI.

If a user's WebSocket disconnects briefly, missing messages are acceptable — the application syncs chat history from a persistent database on reconnection rather than from the Pub/Sub layer. This is a textbook scenario where Pub/Sub's simplicity and speed outweigh its lack of persistence.

Use Case 2: Order Processing with Streams

An e-commerce platform processes orders through a pipeline of microservices: validation, payment, inventory reservation, and shipping notification. Each step is a separate consumer in the same consumer group reading from an "orders.new" stream. Using Streams guarantees that no order is lost if a service crashes mid-processing. The pending entries list ensures the payment service can pick up an order that the inventory service failed to acknowledge, and operations can inspect the stream with XRANGE to replay events for debugging or recovery.

Use Case 3: Hybrid Pattern

A single application can use both patterns simultaneously. Pub/Sub handles real-time dashboard updates (live metrics, user presence) where losing a few update messages is acceptable. Streams handle critical events like user actions for analytics or audit logging, where every event must be processed exactly once or at least once. This layered approach gives you the speed of Pub/Sub for non-critical paths and the reliability of Streams for paths where message durability is non-negotiable.

Production Code Examples

Redis Pub/Sub with Node.js

const { createClient } = require('redis');const publisher = createClient({ url: 'redis://localhost:6379' });const subscriber = createClient({ url: 'redis://localhost:6379' });async function setupPubSub() {  await publisher.connect();  await subscriber.connect();  subscriber.on('message', (channel, message) => {    console.log(`Received on ${channel}: ${message}`);  });  await subscriber.subscribe('notifications', (message) => {    console.log(`Notification: ${message}`);  });  await publisher.publish('notifications', 'New user signed up');  console.log('Message published');}setupPubSub().catch(console.error);

In production, wrap the subscriber connection with reconnect logic. The redis library emits 'error' events on disconnects — handle them to re-subscribe automatically. Also set appropriate socket timeouts so that a stalled subscriber connection is detected rather than consuming resources indefinitely.

Redis Streams with Python

import redisr = redis.Redis(host='localhost', port=6379, decode_responses=True)def produce_order(stream_name, order_id, user_id, amount):    entry_id = r.xadd(stream_name, {        'order_id': order_id,        'user_id': user_id,        'amount': str(amount),        'timestamp': str(int(time.time() * 1000))    })    return entry_iddef consume_orders(stream_name, group_name, consumer_name, count=10):    try:        messages = r.xreadgroup(            groupname=group_name,            consumername=consumer_name,            streams={stream_name: '>'},            count=count,            block=5000        )    except redis.ResponseError as e:        if 'NOGROUP' in str(e):            r.xgroup_create(stream_name, group_name, id='0', mkstream=True)            messages = r.xreadgroup(                groupname=group_name,                consumername=consumer_name,                streams={stream_name: '>'},                count=count,                block=5000            )        else:            raise    for stream, entries in messages:        for entry_id, data in entries:            process_order(data)            r.xack(stream_name, group_name, entry_id)def process_order(data):    order_id = data.get('order_id')    user_id = data.get('user_id')    amount = float(data.get('amount', 0))    print(f"Processing order {order_id} for user {user_id}: ${amount}")if __name__ == '__main__':    produce_order('orders.new', 'ORD-1001', 'USR-42', 149.99)    consume_orders('orders.new', 'order-processing', 'worker-1')

Redis Streams with Go

package mainimport (	"context"	"fmt"	"github.com/redis/go-redis/v9")var rdb *redis.Clientvar ctx = context.Background()func main() {	rdb = redis.NewClient(&redis.Options{		Addr: "localhost:6379",	})	// Produce	eid, err := rdb.XAdd(ctx, &redis.XAddArgs{		Stream: "events.audit",		Values: map[string]interface{}{			"action":  "login",			"user_id": "u-42",			"ip":      "203.0.113.5",		},	}).Result()	if err != nil {		panic(err)	}	fmt.Println("Produced:", eid)	// Consume using consumer group	_, err = rdb.XGroupCreateMkStream(ctx, "events.audit", "audit-processors", "0").Result()	// XGroupCreateMkStream returns NOGROUP error during the first call in some versions — handle gracefully	results, err := rdb.XReadGroup(ctx, &redis.XReadGroupArgs{		Group:    "audit-processors",		Consumer: "worker-1",		Streams:  map[string]string{"events.audit": ">"},		Count:    10,		Block:    5000,	}).Result()	if err != nil {		panic(err)	}	for _, stream := range results {		for _, msg := range stream.Messages {			fmt.Printf("Processing: %s -> %v", msg.ID, msg.Values)			rdb.XAck(ctx, "events.audit", "audit-processors", msg.ID)		}	}}

Comparison Table

Best Practices

When using Pub/Sub, keep channel names consistent and avoid overly broad channels that create unnecessary fan-out. A channel like "global" that every service subscribes to becomes a bottleneck as subscriber count increases. Instead, use hierarchical naming like "service:event:type" to constrain subscription scope naturally.

For Streams, always set a cap on stream length using MAXLEN or the approximately equal modifier (~) to avoid unbounded memory growth. A stream that grows without trimming will consume all available Redis memory and trigger eviction or crashes. For example:

XADD mystream MAXLEN ~ 10000 * event_type user_signup user_id 99

This trims the stream to approximately 10,000 entries, keeping memory predictable while retaining a reasonable replay window.

Use consumer groups for any workload where processing reliability matters. A single consumer reading with XREAD (without a group) gives you at-most-once delivery and no failure recovery. Consumer groups add the overhead of group coordination but give you ack-based durability and pending entry management. Always set a reasonable pending entry timeout — the default of one hour is too long for most applications. A 30-second to 5-minute timeout strikes a good balance between allowing slow consumers to recover and reassigning stuck work quickly.

Common Mistakes

The most common Pub/Sub mistake is relying on it for any workflow where message loss is unacceptable. Because Pub/Sub drops messages when no subscriber is connected, it is never appropriate for order processing, financial transactions, or audit logging. If you find yourself building a persistence layer on top of Pub/Sub to recover missed messages, you have already chosen the wrong tool.

A common Streams mistake is failing to trim streams over time. A high-throughput application writing thousands of events per second can fill memory quickly if streams are left unbounded. Another mistake is not handling the NOGROUP error gracefully when using XREADGROUP — the consumer group must exist before consumers can read from it, and the first consumer should create the group with MKSTREAM if it does not exist. In production, wrap the read operation in error handling that creates the group on demand or pre-creates groups during deployment.

Using Pub/Sub for inter-service communication in a microservices architecture is another frequent error. Pub/Sub has no mechanism to track which messages have been processed, making it impossible to build reliable service-to-service communication on top of it alone.

Performance Tips

For Pub/Sub throughput, minimize the payload size of messages. Redis Pub/Sub delivers messages synchronously to subscriber connections in the same event loop iteration. Large payloads increase the time each PUBLISH command takes, reducing overall throughput. Keep messages small — under 1 KB — for best PUBLISH latency under high fan-out.

For Streams performance, prefer the XADD with auto-generated IDs (using the asterisk) rather than manually constructing IDs. Redis generates IDs with sub-millisecond precision and avoids collisions. When reading streams, use COUNT to limit the number of entries returned per call rather than reading unlimited streams, which can block for extended periods on large streams.

Set the client output buffer limit for Pub/Sub subscribers appropriately. The default server-side buffer limit is 32 MB, but in high-fan-out scenarios where a subscriber temporarily falls behind, buffered messages can consume significant memory. Configure client-output-buffer-limit pubsub in redis.conf to a value appropriate for your subscriber's processing speed and network conditions.

Security Considerations

Pub/Sub channels are not access-controlled. Any client connected to the Redis instance can subscribe to any channel or pattern. If your Redis instance is accessible over an untrusted network, this means sensitive data sent over Pub/Sub channels can be intercepted by any connected client. Always use TLS for Redis connections in production, and prefer Redis ACLs to restrict which users can publish to or subscribe from specific channel namespaces.

Streams do not offer entry-level access control — any consumer in a consumer group can read all entries in the stream. If different consumer groups need access to different subsets of events, create separate streams for each sensitive category rather than relying on a single stream with shared access.

For both patterns, Redis ACLs can restrict commands per user. A monitoring dashboard user might be granted SUBSCRIBE and SSCAN permissions but denied PUBLISH. A logging consumer might be granted XREADGROUP and XACK but denied XADD. Apply the principle of least privilege to every Redis user account.

Deployment Notes

Redis Pub/Sub operates entirely in-memory with no disk persistence requirements for the messaging itself. Streams, however, benefit greatly from AOF (Append-Only File) persistence configured with the appendfsync everysec or yes directive. Without persistence, a Redis restart truncates streams to their last snapshot point, potentially losing unprocessed entries.

In clustered Redis deployments, both Pub/Sub and Streams work across nodes, but consumer group assignments are node-local. When a consumer reconnects to a different node after a failover, Redis redelivers pending entries to the new node using the consumer group's PEL. Ensure your cluster client library handles slot migration and MOVED redirects correctly to avoid missed reads during topology changes.

Configure stream trimming policies as part of your deployment configuration. A MAXLEN cap set at deployment time prevents unexpected memory growth in production. For high-volume streams, consider using a dedicated Redis instance or at minimum isolate high-throughput streams from latency-sensitive Pub/Sub channels on separate Redis instances to avoid resource contention.

Debugging Tips

For Pub/Sub issues, use the CLIENT LIST command filtered by type pubsub to see all active subscribers, their subscription count, and their subscribed channels. If a message is not reaching subscribers, verify that the channel name matches exactly (Pub/Sub channel matching is case-sensitive) and that the subscriber is connected when PUBLISH is called.

For Streams issues, inspect the consumer group state with XINFO GROUPS to see pending entry counts and last delivery IDs per consumer. If entries appear stuck in pending state without being acknowledged, use XPENDING to list specific pending entries and XCLAIM to reassign them to a healthy consumer. Use XRANGE to dump stream entries and verify that producers are writing to the expected stream key.

Monitor Redis memory metrics for Streams growth using INFO memory and MEMORY USAGE on specific stream keys. A stream whose memory grows linearly over time without trimming indicates a missing MAXLEN cap. Check the total number of entries with XLEN and set up alerting if the count exceeds your expected retention window.

FAQ

Can I use Redis Pub/Sub for inter-service communication in a microservices architecture?

Yes, but only for non-critical, ephemeral messaging where losing a message is acceptable. Pub/Sub is well-suited for broadcasting configuration updates, health check signals, or live metrics where occasional drops are tolerable. For inter-service communication requiring guaranteed delivery, use Redis Streams with consumer groups or a dedicated message broker.

What happens to Redis Pub/Sub messages if the Redis server crashes?

All Pub/Sub messages are lost permanently. Pub/Sub messages are delivered synchronously to connected subscribers and never written to disk. Redis server crashes, restarts, or failovers result in complete message loss for Pub/Sub channels. This is the most fundamental limitation of the Pub/Sub pattern and the primary reason to choose Streams for any workflow where reliability matters.

Can multiple consumer groups read from the same Redis stream?

Yes. Each consumer group reads entries independently. A single entry can be delivered to consumer group A and consumer group B simultaneously, making Streams useful for fan-out within the Streams model — one stream feeding multiple independent processing pipelines such as analytics, notification, and audit logging working in parallel from the same source of truth.

How do I prevent Redis Streams from consuming too much memory?

Use the MAXLEN argument with XADD to cap the stream length. The tilde (~) modifier enables approximate trimming, which is more performant than exact trimming under high write throughput. Alternatively, use XTRIM to periodically trim streams from the tail, keeping only the most recent entries. Monitor stream length with XLEN and set alerting thresholds.

Is Redis Streams a replacement for Apache Kafka?

No — Redis Streams is a viable messaging solution for small to medium workloads and teams already operating Redis. It lacks Kafka's partitioning model, built-in replication across data centers, and exactly-once processing semantics. For high-throughput, multi-datacenter event streaming at scale, Kafka remains the stronger choice. Redis Streams excels as an embedded messaging layer where simplicity and in-memory speed are the primary requirements.

Can I pattern-match on stream entries?

No. Redis Pub/Sub supports PSUBSCRIBE for glob-style pattern matching on channel names. Streams do not support pattern matching — consumers read directly from stream keys. If you need pattern-based routing, maintain a separate Pub/Sub channel for routing signals and use Streams for the actual payload delivery.

What is the maximum message size for Redis Pub/Sub and Streams?

The practical limit is governed by Redis's maxmemory and protocol constraints. Messages over 512 MB will be rejected by the client library and protocol. In practice, keep Pub/Sub messages under 1 KB for latency reasons and Streams entries under 512 KB per entry to avoid performance degradation. Large payloads should be stored externally (for example, in object storage) with only a reference key sent through Redis.

When should I choose Pub/Sub over Streams?

Choose Pub/Sub when you need lowest-latency broadcast to multiple subscribers, when message loss is tolerable, and when the simplicity of setup outweighs the need for persistence or acknowledgment tracking. Choose Streams when you need reliable at-least-once delivery, consumer group coordination, message replay, or any workflow where processing durability is a hard requirement.

Conclusion

Redis Pub/Sub and Redis Streams solve different problems, and understanding their architectural differences is essential for building reliable, performant systems. Pub/Sub gives you speed and simplicity for ephemeral broadcast messaging. Streams gives you durability, accountability, and replay capability for workflows where every event must eventually be processed.

The decision is not always binary — many applications benefit from using both patterns strategically, with Pub/Sub for real-time notification fan-out and Streams for durable event processing. Whichever pattern you choose, apply the production practices outlined in this guide: set stream length caps, handle consumer group creation gracefully, configure appropriate pending entry timeouts, and secure your Redis instance with TLS and ACLs.

Start with the code examples and deployment patterns shown here in your next project, and monitor your Redis instance's memory and CPU metrics from day one to catch issues before they impact your users.