Laravel Queue Batch Processing with Redis and Horizon: Advanced Patterns for High-Throughput Applications

Introduction

When your Laravel application graduates from processing hundreds of jobs per day to millions per hour, the standard queue patterns you learned in tutorials begin to crack under pressure. Individual job dispatching creates overwhelming Redis round-trips. Memory leaks accumulate in long-running workers. Failed jobs cascade into unmanageable backlogs. Monitoring becomes a guessing game.

Batch processing transforms this landscape. Instead of dispatching 10,000 individual notification jobs, you dispatch one batch containing 10,000 jobs. Laravel's Bus::batch() combined with Redis and Horizon gives you atomic dispatch, collective failure handling, progress tracking, and retry semantics that individual jobs simply cannot provide.

This article assumes you already understand Laravel queues, Redis basics, and Horizon fundamentals. We'll skip the "hello world" and dive straight into the patterns that distinguish hobby projects from systems processing billions of jobs per month at companies like Laravel Forge, Vapor, and enterprise SaaS platforms.

Table of Contents

• Introduction
• Core Concepts: Beyond Basic Batching
• Architecture Overview: Batch Processing Pipeline
• Step-by-Step Guide: Building Production-Ready Batches
• Real-World Examples: E-commerce Order Processing
• Production Code Examples
• Comparison: Batch vs Individual Job Patterns
• Best Practices for Batch Processing at Scale
• Common Mistakes That Cause Production Incidents
• Performance Tips: Squeezing Every Millisecond
• Security Considerations for Batch Workloads
• Deployment Notes: Horizon in Production
• Debugging Tips: When Batches Go Wrong
• FAQ
• Conclusion

Core Concepts: Beyond Basic Batching

The Batchable Contract

Laravel's Illuminate\Bus\Batchable trait is the gateway to batch processing. But the trait alone doesn't make your jobs batch-aware. The real power emerges when you understand the contract's implications:

• Atomic Dispatch: All jobs in a batch are dispatched in a single Redis transaction. Either all enqueue or none do.
• Collective Completion: The batch finishes only when every job succeeds or reaches its retry limit.
• Progress Aggregation: Batch::progress() gives you real-time completion percentage without polling individual jobs.
• Failure Isolation: One job's failure doesn't cancel siblings by default—configure this via allowFailures().

Batch Lifecycle Hooks

Three callbacks govern batch behavior:

$batch = Bus::batch($jobs)    ->then(function (Batch $batch) {        // All jobs succeeded    })    ->catch(function (Batch $batch, Throwable $e) {        // First job failure (if allowFailures is false)    })    ->finally(function (Batch $batch) {        // Always runs: success or failure    })    ->dispatch();

These hooks execute within the queue worker process, not your HTTP request. This distinction matters for database transactions, external API calls, and timeout management.

Batch ID and Persistence

Every batch receives a UUID stored in Redis under laravel_database_batches:{batch_id}. This key contains serialized batch metadata: job IDs, status counts, callback definitions, and progress data. The TTL defaults to 24 hours but is configurable via queue.batches.ttl in config/queue.php.

Understanding this persistence layer is critical. If your Redis eviction policy is allkeys-lru, batch metadata can vanish mid-execution, breaking progress tracking and callbacks. Use volatile-lru or dedicated Redis instances for queue data.

Horizon's Batch Awareness

Horizon doesn't just monitor queues—it understands batches natively. The dashboard shows batch-level metrics: throughput (jobs/sec), failure rates, retry distributions, and per-batch progress bars. Horizon's horizon:monitor command can alert on batch-specific thresholds like "batch failure rate exceeds 5% over 5 minutes."

Architecture Overview: Batch Processing Pipeline

A production batch pipeline consists of five stages, each with distinct scaling characteristics:

1. Batch Assembly (Application Layer)

Your application code collects work units—database records, API payloads, file paths—and converts them into job instances. This stage runs in web workers or scheduled commands. Memory usage scales with batch size. For 100,000+ jobs, use generators or chunked database cursors to avoid PHP memory exhaustion.

2. Atomic Dispatch (Redis Transaction)

Bus::batch() serializes all jobs and pushes them to Redis using a Lua script for atomicity. The script creates:
- A Redis list for the batch's job queue
- A hash for batch metadata
- Sorted sets for delayed jobs
This single round-trip replaces N individual RPUSH calls.

3. Worker Consumption (Horizon Supervised)

Horizon starts php artisan queue:work processes per your configuration. Each worker:
- Reserves jobs via LPOP (or BRPOPLPUSH for reliability)
- Executes job handle()
- Updates batch progress in Redis
- Triggers callbacks when batch completes

4. Progress Aggregation (Redis + Horizon)

Each job completion increments a Redis counter. Horizon reads this counter for dashboard updates. The aggregation is eventual—expect sub-second delays under load.

5. Callback Execution (Final Worker)

The worker that processes the batch's final job executes then(), catch(), or finally(). These run synchronously in the worker process. Long-running callbacks block that worker from consuming new jobs. Offload heavy callback work to a new batch or job.

Step-by-Step Guide: Building Production-Ready Batches

Step 1: Design Idempotent, Deterministic Jobs

Batch jobs must be safely retryable. Design each job to:
- Accept all required data in constructor (no external lookups)
- Use database transactions for multi-step operations
- Check for existing work before executing (idempotency keys)
- Avoid side effects until the final step

class ProcessOrderItem implements ShouldQueue{    use Batchable, Dispatchable, InteractsWithQueue, Queueable, SerializesModels;    public function __construct(        private readonly string $orderId,        private readonly int $itemId,        private readonly string $idempotencyKey    ) {}    public function handle(OrderService $service): void    {        // Idempotency check - Redis SETNX with 24h TTL        $lock = Redis::setnx("idempotency:{$this->idempotencyKey}", 1);        Redis::expire("idempotency:{$this->idempotencyKey}", 86400);                if (! $lock) {            return; // Already processed        }        DB::transaction(function () use ($service) {            $service->processItem($this->orderId, $this->itemId);        });    }}

Step 2: Configure Batch Size and Chunking

Don't dispatch 500,000 jobs in one batch. Redis memory, Horizon dashboard rendering, and callback timeouts all degrade. Optimal batch sizes:

• Small batches (100-1,000 jobs): Near real-time, low latency
• Medium batches (1,000-10,000): Balanced throughput and observability
• Large batches (10,000-100,000): Maximum throughput, use chunked dispatch

public function handle(): void{    $chunkSize = 5000;    $batches = [];    Order::where('status', 'pending')        ->chunkById($chunkSize, function ($orders) use (&$batches) {            $jobs = $orders->map(fn ($order) => new ProcessOrder($order->id));            $batches[] = Bus::batch($jobs)                ->name("process-orders-{$orders->first()->id}")                ->onQueue('orders')                ->allowFailures()                ->dispatch();        });    // Wait for all batches if needed    foreach ($batches as $batch) {        $batch->await(); // Blocking - use carefully    }}

Step 3: Implement Batch-Level Monitoring

Horizon provides metrics, but your application needs domain-specific alerts. Create a batch observer:

class BatchMonitor{    public function handle(Batch $batch): void    {        $metrics = [            'batch_id' => $batch->id,            'name' => $batch->name,            'total_jobs' => $batch->totalJobs,            'failed_jobs' => $batch->failedJobs,            'progress' => $batch->progress(),            'duration_seconds' => $batch->finishedAt?->diffInSeconds($batch->createdAt),        ];        // Send to your observability stack        DataDog::gauge('laravel.batch.progress', $metrics['progress'], $metrics);                if ($batch->failedJobs > $batch->totalJobs * 0.1) {            Alert::critical("Batch {$batch->name} failure rate: {$metrics['failed_jobs']}/{$metrics['total_jobs']}");        }    }}// Register in AppServiceProviderBatch::finished(function (Batch $batch) {    app(BatchMonitor::class)->handle($batch);});

Step 4: Handle Partial Failures Gracefully

With allowFailures(), the batch completes even if some jobs fail. Your finally() callback must inspect failures and trigger remediation:

->finally(function (Batch $batch) {    if ($batch->failedJobs > 0) {        $failedJobIds = $batch->failedJobs()->pluck('id');                // Dispatch remediation batch        Bus::batch(            $failedJobIds->map(fn ($id) => new RetryFailedJob($id))        )->name("retry-{$batch->name}")->dispatch();                // Alert on-call        Slack::alert("Batch {$batch->name} had {$batch->failedJobs} failures. Retry batch dispatched.");    }})

Step 5: Implement Rate Limiting at Batch Level

Individual job rate limiting (RateLimiter::for()) doesn't coordinate across batch siblings. Implement batch-level throttling using Redis atomic counters:

class BatchRateLimiter{    public function __construct(private Redis $redis) {}    public function acquire(string $batchId, int $maxConcurrent = 100): bool    {        $key = "batch:rate-limit:{$batchId}";        $current = $this->redis->incr($key);                if ($current === 1) {            $this->redis->expire($key, 3600); // 1 hour TTL        }        return $current <= $maxConcurrent;    }    public function release(string $batchId): void    {        $this->redis->decr("batch:rate-limit:{$batchId}");    }}// In your jobpublic function handle(BatchRateLimiter $limiter): void{    $batch = $this->batch();        if (! $limiter->acquire($batch->id, 50)) {        $this->release(30); // Re-queue after 30 seconds        return;    }    try {        $this->doWork();    } finally {        $limiter->release($batch->id);    }}

Real-World Examples: E-commerce Order Processing

Scenario: Black Friday Flash Sale

An e-commerce platform processes 50,000 orders in 10 minutes. Each order requires: inventory reservation, payment capture, confirmation email, warehouse notification, and loyalty points calculation.

Naive Approach (Anti-Pattern)

// DON'T DO THIS - 50,000 x 5 = 250,000 individual job dispatchesforeach ($orders as $order) {    ProcessPayment::dispatch($order);    ReserveInventory::dispatch($order);    SendConfirmation::dispatch($order);    NotifyWarehouse::dispatch($order);    CalculateLoyalty::dispatch($order);}

Batch-Optimized Approach

class ProcessFlashSaleOrders{    public function handle(): void    {        $batches = [];        $chunkSize = 2000;        Order::where('sale_id', $this->saleId)            ->where('status', 'pending')            ->chunkById($chunkSize, function ($orders) use (&$batches) {                $jobs = $orders->flatMap(function ($order) {                    return [                        new ReserveInventory($order->id),                        new CapturePayment($order->id),                        new SendConfirmationEmail($order->id),                        new NotifyWarehouse($order->id),                        new CalculateLoyaltyPoints($order->id),                    ];                });                $batches[] = Bus::batch($jobs)                    ->name("flash-sale-{$this->saleId}-chunk-{$orders->first()->id}")                    ->onQueue('flash-sale')                    ->allowFailures()                    ->then(fn (Batch $b) => $this->onChunkComplete($b))                    ->catch(fn (Batch $b, Throwable $e) => $this->onChunkFailed($b, $e))                    ->dispatch();            });        // Track all batch IDs for final reconciliation        Cache::put("flash-sale:{$this->saleId}:batches",             collect($batches)->pluck('id')->all(), 86400);    }    private function onChunkComplete(Batch $batch): void    {        $saleBatches = Cache::get("flash-sale:{$this->saleId}:batches", []);        $remaining = array_diff($saleBatches, [$batch->id]);                if (empty($remaining)) {            // All chunks done - final reconciliation            ReconcileFlashSale::dispatch($this->saleId);        }    }}

Results Comparison

Production Code Examples

Advanced Batch Job with Checkpointing

For long-running jobs (video processing, ML inference, large file imports), implement checkpointing to survive worker restarts:

class ProcessVideoBatch implements ShouldQueue{    use Batchable, Dispatchable, InteractsWithQueue, Queueable, SerializesModels;    public function __construct(        private readonly string $videoId,        private readonly array $segments,        private readonly int $checkpointInterval = 5    ) {}    public function handle(VideoProcessor $processor, Redis $redis): void    {        $progressKey = "video-progress:{$this->videoId}";        $completed = json_decode($redis->get($progressKey) ?? '[]', true);        foreach ($this->segments as $index => $segment) {            if (in_array($segment['id'], $completed)) {                continue; // Skip already processed            }            $processor->processSegment($this->videoId, $segment);            $completed[] = $segment['id'];            // Checkpoint every N segments            if (count($completed) % $this->checkpointInterval === 0) {                $redis->set($progressKey, json_encode($completed), 'EX', 86400);                $this->updateBatchProgress();            }        }        // Final checkpoint        $redis->set($progressKey, json_encode($completed), 'EX', 86400);        $this->updateBatchProgress();    }    private function updateBatchProgress(): void    {        if ($this->batch()) {            // Custom progress calculation based on segments            $progress = (count($completed) / count($this->segments)) * 100;            // Batch progress is automatic, but we can store custom metadata            Redis::hset("batch:{$this->batch()->id}:metadata", 'video_progress', $progress);        }    }}

Batch Chaining for Multi-Stage Workflows

Chain batches sequentially where each stage depends on the previous stage's completion:

class DataPipeline{    public function handle(): void    {        $stage1 = Bus::batch(            $this->buildExtractionJobs()        )->name('pipeline-extract')->onQueue('pipeline')->dispatch();        $stage2 = Bus::batch(            $this->buildTransformationJobs()        )->name('pipeline-transform')->onQueue('pipeline')->dispatch();        $stage3 = Bus::batch(            $this->buildLoadingJobs()        )->name('pipeline-load')->onQueue('pipeline')->dispatch();        // Chain: stage2 starts when stage1 completes        $stage1->then(function (Batch $batch) use ($stage2) {            if ($batch->failedJobs === 0) {                $stage2->dispatch();            }        });        $stage2->then(function (Batch $batch) use ($stage3) {            if ($batch->failedJobs === 0) {                $stage3->dispatch();            }        });        $stage3->finally(function (Batch $batch) {            PipelineComplete::dispatch($batch->id);        });        // Start the chain        $stage1->dispatch();    }}

Dynamic Batch Scaling with Horizon

Adjust Horizon worker counts based on batch queue depth:

// In a scheduled command running every minuteclass ScaleHorizonWorkers{    public function handle(): void    {        $queue = 'batch-processing';        $pending = Redis::llen("queues:{$queue}");                $targetWorkers = match (true) {            $pending > 50000 => 50,            $pending > 10000 => 20,            $pending > 1000 => 10,            default => 3,        };        $current = Horizon::workers($queue)->count();                if ($targetWorkers !== $current) {            Horizon::scale($queue, $targetWorkers);            Log::info("Scaled {$queue} from {$current} to {$targetWorkers} workers");        }    }}// Horizon config (config/horizon.php)'environments' => [    'production' => [        'supervisor-batch-processing' => [            'connection' => 'redis',            'queue' => ['batch-processing'],            'balance' => 'auto',            'maxProcesses' => 50,            'maxJobs' => 500,            'memory' => 256,            'timeout' => 300,            'tries' => 3,        ],    ],],

Batch Retry with Exponential Backoff

Override default retry behavior for batch-specific patterns:

class BatchRetryStrategy{    public function calculateDelay(int $attempts, Batch $batch): int    {        // Base delay: 60 seconds        $baseDelay = 60;                // Exponential backoff with jitter        $delay = $baseDelay * pow(2, $attempts - 1);        $jitter = random_int(0, $delay * 0.1);                // Cap at 1 hour        return min($delay + $jitter, 3600);    }    public function shouldRetry(Throwable $exception, Batch $batch): bool    {        // Don't retry validation errors        if ($exception instanceof ValidationException) {            return false;        }        // Don't retry if batch failure rate exceeds threshold        if ($batch->failedJobs / max($batch->totalJobs, 1) > 0.5) {            return false;        }        return $batch->attempts() < 5;    }}// Apply in jobpublic function handle(): void{    $strategy = app(BatchRetryStrategy::class);        if (! $strategy->shouldRetry($this->exception, $this->batch())) {        $this->fail($this->exception);        return;    }    $delay = $strategy->calculateDelay($this->attempts(), $this->batch());    $this->release($delay);}

Comparison: Batch vs Individual Job Patterns

Best Practices for Batch Processing at Scale

1. Name Every Batch

Always provide a descriptive name via ->name('descriptive-name'). Horizon displays this name in dashboards, alerts, and logs. Include contextual identifiers: "process-orders-{$saleId}-chunk-{$chunkIndex}".

2. Use Dedicated Queues per Batch Type

Separate queues isolate noisy neighbors. A massive video processing batch shouldn't block critical notification jobs. Configure Horizon supervisors per queue with independent scaling.

3. Set Explicit Timeouts

Default timeout is 60 seconds. Batch jobs often need more. Set per-job via $timeout property or per-batch in Horizon config. Remember: timeout kills the process; retry_after in queue config controls Redis-level job requeue.

4. Implement Graceful Shutdown

Horizon sends SIGTERM before SIGKILL on deployments. Your jobs must handle SIGTERM to checkpoint progress:

class GracefulBatchJob implements ShouldQueue{    use Batchable, Dispatchable, InteractsWithQueue, Queueable, SerializesModels;    private bool $shutdown = false;    public function __construct() {        pcntl_signal(SIGTERM, [$this, 'handleShutdown']);    }    public function handleShutdown(): void    {        $this->shutdown = true;    }    public function handle(): void    {        foreach ($this->workUnits as $unit) {            if ($this->shutdown) {                $this->checkpoint();                $this->release(5); // Quick requeue                return;            }            $this->processUnit($unit);        }    }}

5. Monitor Queue Lag, Not Just Throughput

Throughput (jobs/sec) looks healthy even when a massive batch backs up. Monitor queue lag—the age of the oldest unprocessed job. Horizon's wait_time metric approximates this. Alert when lag exceeds your SLA (e.g., 5 minutes for user-facing batches).

6. Use Database Transactions for Batch Assembly

When creating batches from database records, wrap the query and dispatch in a transaction to prevent duplicate processing on retry:

DB::transaction(function () {    $orders = Order::where('status', 'pending')->lockForUpdate()->get();        if ($orders->isEmpty()) {        return;    }    $jobs = $orders->map(fn ($o) => new ProcessOrder($o->id));    $batch = Bus::batch($jobs)->dispatch();    // Mark as queued atomically    $orders->each->update(['status' => 'queued', 'batch_id' => $batch->id]);});

7. Implement Batch Deduplication

Prevent accidental double-dispatch from retries or race conditions:

class DeduplicatedBatch{    public function dispatchIfNotExists(string $dedupeKey, callable $batchBuilder): ?Batch    {        $lock = Redis::setnx("batch-dedupe:{$dedupeKey}", 1);        Redis::expire("batch-dedupe:{$dedupeKey}", 3600);        if (! $lock) {            return null; // Batch already dispatched        }        try {            return $batchBuilder();        } catch (Throwable $e) {            Redis::del("batch-dedupe:{$dedupeKey}");            throw $e;        }    }}// Usage$batch = app(DeduplicatedBatch::class)->dispatchIfNotExists(    "daily-report-{$date->format('Y-m-d')}",    fn () => Bus::batch($jobs)->name("daily-report-{$date}")->dispatch());

Common Mistakes That Cause Production Incidents

1. Dispatching Batches in HTTP Requests

Large batch assembly + dispatch can exceed PHP's max_execution_time or nginx's proxy_read_timeout. Always dispatch from queued jobs or scheduled commands.

2. Ignoring allowFailures() Defaults

Default is false—one failure fails the entire batch. For 10,000-job batches, this means 9,999 successful jobs get marked failed. Almost always use allowFailures() for large batches.

3. Storing Large Payloads in Job Constructors

Passing entire Eloquent models or large arrays bloats Redis queue entries. Pass only IDs and fetch fresh data in handle(). Use SerializesModels trait properly—it only stores the model's primary key.

4. Forgetting maxJobs in Horizon Config

Without maxJobs, workers never restart, accumulating memory leaks. Set maxJobs: 500 or lower for batch workers. Monitor memory with horizon:terminate if needed.

5. Blocking on await() in Web Contexts

$batch->await() blocks the PHP process until completion. Never use in controllers. Use callbacks or polling endpoints instead.

6. Single Point of Failure in Callbacks

A then() callback that calls an external API can hang the worker indefinitely. Wrap callbacks in timeouts or offload to new jobs.

7. Not Testing Failure Scenarios

Test: Redis connection loss mid-batch, worker OOM kill, Horizon restart during batch, partial job failures, callback exceptions. Chaos engineering your batch pipeline prevents 3 AM pages.

Performance Tips: Squeezing Every Millisecond

Optimize Redis Connection Pooling

Use persistent connections (pconnect) and pipeline batch dispatches. Laravel's Redis client uses phpredis by default—ensure persistent => true in config/database.php.

Reduce Job Serialization Overhead

Default serialization uses serialize(). For high-throughput, implement __serialize() and __unserialize() (PHP 7.4+) or use json_encode with custom hydration:

class OptimizedJob implements ShouldQueue{    use Batchable, Dispatchable, InteractsWithQueue, Queueable;    public function __construct(        private readonly int $orderId,        private readonly string $action,        private readonly array $metadata = []    ) {}    public function __serialize(): array    {        return ['orderId' => $this->orderId, 'action' => $this->action, 'metadata' => $this->metadata];    }    public function __unserialize(array $data): void    {        $this->orderId = $data['orderId'];        $this->action = $data['action'];        $this->metadata = $data['metadata'];    }}

Batch Database Writes in Jobs

If jobs write to the same table, batch those writes using DB::transaction with multiple inserts or insertOrIgnore:

public function handle(): void{    $records = $this->collectRecords(); // From this job's work        // Single multi-row insert instead of N queries    DB::table('order_events')->insert($records);}

Use Redis Pipeline for Batch Metadata Updates

When updating multiple batch metadata fields, pipeline commands:

$redis->pipeline(function ($pipe) use ($batchId, $data) {    $pipe->hset("batch:{$batchId}:metadata", $data);    $pipe->zincrby("batch:metrics:throughput", 1, now()->format('Y-m-d H:i'));    $pipe->expire("batch:{$batchId}:metadata", 86400);});

Horizon Worker Configuration for Batches

// config/horizon.php'batch-workers' => [    'connection' => 'redis',    'queue' => ['batches-high', 'batches-normal', 'batches-low'],    'balance' => 'simple', // Simple often better for batch homogeneity    'maxProcesses' => 30,    'maxJobs' => 200, // Restart frequently to clear memory    'memory' => 512, // MB - batches can be memory intensive    'timeout' => 600, // 10 minutes for long batch jobs    'tries' => 3,    'nice' => 5, // Lower CPU priority],

Security Considerations for Batch Workloads

Job Payload Encryption

Batch jobs may contain PII or sensitive data. Encrypt job payloads at rest in Redis:

class EncryptedBatchJob implements ShouldQueue{    use Batchable, Dispatchable, InteractsWithQueue, Queueable, SerializesModels;    private string $encryptedPayload;    public function __construct(array $payload)    {        $this->encryptedPayload = Crypt::encryptString(json_encode($payload));    }    public function handle(): void    {        $payload = json_decode(Crypt::decryptString($this->encryptedPayload), true);        // Process $payload    }}

Rate Limiting External API Calls

Batches can unleash thundering herds on external APIs. Implement distributed rate limiting with Redis:

class ExternalApiRateLimiter{    public function acquire(string $apiKey, int $limit = 100, int $window = 60): bool    {        $key = "ratelimit:api:{$apiKey}";        $current = $this->redis->eval(            "local c = redis.call('incr', KEYS[1])"            . "if c == 1 then redis.call('expire', KEYS[1], ARGV[1]) end"            . "return c",            [$key],            [$window]        );        return $current <= $limit;    }}

Audit Batch Operations

Log batch dispatch, completion, and failures with structured data for compliance:

Batch::created(function (Batch $batch) {    AuditLog::create([        'event' => 'batch.dispatched',        'batch_id' => $batch->id,        'name' => $batch->name,        'job_count' => $batch->totalJobs,        'user_id' => auth()->id(),        'ip' => request()->ip(),    ]);});

Deployment Notes: Horizon in Production

Zero-Downtime Deployments

Horizon supports graceful reload via php artisan horizon:terminate. On deploy:

1. Send SIGTERM to master process
2. Master stops accepting new work
3. Workers finish current jobs (respecting timeout)
4. Master exits
5. Systemd/supervisor restarts Horizon with new code

Configure your deployment script:

#!/bin/bash# deploy.shphp artisan horizon:terminatesleep 10 # Wait for graceful shutdownphp artisan horizon &# Or let supervisor manage it

Redis High Availability

Use Redis Sentinel or Redis Cluster for production. Configure Laravel's Redis client for sentinel:

// config/database.php'redis' => [    'client' => 'phpredis',    'clusters' => [        'default' => [            ['host' => 'redis-1', 'port' => 6379],            ['host' => 'redis-2', 'port' => 6379],            ['host' => 'redis-3', 'port' => 6379],        ],    ],    'options' => [        'cluster' => 'redis',        'prefix' => 'laravel_queue_',        'persistent' => true,    ],],

Horizon Metrics Persistence

Horizon stores metrics in Redis with 24h TTL by default. For long-term retention, export to Prometheus/DataDog via Horizon's horizon:metrics command or custom listeners.

Container Considerations

In Kubernetes, run Horizon as a Deployment with:
- terminationGracePeriodSeconds: 60
- PreStop hook calling horizon:terminate
- HPA based on custom metrics (queue depth)- Separate Deployment per queue type for independent scaling

Debugging Tips: When Batches Go Wrong

Inspect Batch State in Redis

# Get batch metadataredis-cli HGETALL "laravel_database_batches:batch-uuid-here"# List all batch keysredis-cli KEYS "laravel_database_batches:*"# Check job queue for batchredis-cli LRANGE "queues:batch-processing:batch-uuid" 0 -1

Horizon CLI Debugging

# Show all batches with statusphp artisan horizon:batches# Show specific batch detailsphp artisan horizon:batch batch-uuid-here# Monitor real-timephp artisan horizon:monitor

Common Failure Patterns and Fixes

Replay Failed Batches

Create an artisan command to replay failed batches with fixes applied:

class ReplayFailedBatch extends Command{    protected $signature = 'batch:replay {batchId} {--fix=}';

FAQ

What's the maximum batch size Laravel supports?

There's no hard limit, but practical limits emerge around 100,000 jobs due to Redis memory for metadata, Horizon dashboard rendering performance, and callback timeout constraints. For larger workloads, use chunked batches of 5,000-10,000 jobs each.

Can I add jobs to an existing batch after dispatch?

No. Batches are immutable once dispatched. For dynamic workloads, use a parent batch that dispatches child batches, or implement a "batch coordinator" job that manages work distribution.

How do I handle batch priority? Urgent batches should skip the queue.

Dispatch urgent batches to a dedicated high-priority queue (e.g., batches-urgent) with a separate Horizon supervisor configured with more workers and higher nice value (lower priority number). Use ->onQueue('batches-urgent') when dispatching.

What happens if Redis goes down mid-batch?

Workers will fail to reserve jobs and eventually exit. Horizon will restart them. When Redis recovers, batches resume from the last acknowledged job. Jobs with retry_after expired will be re-queued. Ensure retry_after exceeds your longest job runtime.

Can batches work with database queue driver?

Technically yes, but batch features rely heavily on Redis atomic operations (Lua scripts, sorted sets, hashes). The database driver lacks these primitives. Use Redis for any serious batch workload.

How do I test batch processing locally?

Use Laravel's Queue::fake() with Bus::fake() for unit tests. For integration tests, run real Redis and Horizon in Docker. Use php artisan queue:work --once for single-job debugging.

What's the difference between Bus::batch() and Bus::chain()?

Bus::batch() runs jobs in parallel (subject to worker count). Bus::chain() runs jobs sequentially—each job starts only after the previous succeeds. Combine them: chain of batches for sequential stages, each stage parallelized.

How do I migrate from individual jobs to batches without downtime?

Deploy batch-capable job classes alongside existing ones. Gradually shift traffic using feature flags. Monitor error rates and queue lag. Once stable, remove old job dispatch code. Batches and individual jobs can coexist on the same queues.

Conclusion

Batch processing with Laravel, Redis, and Horizon isn't just a performance optimization—it's an architectural shift that changes how you think about asynchronous work. The patterns covered here—atomic dispatch, collective failure handling, progress aggregation, checkpointing, rate limiting, and horizontal scaling—form the foundation for systems that process millions of jobs reliably.

Start by identifying your highest-volume homogeneous workloads: notification sends, report generation, data synchronization, webhook deliveries. Wrap them in batches. Measure the Redis round-trip reduction, the Horizon dashboard clarity, the deployment confidence. Then expand.

Your next step: audit your current queue usage. Find the top three job types by volume. Design batch equivalents. Deploy to staging with Horizon monitoring enabled. Watch the metrics. The difference will be measurable and immediate.

Ready to go deeper? Explore our guides on Horizon Configuration for Production, Redis Queue Optimization, and Queue Worker Tuning. Share your batch processing wins and war stories in the comments—every production system teaches us something new.