Laravel Queue Workers and Horizon for Background Job Processing

Introduction

Background job processing is a cornerstone of modern web applications. Offloading time‑consuming tasks such as sending emails, generating reports, or interacting with third‑party APIs keeps the HTTP response fast and the user experience smooth. Laravel provides a robust queue abstraction that supports multiple back‑ends — database, Redis, Amazon SQS, and more — while Horizon adds a beautiful dashboard and advanced worker management for Redis‑based queues. In this guide we will explore the full lifecycle of a Laravel queue job, from defining the job class to scaling workers with Horizon, and we will cover production‑grade patterns that you can apply today.

We assume you have a working Laravel 10+ installation and a Redis server available. If you are new to queues, the official Laravel Queues documentation offers a gentle introduction. By the end of this article you will be able to configure workers, monitor them via Horizon, handle failures gracefully, and deploy a resilient background processing pipeline.

Table of Contents

• Introduction
• Core Concepts
• Architecture Overview
• Step‑by‑Step Guide
• Real‑World Examples
• Production Code Examples
• Comparison Table
• Best Practices
• Common Mistakes
• Performance Tips
• Security Considerations
• Deployment Notes
• Debugging Tips
• FAQ
• Conclusion

Core Concepts

Before diving into configuration, it helps to understand the terminology that Laravel uses around queues.

Jobs

A job is a PHP class that implements the ShouldQueue interface. The class contains a handle method where the actual work lives. Jobs can be dispatched synchronously for testing or asynchronously via the queue driver.

Queues and Connections

A connection defines the backend (Redis, database, SQS, etc.) and its credentials. Within a connection you can have multiple named queues (e.g., default, emails, reports) to prioritize work.

Workers

A worker is a long‑running PHP process that pops jobs from a queue, executes them, and marks them as completed or failed. Laravel's queue:work command starts a worker. Workers can be daemonized, limited by memory, and supervised by process managers such as Supervisor or systemd.

Horizon

Horizon is a first‑party package that provides a real‑time dashboard, automatic worker scaling, job analytics, and a clean API for managing Redis queues. It wraps the queue:work command and adds features like balancing strategies, metrics, and a UI for retrying failed jobs.

Failed Jobs

When a job throws an exception or exceeds its retry limit, Laravel stores it in the failed_jobs table. Horizon surfaces these failures in its UI, allowing one‑click retries after the job.

Architecture Overview

At a high level, the request lifecycle looks like this:

1. An HTTP request hits your controller.
2. The controller dispatches a job (e.g., ProcessReport::dispatch($data)).
3. Laravel serializes the job payload and pushes it onto the configured Redis queue.
4. One or more Horizon‑managed workers pull the job, deserialize it, and execute handle().
5. On success the job is removed; on failure it is retried according to tries and backoff settings, eventually landing in the failed jobs table.
6. Horizon's dashboard displays throughput, runtime, memory usage, and failed job counts in real time.

Because workers are independent processes, you can scale horizontally by adding more worker instances on the same server or across multiple servers. Horizon's balance strategy (simple, auto, or custom) decides how many processes to allocate per queue based on load.

Step‑by‑Step Guide

1. Install Horizon

composer require laravel/horizonphp artisan horizon:installphp artisan migrate

The install command publishes the Horizon assets, configuration file (config/horizon.php), and creates the necessary database tables for failed jobs and Horizon's internal metadata.

2. Configure Redis Connection

Edit .env to point to your Redis instance:

REDIS_HOST=127.0.0.1REDIS_PASSWORD=nullREDIS_PORT=6379

Ensure config/database.php defines a redis connection that Horizon will use. The default configuration works for most single‑node setups.

3. Define Queue Names in config/horizon.php

Horizon's configuration maps environments to queue configurations. A typical production block looks like:

'environments' => [    'production' => [        'supervisor-1' => [            'connection' => 'redis',            'queue' => ['high', 'default', 'low'],            'balance' => 'auto',            'maxProcesses' => 10,            'maxTime' => 0,            'maxMemory' => 128,            'tries' => 3,            'nice' => 0,        ],    ],    'local' => [        'supervisor-1' => [            'connection' => 'redis',            'queue' => ['default'],            'balance' => 'simple',            'processes' => 3,            'tries' => 3,        ],    ],],

The balance key controls scaling: simple runs a fixed number of processes, while auto lets Horizon adjust based on queue latency.

4. Create a Job Class

php artisan make:job ProcessReport --queued

Open app/Jobs/ProcessReport.php and implement the handle method:

public function handle(): void{    $report = ReportGenerator::generate($this->data);    Storage::disk('s3')->put("reports/{$this->data['id']}.pdf", $report);    Notification::send($this->user, new ReportReady($report));}

Add public $tries = 3; and public $backoff = [60, 180, 300]; to customize retry behaviour.

5. Dispatch the Job

use App\Jobs\ProcessReport;ProcessReport::dispatch($data)->onQueue('reports');

The onQueue method targets a specific queue name defined in Horizon's configuration.

6. Start Horizon

php artisan horizon

For production you will run Horizon under a process manager (see Deployment Notes). The dashboard becomes available at /horizon (protected by the horizon gate in HorizonServiceProvider).

Real‑World Examples

Email Newsletter Dispatch

A marketing platform sends a weekly newsletter to 100,000 subscribers. The job SendNewsletter chunks the subscriber list into batches of 1,000 and dispatches a SendNewsletterBatch job for each chunk. Horizon's auto‑scaling spins up additional workers when the emails queue latency exceeds the threshold, then scales down after the burst.

Video Transcoding Pipeline

An upload service stores the original file in S3, then dispatches a TranscodeVideo job. The job uses FFmpeg via a PHP wrapper, writes multiple resolutions back to S3, and finally fires a VideoReady event. Because transcoding is CPU‑intensive, the queue is limited to two concurrent processes per server (maxProcesses => 2) to avoid starving the web workers.

Periodic Data Sync

A scheduled command runs every hour and dispatches a SyncExternalApi job for each integration. The job implements ShouldBeUnique to prevent duplicate runs if the scheduler fires twice. Horizon's dashboard shows the unique job lock status and runtime metrics.

Production Code Examples

Supervisor Configuration

[program:laravel-horizon]process_name=%(program_name)s_%(process_num)02dcommand=php /var/www/html/artisan horizonnumprocs=1autostart=trueautorestart=trueuser=www-dataredirect_stderr=truestdout_logfile=/var/www/html/storage/logs/horizon.logstopwaitsecs=3600

This runs a single Horizon master process; Horizon itself forks the configured number of workers.

Health Check Endpoint

// routes/web.phpRoute::get('/health/horizon', function () {    $status = Horizon::isRunning() ? 'ok' : 'down';    return response()->json(['status' => $status], $status === 'ok' ? 200 : 503);});

Load balancers can probe this endpoint to ensure Horizon is alive before routing traffic.

Custom Failed Job Notification

// App\Providers\HorizonServiceProvider.phpuse Illuminate\Support\Facades\Notification;use App\Notifications\JobFailedAlert;Horizon::failed(function ($job) {    Notification::route('mail', config('app.admin_email'))        ->notify(new JobFailedAlert($job));});

The JobFailedAlert notification can include the job payload, exception trace, and a direct link to the Horizon failed jobs page.

Queue Prioritization with Multiple Supervisors

'production' => [    'high-priority' => [        'connection' => 'redis',        'queue' => ['critical', 'high'],        'balance' => 'auto',        'maxProcesses' => 5,        'tries' => 5,    ],    'low-priority' => [        'connection' => 'redis',        'queue' => ['default', 'low'],        'balance' => 'simple',        'processes' => 3,        'tries' => 3,    ],],

Critical jobs get more processes and higher retry counts, while background maintenance jobs run on a smaller pool.

Comparison Table

Best Practices

• Keep jobs small and idempotent. Large jobs increase memory pressure and make retries expensive. Break work into granular steps.
• Use unique jobs for scheduled work. Implement ShouldBeUnique with a lock key that includes the schedule timestamp.
• Configure appropriate timeouts. Set retry_after in config/queue.php slightly higher than your longest expected job runtime to avoid premature re‑queuing.
• Monitor horizon metrics. Set up alerts on queue latency, failed job rate, and worker memory usage.
• Separate queues by priority. Critical user‑facing jobs (password resets, order confirmations) go to a high‑priority queue with dedicated workers.
• Graceful shutdown. Send SIGTERM to Horizon master; it will finish current jobs before exiting. Supervisor's stopwaitsecs should exceed your longest job.
• Version your job payloads. When job structure changes, bump a version property and handle both versions in handle().

Common Mistakes

• Running queue:work without a process manager. The process dies on server reboot or OOM kill, leaving jobs stuck.
• Ignoring retry_after. Too low a value causes duplicate execution; too high delays retries.
• Putting all jobs on the default queue. This defeats prioritization and makes scaling harder.
• Not protecting the Horizon dashboard. The default gate allows any authenticated user; restrict to admins via Horizon::auth.
• Using the database driver for high‑throughput workloads. It creates contention on the jobs table and lacks Horizon's auto‑scaling.
• Forgetting to set maxMemory. Workers can grow unbounded, causing OOM kills.
• Skipping failed job cleanup. The failed_jobs table grows indefinitely; schedule a prune job.

Performance Tips

• Use Redis pipelining. Horizon batches job payloads when pushing, reducing round‑trips.
• Enable queue:listen only for development. Daemon workers (queue:work) avoid bootstrapping the framework on every job.
• Tune maxProcesses per CPU core. A good rule of thumb is 1‑2 worker processes per core for I/O‑bound jobs, fewer for CPU‑heavy work.
• Leverage sleep in maxTime. Setting maxTime to a non‑zero value recycles workers periodically, preventing memory leaks.
• Cache heavy lookups. If a job repeatedly queries the same config, cache it with a short TTL.
• Use bus:chain for sequential dependent jobs. Chains avoid polling and guarantee order.

Security Considerations

• Restrict Horizon access. In HorizonServiceProvider set Horizon::auth(fn ($request) => $request->user()->isAdmin()).
• Sanitize job payloads. Never store secrets (API keys, passwords) directly in the job; retrieve them from a vault at runtime.
• Validate incoming job data. Use Laravel's validation in the job's constructor or handle method.
• Run workers with least privilege. The OS user should not have write access to application code directories.
• Enable TLS for Redis. If Redis is remote, use rediss:// scheme and configure certificates.
• Audit failed jobs. Failed job payloads may contain PII; ensure your retention policy scrubs or encrypts them.

Deployment Notes

Deploying Horizon to production involves a few moving parts:

• Process Manager. Use Supervisor (as shown) or systemd. Ensure the service starts on boot and restarts on failure.
• Environment Variables. Keep .env out of version control; inject via your CI/CD pipeline or server provisioning tool.
• Zero‑Downtime Deployments. Send SIGTERM to the Horizon master before pulling new code. Horizon will drain workers, then the new release starts a fresh master.
• Shared Storage. If workers write temporary files, mount a shared volume (e.g., NFS or EFS) across all worker hosts.
• Log Rotation. Configure logrotate for storage/logs/horizon.log to avoid disk exhaustion.
• Health Checks. Expose the /health/horizon endpoint to your load balancer or orchestration platform (Kubernetes liveness probe, AWS ALB health check).

Debugging Tips

• Run a single worker in foreground. php artisan queue:work redis --queue=high --verbose --tries=1 prints each job payload and execution time.
• Inspect Horizon metrics API. GET /horizon/api/metrics returns JSON with throughput, runtime, and memory per queue.
• Use Log::channel('horizon')->info(). Horizon registers a dedicated log channel; tail it for real‑time worker logs.
• Replay failed jobs locally. Pull the payload from failed_jobs, create a test job with the same data, and run it under Xdebug.
• Check Redis slowlog. redis-cli SLOWLOG GET 10 reveals queue commands that block the event loop.
• Profile memory with memory_get_peak_usage(). Add a temporary line in handle() to log peak memory per job.

FAQ

Can I use Horizon with a database queue driver?

No. Horizon is designed exclusively for Redis‑backed queues. If you need a dashboard for database queues, consider community packages or build a custom UI.

How does Horizon decide when to scale workers?

With the auto balance strategy, Horizon measures the average wait time of jobs in each queue. If latency exceeds the configured threshold, it spawns additional processes up to maxProcesses. When latency drops, it gracefully terminates idle workers.

What happens if a worker crashes mid‑job?

The job is released back onto the queue after the retry_after period. Horizon will eventually retry it according to the job's tries and backoff settings.

How do I prioritize certain jobs over others?

Define multiple queues (e.g., high, default, low) and list them in order in the supervisor's queue array. Workers poll queues sequentially, so jobs in high are always fetched first.

Can I run Horizon on multiple servers for the same application?

Yes. Each server runs its own Horizon master with the same Redis backend. Horizon coordinates via Redis, so workers across servers share the same queues and scaling decisions.

How do I prevent duplicate jobs when using scheduled commands?

Implement ShouldBeUnique on the job and define a unique ID that incorporates the schedule timestamp (e.g., sync-api-2025-09-27-01). Laravel will skip dispatch if a job with that ID is already pending or processing.

What is the recommended way to monitor Horizon in production?

Combine the built‑in dashboard with external monitoring: export Horizon metrics to Prometheus via the /horizon/api/metrics endpoint, set up Grafana dashboards, and configure alerts on failed job rate and queue latency.

Does Horizon support delayed jobs?

Yes. Use MyJob::dispatch($data)->delay(now()->addMinutes(10)). Horizon respects the available_at timestamp and will not make the job visible to workers until that time.

Conclusion

Laravel's queue system, supercharged by Horizon, gives you a production‑grade background processing platform with minimal boilerplate. By following the configuration steps, adopting the best practices, and leveraging Horizon's auto‑scaling and observability, you can handle everything from occasional email sends to high‑throughput video transcoding pipelines. Remember to protect the dashboard, tune your worker limits, and set up proper monitoring so your queue infrastructure remains invisible to users but rock‑solid for your application.

Ready to level up your Laravel background jobs? Start by installing Horizon today, configure a few priority queues, and watch the dashboard come alive. If you found this guide helpful, share it with your team and explore our related articles on queue worker tuning and Horizon security.