php 69 lines · 8 steps

Deduplicating a job queue with Redis

A dispatcher uses an atomic Redis SET NX to ensure each unique job is enqueued only once within a TTL window.

Explained by highlit
Advanced deduplication redis atomicity hashing queueing
1<?php
2 
3namespace App\Queue;
4 
5use Predis\Client as Redis;
6 
7final class DeduplicatingDispatcher
8{
9 private const DEDUP_TTL = 3600;
10 
11 public function __construct(
12 private readonly Redis $redis,
13 private readonly JobQueue $queue,
14 ) {
15 }
16 
17 public function dispatch(string $type, array $payload): ?string
18 {
19 $fingerprint = $this->fingerprint($type, $payload);
20 $dedupKey = "queue:dedup:{$fingerprint}";
21 
22 $acquired = $this->redis->set($dedupKey, '1', 'EX', self::DEDUP_TTL, 'NX');
23 
24 if ($acquired === null) {
25 return null;
26 }
27 
28 try {
29 $jobId = $this->queue->push([
30 'type' => $type,
31 'payload' => $payload,
32 'fingerprint' => $fingerprint,
33 ]);
34 } catch (\Throwable $e) {
35 $this->redis->del([$dedupKey]);
36 throw $e;
37 }
38 
39 return $jobId;
40 }
41 
42 public function release(string $fingerprint): void
43 {
44 $this->redis->del(["queue:dedup:{$fingerprint}"]);
45 }
46 
47 private function fingerprint(string $type, array $payload): string
48 {
49 $this->ksortRecursive($payload);
50 
51 $canonical = json_encode(
52 ['type' => $type, 'payload' => $payload],
53 JSON_THROW_ON_ERROR | JSON_UNESCAPED_SLASHES,
54 );
55 
56 return hash('sha256', $canonical);
57 }
58 
59 private function ksortRecursive(array &$data): void
60 {
61 foreach ($data as &$value) {
62 if (is_array($value)) {
63 $this->ksortRecursive($value);
64 }
65 }
66 
67 ksort($data);
68 }
69}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1An atomic SET with NX and EX turns Redis into a self-expiring lock that guards against duplicate work.
  2. 2Canonicalizing input before hashing ensures logically identical payloads produce the same fingerprint.
  3. 3Releasing the dedup key on failure keeps a transient error from permanently blocking a legitimate retry.

Related explainers

php 
<?php
 
class NameParser
{

Parsing a full name into components in PHP

string-parsing arrays normalization
Intermediate 8 steps
php 
<?php
 
namespace App\Services\Checkout;
 

Validating coupons with Laravel's Pipeline

pipeline chain of responsibility transactions
Intermediate 7 steps
php 
<?php
 
namespace App\Services;
 

How a password strength validator works in PHP

validation regular-expressions data-driven
Intermediate 8 steps
ruby 
class LogAggregator
  BUCKET_FORMAT = "%Y-%m-%dT%H:%M"
 
  def initialize(entries)

Bucketing log entries by the minute in Ruby

aggregation hashing enumerable
Intermediate 5 steps
php 
<?php
 
namespace App\Services;
 

Building a cached daily leaderboard in Laravel

caching aggregation eager-loading
Intermediate 9 steps
go 
package logging
 
import (
	"context"

Deduplicating log attributes in Go's slog

decorator-pattern structured-logging immutability
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Deduplicating a job queue with Redis — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code