java 44 lines · 7 steps

Kafka retry and dead-letter handling in Spring

A Spring config that retries failing Kafka records with backoff, then routes the incurable ones to a dead-letter topic.

Explained by highlit
Spring Intermediate kafka error-handling retry dead-letter-queue backoff
1@Configuration
2@EnableKafka
3public class KafkaErrorHandlingConfig {
4 
5 @Bean
6 public DeadLetterPublishingRecoverer deadLetterRecoverer(KafkaTemplate<Object, Object> template) {
7 return new DeadLetterPublishingRecoverer(template,
8 (record, exception) -> new TopicPartition(record.topic() + ".DLT", record.partition()));
9 }
10 
11 @Bean
12 public DefaultErrorHandler errorHandler(DeadLetterPublishingRecoverer recoverer) {
13 ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(4);
14 backOff.setInitialInterval(500L);
15 backOff.setMultiplier(2.0);
16 backOff.setMaxInterval(10_000L);
17 
18 DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);
19 handler.addNotRetryableExceptions(
20 DeserializationException.class,
21 MethodArgumentResolutionException.class,
22 IllegalArgumentException.class);
23 handler.setRetryListeners((record, ex, attempt) ->
24 log.warn("Retry {} for record from {}-{}@{}: {}",
25 attempt, record.topic(), record.partition(), record.offset(), ex.getMessage()));
26 return handler;
27 }
28 
29 @Bean
30 public ConcurrentKafkaListenerContainerFactory<String, OrderEvent> kafkaListenerContainerFactory(
31 ConsumerFactory<String, OrderEvent> consumerFactory,
32 DefaultErrorHandler errorHandler) {
33 
34 ConcurrentKafkaListenerContainerFactory<String, OrderEvent> factory =
35 new ConcurrentKafkaListenerContainerFactory<>();
36 factory.setConsumerFactory(consumerFactory);
37 factory.setCommonErrorHandler(errorHandler);
38 factory.setConcurrency(3);
39 factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.RECORD);
40 return factory;
41 }
42 
43 private static final Logger log = LoggerFactory.getLogger(KafkaErrorHandlingConfig.class);
44}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Separating retryable from non-retryable exceptions stops you from wasting attempts on errors that can never succeed.
  2. 2Exponential backoff spaces out retries so a struggling downstream isn't hammered while it recovers.
  3. 3A dead-letter recoverer gives permanently failing records a home instead of blocking the partition forever.

Related explainers

rust 
use axum::{
    body::Body,
    http::{Method, StatusCode, Uri},
    response::{IntoResponse, Response},

Nesting routers and JSON fallbacks in Axum

routing http json-responses
Intermediate 7 steps
java 
@Controller
@RequestMapping("/employees")
public class EmployeeController {
 

Customizing form binding in a Spring MVC controller

data-binding validation type-conversion
Intermediate 9 steps
java 
package com.example.config.condition;
 
import org.springframework.context.annotation.Condition;
import org.springframework.context.annotation.ConditionContext;

A custom @Conditional feature flag in Spring

conditional beans feature flags annotations
Intermediate 7 steps
java 
public List<OrderSummary> streamRecentOrders(LocalDateTime since, Consumer<OrderSummary> handler) {
    String sql = """
        SELECT id, customer_id, total_cents, status, created_at
        FROM orders

Streaming large JDBC result sets safely

jdbc streaming resource-management
Intermediate 7 steps
go 
package batch
 
import "fmt"
 

Splitting a slice into batches in Go

generics slices error-handling
Intermediate 6 steps
php 
<?php
 
namespace App\Console\Commands;
 

How a database backup command works in Laravel

artisan-command shell-process error-handling
Intermediate 8 steps

Share this explainer

Here's the card — post it anywhere.

Kafka retry and dead-letter handling in Spring — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code