java 44 lines · 7 steps

Kafka retry and dead-letter handling in Spring

A Spring config that retries failing Kafka records with backoff, then routes the incurable ones to a dead-letter topic.

Explained by highlit

Spring Intermediate kafka error-handling retry dead-letter-queue backoff

1@Configuration
2@EnableKafka
3public class KafkaErrorHandlingConfig {
4 
  @Bean
  public DeadLetterPublishingRecoverer deadLetterRecoverer(KafkaTemplate<Object, Object> template) {
      return new DeadLetterPublishingRecoverer(template,
              (record, exception) -> new TopicPartition(record.topic() + ".DLT", record.partition()));
  }
10 
  @Bean
  public DefaultErrorHandler errorHandler(DeadLetterPublishingRecoverer recoverer) {
      ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(4);
      backOff.setInitialInterval(500L);
      backOff.setMultiplier(2.0);
      backOff.setMaxInterval(10_000L);
17 
      DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);
      handler.addNotRetryableExceptions(
              DeserializationException.class,
              MethodArgumentResolutionException.class,
              IllegalArgumentException.class);
      handler.setRetryListeners((record, ex, attempt) ->
              log.warn("Retry {} for record from {}-{}@{}: {}",
                      attempt, record.topic(), record.partition(), record.offset(), ex.getMessage()));
      return handler;
  }
28 
  @Bean
  public ConcurrentKafkaListenerContainerFactory<String, OrderEvent> kafkaListenerContainerFactory(
          ConsumerFactory<String, OrderEvent> consumerFactory,
          DefaultErrorHandler errorHandler) {
33 
      ConcurrentKafkaListenerContainerFactory<String, OrderEvent> factory =
              new ConcurrentKafkaListenerContainerFactory<>();
      factory.setConsumerFactory(consumerFactory);
      factory.setCommonErrorHandler(errorHandler);
      factory.setConcurrency(3);
      factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.RECORD);
      return factory;
  }
42 
  private static final Logger log = LoggerFactory.getLogger(KafkaErrorHandlingConfig.class);
44}

01 / 01

STEP 01

Walkthrough

Space play ←→ step click any line

Three takeaways

1Separating retryable from non-retryable exceptions stops you from wasting attempts on errors that can never succeed.
2Exponential backoff spaces out retries so a struggling downstream isn't hammered while it recovers.
3A dead-letter recoverer gives permanently failing records a home instead of blocking the partition forever.

Related explainers

rust

use serde::Deserialize;
 
#[derive(Debug, Deserialize)]
#[serde(untagged)]

Parsing flexible JSON shapes with serde

deserialization enums json

Intermediate 6 steps

ruby

require "shellwords"
require "open3"
 
module Backup

Building safe shell commands in Ruby

shell-out subprocess command-injection

Intermediate 7 steps

java

@Component
@Converter
public class EncryptedStringConverter implements AttributeConverter<String, String> {

Transparent column encryption in Spring & JPA

encryption aes-gcm jpa-converter

Advanced 10 steps

java

package com.acme.billing.config;
 
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.ConfigurationProperties;

Feature-flagged beans with Spring @ConditionalOnProperty

feature-flags conditional-beans strategy-pattern

Intermediate 5 steps

rust

use chrono::{Duration, NaiveDate};
 
#[derive(Debug)]
pub struct DateRange {

Parsing and iterating date ranges in Rust

error-handling iterators parsing

Intermediate 7 steps

java

public static Map<String, String> parseCookieHeader(String header) {
    Map<String, String> cookies = new LinkedHashMap<>();
    if (header == null || header.isBlank()) {
        return cookies;