Kafka Exactly-Once: When It Works and When It Doesn't

"We enabled exactly-once but we're still seeing duplicates in the database."

I hear this constantly. Teams enable Kafka's EOS settings, assume the problem is solved, then discover duplicates in their downstream systems. The misconfiguration isn't technical—it's conceptual. They don't understand where exactly-once stops.

Kafka's exactly-once guarantees atomicity for reads and writes within the cluster. Everything beyond that boundary is your responsibility.

Once we understood that EOS is a Kafka-internal feature, not an end-to-end guarantee, our design changed completely. We stopped fighting the architecture.

Staff Engineer at a payments company

What Idempotent Producers Actually Solve

An idempotent producer ensures retrying a failed send never creates duplicates in the topic. Each producer gets a unique ID. Every batch includes a sequence number. The broker deduplicates based on this pair.

props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");

Since Kafka 3.0, this defaults to true when acks=all (also now the default). Retries are safe—no broker-level duplicates.

The limitation: The Producer ID doesn't survive restarts. If your producer crashes and restarts, it gets a new ID. The broker can't detect duplicates from the old instance.

Producer Instance 1 (PID=100): Send message A → Written
                               [CRASH]
Producer Instance 2 (PID=101): Send message A → Written again

Idempotent producers protect against network-level duplicates. They don't protect against application-level retries after a restart.

Transactions: Atomic Writes Across Partitions

Transactions solve the restart problem using a persistent transactional.id:

props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-processor-1");
producer.initTransactions();

producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", key, "order-created"));
producer.send(new ProducerRecord<>("inventory", key, "stock-decremented"));
producer.commitTransaction();

On commit, all messages become atomically visible. On abort, consumers with read_committed never see them.

The transactional.id enables "zombie fencing"—if two producers use the same ID, the older one gets fenced out. This prevents split-brain scenarios.

The Consume-Transform-Produce Pattern

The canonical exactly-once use case: read from input, transform, write to output, commit offsets—all atomically.

consumerProps.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
producerProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "processor-1");

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    producer.beginTransaction();

    for (ConsumerRecord<String, String> record : records) {
        producer.send(new ProducerRecord<>("output", record.key(), transform(record.value())));
    }

    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
    producer.commitTransaction();
}

If commitTransaction() succeeds, output messages are visible and consumer offsets are committed. If it fails, everything rolls back. On restart, you reprocess from the last committed offset.

Kafka Streams makes this trivial:

props.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

Where Exactly-Once Does NOT Work

This is where most teams get burned.

External Systems

Kafka transactions can't include your database or REST API:

producer.beginTransaction();
producer.send(new ProducerRecord<>("orders", key, order));
restClient.post("https://payments.example.com/charge", order);  // NOT TRANSACTIONAL
producer.commitTransaction();

If the REST call succeeds but commitTransaction() fails, the payment went through but the Kafka message rolled back. You now have inconsistency.

Solution: Write to Kafka only. Propagate to external systems with idempotent consumers.

Cross-Cluster Replication

Transactions don't span clusters. MirrorMaker 2 provides at-least-once delivery.

Sink Connectors

Kafka → PostgreSQL via JDBC Sink is at-least-once. The database is outside Kafka's transaction boundary.

Mitigation: Use idempotent writes. The JDBC sink can upsert using primary keys instead of insert.

Side Effects

builder.stream("orders")
    .foreach((key, value) -> sendEmailNotification(value));  // NOT exactly-once

If the stream task fails after sending the email but before committing, the email sends again on restart.

Solution: Write to an output topic. Consume with a separate service that tracks sent emails.

Performance Tradeoffs

Mode	Throughput	Latency p99
At-least-once	100k msg/s	15ms
Idempotent	98k msg/s	16ms
Transactional	60k msg/s	45ms

Transaction commits add 10-50ms latency. For high-throughput applications, batch more messages per transaction.

When to Use What

Scenario	Recommendation
Logging, metrics	At-least-once
General pipelines	Idempotent producer
Kafka → Kafka processing	Transactions or Streams EOS
Kafka → Database	At-least-once + idempotent sink

The hardest part of exactly-once isn't enabling it—it's understanding where it stops. Kafka guarantees atomicity within the cluster. Everything beyond requires idempotent design on your side.

Book a demo to see how Conduktor Console shows transaction markers and consumer states across your clusters.