KRaft Explained: Kafka Without ZooKeeper

ZooKeeper is gone. Kafka 4.0, released March 2025, removed it entirely. KRaft is now the only way to run Kafka.

I've migrated clusters with hundreds of thousands of partitions from ZooKeeper to KRaft. The operational improvements are substantial—faster recovery, simpler configuration, and no more split-brain scenarios between two distributed systems.

We reduced our controller failover time from minutes to seconds. The migration was worth the planning.

Platform Engineer at a fintech company

What KRaft Actually Does

KRaft is Kafka's implementation of the Raft consensus protocol. It handles cluster membership, topic metadata, ACLs, and controller election—everything ZooKeeper did, but inside Kafka itself.

Metadata lives in an internal topic called __cluster_metadata. Every change (topic creation, partition reassignment) is appended as a record. Brokers track their state using offsets. When reconnecting, they catch up by reading from their last known offset instead of loading everything from scratch.

Controller Quorum

A subset of nodes form the controller quorum. One is the active controller, handling all metadata writes. The others are hot standbys with fully replicated state.

Controllers	Tolerable Failures
3	1
5	2

Use 3 controllers for most production clusters. Don't use even numbers—4 controllers tolerates the same failures as 3 with more overhead.

Separated mode (recommended): Controllers and brokers run on different nodes. Best for large clusters.

Combined mode: Single nodes run both roles. Supported for small production clusters since Kafka 3.5, but separated mode is preferred for larger deployments.

Configuration

Generate a cluster ID once:

bin/kafka-storage.sh random-uuid
# dPqzXuANQ5y9rG1mJvBm8A

Controller config:

node.id=0
process.roles=controller
listeners=CONTROLLER://controller-0:9093
controller.quorum.bootstrap.servers=controller-0:9093,controller-1:9093,controller-2:9093

Broker config:

node.id=10
process.roles=broker
listeners=PLAINTEXT://broker-0:9092
controller.quorum.bootstrap.servers=controller-0:9093,controller-1:9093,controller-2:9093

Format storage on each node before first start:

bin/kafka-storage.sh format \
  --cluster-id dPqzXuANQ5y9rG1mJvBm8A \
  --config config/server.properties

Start controllers first, then brokers.

Why KRaft Matters

Partition scalability: ZooKeeper bottlenecked around 200,000 partitions. KRaft handles millions.

Faster recovery: ZooKeeper-based controllers loaded all metadata on startup. With millions of partitions, this took minutes. KRaft controllers already have state in memory. Failover is nearly instant.

Simpler operations: One system instead of two. No ZooKeeper JVM tuning, no separate disk I/O monitoring, unified security.

Metric	ZooKeeper	KRaft
Max partitions	~200,000	2,000,000+
Controller failover	Minutes	Seconds
Broker shutdown	Minutes	Seconds

Monitoring

A broker monitoring dashboard provides visibility into controller state, metadata lag, and quorum health without parsing JMX manually.

bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status

Key JMX metrics:

• kafka.controller:type=KafkaController,name=ActiveControllerCount — Should be 1
• kafka.server:type=BrokerMetadataMetrics,name=CurrentMetadataOffset — Metadata position
• kafka.controller:type=KafkaController,name=OfflinePartitionsCount — Should be 0

Migration from ZooKeeper

Migration is required before upgrading to Kafka 4.0. The process runs online:

1. Upgrade to Kafka 3.9.x first
2. Provision KRaft controllers alongside existing cluster
3. Enable migration mode on brokers
4. Rolling-restart brokers
5. Monitor migration until complete
6. Remove ZooKeeper configuration

For critical clusters, consider a blue-green deployment: build a new KRaft cluster and migrate data using MirrorMaker 2.

Common Issues

"The quorum has not been initialized": Storage wasn't formatted. Run kafka-storage.sh format with the correct cluster ID.

Controllers not forming quorum: Check network connectivity between controller nodes on port 9093.

Broker can't connect to controllers: Ensure controller.quorum.bootstrap.servers matches across all nodes and security protocols align.

KRaft simplifies Kafka operations while dramatically improving scalability. The migration is worth it.

Book a demo to see how Conduktor Console provides unified visibility across KRaft and ZooKeeper clusters.