Broker Won't Start: Common Kafka Startup Failures

Your Kafka broker crashed at 3 AM. You restart it. Nothing happens. The logs show a cryptic error, and your team is staring at a blank terminal.

I've debugged this scenario hundreds of times. The same handful of issues cause most startup failures.

We used to panic when brokers wouldn't start. Now we have a checklist. Most issues resolve in under five minutes.

SRE at a streaming data company

Cluster ID Mismatch

kafka.common.InconsistentClusterIdException: The Cluster ID XYZ123 doesn't match
stored clusterId Some(ABC789) in meta.properties.

ZooKeeper data was deleted while Kafka's log.dirs persisted, or ZooKeeper's dataDir is /tmp (cleared on reboot).

Fix (dev/single-node): rm /var/kafka-logs/meta.properties — the broker recreates it on startup.

⚠️ Production clusters: Never delete meta.properties on multiple brokers simultaneously. This causes cluster ID inconsistency and potential data loss. In multi-broker setups: stop the affected broker, delete meta.properties on that ONE broker only, restart, verify ISR status is healthy, then proceed to the next broker if needed.

Port Already in Use

java.net.BindException: Address already in use
kafka.common.KafkaException: Socket server failed to bind to 0.0.0.0:9092

Another process occupies the port—usually a previous Kafka instance.

lsof -i :9092
kill -9 <PID>

Disk Full

java.io.IOException: No space left on device
ERROR Shutdown broker because all log dirs have failed

Quick fix: Delete old segments from a topic that can tolerate data loss.

Proper fix: Configure retention: kafka-configs.sh --alter --add-config retention.bytes=10737418240

Permission Denied

kafka.common.KafkaException: Could not create directory /var/kafka-logs

chown -R kafka:kafka /var/kafka-logs
chmod 750 /var/kafka-logs

Advertised Listeners Wrong

Broker starts, but clients can't connect:

WARN Connection to node 0 could not be established. Broker may not be available.

advertised.listeners tells clients where to connect. Never use 0.0.0.0:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://your-actual-hostname:9092

Out of Memory

java.lang.OutOfMemoryError: Java heap space

JVM heap too small for loading metadata—especially with thousands of partitions.

Partition Count	Recommended Heap
< 1,000	4 GB
1,000 - 10,000	6 GB
> 10,000	8+ GB

export KAFKA_HEAP_OPTS="-Xms4g -Xmx4g"

ZooKeeper Session Expired

FATAL Could not establish session with zookeeper
org.apache.zookeeper.KeeperException$SessionExpiredException

Old session hasn't expired yet (18-second default). Wait 20 seconds or delete the ephemeral node:

zkCli.sh -server localhost:2181 delete /brokers/ids/0

KRaft Quorum Issues

java.lang.IllegalArgumentException: If process.roles contains broker,
the node ID must not appear in controller.quorum.voters

For broker-only nodes, node.id must NOT appear in controller.quorum.voters.

Quick Diagnostic Checklist

ps aux | grep kafka.Kafka          # Another instance running?
df -h /var/kafka-logs              # Disk full?
ls -la /var/kafka-logs             # Permissions?
lsof -i :9092                      # Port in use?
tail -100 /var/log/kafka/server.log | grep -E "ERROR|FATAL"

Most startup failures fall into these categories. Run through the checklist and you'll find the culprit. For ongoing broker health monitoring, see Conduktor's broker management documentation.

Book a demo to see how Conduktor Console provides real-time broker health monitoring and guided troubleshooting.