Stop Over-Partitioning: You Don't Need 100 Partitions

"Just add more partitions" is the Kafka equivalent of "just add more servers." It sounds reasonable until you hit metadata overhead, slow failovers, and file handle exhaustion.

I've seen teams default to 100 partitions per topic "for future scale" on clusters processing 10 MB/s. The result: 2 GB of pre-allocated index files for a topic that could run fine on 6 partitions.

We audited our cluster and found 80% of topics had 10x more partitions than needed. After right-sizing, controller failover dropped from 45 seconds to 8 seconds.

Platform Engineer at a fintech

The Hidden Costs

Each partition creates:

• Index file (up to 10 MB, grows with segment)
• Time index file (up to 10 MB, grows with segment)
• Open file handles (3-6 per partition)
• Metadata in the controller

100 partitions with full segments ≈ 2 GB in indexes alone.

File handles: 50 topics × 100 partitions = 5,000 partitions per broker = 15,000-30,000 file handles. Most Linux defaults to 1,024.

Controller failover:

Partitions	Failover Time
1,000	~2 seconds
10,000	~20 seconds
100,000	~3+ minutes

Broker failure: 10,000 partitions × 5ms/election = 50 seconds unavailability.

The Actual Formula

Partitions = max(Target Throughput / Producer Rate, Target Throughput / Consumer Rate)

Example:

• Target: 100 MB/s
• Producer rate: 50 MB/s
• Consumer rate: 25 MB/s

Partitions = max(100/50, 100/25) = max(2, 4) = 4 minimum

Add buffer for growth: 6-8 partitions. Not 100.

Test your actual throughput:

kafka-producer-perf-test.sh --topic test --num-records 1000000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=localhost:9092

Common Mistakes

"We might scale later" — You can increase partitions later. You cannot decrease them without recreating the topic. Start small.

"More partitions = more parallelism" — Only if you have matching consumers. 100 partitions with 3 consumers = each consumer handles 33 partitions sequentially. 6 partitions with 6 consumers often outperforms this.

"Our framework requires it" — Kafka Streams creates internal topics matching your input partition count. 100-partition input → potentially 500 partitions across internal topics.

Right-Sizing Guidelines

Throughput	Suggested Partitions
< 10 MB/s	3-6
10-50 MB/s	6-12
50-100 MB/s	12-24
100+ MB/s	Calculate from formula

Cluster limits:

Cluster Size	Max per Broker
Small (3 brokers)	2,000
Large (12+ brokers)	4,000

Auditing Existing Topics

kafka-topics.sh --bootstrap-server localhost:9092 --describe | grep "PartitionCount" | sort -t':' -k2 -n -r | head -20

For each high-partition topic, ask:

1. What's the actual throughput?
2. How many consumers process it?
3. Is key ordering required?

Topic monitoring surfaces these metrics without CLI commands.

If a 100-partition topic processes 5 MB/s with 3 consumers, you're paying for 97 unused partitions.

Migration

You can't reduce partitions in place:

kafka-topics.sh --create --topic events-v2 --partitions 6
# Mirror data, switch producers, drain old topic, switch consumers, delete old

The teams running Kafka most efficiently don't have the most partitions. They have the right number, monitored and adjusted based on actual load.

Book a demo to see how Conduktor Console identifies over-provisioned topics.