Kafka Log Retention and Segments: How It Works

Set a topic's retention to one hour, and on a quiet topic its oldest records can still be readable a week later.

That surprises almost everyone who checks: the retention time is how long Kafka keeps a record at the very least, and deletion can come days later.

Kafka deletes a whole file at a time, once its newest record is older than the retention time. A quiet topic writes to the same file for up to a week, so the first record in it lives that week plus the retention time.

A Partition Is a Directory

A partition is a directory on the broker's disk, named <topic>-<partition>, and its records live in segment files inside it. This is orders-0 on Kafka 4.3.1 after 4,000 records of 1 KB, with segment.bytes lowered to 1 MiB so that segments roll quickly:

$ ls -l /tmp/kraft-combined-logs/orders-0 | awk '{print $5, $9}'
504 00000000000000000000.index
1037120 00000000000000000000.log
132 00000000000000000000.timeindex
504 00000000000000001024.index
1037120 00000000000000001024.log
56 00000000000000001024.snapshot
108 00000000000000001024.timeindex
504 00000000000000002048.index
1037120 00000000000000002048.log
56 00000000000000002048.snapshot
60 00000000000000002048.timeindex
10485760 00000000000000003072.index
939890 00000000000000003072.log
56 00000000000000003072.snapshot
10485756 00000000000000003072.timeindex
8 leader-epoch-checkpoint
43 partition.metadata

Each segment, named after its base offset (the first offset it holds), is a set of files:

• .log. The record batches, in offset order.
• .index and .timeindex. Sparse maps from offset to byte position in the .log (8 bytes per entry) and from timestamp to offset (12 bytes per entry).
• .snapshot. Producer IDs and their last sequence numbers, written at each roll, so that a restarting broker rebuilds idempotent producer state without reading the whole log.
• leader-epoch-checkpoint and partition.metadata. The first offset of each leader epoch, which a replica uses to truncate its log after a leader change, and the topic ID.

Only the newest segment, the active segment, takes appends. Its index files are preallocated to segment.index.bytes (10 MiB) and trimmed to size when it rolls, hence 504 bytes on the closed ones:

On a KRaft node, the log directory also holds __cluster_metadata-0, the cluster metadata log. It follows its own settings: old segments go once a snapshot covers them and the log passes metadata.max.retention.bytes (100 MiB) or metadata.max.retention.ms (7 days).

Indexes Map Offsets to Bytes

To serve a fetch, the broker picks the segment with the largest base offset at or below the requested offset, finds the last .index entry at or below that offset, and reads the .log forward from the entry's byte position.

A lookup by timestamp reads .timeindex first to get the offset.

The index is sparse. The broker adds an entry once index.interval.bytes (4 KiB) have been appended, at most one per batch: the closed segments above hold 1,024 records and 63 index entries, one per producer batch of about 16 KB.

When Segments Roll

The active segment rolls, meaning it closes and a new one opens, in three cases:

• Size. The next batch would push it past segment.bytes (1 GiB).
• Age. The next batch's timestamp is more than segment.ms (7 days) past the segment's first.
• Full index. Not at the defaults: a 10 MiB offset index covers 5 GiB of log, and segment.bytes can't exceed 2 GiB.

The broker checks size and age when a batch arrives.

Since Kafka 4.0, segment.bytes can't go below 1 MiB either (KIP-1030):

$ kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders \
    --config segment.bytes=16384
Error while executing topic command : Invalid value 16384 for configuration segment.bytes: Value must be at least 1048576

How Retention Deletes Data

"Data is deleted one log segment at a time."

Apache Kafka documentation, Log

Every log.retention.check.interval.ms (5 minutes), the broker walks each partition from its oldest segment and deletes segments until it reaches one that doesn't qualify:

• By time. The segment's largest record timestamp is older than retention.ms. The newest record decides for the whole segment.
• By size. The partition would still hold at least retention.bytes without the segment.
• Either rule. With both set, a segment goes as soon as either rule applies to it.

The broker moves the partition's log start offset past the segment, so consumers can't read it anymore. It renames the segment's files to .deleted and removes them file.delete.delay.ms (60 seconds) later.

When every segment qualifies, the active one included, the broker rolls a new empty segment and deletes the rest.

On Kafka 4.3.1, the orders topic from the first section had a one-minute retention. Its 4,000 records stayed readable for four minutes after the last write, then all four segment files went in one retention check.

The broker log, trimmed to the roll and the first and last deletion:

18:53:51,141 [LocalLog partition=orders-0] Rolled new log segment at offset 4000 in 2 ms.
18:53:51,141 [UnifiedLog partition=orders-0] Deleting segment LogSegment(baseOffset=0, size=1037120, …) due to log retention time 60000ms breach based on the largest record timestamp in the segment
18:53:51,142 [UnifiedLog partition=orders-0] Deleting segment LogSegment(baseOffset=3072, …) …

The earliest and latest offsets are then both 4000: the partition is empty.

Default Retention Settings

kafka-configs.sh --describe --all prints a topic's effective values, defaults included:

$ kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics \
    --entity-name events --describe --all \
  | grep -E '^  (retention|segment\.(bytes|ms)|local|remote\.storage|file\.delete)' | awk '{print $1}'
file.delete.delay.ms=60000
local.retention.bytes=-2
local.retention.ms=-2
remote.storage.enable=false
retention.bytes=-1
retention.ms=604800000
segment.bytes=1073741824
segment.ms=604800000

A topic setting you don't override takes the broker default, which has a log. name, such as log.retention.hours or log.segment.bytes:

Setting	Default	Effect
retention.ms	604800000 (7 days)	A segment qualifies once its newest record is older
retention.bytes	-1 (none)	Per partition: oldest segments go while it stays above
segment.ms	604800000 (7 days)	Roll once a batch is this much newer than the first
segment.bytes	1073741824 (1 GiB)	Roll before this size. Minimum 1 MiB since Kafka 4.0
log.retention.check.interval.ms	300000 (5 min)	Broker setting. Read-only: a change needs a restart
file.delete.delay.ms	60000 (1 min)	Wait before deleted files leave the disk
remote.storage.enable	false	Tiered storage for the topic, if the broker allows it
local.retention.ms, local.retention.bytes	-2 (use retention.*)	With tiering, how much stays on the broker's disk

How Long Records Stay Readable

🚫 "Retention is one hour, so nothing in the topic is older than an hour."

The older records in a segment outlive retention.ms. On a payments topic with retention.ms=60000, segment.ms=180000 and one record every five seconds, offset 37 stayed readable for six minutes:

Time after offset 37	Event
0:00	Offset 37 is written and opens a new segment
2:55	Offset 72 is written, the segment's newest record
3:00	Offset 73 arrives past segment.ms, and the segment rolls
3:55	Offset 72 is one minute old: the segment qualifies
6:01	The next retention check deletes it

A record stays readable for as long as its segment stays open (up to segment.ms, or until segment.bytes fills), plus retention.ms, plus up to one check interval. Disk usage grows the same way, most visibly on topics with little traffic:

"If I'm retaining the data for only two days, but in reality, at the disk level, it could be much more."

Platform engineer at a hedge fund

The same mechanics set three more limits:

• retention.bytes is per partition, and a floor. With a 2 MiB limit, each partition kept 2.98 MB. Multiply by partitions and replicas (topic configs).
• Time means the record's timestamp, set by the producer. Records stamped eight days ago were deleted at the next check, three minutes after arriving (timestamp-based data loss).
• No deadline. When data must be gone by a date, size segment.ms for that date.

Tuning Segment Size

For retention that overshoots on a quiet topic, change segment.ms. With a one-hour retention.ms and a 15-minute segment.ms, a record stays readable for at most about 80 minutes.

A smaller segment.bytes shortens the overshoot the same way, at the cost of more files. Each segment keeps its .log open.

The Kafka docs count two memory-map areas per segment for its indexes, against a vm.max_map_count that is "somewhere around 65535" on many Linux systems: about 32,000 segments per broker. The docs also size file handles generously:

"We recommend at least 100000 allowed file descriptors for the broker processes as a starting point."

Apache Kafka documentation, Hardware and OS

On a broker that logs Too many open files, a larger segment.bytes (up to 2 GiB) means fewer files.

Compacted topics are the exception. The cleaner never touches the active segment, so on a quiet compacted topic, segment.ms decides when compaction can start. Everywhere else, leave segment.bytes at 1 GiB unless you've measured a reason.

Tiered Storage Retention

Tiered storage (KIP-405), production-ready since Kafka 3.9, copies closed segments to a remote store and keeps only recent ones on broker disks. local.retention.ms and local.retention.bytes set how long segments stay on the broker's disk, and retention.ms and retention.bytes set how long they stay in total.

"Apache Kafka doesn't provide an out-of-the-box RemoteStorageManager implementation."

Apache Kafka documentation, Tiered Storage

You add a RemoteStorageManager plugin for your object store and set remote.log.storage.system.enable=true on the brokers. Without it, remote.storage.enable=true on a topic fails with Tiered Storage functionality is disabled in the broker.

On a second broker running the docs' test store, LocalTieredStorage, the same 4,000 records went to a topic with remote.storage.enable=true and local.retention.ms=60000. After the copy and the next check:

$ ls /tmp/kraft-combined-logs/orders-0
00000000000000004000.log
00000000000000004000.snapshot
00000000000000004000.timeindex
leader-epoch-checkpoint
partition.metadata
$ ls /tmp/kafka-remote-storage/kafka-tiered-storage/orders-0-3nP4M2h7ThySJmQumZZ4Fw | grep '\.log$'
00000000000000000000-ny__ZweMSVeUedP6IIhY5A.log
00000000000000001024-KKQJxnZSRm-4ihWe_0AEvw.log
00000000000000002048-zlrJVsNrQFKPZlcpRefjww.log
00000000000000003072-7TrsgImkT-KovoDQ63zzcw.log
$ kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders --time earliest-local
orders:0:4000
$ kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders --time earliest
orders:0:0

The broker's disk keeps only the empty active segment, and a consumer reading from offset 0 gets its first record from the remote store:

Tiering inherits the segment mechanics:

• Only closed segments are copied. A segment can't be copied until it rolls, so segment.ms and segment.bytes also decide how long data sits only on the broker's disk.
• Local deletion waits for the upload. A local segment "is eligible for deletion only after it gets uploaded to remote", so if uploads stop, broker disks keep growing.
• Compacted topics stay local. The docs list "No support for compacted topics" among the limitations.

Enforcing Retention Policies

Conduktor Console resource policies check a topic's configuration before it's applied. The documented CEL rule keeps retention.ms between one minute and one hour, and a rule can bound segment.ms too.

Console evaluates the rules when a topic is created or updated, and the form won't create a topic that breaks one:

A record leaves Kafka with its whole segment, once the newest record in that segment passes retention.ms. On a quiet topic at the defaults, that can be up to a week later than retention.ms alone suggests.

The difference matters as soon as a contract or a regulation says how long data may stay. An auditor or the privacy team will ask for the date by which a record is gone, and only segment.ms set alongside retention.ms gives one.

Finance will ask too, when broker disks hold more than the retention setting suggests.

Talk to us about retention guardrails on your clusters →

---

Related: Kafka Retention: When Messages Disappear and Why → · Kafka Tiered Storage: Infinite Retention Without Infinite Disks → · Safeguard Kafka from Timestamp-Based Data Loss →