Audit Logging in Kafka: Who Did What and When

When an auditor asks "who accessed this data in the last 90 days?" and you can't answer, you have a compliance problem. Kafka doesn't provide audit logging out of the box. But it provides the components to build one.

I've been through SOC2 and HIPAA audits where the Kafka access question came up. The teams that had authorizer logging configured passed. The teams that didn't scrambled.

Our auditor asked for 90 days of Kafka access logs. We had 7 days. That finding cost us three months of remediation.

Security Engineer at a healthcare company

What Auditors Actually Want

Framework	Key Requirement
SOC2	Record who accessed systems and what actions they performed
HIPAA	Track all access to electronic Protected Health Information
PCI-DSS	Log all access to cardholder data environments

For Kafka, this means: authentication events, authorization decisions, data access, and administrative actions.

Kafka's Built-in Audit Capabilities

The kafka.authorizer.logger captures all ACL decisions. By default, it logs denials at INFO and allows at DEBUG.

A denial:

[2026-02-03 14:23:15] INFO Principal = User:unauthorized-app is Denied Operation = Read
from host = 10.0.1.45 on resource = Topic:LITERAL:payments (kafka.authorizer.logger)

An allowed operation (DEBUG level):

[2026-02-03 14:23:16] DEBUG Principal = User:consumer-app is Allowed Operation = Read
from host = 10.0.1.42 on resource = Topic:LITERAL:orders (kafka.authorizer.logger)

Enable Authorizer Logging

For Kafka 2.5+ (log4j2): Edit log4j2.properties:

appender.authorizer.type = RollingFile
appender.authorizer.name = authorizerAppender
appender.authorizer.fileName = /var/log/kafka/kafka-authorizer.log
appender.authorizer.filePattern = /var/log/kafka/kafka-authorizer.%d{yyyy-MM-dd-HH}.log
appender.authorizer.layout.type = PatternLayout
appender.authorizer.layout.pattern = [%d] %p %m (%c)%n

logger.authorizer.name = kafka.authorizer.logger
logger.authorizer.level = DEBUG
logger.authorizer.appenderRef.authorizer.ref = authorizerAppender
logger.authorizer.additivity = false

For Kafka < 2.5 (log4j1): Edit log4j.properties:

log4j.appender.authorizerAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.authorizerAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.authorizerAppender.File=/var/log/kafka/kafka-authorizer.log
log4j.appender.authorizerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.authorizerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.logger.kafka.authorizer.logger=DEBUG, authorizerAppender
log4j.additivity.kafka.authorizer.logger=false

The additivity=false prevents authorizer events from appearing in the main Kafka log.

Tradeoff: DEBUG level generates 2 log lines per operation. On a busy cluster with 10,000 requests/second, expect gigabytes per hour.

Centralize Your Logs

Broker-local files don't satisfy auditors. Ship to centralized storage.

Approach	Consideration
ELK Stack	Flexible queries, operational overhead
CloudWatch/Stackdriver	Managed, cost at scale
Splunk/Datadog	Enterprise features, licensing cost

Example Filebeat config:

filebeat.inputs:
  - type: log
    paths:
      - /var/log/kafka/kafka-authorizer.log
    fields:
      log_type: kafka_authorizer
      cluster: production-us-east

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  index: "kafka-audit-%{+yyyy.MM.dd}"

Investigating Incidents

Who accessed topic X?

grep "resource = Topic:LITERAL:payments" /var/log/kafka/kafka-authorizer.log | \
  grep -oP "Principal = \K[^,]+" | sort | uniq -c | sort -rn

What did principal Y do?

grep "Principal = User:suspicious-app" /var/log/kafka/kafka-authorizer.log | \
  grep -oP "Operation = \K\w+" | sort | uniq -c

Compliance-Specific Requirements

SOC2: 90 days minimum retention, 1 year recommended.

HIPAA: 6 years minimum. Mark PHI topics with naming conventions (phi-*) and filter logs accordingly.

PCI-DSS: 1 year minimum, 3 months immediately available.

Gaps in Open-Source Kafka

Apache Kafka's authorizer logging has limitations:

• No record-level audit—you know someone read from payments but not which records
• No data classification—all topics treated equally
• No real-time alerts—post-hoc analysis only
• No tamper protection—local logs can be modified

For regulated industries, augment with schema registry for field metadata, proxy layers for record-level logging, and external aggregation with immutable storage. Role-based access control in Conduktor provides additional audit trail capabilities.

Audit logging is a requirement, not a feature. The time to configure it is before the auditor arrives, not after the breach.

Book a demo to see how Conduktor Gateway provides built-in compliance logging with topic-level access tracking.