Kafka Security: Beyond TLS

Without encryption, anyone on the network or with disk access can read your Kafka messages. You end up needing encryption in three places: in traffic between clients and brokers, in whatever Kafka writes to disk, and inside the payload itself for anything sensitive.

Each layer stops a different kind of attacker, and none of them substitutes for the others.

Data in transit: TLS and mTLS

Kafka supports TLS out of the box. Clients negotiate a TLS session with the broker, exchange certificates, and encrypt traffic over the wire. Once TLS is enforced on a listener, unencrypted connections get rejected.

Mutual TLS (mTLS) extends this. Instead of only the server presenting a certificate, the client presents one too. The broker validates the client certificate against a trusted CA. You get encrypted traffic and authentication in a single handshake.

A minimal broker config that enforces TLS 1.3 and requires client certificates:

# broker.properties
listeners=SSL://:9093
listener.security.protocol.map=SSL:SSL
ssl.keystore.location=/var/kafka/kafka.server.keystore.jks
ssl.keystore.password=<redacted>
ssl.truststore.location=/var/kafka/kafka.server.truststore.jks
ssl.truststore.password=<redacted>
ssl.client.auth=required
ssl.enabled.protocols=TLSv1.2,TLSv1.3
ssl.protocol=TLSv1.3

Set ssl.client.auth=required to reject any client that can't present a valid certificate. Drop the PLAINTEXT listener entirely; there's no production reason to keep one.

Data at rest: disk-level encryption

Kafka doesn't encrypt the messages it stores on disk. That's on the OS or your cloud provider. Common options:

• Linux Unified Key Setup (LUKS) for self-managed brokers
• Amazon EBS encryption for AWS
• Azure Disk Encryption for Azure
• Google Cloud persistent disk encryption for GCP

Disk-level encryption protects against stolen disks and misconfigured backups. It does nothing about a process that already has broker access; that process sees plaintext.

Field-level and message-level encryption

For sensitive data like credentials, payment details, or health records, disk-level encryption isn't enough. The payload itself needs to be encrypted before it reaches the broker. This is called application-level encryption and comes in two forms.

Message-level encryption encrypts the entire payload. The broker sees only ciphertext. It cannot route or filter based on content. Maximum confidentiality, but Kafka loses some capabilities.

Field-level encryption encrypts only sensitive fields, such as PII or payment data. Routing and filtering on non-sensitive fields still work. The broker never sees the protected values in plaintext.

Layer	TLS only	Disk-level	Field-level	Message-level
On the wire	Encrypted	Plaintext unless TLS is also enabled	Encrypted with TLS	Encrypted with TLS
On broker disk	Plaintext	Encrypted by the OS or cloud	Ciphertext only	Ciphertext only
Readable by broker admins	Yes	Yes, through a Kafka client	No	No
Routing and filtering	Preserved	Preserved	Works on non-encrypted fields	Lost — the full payload is opaque
Typical use	Baseline	Stored-data compliance	PII, PCI, HIPAA	End-to-end confidentiality
Overhead	Low	Low	Moderate	Higher

Native Kafka gives you TLS. The other three layers are on you. Either you write encryption code in every producer and consumer, or you use a proxy layer that enforces it centrally.

Go deeper: full Kafka encryption guide → · how Conduktor Gateway handles encryption across apps →

Authentication verifies the identity of every client connecting to Kafka. Without it, anyone on the network can produce or consume messages. There are three common approaches.

68% of breaches involve the human element: errors, stolen credentials, social engineering. Not exotic exploits. (Verizon 2024 DBIR)

Mutual TLS (mTLS)

Client and broker each present a certificate signed by a trusted CA. Once validated, the TLS session carries an authenticated identity. Common in service-to-service scenarios where certificate issuance is already part of the infrastructure.

SASL (Simple Authentication and Security Layer)

A plug-in framework. Kafka supports four SASL mechanisms:

• SASL/PLAIN — username and password. Simple. Credentials travel with every connection, so use only with TLS.
• SASL/SCRAM — challenge-response with hashed passwords. Credentials never travel in plaintext.
• SASL/GSSAPI (Kerberos) — ticket-based auth against a KDC. Common in large enterprises with existing Kerberos.
• SASL/OAUTHBEARER — token-based auth for OAuth2 and OpenID Connect providers.

SSO: OIDC and LDAP

SSO isn't native to Kafka, but it's what operators end up needing. Humans don't manage their own certificates. They log into Okta, Azure AD, or Google Workspace. A proxy or control plane in front of Kafka terminates the OIDC or LDAP handshake and maps the identity to a Kafka principal.

A minimal SASL/SCRAM configuration:

# broker.properties
listener.security.protocol.map=SASL_SSL:SASL_SSL
sasl.enabled.mechanisms=SCRAM-SHA-512

# client.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=\
  org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="alice" password="alice-secret";

Kafka validates the SCRAM handshake against credentials stored in KRaft (or ZooKeeper on older clusters) and binds the connection to a principal like User:alice.

Go deeper: SSO for humans in Conduktor Console → · service auth translation at the wire in Gateway →

Authentication tells you who is connecting. Authorization decides what they're allowed to do. Kafka has two main approaches: native ACLs, and role-based access control layered on top.

Access Control Lists (ACLs)

ACLs are off by default. Turn them on in broker.properties before anything else, otherwise every authenticated principal can do everything:

# broker.properties (KRaft)
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin
allow.everyone.if.no.acl.found=false

On ZooKeeper-backed clusters, swap the authorizer for kafka.security.authorizer.AclAuthorizer. The other two lines are the same.

With allow.everyone.if.no.acl.found=false, the broker denies any request that isn't explicitly permitted. Safer default than the alternative.

ACLs are then managed through the kafka-acls.sh CLI. Each ACL is an allow rule mapping a principal to a resource and operation:

bin/kafka-acls.sh --bootstrap-server localhost:9092 \
  --add --allow-principal User:alice \
  --operation Read --topic customer-events

ACLs are fine-grained but have practical limits at scale:

• No groups. Add a new team member, add ACLs one at a time.
• No role inheritance. Every principal's permissions are managed directly.
• No audit of changes. Who added this ACL last Tuesday? Hard to answer.
• No centralized visibility. Listing permissions across clusters means running the CLI per cluster.

For dozens of users and topics, ACLs work. For hundreds or thousands across multiple clusters, they become unmanageable.

Role-Based Access Control (RBAC)

RBAC adds a layer of indirection between users and permissions. Permissions attach to roles. Users join groups. Groups get roles. Change the role once, and everyone in the group picks up the new permissions.

This makes onboarding and offboarding one step. A new data scientist joining data-science-team inherits every permission that team has. A leaver removed from the group loses access immediately.

	ACLs	RBAC
Unit of control	Per-resource, per-operation	Per-role, applied across resources
Groups	Not supported	First-class
Role inheritance	Not supported	Nested roles and hierarchies
Centralized visibility	CLI per cluster	Unified UI or API
Change audit	Limited, no native history	Full history of role and assignment changes
Scale	Small deployments	Hundreds of users and topics
Standards alignment	Kafka-specific	NIST RBAC

ACLs aren't going away. They're still the enforcement layer at the broker. RBAC sits on top to manage them. The combination is what you want: roles and groups for humans, ACLs for the broker.

Go deeper: Kafka ACL guide → · Kafka RBAC guide → · group-based RBAC in Conduktor Console →

When an auditor asks "who consumed customer PII on March 3 at 2 PM?", the answer should take minutes, not weeks. Auditing captures every action with enough context to reconstruct what happened.

What Kafka logs natively

Kafka uses Log4j to emit authorization and authentication events. With the right configuration, you get records of:

• Successful and failed authentication attempts
• Authorization decisions (allow or deny)
• Admin API calls (topic creation, ACL changes, config updates)

# log4j.properties
log4j.logger.kafka.authorizer.logger=INFO, authorizerAppender
log4j.logger.kafka.request.logger=INFO, requestAppender

What you don't get natively: consumer-side activity (who actually read which message from which partition), full payload context, and cross-cluster correlation. Turning raw broker logs into a proper audit trail is a separate build.

SIEM integration

Kafka's Log4j output ships to any standard SIEM or log aggregator:

• Splunk — via universal forwarders or HEC
• ELK / OpenSearch — Filebeat or Logstash
• Datadog — Datadog Agent with the Kafka integration
• Sumo Logic, New Relic — via their Log4j appenders

Once in the SIEM, detection rules do the actual security work: repeated auth failures, unusual IP patterns, privilege escalation, admin API spikes.

Compliance reporting

Auditors don't want raw logs. They want evidence: "show me every access to PCI data in Q3." That means:

• Retention policies aligned with regulations (GDPR: case-by-case; PCI-DSS: at least one year; HIPAA: six years)
• Tamper-evidence — signed or write-once audit logs
• Searchable reports filtered by user, resource, time range, or action type
• Export formats your auditors will accept, usually CSV, JSON, or PDF

Native Kafka produces the raw events. Turning them into audit-ready evidence is work most teams end up doing themselves.

Go deeper: audit trails and SIEM integration in Console → · wire-level auth logging in Gateway →

---