Home / Glossary

Zero Trust Architecture for Kafka

Traditional network security models assume that everything inside the corporate perimeter is trustworthy. Kafka clusters deployed in private networks often rely on this assumption: if a client reaches the broker, it must be authorized. This "castle and moat" approach fails when attackers breach the perimeter, when internal applications are misconfigured, or when multi-tenant environments share infrastructure across teams with different security clearances.

Zero Trust Architecture rejects implicit trust based on network location. Instead, it requires continuous verification of every access request, regardless of where it originates. For Kafka, this means authenticating every producer and consumer, authorizing every operation, encrypting all traffic, and logging all access patterns for continuous monitoring.

This approach is particularly critical for streaming platforms because Kafka often serves as the central nervous system of an organization, carrying financial transactions, customer PII, operational metrics, and inter-service communication across a single cluster. A breach in one area can cascade across the entire data fabric.

Core Zero Trust Principles for Streaming

Zero Trust is not a product but a security philosophy built on five core principles, each with specific implications for Kafka deployments:

  • Never trust, always verify: Authenticate every connection, even from internal networks. A producer running in the same data center as the broker is not inherently more trustworthy than an external client.
  • Least privilege access: Grant the minimum permissions required for each client. A consumer reading from user-events should not have access to payment-transactions. ACLs should be topic-specific, not cluster-wide.
  • Assume breach: Design security assuming attackers may already be inside your network. Encrypt all traffic, including broker-to-broker replication within the same availability zone.
  • Verify explicitly: Use strong authentication (mTLS, OAuth 2.0, SCRAM-SHA-512) rather than relying on IP addresses or network segments for access decisions.
  • Continuous monitoring: Log all authentication attempts, authorization decisions, and data access patterns. Detect anomalies that may indicate compromise or misconfiguration.

Zero Trust Architecture for Kafka

Authentication: Verifying Every Identity

Zero Trust requires strong authentication for every client connection. Network location is not identity. An application running in your VPC is not inherently trustworthy because it can reach the Kafka port.

Mutual TLS (mTLS) provides cryptographic identity verification for machine-to-machine communication:

  • Both client and broker present certificates during the TLS handshake
  • The certificate's Distinguished Name (DN) becomes the principal for authorization
  • No passwords to manage or rotate; certificate lifecycle replaces credential management
  • Eliminates credential theft via network sniffing
# Broker configuration enforcing mTLS
listeners=SSL://kafka-broker:9093
security.inter.broker.protocol=SSL
ssl.client.auth=required

ssl.keystore.location=/var/private/ssl/kafka.server.keystore.p12
ssl.keystore.password=${KEYSTORE_PASSWORD}
ssl.truststore.location=/var/private/ssl/kafka.server.truststore.p12
ssl.truststore.password=${TRUSTSTORE_PASSWORD}
ssl.enabled.protocols=TLSv1.3

For detailed mTLS implementation patterns, certificate lifecycle management, and KRaft-specific configuration, see mTLS for Kafka.

OAuth 2.0/OIDC integrates Kafka with enterprise identity providers (Okta, Azure AD, Keycloak):

  • Short-lived tokens reduce exposure window from compromised credentials
  • Centralized identity management across all systems
  • Token claims can drive fine-grained authorization decisions
  • Supports service accounts and human users with different policies

For organizations with existing identity infrastructure, OAuth 2.0 provides zero-trust authentication without deploying separate certificate infrastructure. See Kafka Authentication: SASL, SSL, OAuth for implementation details.

Authorization: Continuous Access Verification

Authentication proves identity; authorization determines what that identity can do. Zero Trust demands fine-grained, continuously evaluated authorization, not one-time permission grants.

Kafka ACLs enforce permissions at the topic, consumer group, and cluster level:

# Grant read-only access to specific topic
kafka-acls --add --allow-principal "User:CN=analytics-service,O=Acme" \
  --operation Read --topic customer-events

# Consumer group access (required for consumers)
kafka-acls --add --allow-principal "User:CN=analytics-service,O=Acme" \
  --operation Read --group analytics-group

Zero Trust ACL patterns:

  • Default deny: Without explicit ACLs, all access is blocked
  • Topic-specific permissions: Never use wildcard (*) ACLs in production
  • Separate read/write principals: Producers and consumers use different identities
  • Consumer group isolation: Each application owns its consumer groups

For comprehensive authorization patterns including RBAC and ABAC models, see Kafka ACLs and Authorization Patterns.

Proxy-Based Zero Trust with Conduktor Gateway

Native Kafka security requires configuring each broker and managing ACLs directly on the cluster. This approach becomes unwieldy at scale: certificate distribution across thousands of clients, ACL sprawl across hundreds of topics, and no visibility into access patterns.

Conduktor Gateway implements zero trust as a Kafka-protocol-compatible proxy, providing authentication and authorization at the network edge without modifying broker configuration:

Gateway authentication modes support gradual adoption:

  • Gateway-managed mode: Gateway handles all authentication with local or external service accounts. Supports mTLS, SASL (PLAIN, SCRAM, OAUTHBEARER), enabling zero trust without touching Kafka security configuration.
  • Kafka-managed mode: Authentication delegated to backing Kafka cluster for existing deployments migrating to zero trust. See Gateway authentication and authorization for configuration.

Service accounts in Gateway provide identity management for clients:

  • Local service accounts: Gateway-managed credentials with configurable TTL, ideal for external partners or temporary access
  • External service accounts: Map identities from OIDC providers or mTLS certificates to friendly names for audit trails

See Gateway service accounts for implementation details.

Virtual Clusters for Microsegmentation

Zero Trust demands microsegmentation: isolating workloads so compromise of one area cannot cascade across the infrastructure. Traditional Kafka achieves this through separate physical clusters, which is operationally expensive.

Virtual Clusters provide logical isolation within a single physical cluster:

  • Each virtual cluster operates as an independent namespace
  • Topics, consumer groups, and ACLs are scoped to the virtual cluster
  • Clients in one virtual cluster cannot see or access another's resources
  • Different security postures per virtual cluster (mTLS for payments, OAuth for analytics)

A payments team operating in a virtual cluster with strict mTLS requirements cannot be affected by a compromised credential in a less-sensitive development virtual cluster. This reduces blast radius, a core zero trust principle.

For Virtual Cluster configuration and multi-tenancy patterns, see Virtual Clusters documentation.

Interceptors for Policy Enforcement

Zero Trust is not just authentication and authorization; it includes continuous enforcement of security policies. Gateway Interceptors apply policies to Kafka traffic in real-time:

Security-relevant interceptors:

  • Encryption: Field-level or full-payload encryption via KMS integration (AWS KMS, HashiCorp Vault, Azure Key Vault)
  • Data masking: Hide sensitive fields from unauthorized consumers
  • Audit logging: Track specific Kafka operations (PRODUCE, FETCH, CREATE_TOPICS) for compliance
  • Header injection: Add metadata (client IP, user identity, timestamp) to messages for lineage tracking
# Audit interceptor for sensitive topics
pluginClass: io.conduktor.gateway.interceptor.AuditPlugin
config:
  topic: "payments.*"
  apiKeys: [PRODUCE, FETCH]

Traffic Control Policies enforce operational guardrails:

  • Rate limiting per service account or virtual cluster
  • Topic creation policies (minimum replication factor, naming conventions)
  • Schema ID requirements for data quality
  • Read-only topic enforcement

See Interceptors and Data security Interceptors for configuration reference. Zero Trust Implementation Layers

Audit Logging for Continuous Verification

Zero Trust assumes breach. Continuous monitoring and audit logging detect anomalies that indicate compromise or misconfiguration.

Gateway captures security-relevant events to a dedicated audit topic:

  • Authentication events: Successful and failed login attempts with client details
  • Authorization failures: Denied operations with principal, resource, and attempted action
  • Interceptor events: Encryption/decryption errors, policy violations
  • API key requests: PRODUCE, FETCH, CREATE_TOPICS operations on sensitive topics
{
  "id": "a3b7355b-dbe6-4db4-90a8-f41e0111710e",
  "source": null,
  "type": "AUTHENTICATION",
  "authenticationPrincipal": "payments-service",
  "userName": "payments-svc-account",
  "connection": {
    "localAddress": "/gateway:6969",
    "remoteAddress": "/10.0.1.42:52341"
  },
  "specVersion": "0.1.0",
  "time": "2025-07-18T16:07:06.570723674Z",
  "eventData": "SUCCESS"
}

Zero Trust monitoring patterns:

  • Alert on repeated authentication failures from same principal
  • Detect access to topics outside normal application scope
  • Monitor for ACL changes indicating privilege escalation
  • Track consumer lag anomalies that may indicate data exfiltration

For complete audit event types and configuration, see Configure audit logs.

Encryption at All Layers

Zero Trust mandates encryption everywhere, not just at network boundaries:

In-transit encryption:

  • TLS 1.3 for all client-to-Gateway and Gateway-to-broker communication
  • mTLS provides both encryption and authentication in a single mechanism
  • Broker-to-broker replication uses TLS even within the same data center

At-rest encryption:

  • Disk encryption (LUKS, dm-crypt, cloud provider encryption) for broker storage
  • Gateway KMS integration for field-level encryption of sensitive data

Field-level encryption via Gateway Interceptors protects data even if broker storage is compromised:

pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin
config:
  topic: "customer-data"
  kmsConfig:
    vault:
      uri: https://vault:8200
      token: ${VAULT_TOKEN}
  fields:
    - fieldName: ssn
      keySecretId: vault-kms://vault:8200/transit/keys/pii-key
    - fieldName: credit_card
      keySecretId: vault-kms://vault:8200/transit/keys/pii-key

For encryption details and KMS integration, see Encrypting Kafka with Conduktor.

Implementation Strategy

Zero Trust adoption is a journey, not a one-time project. A phased approach reduces risk and allows teams to build operational maturity:

Phase 1: Visibility

  • Deploy Gateway in passthrough mode (Kafka-managed authentication)
  • Enable audit logging to understand current access patterns
  • Identify which clients access which topics
  • Document existing implicit trust relationships

Phase 2: Authentication

  • Enable Gateway-managed authentication for new clients
  • Migrate existing clients to mTLS or OAuth 2.0
  • Retire legacy SASL/PLAIN credentials
  • Implement certificate automation (cert-manager, Vault PKI)

Phase 3: Authorization

  • Define ACLs based on observed access patterns
  • Enable default-deny for new virtual clusters
  • Implement traffic control policies for production topics
  • Remove overly permissive wildcard ACLs

Phase 4: Continuous improvement

  • Implement field-level encryption for sensitive data
  • Deploy data masking for unauthorized consumers
  • Automate security testing with chaos engineering
  • Regular ACL audits and principal lifecycle reviews

Sources and References

Written by Stéphane Derosiaux · Last updated February 18, 2026