Kafka Encryption Beyond TLS
Enabling TLS doesn't encrypt your Kafka data. It encrypts the connection. Message payloads remain readable by anyone with broker access. Conduktor adds field-level encryption: specific fields are encrypted before reaching the broker, decrypted only for authorized consumers.
Kafka encryption has three distinct layers, each protecting against different threats.
Encryption in Transit (TLS)
Protects data moving between clients and brokers. Prevents network eavesdropping. TLS 1.2/1.3 configured on broker listeners. mTLS for mutual authentication. Required baseline for any production deployment.
Encryption at Rest (Disk)
Protects data on broker disks from physical theft or unauthorized storage access. Managed by your cloud provider (EBS encryption on AWS, persistent disk encryption on GCP) or OS-level disk encryption. Doesn't protect against broker compromise.
Field-Level Payload Encryption
Encrypts the actual content of specific message fields before they're written to the broker. The broker stores and replicates ciphertext. Even with full broker access, sensitive fields are unreadable without the decryption key. This is where Conduktor focuses.
Encryption at the Proxy
Conduktor Gateway intercepts produce requests and encrypts specified fields before writing to the broker. Algorithm is configurable (AES-128-GCM by default, AES-256-GCM and ChaCha20-Poly1305 also supported). No changes to producer application code required.
Key Management Integration
Encryption keys are stored in your KMS (AWS KMS, Azure Key Vault, GCP Cloud KMS, HashiCorp Vault, or Fortanix). Conduktor never stores keys. It retrieves and caches them with a configurable TTL.
Per-Consumer Decryption
Decrypt requests are intercepted at consume time. Gateway checks consumer permissions, retrieves the decryption key from KMS, and delivers decrypted fields to authorized consumers only.
Schema-Aware Field Selection
Define which fields to encrypt in each topic using field path notation (e.g., payload.creditCardNumber, headers.ssn). Supports Avro, JSON Schema, and Protobuf.
GDPR Right to Erasure
Implement: encrypt each user's data with a per-user key. To erase a user's data, destroy their key. Instant, verifiable data deletion without topic compaction or reprocessing.
crypto shredding →PII Protection
Encrypt PII fields (names, emails, SSNs, credit card numbers) in Kafka messages. Downstream consumers that don't need the raw value receive the encrypted field or a masked version.
Multi-Tenant Data Isolation
Each tenant's data encrypted with a separate key. Even if one tenant's data is accessed, other tenants' data remains protected. Required for shared-cluster multi-tenancy.
Regulated Industry Compliance
HIPAA, PCI-DSS, and financial regulations require encryption of sensitive data at rest. Field-level encryption satisfies this requirement for Kafka-based pipelines.
How is field-level encryption different from TLS?
TLS encrypts the connection between client and broker. Data is decrypted when it reaches the broker. Field-level encryption encrypts the message payload. Data remains encrypted in the broker's storage. TLS and field-level encryption are complementary, not alternatives.
Which KMS providers does Conduktor support?
AWS KMS, Azure Key Vault, GCP Cloud KMS, HashiCorp Vault, and Fortanix. Keys are retrieved at encryption/decryption time and cached with a configurable TTL.
Does this require changes to producer or consumer code?
No. Producers write to Conduktor Gateway instead of directly to Kafka. Encryption is applied transparently. Consumers that aren't authorized for decryption receive the encrypted ciphertext or an error, depending on policy configuration.
What happens if the KMS is unavailable?
Gateway caches keys with a configurable TTL, so short KMS outages don't immediately affect encryption or decryption. If the KMS is unreachable and the cache has expired, requests fail with an error rather than passing through unencrypted. For compliance use cases, this fail-closed behavior is correct.
Can I encrypt only specific topics?
Yes. Encryption policies are configured per topic or topic pattern. Most organizations start with topics containing PII or financial data and expand from there.
See Kafka Encryption in Production
Stop relying on TLS as your only Kafka encryption. 30-minute demo: configuring field-level encryption on a topic, integrating with your KMS, and verifying broker-side ciphertext.