If the analytics team needs the orders topic but must never see a card number, Kafka gives you two answers. Grant READ on orders and analytics receives every field of every record, card number included. Refuse it and analytics receives nothing.
A Kafka ACL is a decision about a topic, and the card number is a field inside a record the broker never decodes. A rule about fields needs a component that decodes records: a Kafka proxy between clients and brokers, which masks the card number for the readers who shouldn't see it.
orders. The proxy decides which fields each one gets back.The ACL still decides who reads orders. The proxy decides what each of those readers receives.
What a Kafka ACL Decides
An ACL has four parts: a principal, an operation, a resource and a host. Granting analytics read access on Kafka 4.3.1 prints the whole rule back:
$ kafka-acls.sh --bootstrap-server localhost:9092 --add \
--allow-principal User:analytics --operation Read --topic orders
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=orders, patternType=LITERAL)`:
(principal=User:analytics, host=*, operation=READ, permissionType=ALLOW) Every ACL has this shape: its resource is a topic, a group, a transactional ID, a delegation token, a user or the cluster. No resource type points inside a record, and the other controls Kafka ships stop at the same level:
| Control | What it decides | What it can't express |
|---|---|---|
| ACLs | Which principal may read, write, create or delete a topic, group or cluster | Which fields of a record that principal receives |
| Quotas | Bytes per second or request-thread time for a user or client ID, on each broker | A limit for one topic |
| TLS and SASL | Who the client is, and encryption on the wire | What an authenticated client may see inside a record |
| Schema ID validation (Confluent Server, Redpanda) | That a record's schema ID is registered | Whether the payload matches that schema |
The Broker Sees Only Bytes
To the broker, a record's key and value are byte arrays. Serialization happens in the client, and with Schema Registry the serializer writes a magic byte and a four-byte schema ID in front of the payload, so that a reader can fetch the schema and decode the rest. The broker validates the batch framing and stores the bytes. It never fetches a schema, so it has no field names to apply a rule to:
Kafka's designers chose this on purpose:
"Leaving the key and value opaque is the right decision"
Apache Kafka documentation, Messages
Their reason is that no single serialization format suits every application. So any rule that names a field has to run in a component that decodes the record.
🚫 "Our ACLs are tight, so only the right people see card numbers."
Tight ACLs shorten the list of principals that read orders. Every principal on that list still receives the card number, in clear, in every record.
Masking Without a Proxy
Teams that need field-level rules put them in one of four places, each further from the data than the record itself:
| Workaround | Who still receives the card number | What it costs |
|---|---|---|
| Keep the field out of Kafka | Nobody | The fraud service loses it too, unless the producer writes a second topic |
| A filtered copy of the topic | Every reader of the original topic | A stream processing job and a second topic per audience, both lagging the source |
| Masking in each consumer or client library | Every consumer, before its own code masks the field | One library per language, applied separately by each team |
| Masking in a Kafka UI | Every application consuming the topic | Covers people browsing data in that UI, and only them |
In our calls with platform teams, the common answers are to keep PII out of Kafka or to mask it downstream, in the warehouse or in each consumer:
"We don't mask it in Kafka itself, but we do in most places that access it… but not everywhere for sure."
Senior software engineer at a fintech
A rule that each consumer applies holds only in the consumers that implement it, and every new consumer starts without it.
"So far… we've just been saying it cannot be solved, but potentially… with the proxy in the middle, it can be solved."
Platform engineering manager at a large fintech
A Proxy Decodes Each Record
A Kafka proxy speaks the Kafka protocol on both sides. Clients use it as their bootstrap server, and it rewrites the broker addresses in Metadata responses so that every later connection comes back through it (how that rewrite works).
It can authenticate clients with the SASL or mTLS identity your ACLs already use, so User:analytics names the same principal in both places. It reads each record's schema ID, fetches the schema from Schema Registry, decodes the value and applies rules by field name. The consumer receives a record of the same shape with the masked value in place.
The open-source Kroxylicious ships record encryption and validation filters, and Confluent Private Cloud Gateway, Kong Event Gateway and IBM Event Endpoint Management enforce policies in a Kafka-protocol proxy too. Kai Waehner's survey counts compliance and schema validation among their main uses.
Mask, Encrypt, Validate
The example below uses Conduktor Gateway, where each rule is an interceptor: a plugin with a scope (a user, a group or a virtual cluster), applied through the API, the CLI, Terraform or Console.
Mask fields per reader
Masking runs on fetch. This interceptor masks the first twelve digits of cardNumber and all of email when User:analytics reads orders; principals outside the scope, fraud-service included, receive the original values:
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: mask-pii-for-analytics
scope:
username: analytics
spec:
pluginClass: io.conduktor.gateway.interceptor.FieldLevelDataMaskingPlugin
priority: 100
config:
topic: orders
schemaRegistryConfig:
host: http://schema-registry:8081
policies:
- name: Keep the last four card digits
rule:
type: MASK_FIRST_N
numberOfChars: 12
fields:
- cardNumber
- name: Hide the email address
rule:
type: MASK_ALL
fields:
- email Masking replaces the value and leaves the field in place, on Avro, Protobuf and JSON Schema records through Schema Registry and on schemaless JSON (masking reference).
Encrypt fields the brokers shouldn't hold
Masking applies on the way out, so the brokers still store the clear card number. When the people operating the cluster must not read it either, the proxy encrypts the field on produce and decrypts it on fetch only for the principals in the decrypt scope, with keys in your own KMS (encryption guide).
Validate the payload on produce
Schema ID validation checks that an ID is registered. The proxy can decode the payload and check it against the schema before the broker stores it: with action: BLOCK the produce request is rejected, and INFO writes the data and logs the violation, the safer first setting on a busy topic. Rules on client behaviour such as acks or commit rates work the same way (Kafka Has Rate Limits. Nobody Turns Them On.).
Gateway writes an audit trail to a Kafka topic, with the principal behind each connection, ACL denial, and block or throttle decision its policies take. Asked which principals can read card numbers on orders, you find the answer in the masking interceptor's scope, a single resource you can review.
A Proxy's Costs and Limits
Every request now crosses one more component, and Kai Waehner's survey names the first cost:
"A proxy adds an extra network hop between clients and brokers, which can slightly increase end-to-end latency."
Kai Waehner, Kafka Proxy Demystified
- Latency. In our Gateway benchmark over SASL and TLS, one topic at 120,000 messages of 1 KiB per second, end-to-end p99 went from 45 ms to 52 ms with unchanged throughput. Masking wasn't in that run and decodes every record it touches: measure it on your own schemas.
- A tier to run. A sidecar per application avoids the extra hop but covers only the applications that deploy it. A central tier covers every client and needs the care you give brokers: Gateway's instances are stateless, the docs recommend at least three in production, and a wrong rule reaches every client at once.
- Records it can't decode. When masking can't decode a record, the fetch fails by default (
errorPolicy: fail_fetch);skip_maskingreturns it unmasked. - Whole batches. In
BLOCKmode, payload validation rejects the whole produce request when one record fails. - Filtering at the source. A solution architect at a health insurer argued on a call with us that data a third party shouldn't see should be filtered before it's produced. When no consumer needs a field, don't produce it.
Close the direct path to the brokers first, because nothing reports it. Masking applies only to traffic through the proxy, and a client connected to the brokers directly reads the clear value:
Restrict the brokers at the network level so that they accept connections from the proxy only.
Put Field Rules in the Proxy
Conduktor Gateway implements this layer: masking, field-level encryption and payload validation as interceptors scoped to the same principals as your ACLs, configured from Console, the CLI or Terraform and enforced on every request (interceptor reference).
Keep your ACLs: they decide who reads a topic. A rule that names a field needs a component that decodes the record before the consumer receives it. Put that rule in a proxy every client goes through, and it covers every consumer of the topic, including the ones that don't exist yet.
Talk to us about field-level rules on your clusters →
Related: What a Kafka Proxy Can Do → · Kafka Encryption: Transit, Rest, End-to-End → · Real-Time Data Sharing with External Partners →
