Kafka ACLs Stop at the Topic

Stéphane Derosiaux October 5, 2025 9 min read
Isometric wireframe line-art on dark teal: a single file of flat record tiles, each engraved with four field lines, glides left to right along a track under one small lime lamp. Past the lamp, the third line on every tile is a row of hollow dots.

If the analytics team needs the orders topic but must never see a card number, Kafka gives you two answers. Grant READ on orders and analytics receives every field of every record, card number included. Refuse it and analytics receives nothing.

A Kafka ACL is a decision about a topic, and the card number is a field inside a record the broker never decodes. A rule about fields needs a component that decodes records: a Kafka proxy between clients and brokers, which masks the card number for the readers who shouldn't see it.

checkoutproducerfraud-serviceACL: READ ordersanalyticsACL: READ ordersKafka brokerstopic ordersfull recordsKafka proxyauthenticatedecode recordmask per readerproducefull recordcard 4111111111111234card ************1234
Both consumers hold the same ACL on orders. The proxy decides which fields each one gets back.

The ACL still decides who reads orders. The proxy decides what each of those readers receives.

What does this post change for your team?ClaudeChatGPT

What a Kafka ACL Decides

An ACL has four parts: a principal, an operation, a resource and a host. Granting analytics read access on Kafka 4.3.1 prints the whole rule back:

$ kafka-acls.sh --bootstrap-server localhost:9092 --add \
    --allow-principal User:analytics --operation Read --topic orders
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=orders, patternType=LITERAL)`:
    (principal=User:analytics, host=*, operation=READ, permissionType=ALLOW)

Every ACL has this shape: its resource is a topic, a group, a transactional ID, a delegation token, a user or the cluster. No resource type points inside a record, and the other controls Kafka ships stop at the same level:

ControlWhat it decidesWhat it can't express
ACLsWhich principal may read, write, create or delete a topic, group or clusterWhich fields of a record that principal receives
QuotasBytes per second or request-thread time for a user or client ID, on each brokerA limit for one topic
TLS and SASLWho the client is, and encryption on the wireWhat an authenticated client may see inside a record
Schema ID validation (Confluent Server, Redpanda)That a record's schema ID is registeredWhether the payload matches that schema
Schema ID validation is licensed on both platforms, and both vendors document that the check doesn't inspect the payload.

The Broker Sees Only Bytes

To the broker, a record's key and value are byte arrays. Serialization happens in the client, and with Schema Registry the serializer writes a magic byte and a four-byte schema ID in front of the payload, so that a reader can fetch the schema and decode the rest. The broker validates the batch framing and stores the bytes. It never fetches a schema, so it has no field names to apply a rule to:

what the broker seesACL: User:analytics · READ · Topic:orders → ALLOW0000 00 00 2a10 6f 72 64 2d 39 31 38 34 c8 01 20 34 31 31 31 31 31 …value: bytes, never decodeddecode with schema 42from Schema Registrywhat a schema-aware proxy seesmagic0schema id42orderIdord-9184amount100cardNumber4111111111111234emaila@example.commasked for User:analytics
The broker checks the ACL and stores bytes. Field names exist only for a reader that fetches schema 42 and decodes the payload.

Kafka's designers chose this on purpose:

"Leaving the key and value opaque is the right decision"

Apache Kafka documentation, Messages

Their reason is that no single serialization format suits every application. So any rule that names a field has to run in a component that decodes the record.

🚫 "Our ACLs are tight, so only the right people see card numbers."

Tight ACLs shorten the list of principals that read orders. Every principal on that list still receives the card number, in clear, in every record.

Masking Without a Proxy

Teams that need field-level rules put them in one of four places, each further from the data than the record itself:

WorkaroundWho still receives the card numberWhat it costs
Keep the field out of KafkaNobodyThe fraud service loses it too, unless the producer writes a second topic
A filtered copy of the topicEvery reader of the original topicA stream processing job and a second topic per audience, both lagging the source
Masking in each consumer or client libraryEvery consumer, before its own code masks the fieldOne library per language, applied separately by each team
Masking in a Kafka UIEvery application consuming the topicCovers people browsing data in that UI, and only them
Lenses and Kpow document their masking as applying to people using their interfaces, and applications consuming the topic still read raw records. SecuPi productizes the consumer-side option, with central policies enforced inside each producer and consumer process.

In our calls with platform teams, the common answers are to keep PII out of Kafka or to mask it downstream, in the warehouse or in each consumer:

"We don't mask it in Kafka itself, but we do in most places that access it… but not everywhere for sure."

Senior software engineer at a fintech

A rule that each consumer applies holds only in the consumers that implement it, and every new consumer starts without it.

"So far… we've just been saying it cannot be solved, but potentially… with the proxy in the middle, it can be solved."

Platform engineering manager at a large fintech

A Proxy Decodes Each Record

A Kafka proxy speaks the Kafka protocol on both sides. Clients use it as their bootstrap server, and it rewrites the broker addresses in Metadata responses so that every later connection comes back through it (how that rewrite works).

It can authenticate clients with the SASL or mTLS identity your ACLs already use, so User:analytics names the same principal in both places. It reads each record's schema ID, fetches the schema from Schema Registry, decodes the value and applies rules by field name. The consumer receives a record of the same shape with the masked value in place.

The open-source Kroxylicious ships record encryption and validation filters, and Confluent Private Cloud Gateway, Kong Event Gateway and IBM Event Endpoint Management enforce policies in a Kafka-protocol proxy too. Kai Waehner's survey counts compliance and schema validation among their main uses.

Mask, Encrypt, Validate

The example below uses Conduktor Gateway, where each rule is an interceptor: a plugin with a scope (a user, a group or a virtual cluster), applied through the API, the CLI, Terraform or Console.

Mask fields per reader

Masking runs on fetch. This interceptor masks the first twelve digits of cardNumber and all of email when User:analytics reads orders; principals outside the scope, fraud-service included, receive the original values:

apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: mask-pii-for-analytics
  scope:
    username: analytics
spec:
  pluginClass: io.conduktor.gateway.interceptor.FieldLevelDataMaskingPlugin
  priority: 100
  config:
    topic: orders
    schemaRegistryConfig:
      host: http://schema-registry:8081
    policies:
      - name: Keep the last four card digits
        rule:
          type: MASK_FIRST_N
          numberOfChars: 12
        fields:
          - cardNumber
      - name: Hide the email address
        rule:
          type: MASK_ALL
        fields:
          - email

Masking replaces the value and leaves the field in place, on Avro, Protobuf and JSON Schema records through Schema Registry and on schemaless JSON (masking reference).

Encrypt fields the brokers shouldn't hold

Masking applies on the way out, so the brokers still store the clear card number. When the people operating the cluster must not read it either, the proxy encrypts the field on produce and decrypts it on fetch only for the principals in the decrypt scope, with keys in your own KMS (encryption guide).

Validate the payload on produce

Schema ID validation checks that an ID is registered. The proxy can decode the payload and check it against the schema before the broker stores it: with action: BLOCK the produce request is rejected, and INFO writes the data and logs the violation, the safer first setting on a busy topic. Rules on client behaviour such as acks or commit rates work the same way (Kafka Has Rate Limits. Nobody Turns Them On.).

Gateway writes an audit trail to a Kafka topic, with the principal behind each connection, ACL denial, and block or throttle decision its policies take. Asked which principals can read card numbers on orders, you find the answer in the masking interceptor's scope, a single resource you can review.

A Proxy's Costs and Limits

Every request now crosses one more component, and Kai Waehner's survey names the first cost:

"A proxy adds an extra network hop between clients and brokers, which can slightly increase end-to-end latency."

Kai Waehner, Kafka Proxy Demystified
  • Latency. In our Gateway benchmark over SASL and TLS, one topic at 120,000 messages of 1 KiB per second, end-to-end p99 went from 45 ms to 52 ms with unchanged throughput. Masking wasn't in that run and decodes every record it touches: measure it on your own schemas.
  • A tier to run. A sidecar per application avoids the extra hop but covers only the applications that deploy it. A central tier covers every client and needs the care you give brokers: Gateway's instances are stateless, the docs recommend at least three in production, and a wrong rule reaches every client at once.
  • Records it can't decode. When masking can't decode a record, the fetch fails by default (errorPolicy: fail_fetch); skip_masking returns it unmasked.
  • Whole batches. In BLOCK mode, payload validation rejects the whole produce request when one record fails.
  • Filtering at the source. A solution architect at a health insurer argued on a call with us that data a third party shouldn't see should be filtered before it's produced. When no consumer needs a field, don't produce it.

Close the direct path to the brokers first, because nothing reports it. Masking applies only to traffic through the proxy, and a client connected to the brokers directly reads the clear value:

analyticsconsumerKafka brokerstopic ordersKafka proxyvia proxy: card ************1234direct: card 4111111111111234close it: brokers accept connections from the proxy only
Masking holds for traffic through the proxy. A direct connection to the brokers returns the clear value.

Restrict the brokers at the network level so that they accept connections from the proxy only.

Put Field Rules in the Proxy

Conduktor Gateway implements this layer: masking, field-level encryption and payload validation as interceptors scoped to the same principals as your ACLs, configured from Console, the CLI or Terraform and enforced on every request (interceptor reference).

Keep your ACLs: they decide who reads a topic. A rule that names a field needs a component that decodes the record before the consumer receives it. Put that rule in a proxy every client goes through, and it covers every consumer of the topic, including the ones that don't exist yet.

Talk to us about field-level rules on your clusters →


Related: What a Kafka Proxy Can Do → · Kafka Encryption: Transit, Rest, End-to-End → · Real-Time Data Sharing with External Partners →