Kafka Proxy: Beyond ACLs for Governance

ACLs grant topic-level access. PII compliance requires field-level control. For a deeper look at Kafka encryption options and their tradeoffs, see our encryption guide.

If your data governance requirement is "analysts can read customer orders but not credit card numbers," Kafka ACLs can't help. ACLs operate at topic and operation level: you can read the topic or you can't. They don't inspect message contents, don't understand schemas, and can't mask sensitive fields.

This leaves teams with bad options: duplicate topics with PII removed (data consistency nightmare), force every consumer to implement field filtering (error-prone and unenforceable), or deny access entirely (blocking legitimate use cases). For compliance-heavy enterprises, Conduktor Gateway's proxy architecture enables field-level encryption, data masking, and policy enforcement at the wire level without modifying producer or consumer applications.

A proxy layer sits between clients and Kafka, intercepting messages and applying policies: mask PII fields for analytics teams, enforce rate limits to protect clusters, validate schemas before allowing production, or route traffic across multiple clusters. The proxy speaks the Kafka protocol, so clients don't change—but governance capabilities expand beyond what Kafka natively supports.

The Kafka Security Model's Limits

Kafka's native security model has three layers: authentication (SASL or mTLS), authorization (ACLs), and encryption (TLS). This works well for topic-level control but fails for field-level requirements.

Topic-level authorization grants or denies access to entire topics. If a topic contains customer orders with both order details and credit card numbers, ACLs can't grant access to order details while denying access to card numbers. It's all-or-nothing.

No content inspection means Kafka brokers don't parse message contents. A topic might contain PII in 5 of 20 fields, but Kafka can't selectively mask those fields based on the consumer's role. The broker sees bytes, not structured data.

No rate limiting per consumer or application. Kafka has quotas (bytes/sec per client ID), but these are cluster-level protections, not per-topic or per-application limits. You can't enforce "analytics team gets 1000 msg/sec from orders topic while the fraud detection service gets unlimited access."

No schema validation at write time means producers can publish malformed messages that break consumers. Schema Registry validates at the client level, but if a producer bypasses it (or uses an incompatible serializer), invalid messages reach Kafka.

These limitations aren't bugs—they're design decisions. Kafka prioritizes throughput and horizontal scale. Content inspection and field-level access would compromise both. But compliance requirements don't care about Kafka's design constraints.

What a Proxy Layer Adds

A Kafka proxy sits between clients and brokers, intercepting Kafka protocol traffic and applying policies before forwarding messages. Kafka proxies can enforce policies such as record encryption, tenant isolation, or filtering at the event level.

Field-level filtering inspects message payloads and removes sensitive fields based on consumer identity. The analytics team consumes the orders topic, but the proxy strips creditCardNumber and ssn fields before returning messages. The fraud detection team gets full messages, including sensitive fields, because their service account has different permissions.

This works because the proxy understands schema (Avro, Protobuf, JSON) and can parse message structures. Tools like SecuPi provide fine-grained, field or value level access controls, encryption, masking, or filtering based on various user or data attributes.

Data masking replaces sensitive data with anonymized values. Credit card numbers become ---1234, email addresses become u@example.com, names become J D. Consumers see realistic data shapes for testing or analytics without accessing real PII.

Lenses Blog describes how server-side masking ensures PII never reaches the browser, satisfying PCI-DSS and HIPAA requirements. Kpow's data policies enable masking based on key names or patterns.

Rate limiting per consumer or application protects clusters from runaway consumers. The proxy enforces "service-x can consume 1000 msg/sec from topic-y" independently of Kafka's cluster-level quotas. This prevents one misbehaving consumer from degrading cluster performance for others.

Schema enforcement validates messages before they reach Kafka. If a producer tries to publish a message that violates the registered schema, the proxy rejects it with a clear error. This prevents malformed messages from breaking downstream consumers.

Multi-tenancy without separate clusters routes traffic based on virtual topics. Team A and Team B both use a topic named orders, but the proxy routes them to team-a.orders and team-b.orders in the underlying cluster. Clients see simple topic names; the proxy handles namespace isolation.

Use Cases

PII Masking for Analytics: Analysts need to query customer orders for reporting but shouldn't see credit card numbers, Social Security numbers, or home addresses. The proxy allows analysts group to read the orders topic but masks ccNumber, ssn, and address fields before returning messages. Analysts get realistic data for aggregation (order counts, revenue trends) without accessing PII.

Without a proxy, this requires: duplicating the topic with PII removed (data consistency issues), forcing every consumer to filter fields (error-prone, unenforceable), or writing a separate anonymization pipeline (operational overhead).

Multi-Tenancy in Shared Clusters: Instead of running separate Kafka clusters for each team (expensive, operationally complex), use one cluster with proxy-enforced namespace isolation. Team A's producers write to virtual topic orders; the proxy routes it to team-a.orders. Team B's consumers read from orders; the proxy routes to team-b.orders.

Each team sees their own orders topic. The proxy handles physical topic routing and ensures teams can't access each other's data. This reduces infrastructure costs while maintaining isolation guarantees.

Rate Limiting for Protection: A misbehaving consumer (infinite loop, retry storm, misconfigured parallelism) can overwhelm a Kafka cluster by consuming messages faster than brokers can serve them. Kafka quotas limit bytes/sec per client, but they're cluster-wide.

The proxy enforces per-application limits: service-x can consume 10 MB/sec from high-traffic-topic, while service-y gets 1 MB/sec. This protects cluster stability while allowing high-priority consumers to operate at full speed.

Schema Validation at Write Time: Schema Registry validates schemas when producers register them, but producers can bypass validation by using a custom serializer or disabling schema checks. The proxy enforces validation at the wire level: messages that don't match the registered schema are rejected before reaching Kafka.

This prevents invalid messages from breaking consumers. Instead of discovering schema violations when consumers fail, the proxy catches them at write time with actionable error messages.

Cross-Cluster Routing: Organizations running multiple Kafka clusters (different regions, different compliance zones, separate prod/staging) need to route traffic dynamically. The proxy acts as a unified endpoint: producers write to orders, and the proxy routes EU customer data to the EU cluster, US customer data to the US cluster, based on message contents.

This satisfies data residency requirements (GDPR mandates EU data stays in EU) without forcing producers to manage cluster routing logic.

Architecture: Sidecar vs. Centralized Proxy

Proxies deploy in two patterns: sidecar (one proxy instance per producer/consumer) or centralized (shared proxy layer).

Sidecar deployment runs the proxy as a process alongside each producer or consumer. Producers connect to localhost:9092, which is the proxy, which forwards to the real Kafka cluster. This pattern minimizes network hops (proxy is local) and scales naturally (each application brings its own proxy capacity). The downside is operational overhead—every application deployment includes proxy configuration.

Centralized deployment runs the proxy as a shared service. All producers and consumers connect to the proxy cluster, which forwards to Kafka. This centralizes policy management (update policies once, not per application) and reduces client-side complexity (applications just need proxy URL). The downside is the proxy becomes a chokepoint—it must handle aggregate throughput across all applications.

The choice depends on operational constraints. Sidecar fits teams that already run service meshes (Istio, Linkerd) with sidecar patterns. Centralized fits teams that prefer fewer moving parts and centralized policy control. Conduktor Gateway takes the centralized approach, focused on governance, security, and observability with field-level filtering and data masking capabilities.

Performance Considerations

Proxies add latency—the question is how much and whether it's acceptable.

Latency overhead comes from two sources: network hop (client → proxy → broker instead of client → broker) and message processing (parsing, filtering, masking). Network hop adds 1-5ms depending on proximity. Message processing adds 0.5-5ms depending on complexity (simple filtering vs. complex masking).

Total overhead is typically 2-10ms per message. For use cases where end-to-end latency is 100-500ms, this is negligible. For ultra-low-latency use cases (sub-10ms), proxy overhead might be prohibitive.

Throughput impact depends on proxy capacity. A well-configured proxy layer can handle 100K+ messages/second per instance. Scaling horizontally (adding proxy instances) increases aggregate throughput. The proxy should never be the bottleneck—if Kafka can handle 1M msg/sec, the proxy should too.

Resource usage includes CPU (for message parsing and policy evaluation), memory (for buffering and caching), and network (proxy instances need bandwidth for inbound and outbound traffic). Right-sized proxy instances (8 cores, 16GB RAM) handle 50-100K msg/sec comfortably.

Monitor proxy resource usage independently of Kafka cluster metrics. If proxy CPU hits 80% but Kafka brokers are at 40%, the proxy needs scaling, not the cluster.

Policy Examples

Conduktor Gateway uses interceptors—plugins that process requests at the proxy layer. Each interceptor targets a scope (virtual cluster, service account) and applies a specific transformation.

Mask PII fields for analytics team:

apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: mask-pii-for-analytics
  scope:
    vCluster: passthrough
    username: analytics-team
spec:
  pluginClass: io.conduktor.gateway.interceptor.FieldLevelDataMaskingPlugin
  priority: 100
  config:
    schemaRegistryConfig:
      host: http://schema-registry:8081
    policies:
      - name: Mask sensitive fields
        rule:
          type: MASK_ALL
        fields:
          - creditCardNumber
          - ssn
          - email

Rate limit per application:

apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: throttle-batch-processor
  scope:
    username: batch-processor
spec:
  pluginClass: io.conduktor.gateway.interceptor.safeguard.ConsumerRateLimitingPolicyPlugin
  priority: 100
  config:
    maximumBytesPerSecond: 5242880  # 5 MB/sec

Enforce schema validation:

apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: enforce-schema-validation
  scope:
    vCluster: production
spec:
  pluginClass: io.conduktor.gateway.interceptor.safeguard.SchemaPayloadValidationPolicyPlugin
  priority: 100
  config:
    topic: ".*"
    schemaIdRequired: true
    validateSchema: true
    schemaRegistryConfig:
      host: http://schema-registry:8081

These interceptors are defined as YAML resources and applied via Console, CLI, or Terraform. Interceptors can be chained by priority: validate schema first (priority 10), then mask PII (priority 20), then enforce rate limits (priority 30).

Enforcement and Auditability

Proxies generate audit trails for policy enforcement. Every message masked, every request throttled, every schema violation—all logged with principal, timestamp, and policy ID.

This satisfies compliance requirements: "prove that PII was masked for non-privileged consumers" becomes a query against proxy audit logs. Kafka itself doesn't log content-level operations (it can't—it doesn't inspect payloads), but the proxy does.

Audit logs answer: who accessed what topic? What fields were masked? When did rate limiting activate? Which messages were rejected for schema violations?

The Path Forward

Kafka proxies fill governance gaps that ACLs can't address. Field-level filtering, data masking, rate limiting, schema enforcement, and multi-tenancy enable compliance-heavy organizations to use Kafka without sacrificing data protection.

Conduktor Gateway provides proxy-layer governance with field-level encryption, data masking, and policy enforcement. Organizations use Gateway to satisfy PCI-DSS, HIPAA, and GDPR requirements while giving developers self-service access to data through automated guardrails.

If your governance requirement is "analysts can access orders but not credit card numbers," and your current answer is "duplicate the topic with PII removed," the problem isn't Kafka—it's the lack of a proxy layer.

---

Related: Conduktor Gateway → · Kafka Encryption → · Data Sharing Without Replication →