Kafka Schema Registry: How It Works & Why You Need It

Stéphane Derosiaux November 24, 2022 9 min read

Ask about this post

Isometric wireframe line-art on dark teal: a low library card-catalogue cabinet with three rows of small drawers. One drawer is pulled open and lit lime from inside, and blank index cards fly out of it to the right along dashed trails.

Ask about this post

Kafka lets teams share data without meeting. The checkout team writes orders to a topic, and the warehouse, billing and fraud teams each read them on their own release schedule.

That independence is also how a field renamed in checkout can stop three services run by people the checkout team has never met. A schema registry is the contract those teams never sat down to write.

It records the shape of every message, checks each change against rules the topic's owner picks, and lets consumers read records written months earlier. It protects a topic only while every client uses it, and by default any caller can switch its checks off.

Kafka stores only bytes. The prefix points to a schema in the registry, and a consumer needs that schema to turn the rest back into a record.

Why a Schema Registry Exists

Kafka treats a record's key and value as byte arrays. Serialization happens in the client: the producer turns an object into bytes and the consumer turns them back.

The Apache Kafka docs call leaving the key and value opaque the right decision, because no single format suits every application. When a producer changes a field type, the broker stores the new bytes like the old ones.

Producers and consumers ship on their own schedules, often from different teams. A consumer learns about a change when it can't decode a record: a poison pill that stops it until someone skips the record.

The two older ways to share a contract both cost something:

Embed the full schema in every record. Every record grows by the size of its schema.
Share generated classes between teams. Their releases are tied together.

A registry is an HTTP service that stores schemas, gives each one a numeric ID and keeps a version history per name. Other registries implement Confluent Schema Registry's REST API, so Confluent's serializers work against them too.

Confluent Schema Registry keeps its state in _schemas, a single-partition compacted Kafka topic:

The producer and the consumer each call the registry. The broker stores _schemas like any other topic and never looks up a schema.

Wire Format and Schema IDs

An Order record, read back from orders and printed in hex, on Confluent Schema Registry 8.4.0 and Apache Kafka 4.3.1:

$ kcat -b kafka:9092 -C -t orders -o beginning -c 1 -e -f '%s' | xxd
00000000: 0000 0000 020a 6f72 642d 3100 0000 0000  ......ord-1.....
00000010: 4045 4006 4555 52                        @E@.EUR

The first byte, the magic byte, is a format version: 0. The next four hold the schema ID in big-endian order, then comes plain Avro binary, which carries values and no field names:

Five bytes point to the contract. The payload decodes only with schema 2.

Confluent documents this wire format, plus one addition for Protobuf: a list of message indexes sits between the ID and the data.

Since Confluent Platform 8.1.1, a serializer can write a 16-byte schema GUID to a __value_schema_id header instead and leave the payload unprefixed. The deserializer reads the header first and falls back to the prefix.

Subject Naming Strategies

A subject is a named version history: the registry assigns versions and checks compatibility within one subject. The serializer picks the subject with a naming strategy. The default, TopicNameStrategy, files the value schema of orders under orders-value and the key schema under orders-key.

Confluent's serializers ship three strategies. For an Order record (com.example.Order) on the orders topic:

Strategy	Subject	Fits	Consequence
`TopicNameStrategy` (default)	`orders-value`	One record type per topic	Two topics with the same record keep two histories
`RecordNameStrategy`	`com.example.Order`	Several event types per topic	One history for every topic that carries `Order`
`TopicRecordNameStrategy`	`orders-com.example.Order`	Several event types per topic	One history per topic and record type

Keep the default unless a topic carries several event types. Then use TopicRecordNameStrategy: under RecordNameStrategy, a version registered for one topic becomes the latest Order for every other topic that carries it.

Schema Compatibility Modes

Each subject has a compatibility mode, set globally or per subject. When a client registers a version, the registry compares it with the latest one and refuses it if the mode's reading guarantee doesn't hold.

The three _TRANSITIVE modes compare the new version with every earlier version instead. The modes and the Avro changes each accepts, from Confluent's compatibility reference:

Mode	Guarantee	Avro changes allowed	Upgrade first
`BACKWARD` (default), `BACKWARD_TRANSITIVE`	The new schema reads data written with the old one	Delete fields, add fields with a default	Consumers
`FORWARD`, `FORWARD_TRANSITIVE`	The old schema reads data written with the new one	Add fields, delete fields with a default	Producers
`FULL`, `FULL_TRANSITIVE`	Both directions	Add or delete fields with a default	Either, in any order
`NONE`	None	Anything	All at once, or a new topic

Confluent gives the reason for the default:

"The main reason that BACKWARD compatibility mode is the default […] is so that you can rewind consumers to the beginning of the topic."
Confluent, Schema Evolution and Compatibility

BACKWARD compares a new version with the latest one only. A consumer that rewinds across several versions needs BACKWARD_TRANSITIVE (or FULL_TRANSITIVE), so set it on topics you replay from the start.

Here version 2 of Order adds currency with a default of "EUR", and version 3 adds customerId without a default:

$ curl -s -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
    --data @order-v2.json http://schema-registry:8081/subjects/orders-value/versions
{"id":2,"version":2,"guid":"d1c06f8f-ad67-4a58-16be-be236d7aff86","schemaType":"AVRO","schema":"…"}

$ curl -s -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
    --data @order-v3.json http://schema-registry:8081/subjects/orders-value/versions
{"error_code":40901,"message":"Schema being registered is incompatible with an earlier schema for subject \"orders-value\",
details: [{errorType:\"READER_FIELD_MISSING_DEFAULT_VALUE\", … {validateFields: 'false', compatibility: 'BACKWARD'}]"}

A consumer on version 3 would have no value for customerId in records written with version 2, so the registry answers 409 and keeps version 2 as the latest.

Limits of a Schema Registry

A record reaches the broker the same way whether a registry was involved or not. Say a second producer writes plain JSON text to orders, here with the console producer. The broker stores the record at offset 1, and the Avro consumer stops on it:

$ echo '{"id":"ord-2","amount":"12.00"}' | kafka-console-producer.sh \
    --bootstrap-server kafka:9092 --topic orders
$ kafka-avro-console-consumer --bootstrap-server kafka:9092 --topic orders --from-beginning \
    --formatter-property schema.registry.url=http://schema-registry:8081
{"id":"ord-1","amount":42.5,"currency":"EUR"}
org.apache.kafka.common.errors.SerializationException: Error deserializing schema ID
Caused by: java.lang.IllegalArgumentException: Unknown magic byte!

The producer's choice of serializer decided which record carries the contract, and the broker accepted both:

The broker stores what each producer's serializer wrote. The consumer finds out at read time.

🚫 "We run a schema registry, so every record on our topics matches a schema."

The registry checked the schemas it was sent, and offset 1 never went near it. Apache Kafka has no setting that makes the broker check.

Some commercial Kafka distributions validate schema IDs on the broker as a licensed feature, which confirms that an ID is registered and leaves the payload unchecked.

Each check runs in one place, and each has a way around it:

Check	Runs in	What bypasses it
The record carries a registered schema ID	The producer's serializer	Any other serializer
A new version is compatible	The registry, at registration	Any caller setting the mode to `NONE`
Someone reviewed the new version	Nowhere, by default	Auto-registration, on by default

Securing the Registry API

Confluent documents http://0.0.0.0:8081 as the default listener and notes that, by default, clients make REST calls over HTTP. With no authentication configured, any caller that reaches the port can switch a subject's check off and register the version 3 that BACKWARD refused:

$ curl -s -X PUT -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
    --data '{"compatibility":"NONE"}' http://schema-registry:8081/config/orders-value
{"compatibility":"NONE"}

$ curl -s -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
    --data @order-v3.json http://schema-registry:8081/subjects/orders-value/versions
{"id":3,"version":3,"guid":"2c6ac447-5b65-40c8-4dbe-3547fb0e4712","schemaType":"AVRO","schema":"…"}

The requirement, from a team that runs a shared registry:

"It's not okay for anyone to delete… or modify schemas of others."
Kafka competence lead at an energy company

Each registry adds authentication and per-subject rules its own way, through a plugin or a config file. Kafka Schema Registry: Open by Default compares the options, including an HTTP proxy in front of the registry.

Register Schemas From CI

Confluent's serializers and Kafka Connect converters register schemas by default (auto.register.schemas). The first producer to send a new shape creates the next version in production, if it passes the compatibility check.

That version becomes the baseline for every later change. Move registration out of the producers:

Turn off auto-registration. With auto.register.schemas=false, an unregistered schema fails with error code: 40403 instead of becoming a new version. Add use.latest.version=true.
Check and register in the pipeline. A CI step posts the schema to /compatibility/subjects/orders-value/versions/latest, fails on "is_compatible":false, and registers it after the merge.
Restrict who can write. Authentication on the listener plus per-subject write rules keep /config, deletes and registrations with each subject's owner.

Enforcing Schemas Centrally

Two Conduktor components enforce schemas for every client:

Schema Registry Proxy. Sits in front of the registry and checks Read or Write on the subject for every request, derived from your topic ACLs (docs).
Gateway. Can reject produce requests whose records lack a registered schema ID or don't match their schema (docs).

In Console, each subject shows its versions, its compatibility mode and a diff between any two versions:

Subject orders-value in Conduktor Console: current version 3 of 3, format Avro, compatibility BACKWARD, with a diff of version 3 against version 2 highlighting the added channel field with its default value web — The orders-value subject in Console, comparing version 3 with version 2: the new field carries a default, which BACKWARD requires.

A schema registry stores each topic's contract and checks each new version before accepting it. The broker never reads that contract, so the contract holds only if a client, a proxy or a gateway checks records on the way in.

This matters more once a topic's producers include clients the owning team didn't write. Say an AI agent gets a tool that produces to orders:

The tool uses a registry serializer with auto-registration on. The agent's first new shape becomes the next version of orders-value with nobody reviewing it.
The tool doesn't use one. Its records skip the registry and stop every Avro consumer that reads them.

The consuming teams will ask for a check that producers can't skip, and the security team will ask for every new version to carry its owner's approval.

Talk to us about schema governance on your clusters →