Set Up a Kafka Platform with an AI Agent

Spinning up a local Conduktor environment with Kafka

I started from nothing. Docker running, no Kafka, no Conduktor, no config files. I wanted a local environment where I could learn Kafka, test things, and break stuff without consequences.

I told the agent: "install Conduktor and set it up so I can login."

The agent checked my environment, asked what I was trying to do, and made every decision from there. It hit an error along the way, self-corrected, and handed me a working platform. A quickstart guide would have gotten me here too, but I would have needed to understand what I was installing. The agent didn't ask me to.

What platform engineers get

Having Kafka running isn't the hard part. The hard part is making it safe for a team to use without constant supervision. Naming conventions, ownership boundaries, policies. The stuff that prevents a shared cluster from turning into a junkyard.

I told the agent: "set up governance for two teams, Payments and Analytics, with topic policies and cross-team permissions."

It worked through the problem in stages, figuring out dependency ordering on its own. When it hit errors, it read the rejection, restructured the YAML, and retried. Here's what it built:

"How do I enforce naming conventions?" / "What about retention and required labels?"

The agent generated two TopicPolicies: one that locks down naming per team, and one that enforces safe defaults (retention, replication, required labels) across all topics. It picked replication factor 1 based on the single-broker setup.

"How do I set up ownership and cross-team permissions?"

The agent created two Applications with non-overlapping resource boundaries, five topics with descriptions and labels in the Topic Catalog, and one cross-team permission granting Analytics read access to payments.orders.*.

Thirteen resources total in one conversation. Here's the full session — the agent working through dependency ordering, hitting errors, self-correcting, and applying:

This is federated ownership in practice: the platform team defines the boundaries, developers operate freely within them. The kind of knowledge that takes months to accumulate and lives in one person's head. Now it lives in a skill file.

What developers get

I set up the guardrails. Now what happens when a developer on the Payments team installs the same skill and starts asking questions?

They don't need to know any of what I just did. They don't need to know about ApplicationInstances or TopicPolicies or YAML syntax. They just talk to their agent. Here's what that looks like:

Let's break that down.

"What topics do we have?"

The agent runs conduktor get Topic and shows them the catalog: descriptions, ownership, labels, visibility. They can see that payments.orders.created exists, who owns it, what's in it, and that it's public.

Without the skill, this question goes to Slack. "Hey, does anyone know if we have a topic for order events?" Three people respond with three different answers. Someone links a wiki page from six months ago that's half-wrong.

"I need a topic for my service."

The agent checks their ApplicationInstance, reads the TopicPolicy constraints, and knows:

• Naming prefix is payments.*
• Name must match payments..
• Retention between one and seven days
• data-criticality label required

It asks: "What's the topic for? What data criticality?" Then it generates the YAML, validates it against the policies, runs a dry-run, and applies.

The developer never saw the policy. They just got a topic that's compliant by default.

Topic/payments.fulfillment.shipped: Created

Without the skill, this is a ticket. Or a Slack message. Or thirty minutes of reading docs to figure out the naming convention, the required labels, and the right YAML structure.

"How do I produce to this topic from my app?"

Earlier I used the CLI to test. Now the developer needs production code. The agent reads the cluster config, gets the bootstrap server, and generates a working Python producer with real connection details:

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "localhost:19092",
})

producer.produce("payments.fulfillment.shipped",
    key="ord-123",
    value='{"orderId": "ord-123", "status": "shipped"}')
producer.flush()

Copy. Paste. Run. Data flows.

"I need to read from the analytics team's clickstream data."

The agent discovers that analytics.clickstream.pageviews is owned by the Analytics team's ApplicationInstance. It generates a permission granting read access, scoped to that specific topic, with read-only at both the Kafka and Console layers.

The developer doesn't know what an ACL is. They don't know what patternType: LITERAL means. They asked a question in English and got access to data they needed.

Without the skill, this is an email chain. Developer asks their lead, lead asks the Analytics team, someone writes the permission YAML, someone applies it, someone tells the developer it's done. Three days if everyone's responsive.

What else can the skill do?

This post showed governance setup and developer onboarding. But the skill covers the full platform: Gateway encryption, data quality rules, Terraform export, and CI/CD scaffolding.

Try it. One install, same conversation whether you're a platform engineer or a developer.

npx skills add conduktor/skills

Contribute. The skill is open source. If you find a use case that's missing or a workflow that could be better, open a PR.

New to Conduktor? The Community Edition is free and self-hosted. The skill handles the install for you.