Terraform automates the provisioning and management of resources via infrastructure-as-code (IaC). As a multi-cloud, open source technology, Terraform is one of the biggest IaC platforms today, with 4,000+ integrations (or providers in Terraform parlance), over 250 partners, and a thriving community of contributors and committers. 

As environments became more dynamic and complex with a greater degree of interdependent services, Terraform became the preferred solution for standardizing and automating infrastructure across organizations and clouds. With Terraform, teams could create repeatable processes as declarations—essentially specifying the desired end state without having to learn different toolsets or even to outline step-by-step directions. 

Terraform needs Conduktor to control Kafka

Apache Kafka is powerful and versatile, but lacks advanced governance and security features. Moreover, massive Kafka deployments can be complicated to manage and configure, requiring lots of manual work, custom scripts, and sprawling configs. The problem is compounded if a Kafka environment is distributed across multiple departments, projects, and teams.

In addition, manually implementing governance across large Kafka environments will introduce inconsistencies and additional overhead. If a single administrator is tasked with managing and governing hundreds of topics across multiple Kafka clusters (and hundreds to thousands of Kafka applications), they will be a bottleneck for Kafka operations. In particular, onboarding times for new Kafka resources will increase drastically, severely impacting developer productivity.

As a result, teams integrate an enterprise data management solution like Conduktor directly into their Terraform automation pipelines. This way, teams can enforce security policies, implement access controls, and enable self-service abilities through Conduktor—while templatizing and automating these procedures through Terraform. 

At the same time, using these two technologies in tandem also removes the need for platform engineers to learn a new infrastructure solution from scratch, and reduces the time required for Conduktor’s time to value. More importantly, it also provides an easy way to standardize guidelines and best practices across all Kafka clusters, regardless of cloud, owner, or project. 

Help developers move fast—without breaking things

Platform teams face a difficult dilemma: how can they empower developers to move quickly and boost innovation—without sacrificing governance, security, or consistency? The answer comes down to creating (and automating) repeatable processes across a diverse Kafka environment. 

By pairing Terraform with Conduktor, platform teams can codify policies, define access controls, and templatize user provisioning as code—thus preventing configuration drift. Conduktor and Terraform will also log any changes, in case they need to be reviewed, reverted, or audited at a later date. 

Platform teams can also implement self-service features. First, platform teams can codify preset guardrails for security and access policies, enabling developers to onboard themselves, provision resources, and ship faster, all without having to file tickets or wait on manual approvals. This frees platform teams from tedious, administrative tasks and accelerates developer productivity—while simultaneously preserving security and stability. 

Ultimately, platform teams become enablers, rather than gatekeepers, while developer teams can move faster and more confidently. 

Conduktor x Terraform in the real world

With their Conduktor provider, platform teams can codify Conduktor resources, such as access policies, roles, clusters, and interceptors, and enforce this state across their entire Kafka environment. This brings predictability and uniformity to Kafka operations, and automates critical workflows. It also removes the need for platform teams to manually execute actions like onboarding, monitoring, or provisioning.

Here’s an example: one major retailer’s platform teams were overwhelmed with requests for onboarding, resource provisioning, and access configuration. Without Conduktor, they lacked an effective process, instead relying on ticket-based workflows and custom scripts to provision every single ACL, topic, and group assignment. 

As Kafka usage grew to a high of nearly 300 users, this approach couldn’t scale—and instead created long wait times for new users and resources, complicated workflows, and monitoring and troubleshooting issues. Developers were blocked by unclear procedures and limited visibility, while platform teams struggled with consistency, security, and compliance across environments. 

By using Conduktor’s Terraform provider, this retailer can now standardize security across Kafka clusters, such as for sharing encrypted data with customers or external partners. Platform teams can also define roles, groups, and policies, ensuring that users can access only the data they are authorized for, and implementing these boundaries across entire organizations. 

resource "conduktor_console_group_v2" "project-group" {

  name = "project-group"

  spec = {

    display_name    = "project-group"

    members         = [conduktor_console_user_v2.user1.name]

    permissions = [

      {

        permissions   = ["userView", "datamaskingView", "auditLogView"]

        resource_type = "TOPIC"

        permissions   = ["topicViewConfig", "topicConsume", "topicProduce"]

      }

    ]

  }

}

Lastly, combining Conduktor with Terraform also lessens operational burdens. Now, the platform team is no longer a gatekeeper for even minor operations like onboarding users to existing teams or provisioning standard resources that adhere to policies. Instead, application teams can now act autonomously but within the parameters set by the platform team, developers can now move faster and more efficiently. 

Get started with Conduktor and Terraform

Large-scale Kafka infrastructure can be difficult to manage, requiring manual work across multiple teams; creating overhead for developers and platform administrators alike; and causing vulnerabilities in policies and management.

Terraform’s Conduktor provider enforces security, governance, and control across these distributed, massive Kafka environments. Platform teams can predefine permissions and automate configurations for ease of mind and standardization; developers can autonomously provision their own resources and accounts; and all sides can rest easy knowing they won’t risk introducing new inconsistencies or issues into their Kafka infrastructure.

Bring order to your Kafka environments today with the Conduktor provider. Visit the Conduktor’s documentation pages on Terraform to learn more.