Deploying Apache Kafka on AWS is very easy, thanks to AWS MSK (Amazon Managed Streaming for Apache Kafka — Amazon MSK). You get a production-ready, self-managed Kafka cluster in a few clicks on which you can run your applications, create data streams and manage connectors.

Conduktor's mission is to provide simple, flexible and powerful tooling for Kafka developers and infrastructure. Our product is composed of two parts: a UI dedicated to developers efficiency and Conduktor Gateway, a Kafka proxy to embrace DevSecOps needs in organizations (enforcing best-practices, end-to-end encryption auditability, multi-tenancy, seamless failover, and much more).

Conduktor is now available on the AWS Marketplace. You can deploy it in a few clicks and connect it with your AWS MSK cluster. Within moments, you will be able to view and manage your Kafka data, perform Kafka operations, troubleshoot connectors, consumer groups and more.

AWS provides an exceptional managed Kafka platform, while Conduktor provides the best tooling to work with Kafka! What if we combine both?

Late 2022, we showed how to deploy Conduktor on AWS using CloudFormation to gain visibility on your Amazon MSK, and solve Kafka issues around monitoring, governance and data security. It's time for some update, as we have a lot of new things to share!

Conduktor: How to deploy it?

At Conduktor, we value simplicity, quick feedback loops, and an easily deployable solution. Conduktor is distributed as a Docker container. It can run on everything supporting Docker on AWS:

• ECS

• EKS

• Fargate

• or a single EC2 instance

We recommend installing it inside of your VPC that also hosts AWS MSK (Apache Kafka), this way you avoid any connectivity and security issues. You just need to expose one port for HTTP (:8080) and be sure Conduktor can connect to your Kafka clusters (:9092).

You can also simply start Conduktor on your laptop with an embedded Kafka to get a first grasp, thanks to this one-liner:

curl -L https://releases.conduktor.io/quick-start -o docker-compose.yml && docker compose up

Or directly connect to your AWS MSK clusters (no need for an embedded Kafka in this case) with this simplified docker-compose version:

curl -L https://releases.conduktor.io/console -o docker-compose.yml && docker compose up

After your evaluation, you can deploy Conduktor more globally on your AWS infrastructure to make it self-managed and available for all of your team. The real value of Conduktor comes from a centralized deployment that all your developers and join and collaborate together.

Next, we'll see how to start Conduktor using dedicated AWS infrastructure and connect to AWS MSK securely from anywhere.

AWS MSK: An enterprise-grade Kafka streaming platform

AWS MSK clusters come in two types: provisioned or serverless.

• Provisioned: allows you to specify the number of brokers and the amount of storage per broker.

• MSK Serverless: is recommended in cases where the throughput requirements of client applications are variable and hard to predict. MSK Serverless scales cluster capacity automatically in response to throughput needs.

When creating AWS MSK clusters, AWS automatically deploys Kafka broker to different and isolated Availability Zones (AZs) to provide high availability (in case a Zone fails, see here for the latest outages).

AWS MSK comes with everything you need to do efficient data streaming:

• Auto-scaling Kafka clusters: to suit your business needs automatically

• AWS Glue Schema Registry: to manage your schemas

• MSK Connect: to connect AWS MSK to your data sources and sinks easily (S3, OpenSearch, DynamoDB, Redshift, etc.)

Obviously, everything is secured by AWS IAM and VPCs which are the best in class to ensure your resources are secured.

We are working closely with AWS to ensure we support all their technologies and use-cases.

Conduktor supports AWS MSK

Conduktor is compatible with your AWS MSK ecosystem:

• MSK Dedicated or Serverless clusters

• IAM (authentication and authorization)

• AWS Glue Schema Registry

• MSK Connect: it's on our roadmap! Give us feedback if you need this!

Conduktor needs a database to save its configuration, and can rely on Amazon RDS. The Amazon RDS Proxy support was introduced in 1.17.1. Note that the PostgreSQL engines within RDS should be either 14.8 or 15.3 as other versions are not fully supported.

Connect Conduktor to AWS MSK using IAM

To connect Conduktor to your AWS MSK clusters, you can create dedicated AWS access keys as shown below or rely on IAM roles inherited from the host where Conduktor is deployed (ECS, EC2, etc.).

Using dedicated AWS access keys

In your Conduktor cluster configuration, select 'I will set the IAM credentials myself' to use the dedicated AWS access keys you've created.

Conduktor will automatically use the right SASL mechanism (AWS_MSK_IAM) and the AWS IAM module (IAMLoginModule) to enable the use of AWS Identity and Access Management (IAM) when connecting to Amazon MSK clusters. For more details and configuration (alternative profiles, roles, ...), please refer to the documentation.

Using IAM roles

You can ask Conduktor to rely on the authorization provided by the environment itself: select 'Inherited IAM credentials from the environment' in your cluster configuration. Conduktor will automatically use the standard AWS Default Credentials Provider Chain to find the right IAM role to use.

Connect Conduktor to AWS Glue

Conduktor is one of the only products to fully support AWS Glue (the Schema Registry used to store and retrieve schemas) with the same AWS IAM strategies as connecting to an MSK cluster.

Note that AWS Glue has an API entirely different from the official Confluent Schema Registry (which is also fully supported by Conduktor) and does not support all the same features (no schema strategy like TopicNameStrategy).

In your cluster configuration, select AWS Glue, pick your region and registry name, and select the right IAM strategy (dedicated keys or inherited from the environment).

Connecting to your AWS MSK securely from anywhere

Large organizations may use several Cloud providers and multi-regions/zones to ensure the continuity of their operation in case of technical issues (like when a Cloud zone is down for hours). This flexibility creates a more complex networking topology and adds more work for users and applications to communicate seamlessly across various networks (VPC), while ensuring the whole security, compliance, and traceability of the data is in check.

Conduktor Gateway is a Kafka proxy: it's a small artifact that can be deployed anywhere, across VPC networks, and make the bridge between various parts of the business.

It can take care of doing simple or complex Kafka routing: sending the traffic to the right cluster/topic and passing the traffic from the clusters to the users/applications. See how to build Virtual Clusters and Federate multiple Kafka clusters. All of this can be configured at runtime without impacting any applications. If your traffic has multiple hops to traverse your networking layout, Conduktor Gateway is your Kafka router through all of this. Think of it as the nginx for the Kafka protocol (and not HTTP).

In the end, Conduktor Gateway is a layer you can put on top of your AWS MSK to enforce security, data encryption, and safeguarding rules at scale and across multiple clusters. It can also drive cost reductions and introduce a mechanism for managing failovers without without impacting your applications and your infrastructure decisions.

It's seamless for your developers (nothing to change) and as an Ops, you have the flexibility to update your infrastructure. You can use AWS MSK to manage your streaming infrastructure, and Conduktor Gateway to connect to your Kafka clusters from anywhere, including your on-premise datacenter, or your other Cloud providers.

Conduktor on AWS Marketplace

We are happy to announce that we are now present on the AWS marketplace: https://aws.amazon.com/marketplace/pp/prodview-vbf5jztb7cwlc.

This is the first step of a more global partnership between Conduktor and AWS to support all the AWS technologies and use-cases.

• Go to the AWS marketplace and 'Subscribe' to Conduktor

• Accept terms

• Configure your fulfillment

Continue, and you are ready to install!

AWS provides the command line to install Conduktor from ECR:

$ aws ecr get-login-password --region us-east-1 | docker login \
    --username AWS \
    --password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com
Login Succeeded
$ docker pull 709825985650.dkr.ecr.us-east-1.amazonaws.com/conduktor/conduktor-selfhosted:1.17.1
1.17.1: Pulling from conduktor/conduktor-selfhosted

You can list all the tags available:

$ aws ecr describe-images --registry-id 709825985650 --repository-name conduktor/conduktor-selfhosted --region us-east-1 | jq -r '.imageDetails[].imageTags[0]'
1.17.1-arm64
1.17.1-amd64
1.17.1
...

Finally, because it's a simple Docker container, you can use these images to run Conduktor wherever you want (EC2, ECS, EKS, ...) safely in your AWS ecosystem, or just on your laptop:

$ docker run -p 8080:8080 709825985650.dkr.ecr.us-east-1.amazonaws.com/conduktor/conduktor-selfhosted:1.17.1
2023-08-03T07:11:01Z [entrypoint] INFO - Welcome to Conduktor Platform

We provide a guide to deploying Conduktor on AWS using CloudFormation: Getting Started on AWS.

What's next?

We're excited to work with AWS MSK to support all their use-cases and be intimately linked to the AWS ecosystem.

Our integration on the AWS Marketplace is the first step of a more global partnership with AWS to make sure we remove all the frictions to deploy Conduktor and connect to AWS MSK. We will also continue improving our integration with AWS MSK ecosystem to support all their latest features (like MSK Connect) and make them easily accessible in our products.

AWS MSK cares a lot about the experience of its users. It has to be frictionless and helpful to lead customers to focus more on their business and spend less time on the technical pitfalls. We share the same vision so we're excited to work with AWS to offer the same experience to all our users.