Coming from Apache Kafka Series -
Learn Apache Kafka for Beginners v3?
Keep Reading! This is important as we've removed Conduktor Playground, you need to change your playground.config to follow the course without interruption.
Conduktor Free Kafka Playground has shut down.
We are sorry to announce that our Free Kafka Playground has been removed. This was used to get access to a free Kafka available in seconds and run some tests, without having to start Kafka on your laptop.
But fear not, there are free alternatives out there from Kafka providers. This page details two options for you. Option 1 involves running a Kafka cluster & Conduktor locally on your laptop. This is as simple as one line command if/once you have Docker installed on your machine.
Option 2 involves connecting to a free cloud provider, aiven.io, but requires a few more steps as it is more secure due to being publicly hosted on the internet. The steps are detailed below but please bear in mind these are subject to change by the provider.
Note: If you are already quite comfortable with Kafka, you can use any other alternative to get a Kafka cluster such as: Confluent, Aiven, AWS MSK, Strimzi, Redpanda and skip this guide.
To summarize the two options:
Option 1: Run a local Kafka cluster on your machine and Conduktor UI
Option 2: Aiven will provide you a Free trial of a cloud Kafka cluster
You can keep using Conduktor UI to learn and view everything about Kafka.
Let's explore what to do!
Option 1: Run a local Kafka cluster
1) Run Conduktor (and Kafka) with Docker
Follow the course to install Conduktor UI with Docker (lecture #57) or directly follow the instructions on our website here: Get Started with Conduktor. You will need to install Docker as a prerequisite.
Note: Running the Conduktor get-started to spin up a local Kafka will be much quicker and simpler to connect to than the steps detailed in Option 2 below. Option 2 involves setting up a cloud cluster and it's connection using SSL.
2) Connect to the local Kafka cluster
After successfully running the command for the Get Started you will have a local Kafka cluster running inside Docker, which can be accessed on localhost:19092. As this cluster is setup for no security, you should be able to interact without any additional configuration.
You should see returned a list of topics created by the rest of the services as part of the Conduktor Get Started stack. Return to the course.
3) Destroy the local stack
Once you are finished using this local Kafka cluster run the below command in the same directory as the docker-compose.yml file you downloaded. Note this will destroy the topic data.
Option 2: Create and connect to a cloud Kafka cluster
1) Create your Aiven account
Go to https://console.aiven.io/signup register and verify your account.
2) Create your free Kafka cluster on Aiven
Once registered you can create your free Kafka cluster in seconds, choose the Apache Kafka service to begin creating a cluster.
Choose the default options for creation and you'll be greeted with the configuration screen. Change the authentication method to SASL, this toggle is towards the bottom of the screen. Enable SASL.
In the next step, we'll see how to connect Conduktor and other tools to this free Kafka cluster.
3) Create the kafka-client.config file
The Aiven cluster is secured using SSL, so requires some more advanced configuration. To connect, copy & paste the details from Aiven into your favourite text editor, save this as a new file kafka-client.config. Reference this file in your commands as in the example below. The Service URI is the bootstrap-server.
4) Create the certificate.pem file
For the Aiven cluster you'll also need a certificate as part of the connection. In the connection details find the CA certificate, and copy out this value, save this to a file, certificate.pem. Note this is what is referenced in the previous config step.
5) Connect to the cloud Kafka cluster
Use the service-uri as the bootstrap server you can now query the Kafka cluster, including the command configuration you have saved earlier.
6) Connect Conduktor to Local
You now have Conduktor running locally. You have a free Kafka cluster. Time to connect both! Add your cluster configuration to Conduktor running on: http://localhost:8080/settings/clusters.
Copy/paste your values into the appropriate fields or follow Conduktor's guide.
Test the connection and it should work! You're ready to use Conduktor with Aiven.
You're ready to continue using Conduktor!
To summarize, we have updated how to connect Conduktor and the CLI to another Kafka cluster, be that running locally or on Aiven's cloud.
If some steps are unclear, please tell us: support@conduktor.io. Thanks!
Happy learning :)