There is a good reason for Apache Kafka's rapid spread around the world's major organizations. It is a great system, flexible and powerful enough to handle various real-time data needs. However, the wide usage of Kafka in organizations will also introduce problems related to security and governance. Topic as a Service for the Conduktor Platform was built to help solve these issues.

Dashboard for topic as a service

Why Do You Need Topic as a Service?

When starting something new, most don't take the time to check everything. Few people read terms & conditions or do test runs. People want to get started. So it is with Apache Kafka; deciding on naming conventions or figuring out the best topic configuration will slow you down, and you can always look at them later. Not a problem when you're getting started, but as Kafka scales, it will become important.

At some point, organizations will want to combine their different Kafka flavors and teams. All of the different approaches will not work together without a serious amount of effort to integrate them. Soon enough, you are spending more time building and maintaining these integrations than you are actually working with Kafka!

There is one potential solution to this: just get Ops to handle the foundational topic processes. However, given how much of Kafka is built on topics, your Ops will change from occasional gatekeepers to permanent topic robots, doing nothing but approving requests. It's fair to say that this is not the best use of your skilled employees.

What Topic as a Service Can Bring

The appeal of Topic as a Service is thus quite simple: it becomes your Kafka gatekeeper, standardizing topic access and creation right from the start. It's an effortless approach to ensuring that disparate teams can create, share and control Kafka resources.

Let's take a look at how Topic as a Service can simplify topic ownership and access. In the clip below, the "Wikipedia" application takes ownership of any topic prefixed with "wikipedia", auto-generating ACLs for each topic. If any other applications need access to these topics, all that is needed is to send a request and have it granted:

Upcoming Features

• Automated Topic Creation

  • Applications and Teams created in Topic as a Service will be able to define naming schemes and topic configurations, allowing authorized users to instantly create their own topics with full approval of application owners

• Topic Creation bound to Identity

  • With Topic as a Service, all topic creation is connected back to the individual user's identity, allowing you to understand the flow of requests, who created what, who asked for what, and how it is all being shared

How do I get Started

Topic as a Service is available to Enterprise subscribers for the Conduktor Platform. Before you start creating applications, there are two steps you'll need to take:

• Create your environments (Prod, Staging, etc.) and assign them to clusters. This is done from within Topic as a Service.

• Create the teams that will own each application. Teams can be created during application setup, but you will need to go to the Admin panel to assign users to each team.

Summary

Topic as a Service is one of the simpler solutions available in the Conduktor Platform, but it is no less important for that. The very best advances in productivity have never been highly complicated; instead, they take what used to be tedious human work and automate it, freeing up time to focus on other, more important things. So it is with Topic as a Service, giving time back to your teams and letting them work on what matters, instead of worrying about procedures and rules.