Here we are, one week after the Kafka Summit conference that took place in London. It's a wonderful event where the whole Kafka community gathers and discusses the latest evolution in the domain.

As a company leading the Kafka tooling space, we were in the place fully present, enthusiastic, motivated, and ready to engage with you to talk about our latest developments and what's next.

Typically, in a blog like this, we would praise how extraordinary the conferences were and so on. However, we couldn't attest to their greatness because we didn't attend ... a single one!

That being said, our Tom had the opportunity to run a session where he talked about Safeguarding your Kafka Infrastructure in a proactive manner! Definitely a must-watch.

Booth on fire 🔥

The reason behind our absence was that enthusiastic visitors constantly besieged our stand: that's what we expected!

The whole team spent the entire exhibition engaging in non-stop conversations. It was an immense pleasure to showcase our achievements, but more importantly, we learned a great deal. We gained insights into the current market state, the status of ongoing projects, and, most notably, the topics that sparked the strongest reactions from all of you. And that brings us to the purpose of this blog.

Let us present some of the most existing features we talked with our users at length.

End to End Encryption

Of all our features, this one undoubtedly captured the most interest.

Field-level encryption is a concrete problem, widely recognized, and lacks satisfactory solutions. GPDR, PII, HIPAA, and other regulator things are no joke. Being able to encrypt sensitive data while seamlessly integrating this feature into your existing ecosystem with zero change to the code in your applications (meaning regardless of the programming language, version, or operating system), was a particularly successful achievement we're proud of. 🎯

Documentation on our Gateway Marketplace!

See below for the demo!

Multi-tenancy

Multi-tenancy is another feature that received an overwhelming praise at our booth. To be honest, this is something we are also super fan at Conduktor, it's just something magical. It's as powerful as what happened in the 2000s with the Cloud movement, where we started to run our applications on Cloud's Virtual Machines (AWS EC2 for instance), to have more flexibility, scale more easily/quickly, be more in control, and be able to isolate them. We provide the same for Kafka clusters here: Virtual Clusters, aka Multi-tenancy!

It's fascinating to see how quickly our visitors grasped its potential for significant cost reduction.

Fun fact: our Conduktor Free Playground is running over ... 4000 Virtual Clusters on a standard Confluent Cloud instance. 🚀

Such a massive number of clusters could operate seamlessly on a single instance was nothing short of magical. Who doesn't like cost reduction?

Documentation on our Gateway Marketplace!

Topic as a Service

Topic as a Service in your Kafka UI is the favorite of many of us here: data owners now have the power to grant rights and permissions directly to their users, bypassing the traditional dependency on some Platform Ops team intervention.

We've all been there: asking on Slack for some access, explaining which team you are part on, explaining that your team leader is on holidays and you quick need some access, explaining that your team has a new name and many things should change etc.

As soon as you put humans in the middle of some process: it will take 'indefinite' time. Best developers are lazy developers.

Giving back autonomy to developers not only streamlines processes but also aligns with the principles of data ownership and governance. The realization of giving data owners the ability to control access and coordination brings a sense of clarity and purpose.

It simplifies the flow of information, eliminates unnecessary bottlenecks, and fosters a more efficient and collaborative environment. Who does not want more autonomy? We are grown ups, let's act like it. 👯

Documentation about Topic as a Service

RBAC: Roles & Permissions

Another good one is the presentation of our RBAC module for Kafka users and Kafka applications.

Today we know about ACLs, but it's not enough. Let's admit it, managing and implementing ACLs at scale is cumbersome and Kafka should have provided RBAC natively.

Well Kafka did not, but you now have it with Conduktor:

• For users: I want to access all these topics because I'm part of the team @delivery

• For applications: I want my applications to have a Service Account to access all these topics

RBAC is isolation is already good. But then, you can combine it with our Audit feature to create an amazing combo: you can now explore who (users or apps) accessed what, when, and how! Having all this power in your Kafka UI makes it a breeze and easily accessible for everyone.

This granular level of visibility enhances compliance, strengthens accountability, and empowers you with the necessary insights to optimize your Kafka environment. 🔐

Documentation about all-Security things and RBAC on our Gateway Marketplace !

Virtual Topics: what is this?!

Last interesting bit was your feedback on our "SQL filter" feature that creates "Virtual topics" in our world. Like a VIEW in PostgreSQL, but on a topic. Easy concept right? And so powerful!

This is truly one of Conduktor's gems, enabling developers and organizations to simplify drastically any architecture, remove moving parts, accelerate your delivery... while reducing costs. A dream coming true?

Stay tuned for this one! ☝️

One demo to rule them all

The best part is that nothing was BS. Everything we said were demonstrated live - no fuss, just reality.

Understanding the whole Gateway journey, starting with multitenancy support, then policy, then virtual topics, followed by data masking, field-level encryption, cold storage etc. was totally making sense: it's expected, as this is our goal!

Long demo ahead, be ready!

Then what?

We were not shy about talking about what we are currently building.

The feature of merging multiple clusters as one caught your attention immediately.

Some were in awe when they realized what it meant:

• Bringing Data Mesh of your multiple Kafkas to life!

• Automatic failover management without requiring any changes to your applications! 🤯

• No more replicator, mirror-maker, and other means to duplicate data

• With the caching feature, you have almost a CDN for Kafka (.... psst Kafka Cache Documentation !)

You were as excited as us! 😍

A resounding success!

Once again, we want to express our gratitude for all the enlightening conversations we had.

It was a resounding success, and we thoroughly enjoyed being with you, answering your most complex questions, and providing a glimpse under the hood of Conduktor.

Once again, thank you, you made this moment awesome!

Next step: Current 2023 in California! See you there!