Happy holidays everyone! Our team has been working hard to do one final release this year and it’s packed with exciting new features and improvements.

In this update, we have a big focus on improving visibility and making real-time data accessible to more and more people across your organization, including a very powerful capability that simplifies SQL querying on Kafka to transform the way you do real-time data analysis.

But that’s not all, we also continue improving team collaboration with shareable filters, give you control on what topics to share in the topic catalog, we made our audit logs actionable, and much more.

Let’s look at some of this in more detail.

Unlock Real-Time Insights with SQL on Kafka

Apache Kafka powers real-time event-driven architectures, but querying its data often requires duplicating it into external systems, leading to operational overhead, misaligned security policies, and delays. 

Our new SQL on Kafka capability can solve most of these issues and empower teams to quickly extract actionable insights from both real-time and historical data streams. Conduktor enhances this experience with features like granular RBAC to enforce least privilege access, configurable control on what topics and how much data is indexed for querying, and a seamless transformation of Kafka data into a columnar table format for easy querying.

By bringing the accessibility of SQL to Kafka, integrated with our enterprise-grade security model, you can safely empower more teams to innovate faster and smarter. After all, SQL is the universal language of data, understood across engineering, analytics, and business teams.

Data engineers can now use SQL for tasks like exploring raw data streams, troubleshooting data pipelines and debugging Kafka messages, all while ensuring security and compliance with masking and access controls.

Business teams can analyze business critical events such as orders and customer interactions using an agile, on-demand approach without permanent storage, making real-time analytics agile and cost-effective.

Ops teams can query Kafka metadata like timestamps, offsets and compression type to inform performance optimization, monitor Kafka resource usage, and troubleshoot issues like identifying duplicates.

Read our recent blog on Transforming Real-Time Data Into Instant Insights, to learn more about SQL over Kafka and see it in action.

We have revamped the homepage and consumer groups in our UI to provide a more informative and useful experience. 

Your Kafka Cluster Health at a Glance

Our redesigned homepage puts everything you need in one view! 

• Instantly check your Kafka Cluster’s health with key metrics and graphs

• Monitor the state of Console Indexing modules

• Quickly access your most recently viewed resources. 

It's all about keeping critical insights at your fingertips!

Enhanced Consumer Group Insights

We've also revamped the Consumer Group page for better clarity and control. The Topics tab groups data by subscribed topics, and the Members tab organizes info by active members. Drill down to individual member or topic-partition assignments with ease. 

Take Control with Self-Service Topic Visibility

Sharing data just got smarter! Now, you can choose exactly which topics to share in the topic catalog, keeping full control over your internal data while making only the right information accessible to other teams or trusted partners. 

---
apiVersion: kafka/v2
kind: Topic
metadata:
  cluster: shadow-it
  name: click.event-stream.avro
  catalogVisibility: PUBLIC # or PRIVATE
spec:
  ...

Collaborate Better and Remove Silos with Shareable Filters

Filters in the Topic Consume view are now shareable, making teamwork effortless. Share pre-defined views to ensure everyone works with the same data subset, save time with reusable filters, and maintain consistency across teams. Easily save filters as Private or Organization-wide to streamline troubleshooting and analysis.

Make Audit Events Actionable with Exportable Audit Logs

Easily publish Audit Log events directly into a Kafka topic to export or maintain your own audit trail in external systems. This feature makes audit events actionable—consume the stream as an application and trigger actions like raising a ServiceNow ticket for specific events, giving you greater control and operational efficiency.

The events conform to the CloudEvents specification, a vendor-neutral format. Here an example:

{
	"source": "//kafka/kafkacluster/production/topic/website-orders",
	"data": {
		"eventType": "Kafka.Topic.Create",
		// Additional event specific data...
		"metadata": {
			"name": "website-orders",
			"cluster": "production"
		}
		// Additional event specific metadata...
	},
	"datacontenttype": "application/json",
	"id": "ad85122c-0041-421e-b04b-6bc2ec901e08",
	"time": "2024-10-10T07:52:07.483140Z",
	"type": "AuditLogEventType(Kafka,Topic,Create)",
	"specversion": "1.0"
}

Strengthen Reliability with Broader High-Availability Support

Conduktor’s supporting database—storing key configurations like users and permissions—now supports a multi-host PostgreSQL high-availability setup, to ensure you enjoy uninterrupted access and consistent performance, even in the face of unexpected challenges.

These are some of the highlights! For a full list of all the features, fixes, and improvements please read the change log.