Introduction

So, you're an existing Desktop customer and are considering whether to make the leap towards Conduktor's shiny, web-based 'Console' (Kafka UI). But as usual, your core business remains your primary focus, Conduktor Desktop 'does the job', and you need to be convinced that migrating will benefit you.

In this blog post, we'll look at the fundamental differences and why it might make sense for you and your organization to migrate.

Console is just one deployment away

Desktop is a pain to onboard, upgrade, control and secure in an organization

One obvious difference between Desktop and Conduktor Console is how it's deployed and maintained. Desktop is a desktop application that must be installed on every users machine.

This means every time there is a patch or update, each user must manually complete the upgrade on their machine. This is painful and difficult for all of your users.

Performance is degraded as there are more network hops between a user's machine and where your clusters are located. Don't even get me started on Bastion hosts! (if your Kafka is protected behind a secure server).

Conduktor Console is a safe, central and up-to-date deployment

Conduktor Console is built on the premise of a single, centralized deployment that is hosted for all your users to have their Kafka UI.

Rather than be concerned with each user manually importing certificates for SSL/TLS secured clusters, you can govern certificates centrally with control.

• one source of truth

• one deployment to patch/upgrade

• watchtower-like visibility of your users activity

Deploy it close to your clusters, inside your VPC, and get improved performance when interacting with Kafka.

We offer flexibility in deployments with our quick-starts for Docker and Helm chart for Kubernetes. You also have the expert support of our customer success team to support your migration!

User Experience at the heart of Console development

An improved UX with ReactJS interface

It's no secret that Conduktor Console is our second-generation interface for troubleshooting, exploring, and operating your Kafka data. Utilizing a react-based interface allows for a much more compelling experience than JavaFX! That's a fact; we are in the 21st century, after all.

But beneath the purely technical differences, the Conduktor Design team is meticulously optimizing the Console to provide an excellent user experience. This emphasis on design has resulted in differences that dramatically reduce the time to complete everyday tasks.

Work across several clusters with multiple tabs open

Take this example; instantly switch between your dev, staging, and prod clusters using the dropdown selector. The same process in Conduktor Desktop involves 3 distinct actions:

• loading the cluster view

• exiting the window

• selecting a different cluster from the main navigation.

As this is a very regular operation, this improved UX helps you complete core jobs faster, allowing you to minimize time spent in Conduktor, and maximize time spent on business priorities.

Moreover, as this is a web interface, you can have many tabs open to work with multiple clusters simultaneously, something impossible with Conduktor Desktop.

Introducing a multi-player experience

While Desktop will ensure you get the job done, we have opened the door to new possibilities with our shift towards a web-based application. Consider:

• Graphing libraries for live updating time-series data

• Automation that reduces manual intervention

• Sharing views and filters with colleagues for collaboration

• Metadata 'tagging' to highlight resource ownership

• Integrations with third-party apps and identity providers

Fundamentally we are focusing on a 'multi-player' experience rather than a tool developed for an individual developer. This shared experience can help nurture team members if they are new to Kafka and promotes collaboration and visibility of your Kafka resources across domains and business units. After all, Kafka IS organization software, not something used in isolation.

While we will continue to support Desktop, all of our product development of new features will gravitate towards Console moving forwards.

Console brings a stronger security posture

More granular permissions (RBAC)

While Conduktor Desktop had a lightweight RBAC (Role-Based Access Control) implementation for Kafka resources, it is limited to assigning topic-level permissions only.

Conduktor Console goes one step further and permits Admin users to apply resource-based permissions for users and groups on:

• Clusters (ACLs, Registry Compatibility)

• Topics

• Subjects

• Consumer Groups

• Connectors

Now you can provide even finer-grained controls across the entire Kafka ecosystem; ensuring Software Developers, Product Owners, and Data Engineers have the least privilege access to fulfill their responsibilities.

Mask sensitive and PII data from the interface

RBAC solves the read/write access to Kafka resources but leaves one major problem unsolved.

What if Team A requires access to production topics for debugging purposes but can't risk being exposed to sensitive customer data inside those topics? How can you solve something like that?

Conduktor Console provides a mechanism for field-level obfuscation of data in topics. eg: you can provide Team A access to production topics but define explicit policies to mask data in fields containing sensitive or PII data (emails, addresses, credit cards, etc.).

To ensure full flexibility, policies can be set across multiple clusters and refined to exclude specific users or groups if you do not want to configure global rules. Pretty neat, right? Your sensitive data is secure without compromising on developer troubleshooting needs.

All actions are now Auditable

Desktop was built to be operated as a single-player tool, meaning there is no organizational traceability of Kafka operations, actions, and data access. Do you know who did what, when and how? Unfortunately, you do not!

Conversely, Conduktor Console provides a persistent record of all user-related and resource-related events relating to your Kafka infrastructure and associated data. This is critical for enabling operational and risk auditing, governance, and compliance.

Conduktor Console will ensure your organization meets security and operational best practices. You will have full visibility of who performed a particular operation, which resources were acted upon, when the event happened, and other contextual details relating to the event. You also get helpful filtering tools to easily view, search and analyze your logs, minimizing time spent trawling logs when response time is critical.

Console adapts to your internal tooling

Complete support for all Identity Providers (SSO)

Both Desktop and Console support some degree of Single Sign-On mechanism. This allows users to login to Conduktor with their existing credentials from your source of truth database. It ensures you can control authentication policies and maintain within your IdP are allowed to access Conduktor.

Below outlines SSO protocol and feature support:

Desktop does NOT support any form of offline SSO which is often a major issue.

Note that Console provides enhanced integration for external group synchronization. By linking a Conduktor group to an external group in your IdP, it ensures a user inherits necessary group permissions and that they are removed accordingly from Conduktor groups if their external membership changes.

Be connected: Slack, MS Teams

Console brings real-time notifications to users, for example, when a specific threshold is met regarding a lagging consumer group, or when you have under-replicated partitions.

You have this new power with Console: you can integrate with your MS Teams or Slack to receive alerts directly there. This way, you minimize your Mean Time To Resolve (MTTR) and reduce your downtime.

Full AWS support: AWS + Glue Schema Registry

Desktop provides minimum support for AWS IAM, allowing you to at least setup your cluster connection with IAM access. It also has baked-in support for AWS Glue Registry deserializer since 2.21.9.

Console goes a step further with a full AWS integration:

• Connect to an MSK cluster using IAM credentials inherited from the environment where Conduktor is deployed, or from using custom AWS Access Key and Secret.

• Manage your AWS Glue Schema Registry by providing all the operations to manage the schemas (create, update, delete, change compatibility mode, etc.)

To deploy Console on AWS, read our blog post on AWS Big Data Blog.

Protect your infra and enrich your data: Console + Gateway

One significant advantage of migrating to Conduktor Console is the integration opportunities with Conduktor Gateway.

Conduktor Gateway is a Kafka proxy that sits between your client applications and your Kafka clusters. This transport layer is enriched with interceptors to add your business and technical rules and policies against the data being produced and consumed. We offer an Open-Source version with its marketplace (extensible to your needs) and an Enterprise version for production deployments.

For example, it takes 1 minute to configure a centralized encryption policy agnostic to the producing applications (and the programming languages they are written in). How awesome!

With this new power comes new responsibilities.

To provision such rules, it begs the question of how to deploy and govern your Gateway configurations. Conduktor Console can be your single control plane for Kafka and Gateway. You will be able to deploy and manage Gateway interceptors from within the Console itself.

Example:

• Scenario: you want to test your application resilience by simulating 'real-world' Kafka anomalies: slow brokers, leader election errors, invalid Schema Id etc.

• How to: Conduktor Gateway provides a 'chaos' interceptor. With just a few clicks inside Console, you can throw Chaos Engineering tactics at your applications to test their robustness and response.

This is just one thing among thousand new possibilities. We won't deep-dive into this subject here, but I encourage you to read more about Gateway to comprehend why Conduktor is not just 'another beautiful Kafka UI'.

Console provides enhanced developer features

As well as the fundamental differences we have already covered, there are a host of new features that Conduktor Console brings.

Automatic Connector Restarts

Console brings automation that improves your experience as a developer or DevOps person interacting with Kafka.

If you already used Debezium, JDBC or Elasticsearch connectors you know what we are talking about. You probably had connectors failing 'randomly', without even mentioning network issues, timeouts in the source system.

Now you can sleep tight knowing that Conduktor will restart them automatically and let you know through its notifications system (Slack, MS Teams).

Monitoring & Alerting included

Experiencing consumer lag (latency) in your real-time applications can have a detrimental impact on the end-users experience, or worse, directly impact your business' bottom line.

Lag is one of many metrics you should be actively monitoring to ensure that:

• Your streaming applications are performing optimally

• Your cluster itself is healthy

Monitoring Kafka should not just be for Ops but also for Product teams that are responsible for their applications. This is why our Kafka UI provides a embedded monitoring and alerting solution for your Kafka ecosystem.

While Desktop provides a 'snapshot' for a few metrics, Console has an opinionated solution for monitoring and alerting fundamental Kafka metrics and beyond.

• Console comes with time-series graphs to demonstrate the full picture, and show the history, rather than just showing the latest point in time like Desktop does.

• Console will also periodically run a cluster 'health check', which assesses the below metrics at regular intervals to determine if your cluster is healthy:

  • Under-Replicated Partitions

  • Active Controllers

  • Min In-Sync-Replicas

  • Unclean Elections

  • Offline Partitions

The best part is that cluster monitoring works 'out of the box', requiring no additional agent configuration. It comes with baked-in alerts to ensure you identify anomalous metrics as soon as thresholds are breached, allowing you to rectify them immediately.

You can tag resources to define ownership

Significant Kafka adoption also brings the headache of sprawling resources. Unless you implemented strong internal governance from day 1, you're probably questioning who owns a set of resources and which applications they relate to.

To combat this common challenge, Console brings resource tagging to your fingertips:

Kafka is a collaborative software. Console focuses provides features to nurture this collaboration. Adding additional metadata to resources allows you to attribute them to a specific team, project, or business unit. This improves ownership clarity for all company parties with a stake in your Kafka ecosystem.

New Service Account Management

Console introduces service account management, something not available in Conduktor Desktop. Service Accounts provides access management for applications that require programmatic access to Kafka.

This feature provides:

• a consolidated view of created service accounts

• associated ACLs

• quotas

It will also be integrated with Confluent and Aiven (they provide an API) to unify your Kafka application access management across multiple clusters and providers in Conduktor Console.

GitOps and an API to interact with Conduktor

A frequent ask from our customers is a mechanism for interacting with Conduktor outside the interface itself, like:

• Synchronizing user or group-based permissions via an external application

• Application deployment performed through GitOps process

This comes with custom configurations bespoke to the customers environment. To match these requirements, Console provides a public API that allows third-party developers to access Conduktor and build applications and services that interact with it.

The public API currently focuses on Users, Groups, Permissions, and Clusters management and will be extended to support new use cases in the future.

Optimized for performance

A massive amount of work has been done regarding performance. As we acquired more and more customers with larger environments, we had to provide top-notch performance when navigating resources: topics, consumer groups, schema registry, connectors, etc. It's very common to have tens of thousands of each.

Desktop is not tailored to support such scale and is really not recommended, at the risk of creating a big productivity loss, as screens might take minutes to render.

Console is the solution to this. Working with large customers drives us to optimize every tiny bit of logic we do. We strive to provide the best user experience possible and regularly test our product against large clusters to ensure we can meet any customers demands.

Conclusion

As highlighted here, our Kafka UI, the Console moves away from isolated, individual development that lacks organization-wide controls. At the same time, benefits are not just organizational, as it brings many new features and automation to benefit developers greatly.

• As a Platform Architect or Ops Lead, you're looking to solidify your company's position regarding Kafka user security, governance, and compliance: Conduktor Console is a tremendous leap in that direction. It brings a security and governance wrapper around a centralized deployment for all Kafka operations and data access. This dispels the 'open bar' environment, which comes with inherent risk when Kafka is critical for your business.

• As a Developer, you value speed, repeatability, and collaboration. You want to troubleshoot, operate and explore your Kafka data as practically as possible: you will find the overall experience of driving the Console more seamless.

Enhanced support for programmatic access, real-time notifications, shareable URLs, and features that reduce manual intervention help to extend flexibility and improve teamwork. Finally, with more extensive RBAC and data masking controls, developers can operate more freely than ever, banishing blanket access rules that slow down productivity and incident resolution times.

If you are a Conduktor Desktop user, we encourage you to switch to Conduktor Console as it is superseding Conduktor Desktop in every direction. Whether you are a single developer or an engineering team, you will find it more helpful and enjoyable to use.