Kafka Self-service

There's a host of new functionality available providing a truly powerful self-service release. This comes from the addition of two new resources: the Subject and the ApplicationGroup.

Define how teams access your Application resources

Represent the different teams that use your Application by setting group permissions with the Application Group, it's a form of delegated RBAC.

Create as many groups as are required to restrict or represent the different teams such as, a support team that are only allowed Read access in Production, a DevOps team with more privileged permissions across environments, or the developers who maintain the product.

apiVersion: self-service/v1
kind: "ApplicationGroup"
metadata:
  application: "clickstream-app"
  name: "clickstream-support"
spec:
  title: Support Clickstream
  description: |
    Members of the Support Group are allowed:
    Read access on all the resources
    Can restart owned connectors
    Can reset offsets
  permissions:
    - appInstance: clickstream-app-dev
      resourceType: TOPIC
      patternType: "LITERAL"
      name: "*" # All owned & subscribed topics
      permissions: ["topicViewConfig", "topicConsume"]
    - appInstance: clickstream-app-dev 
      resourceType: GROUP
      patternType: "LITERAL"
      name: "*" # All owned consumer groups
      permissions: ["consumerGroupCreate", "consumerGroupReset", "consumerGroupView"] 
members:
  - user1@company.org
  - user2@company.org
externalGroups:
  - GP-COMPANY-CLICKSTREAM-SUPPORT

Have application teams manage their Subject resource lifecycle

Schema subjects are now supported within self-service, expanding what Kafka resources can now be managed within the Conduktor self-service framework. Empower application teams with the automated deployment of schema changes and improved collaboration.

apiVersion: v1
kind: Subject
metadata:
  cluster: shadow-it
  name: myPrefix.topic-value
spec:
  schemaFile: schemas/topic.avsc # relative to conduktor CLI execution context
  format: AVRO
  compatibility: FORWARD_TRANSITIVE

To get these benefits you'll need Console 1.24 and to upgrade the Conduktor CLI to 0.2.5.

Automate user & group provisioning for on/off boarding

Whether you are yet to start getting the benefits from delegated resource creation and permissions management, you can still take advantage of the latest changes when managing groups.

Create your groups and set permissions through IaC for Conduktor Console. By declaring the Group with its corresponding User(s) you have more flexibility to further automate the on/off-boarding experience.

apiVersion: iam/v2
kind: "Group"
metadata:
  name: "devops"
spec:
  displayName: "DevOps team"
  description: "Making devs happy"
  permissions:
    - resourceType: PLATFORM
      permissions: ["userManage"]

  - type: CLUSTER
    name: "my-cluster"
    permissions: ["clusterViewBroker"]

  - type: TOPIC
    cluster: "my-cluster"
    patternType: "LITERAL"
    name: "*" # All topics
    permissions: ["topicViewConfig", "topicConsume"]

  - type: SUBJECT
    cluster: "my-cluster"
    patternType: "LITERAL"
    name: "subject-a"
    permissions: ["subjectCreateUpdate", "subjectDelete"]

members:
  - bob@conduktor.io
  - jane@conduktor.io
externalGroups:
  - GP-COMPANY-DEVOPS

Checkout the reference docs for the full config and all other reference pieces within Conduktor.

Topic Catalog Filtering

The topic catalog now has filters! Finding and discovering your topics is even easier than before by filtering on the business metadata you have added to the topics. Need to request access to the topic? Request access by adding the generated snippet to a PR for approval by the owner.

# Topic annotated with useful metadata, see the labels section
---
apiVersion: kafka/v2
kind: "Topic"
metadata:
  cluster: "shadow-it"
  name: clickstream.events
  labels:
    description: "A description for what kind of data this topic contains."
    business-data-classification: C2
    business-doc-url: "https://confluence.company.org/display/CLICK/Kafka"
    application-code: CLK
    environment-code: dev
spec:
  replicationFactor: 3
  partitions: 6
  configs:
    min.insync.replicas: "2"
    cleanup.policy: "delete"
    retention.ms: "60000"

Is there anything else you'd like to see, how would you want the topic catalog to work in your org? Let us know through the public roadmap or drop us an email at product@conduktor.io, we'd love to hear from you.

Active Data Policies in Topic Consume Page

When exploring topics, fields masked by active Data Policies are now displayed in a different color, while the policy name is also now visible on hover.

Quality of Life Improvements

Quickly Analyze Stale and Noisy Topics

• Sort topics by produce rate in msg/s

• Sort topics by the last active

Understand More About Messages

• Navigate between messages with new Next & Previous buttons

• See the Subject name (Confluent SR only) and compression information (metadata) for messages

Rebalance Consumer Groups

• Rebalance consumer groups by removing all static members with the click of a button

Improve Mock Data Generation

The "Generate once" feature in the Produce tab now generates much more realistic, randomized messages, especially for Registry schemas and JSON.

Clarify when Kafka Connect Auto-restart is Active

Added an icon in the Kafka Connect list to inform that auto-restart feature is active.

Conclusion

In summary, we've explored the upgrades from Conduktor's latest release, bringing the powerful group abilities that are now available to manage through infrastructure as code (IaC), be that in relation to your Application or general Console groups. We also looked at the new filtering available in the topic catalog to quickly find relevant topics and some experience updates when working with messages and topics in Console.

Check out our latest changelog for all the details on the 1.24 product release, or if you’re new here, it’s easier than ever to get started with Conduktor, for free.