Kafka Monthly Digest – August 2021

This is the 43rd edition of the Kafka Monthly Digest. In this edition, I’ll cover what happened in the Apache Kafka community in August 2021.

For last month’s digest, see Kafka Monthly Digest: July 2021.

Kafka Summit

The Kafka Summit Americas is on September 14 and 15. It is a virtual event with dozens of sessions covering all aspects of the Kafka ecosystem. You can register for free on the Kafka Summit website.

Releases

3.0.0

Kafka 3.0.0 is currently in the stabilization period following code freeze. After a few more Blocker JIRAs (KAFKA-13198, KAFKA-13219), Konstantine published the first Release Candidate, 3.0.0 RC0, on August 27. A vote on this RC is currently on going.

2.8.1

On August 12, David Jacot volunteered to run this bugfix release. The release date is currently targetted for September. The release plan is available on the wiki.

Last month, the community submitted 5 KIPs (KIP-766 to KIP-770), and these are the ones that caught my eye.

• KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC. Kafka supports OAuth authentication since 2.0 (KIP-255). However, KIP-255 only introduced a framework and organizations have to provide their own implementations. This KIP proposes adding a production ready implementation of the interfaces defined in KIP-255 to authenticate with Open ID identity providers.

• KIP-770: Replace “buffered.records.per.partition” with “input.buffer.max.bytes”. This KIP aims at controlling the memory used by a Kafka Streams application. It proposes adding a new configuration input.buffer.max.bytes to specify the total memory that can be used to buffer records. The existing configuration buffered.records.per.partition which only controlled memory per partition will be deprecated.

Community Releases

In this section, I will cover releases of some community projects. This only includes projects that are open source.

• kafka-connect-file-pulse 2.2.0. FilePulse is a connectors to stream any file into Kafka. This new version improves scenarios with a very large number of small files, improves support for XML files and introduces a new pluggable Partitioner interface to distribute files across tasks.

• kcat 1.7.0. Previously called kafkacat, kcat is a CLI client for Kafka. This release improves multibytes delimiters and adds support for incremental rebalancing.

• node-rdkafka 2.11. This new release of node-rdkafka is now based on librdkafka 1.6.1. It now supports Nodejs v16 and also the idempotent and transactional producer.

• kcctl 1.0.0-EA. kcctl is a command line client for Kafka Connect. It allows to easily interact (deploy, stop, restart, describe, etc) with connectors.

Blogs

Get started with Kafka

IBM Event Streams for Cloud is Apache Kafka-as-a-Service for IBM Cloud. Get started with IBM Event Streams today.