Blog

Apache Kafka 3.6.0 Release Announcement

We are proud to announce the release of Apache Kafka 3.6.0. This release contains many new features and improvements. This blog post will highlight some of the more prominent features. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation for the list of notable changes and detailed upgrade steps.

The ability to migrate Kafka clusters from a ZooKeeper metadata system to a KRaft metadata system is now ready for usage in production environments. See the ZooKeeper to KRaft migration operations documentation for details. Note that support for JBOD is still not available for KRaft clusters, therefore clusters utilizing JBOD can not be migrated. See KIP-858 for details regarding KRaft and JBOD.

Support for Delegation Tokens in KRaft (KAFKA-15219) was completed in 3.6, further reducing the gap of features between ZooKeeper-based Kafka clusters and KRaft. Migration of delegation tokens from ZooKeeper to KRaft is also included in 3.6.

Tiered Storage is an early access feature. It is currently only suitable for testing in non production environments. See the Early Access Release Notes for more details.

Note: ZooKeeper is marked as deprecated since 3.5.0 release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more information, please see the documentation for ZooKeeper Deprecation

Kafka Broker, Controller, Producer, Consumer and Admin Client

• KIP-405: Kafka Tiered Storage (Early Access):
  Introduces Tiered Storage to Kafka. Note that this is an early access feature only advised for use in non-production environments (see the early access notes for more information). This feature provides a separation of computation and storage in the broker for pluggable storage tiering natively in Kafka Tiered Storage brings a seamless extension of storage to remote objects with minimal operational changes.
• KIP-890: Transactions Server Side Defense (Part 1):
  Hanging transactions can negatively impact your read committed consumers and prevent compacted logs from being compacted. KIP-890 helps address hanging transactions by verifying partition additions. Part 2 of KIP-890 will optimize verification, which currently adds an extra hop.
  In 3.6.0, transaction verification will prevent hanging transactions on data partitions. In the next release, transactional offset commits will also be covered.
• KIP-797: Accept duplicate listener on port for IPv4/IPv6:
  Until now, Kafka has not supported duplicate listeners on the same port. This works when using only a single IP stack, but presents an issue if you are working with both IPv4 and IPv6. With KIP-797, brokers can be configured with listeners that have the same port on different IP stacks. This update does not affect advertised listeners, which already have this feature.
• KIP-863: Reduce CompletedFetch#parseRecord() memory copy:
  Reduces memory allocation and improves memory performance during record deserialization by using a ByteBuffer instead of byte[] for deserialization, which improves efficiency. Updated public interfaces include the Deserializer class, ByteBufferDeserializer class, and StringDeserializer class.
• KIP-868: Metadata Transactions:
  Improves the overall durability of the KRaft layer by adding metadata transactions that consist of:
  • BeginTransaction
  • Number of records
  • EndTransaction or AbortTransaction
  KRaft uses record batches as a mechanism for atomicity. Typically, there was a limit to the fetch size on the Raft consensus layer, and the controller could generate a set of atomic records that exceeded this limit. This update introduces marker records that allow larger sets of atomic records to be sent to the Raft consensus layer in multiple batches. This bypasses the fetch limit.
• KIP-902: Upgrade Zookeeper to 3.8.2:
  This upgrades the ZooKeeper version that is bundled with Kafka to version 3.8.2. The new version includes several updates and security improvements.
• KIP-917: Additional custom metadata for remote log segment:
  It introduces having an optional custom metadata as part of remote log segment metadata. RemoteStorageManager returns the optional custom metadata when copyLogSegmentData() is invoked. It will be passed along with remote log segment metadata.
• KIP-937: Improve Message Timestamp Validation:
  It improves data integrity and prevents potential pitfalls caused by inaccurate timestamp handling by adding more validation logic for message timestamps. While past timestamps are a normal occurrence in Kafka, future timestamps might represent an incorrectly formatted integer. KIP-937 rejects messages with future timestamps and provides a descriptive exemption.
• KIP-938: Add more metrics for measuring KRaft performance:
  Adds new controller, loader, and snapshot emitter KRaft performance metrics.

Kafka Streams

• KIP-923: Add A Grace Period to Stream Table Join:
  Adds a grace period to stream-table joins to improve table-side out-of-order data handling. The joined object has a new method called "withGracePeriod" that will cause the table side lookup to only happen after the grace period has passed.
• KIP-941: Range queries to accept null lower and upper bounds:
  Previously, RangeQuery did not support null to specify “no upper/lower bound”. KIP-941 allows users to pass null into withRange(...) for lower/upper bounds to specify a full or half-open range:
  • `withRange(null, null)` == `withNoBounds()`
  • `withRange(lower, null)` == `withLowerBound(lower)`
  • `withRange(null, upper)` == `withUpperBound(upper)`

Kafka Connect

• KIP-793: Allow sink connectors to be used with topic-mutating SMTs:
  Adds support for topic-mutating SMTs for async sink connectors. This is to address an incompatibility between sink connectors overriding the SinkTask::preCommit method and SMTs that mutate the topic field of a SinkRecord .
• KIP-875: First-class offsets support in Kafka Connect:
  Provides first-class admin support for offsets in Kafka Connect. KIP-875 Part 1 added endpoints to get offsets and a new STOPPED state for connectors. The alter offsets and reset offsets endpoints have now been added.

  Action	Description
  GET /connectors/{connector}/offsets	Retrieve the offsets for a connector; the connector must exist
  PATCH /connectors/{connector}/offsets	Alter the offsets for a connector; the connector must exist, and must be in the STOPPED state
  DELETE /connectors/{connector}/offsets	Reset the offsets for a connector; the connector must exist, and must be in the STOPPED state
  PUT /connectors/{connector}/pause	Pause the connector; the connector must exist

• KIP-898: Modernize Connect plugin discovery:
  With KIP-898, Connect workers can now read from ServiceLoader manifests and module info directly during startup for more efficient plugin class discovery. Note that this update allows connector developers to add service declarations to their plugins.

Summary

Ready to get started with Apache Kafka 3.6.0? Check out all the details in the release notes and download Apache Kafka 3.6.0.

This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 139 contributors:
A. Sophie Blee-Goldman, Aaron Ai, Abhijeet Kumar, aindriu-aiven, Akhilesh Chaganti, Alexandre Dupriez, Alexandre Garnier, Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield, Andrew Grant, Aneel Kumar, Anton Agestam, Artem Livshits, atu-sharm, bachmanity1, Bill Bejeck, Bo Gao, Bruno Cadonna, Calvin Liu, Chaitanya Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190, Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur, David Jacot, David Mao, dengziming, Deqi Hu, Dimitar Dimitrov, Divij Vaidya, DL1231, Dániel Urbán, Erik van Oosten, ezio, Farooq Qaiser, Federico Valeri, flashmouse, Florin Akermann, Gabriel Oliveira, Gantigmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin, iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson, Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio, Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True, Lianet Magrans, LinShunKang, Liu Zeyu, lixy, Lucas Bradstreet, Lucas Brutschy, Lucent-Wong, Lucia Cerchie, Luke Chen, Manikumar Reddy, Manyanda Chitimbo, Maros Orsak, Matthew de Detrich, Matthias J. Sax, maulin-vasavada, Max Riedel, Mehari Beyene, Michal Cabak (@miccab), Mickael Maison, Milind Mantri, minjian.cai, mojh7, Nikolay, Okada Haruki, Omnia G H Ibrahim, Owen Leung, Philip Nee, prasanthV, Proven Provenzano, Purshotam Chauhan, Qichao Chu, Rajini Sivaram, Randall Hauch, Renaldo Baur Filho, Ritika Reddy, Rittika Adhikari, Rohan, Ron Dagostino, Sagar Rao, Said Boudjelda, Sambhav Jain, Satish Duggana, sciclon2, Shekhar Rajak, Sungyun Hur, Sushant Mahajan, Tanay Karmarkar, tison, Tom Bentley, vamossagar12, Victoria Xia, Vincent Jiang, vveicc, Walker Carlson, Yash Mayya, Yi-Sheng Lien, Ziming Deng, 蓝士钦

Apache Kafka 3.5.1 Release Announcement

We are proud to announce the release of Apache Kafka 3.5.1. This is a security patch release. It upgrades the dependency, snappy-java, to a version which is not vulnerable to CVE-2023-34455. You can find more information about the CVE at Kafka CVE list. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.5.1 from any version 0.8.x through 3.4.x section in the documentation for the list of notable changes and detailed upgrade steps.

Kafka Broker, Controller, Producer, Consumer and Admin Client

• Upgraded the dependency, snappy-java, to a version which is not vulnerable to CVE-2023-34455. You can find more information about the CVE at Kafka CVE list.
• Fixed a regression introduced in 3.3.0, which caused security.protocol configuration values to be restricted to upper case only. After the fix, security.protocol values are case insensitive. See KAFKA-15053 for details.

Summary

This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 22 authors. Please report an unintended omission.

Alyssa Huang, Aman Singh, andymg3, Bo Gao, Calvin Liu, Chia-Ping Tsai, Chris Egerton, d00791190, Damon Xie, David Arthur, David Jacot, Divij Vaidya, DL1231, ezio, Manikumar Reddy, Manyanda Chitimbo, Mickael Maison, minjian.cai, Proven Provenzano, Sambhav Jain, vamossagar12, Yash Mayya

Apache Kafka 3.5.0 Release Announcement

We are proud to announce the release of Apache Kafka 3.5.0. This release contains many new features and improvements. This blog post will highlight some of the more prominent features. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.5.0 from any version 0.8.x through 3.4.x section in the documentation for the list of notable changes and detailed upgrade steps.

The ability to migrate Kafka clusters from ZK to KRaft mode with no downtime is still an early access feature. It is currently only suitable for testing in non production environments. See KIP-866 for more details.

Note: ZooKeeper is now marked deprecated in this release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)

Kafka Broker, Controller, Producer, Consumer and Admin Client

• KIP-881: Rack-aware Partition Assignment for Kafka Consumers: Kafka 3.4.0 only contained the protocol changes for KIP-881. The built-in assignors have now been updated to support rack-awareness.
• KIP-887: Add ConfigProvider to make use of environment variables: KIP-887 introduces a new ConfigProvider implementation, EnvVarConfigProvider, to retrieve configurations from environment variables.
• KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for Kafka Brokers: KIP-900 updates the kafka-storage tool and adds a mechanism to configure SCRAM for inter broker authentication with KRaft.
• KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR: KIP-903 addresses a limitation of the inter broker replication protocol which could lead to data loss in case of a broker failing while another broker had an unclean shutdown.

Kafka Streams

• KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions: KIP-399 adds a method, handleSerializationException(), to the ProductionExceptionHandler interface to handle any serialization errors encountered while producing records.
• KIP-889: Versioned State Stores: KIP-889 introduces versioned state stores to improve the accuracy of joins when out of order records are processed.
• KIP-907: Add Boolean Serde to public interface: Kafka Streams includes built-in Serdes for most primitive types. KIP-907 adds a new one for booleans.

Kafka Connect

• KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters: KIP-710 improves the dedicated mode of MirrorMaker. It now supports running multiple instances and handling automatic reconfigurations.
• KIP-875: First-class offsets support in Kafka Connect: KIP-875 adds REST API endpoints for managing offset of Connectors. 3.5.0 only contains endpoints for listing offsets. Endpoints for updating and deleting offsets will come in a future release.
• KIP-894: Use incrementalAlterConfig for syncing topic configurations: With KIP-894, MirrorMaker can use the IncrementalAlterConfig API when mirroring topic configuration between clusters.
• KAFKA-14021: MirrorMaker 2 should implement KIP-618 APIs: KAFKA-14021 adds support for exactly-once semantics to MirrorSourceConnector.

Summary

This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 103 authors: A. Sophie Blee-Goldman, Akhilesh Chaganti, Alex Sorokoumov, Alexandre Dupriez, Alyssa Huang, Anastasia Vela, Andreas Maechler, andymg3, Artem Livshits, atu-sharm, bachmanity1, Bill Bejeck, Brendan Ribera, Calvin Liu, Chaitanya Mukka, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Colin P. McCabe, csolidum, Daniel Scanteianu, David Arthur, David Jacot, David Karlsson, David Mao, Dejan Stojadinović, Divij Vaidya, dorwi, drgnchan, Dániel Urbán, Edoardo Comar, egyedt, emilnkrastev, Eric Haag, Farooq Qaiser, Federico Valeri, Gantigmaa Selenge, Greg Harris, Guozhang Wang, Hao Li, Hector Geraldino, Himani Arora, Hoki Min, hudeqi, iamazy, Iblis Lin, Ismael Juma, Ivan Yurchenko, Jakub Scholz, Jason Gustafson, Jeff Kim, Jim Galasyn, Jorge Esteban Quilcate Otoya, Josep Prat, José Armando García Sancio, Juan José Ramos, Junyang Liu, Justine Olshan, Kamal Chandraprakash, Kirk True, Kowshik Prakasam, littlehorse-eng, liuzc9, Lucas Brutschy, Lucia Cerchie, Luke Chen, Manikumar Reddy, Manyanda Chitimbo, Matthew Wong, Matthias J. Sax, Matthias Seiler, Michael Marshall, Mickael Maison, nicolasguyomar, Nikolay, Paolo Patierno, Philip Nee, Pierangelo Di Pilato, Proven Provenzano, Purshotam Chauhan, Qing, Rajini Sivaram, RivenSun, Robert Young, Rohan, Roman Schmitz, Ron Dagostino, Ruslan Krivoshein, Satish Duggana, Shay Elkin, Shekhar Rajak, Simon Woodman, Spacrocket, stejani-cflt, Terry, Tom Bentley, vamossagar12, Victoria Xia, Viktor Somogyi-Vass, Vladimir Korenev, Yash Mayya, Zheng-Xian Li