Blog

Apache Kafka 3.9.0 Release Announcement

6 November 2024 - Colin P. McCabe

We are proud to announce the release of Apache Kafka 3.9.0. This is a major release, the final one in the 3.x line. This will also be the final major release to feature the deprecated Apache ZooKeeper mode. Starting in 4.0 and later, Kafka will always run without ZooKeeper.

Dynamic KRaft Quorums

When in KRaft mode, Kafka relies on a Raft quorum of controller processes to store its metadata log. Previously, controller quorums were specified by a static configuration that needed to be set on all brokers and controllers. This made changing the hostnames of controllers, or adding or removing them, very difficult to do without downtime.

KIP-853: KRaft Controller Membership Changes makes quorum membership dynamic. Administrators can now add and remove controller nodes by running the kafka-metadata-quorum.sh tool or using the associated AdminClient API.

This feature has been eagerly awaited by the community, and we are excited to deliver it in 3.9. The main limitation in 3.9 is that we do not support converting over static metadata quorums to dynamic ones. However, this ability is coming soon.

For more information about using the new dynamic quorums feature, see the KIP-853 documentation.

Improved ZooKeeper Migration

Users that are using ZooKeeper mode today need to go through a process called ZooKeeper migration before they will be able to use KRaft mode. The migration process involves running a series of commands to copy the metadata stored in ZooKeeper into a KRaft quorum.

ZK migration is not a new feature; indeed, it has been under development since Kafka 3.4. However, I wanted to highlight that Kafka 3.9 is the final and best iteration of our ZK migration feature. As we migrated thousands of clusters (big and small), we found and fixed many bugs. We also closed all of the remaining feature gaps that kept some users tied to ZooKeeper.

Kafka 4.0 will be fully saying goodbye to ZooKeeper. There will be no support for running in ZK mode, or migrating from ZK mode. This means that administrators that are still using the deprecated ZK mode who need to upgrade to 4.0 and beyond will need to make a stop at a "bridge release." For example, if you wanted to upgrade from Kafka 3.0 to Kafka 4.0, you might do the following:

  • Upgrade to Kafka 3.9.
  • Perform ZK migration.
  • Upgrade to Kafka 4.0.

In this example, Kafka 3.9 serves as the "bridge" to 4.0

As you migrate your older Kafka clusters, also keep in mind that Kafka 3.5 and later use a version of ZooKeeper that is not wire-compatible with Kafka versions older than 2.4. Therefore, if you want to migrate a cluster older than that, you will have to make an additional stop at a release with a ZK version supported by both Kafkas. See KIP-902 for details.

As an example, if you wanted to migrate from Kafka 1.0 to Kafka 4.0, you might do the following:

  • Upgrade to Kafka 3.4 (a newer version will not work due to ZK incompatibilites)
  • Upgrade to ZooKeeper 3.8
  • Upgrade to Kafka 3.9.
  • Perform ZK migration.
  • Upgrade to Kafka 4.0.

These multi-step migrations should be quite rare. Running such an old Kafka version raises security concerns, after all. However, I wanted to mention it for completeness.

Tiered Storage

Tiered storage is a feature that has been under development since Kafka 3.6. It allows Kafka to seamlessly offload data to pluggable external storage systems, such as cloud object stores. (See KIP-405 for more details.)

Tiered storage is now production-ready in Kafka 3.9. In addition, we added the following improvements:

Kafka Streams

There are several Kafka Streams improvements in Apache Kafka 3.9.

Kafka Connect

Kafka Connect received several improvements in 3.9 as well.

Conclusion

I hope that this post has given you a sense of all the exciting things that are going on in Kafka 3.9. For more details, take a look at the release notes, or simply download the release for yourself.

This was a community effort, so thank you to everyone who contributed to this release:
A. Sophie Blee-Goldman, abhi-ksolves, Abhijeet Kumar, Abhinav Dixit, Adrian Preston, Alieh Saeedi, Alyssa Huang, Anatoly Popov, Andras Katona, Andrew Schofield, Andy Wilkinson, Anna Sophie Blee-Goldman, Antoine Pourchet, Apoorv Mittal, Arnav Dadarya, Arnout Engelen, Arpit Goyal, Arun Mathew, Ayoub Omari, bachmanity1, Bill Bejeck, brenden20, Bruno Cadonna, Chia Chuan Yu, Chia-Ping Tsai, ChickenchickenLove, Chirag Wadhwa, Chris Egerton, Christo Lolov, Ming-Yen Chung, Colin P. McCabe, Cy, David Arthur, David Jacot, Demonic, dengziming, Dimitar Dimitrov, Dmitry Werner, Dongnuo Lyu, dujian0068, Edoardo Comar, Farbod Ahmadian, Federico Valeri, Fiore Mario Vitale, Florin Akermann, Francois Visconte, Ganesh Sadanala, Gantigmaa Selenge, Gaurav Narula, gongxuanzhang, Greg Harris, Gyeongwon Do, Harry Fallows, Hongten, Ian McDonald, Igor Soarez, Ismael Juma, Ivan Yurchenko, Jakub Scholz, Jason Gustafson, Jeff Kim, Jim Galasyn, Jinyong Choi, Johnny Hsu, José Armando García Sancio, Josep Prat, Jun Rao, Justine Olshan, Kamal Chandraprakash, Ken Huang, Kevin Wu, Kirk True, Kondrat Bertalan, Krishna Agarwal, KrishVora01, Kuan-Po (Cooper) Tseng, Lee Dongjin, Lianet Magrans, Logan Zhu, Loïc GREFFIER, Lucas Brutschy, Luke Chen, Maciej Moscicki, Manikumar Reddy, Mason Chen, Matthias J. Sax, Max Riedel, Mickael Maison, Murali Basani, Nancy, Nicolas Guyomar, Nikolay, Okada Haruki, Omnia Ibrahim, PaulRMellor, Pavel Pozdeev, Phuc-Hong-Tran, Piotr Rzysko, PoAn Yang, Ritika Reddy, Rohan, Ron Dagostino, Sanskar Jhajharia, Satish Duggana, Sean Quah, Sebastien Viale, Shawn Hsu, ShivsundarR, Sushant Mahajan, TaiJuWu, TengYao Chi, TingIāu "Ting" Ki, vamossagar12, Vedarth Sharma, Vikas Balani, Vikas Singh, Viktor Somogyi-Vass, Vinay Agarwal, Vincent Rose, Volk, Wang Xiaoqing, Xiduo You, xijiu, Xuan-Zhang Gong, Yash Mayya, Zhengke Zhou.

Apache Kafka 3.8.1 Release Announcement

29 October 2024 - Josep Prat (@jlprat)

We are proud to announce the release of Apache Kafka 3.8.1. This is a bugfix release. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.8.1 from any version 0.8.x through 3.7.x section in the documentation for the list of notable changes and detailed upgrade steps.

Summary

This was a community effort, so thank you to everyone who contributed to this release:
Andrew Schofield, Apoorv Mittal, Bill Bejeck, Bruno Cadonna, Chia-Ping Tsai, Chris Egerton, Colin P. McCabe, David Arthur, Guillaume Mallet, Igor Soarez, Josep Prat, Justine Olshan, Ken Huang, Kondrat Bertalan, Kuan-Po Tseng, Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael Maison, PoAn Yang, Rohan, TengYao Chi, Vikas Singh

Apache Kafka 3.8.0 Release Announcement

29 July 2024 - Josep Prat (@jlprat)

We are proud to announce the release of Apache Kafka 3.8.0. This release contains many new features and improvements. This blog post will highlight some of the more prominent features. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.8.0 from any version 0.8.x through 3.7.x section in the documentation for the list of notable changes and detailed upgrade steps.

In a previous release, 3.6, tiered storage was released as early access feature. In this release, Tiered Storage now supports clusters configured with multiple log directories (i.e. JBOD feature). This feature still remains as early access.

In the last release, 3.7, KIP-858 was released in early access. Since this version, JBOD in KRaft is no longer considered an early access feature.

Up until now, only the default compression level was used by Apache Kafka. From this version on, a configuration mechanism to specify compression level is included. See KIP-390 for more details.

KIP-848 The Next Generation of the Consumer Rebalance Protocol is available as preview in 3.8. This version includes numerous bug fixes and the community is encouraged to test and provide feedback. See the preview release notes for more information.

The configuration value offsets.commit.required.acks is deprecated in this version and it will be removed in Kafka 4.0. See KIP-1041 for more details.

Note: ZooKeeper is marked as deprecated since the 3.5.0 release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more information, please see the documentation for ZooKeeper Deprecation.

Kafka Broker, Controller, Producer, Consumer and Admin Client

Kafka Streams

  • KIP-989: Improved StateStore Iterator metrics for detecting leaks:
    This KIP introduces several new metrics to aid users in finding leaked Iterators, as well as identifying the cause of a high number of pinned blocks, or other kinds of memory leaks and performance problems.
  • KIP-924: customizable task assignment for Streams:
    This KIP adds a new group of configurable interfaces for plugging custom behaviour into the Streams Partition Assignor. This configuration supplants the existing internal task assignor config. Additionally, it limits the scope of these configs to supplying a custom task assignor. This opens the door for future KIPs to add further configs in which a user can set to plug in custom behavior.
  • KIP-813: Shareable State Stores:
    This KIP adds Shareable State Stores, introducing the ability to use data within a state store across multiple applications without duplicating it on topic level.

Kafka Connect

Summary

Ready to get started with Apache Kafka 3.8.0? Check out all the details in the release notes and download Apache Kafka 3.8.0.

This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 196 contributors:
Aadithya Chandra, Abhijeet Kumar, Abhinav Dixit, Adrian Preston, Afshin Moazami, Ahmed Najiub, Ahmed Sobeh, Akhilesh Chaganti, Almog Gavra, Alok Thatikunta, Alyssa Huang, Anatoly Popov, Andras Katona, Andrew Schofield, Anna Sophie Blee-Goldman, Antoine Pourchet, Anton Agestam, Anton Liauchuk, Anuj Sharma, Apoorv Mittal, Arnout Engelen, Arpit Goyal, Artem Livshits, Ashwin Pankaj, Ayoub Omari, Bruno Cadonna, Calvin Liu, Cameron Redpath, charliecheng630, Cheng-Kai, Zhang, Cheryl Simmons, Chia Chuan Yu, Chia-Ping Tsai, ChickenchickenLove, Chris Egerton, Chris Holland, Christo Lolov, Christopher Webb, Colin P. McCabe, Colt McNealy, cooper.tseng@suse.com, Vedarth Sharma, Crispin Bernier, Daan Gerits, David Arthur, David Jacot, David Mao, dengziming, Divij Vaidya, DL1231, Dmitry Werner, Dongnuo Lyu, Drawxy, Dung Ha, Edoardo Comar, Eduwer Camacaro, Emanuele Sabellico, Erik van Oosten, Eugene Mitskevich, Fan Yang, Federico Valeri, Fiore Mario Vitale, flashmouse, Florin Akermann, Frederik Rouleau, Gantigmaa Selenge, Gaurav Narula, ghostspiders, gongxuanzhang, Greg Harris, Gyeongwon Do, Hailey Ni, Hao Li, Hector Geraldino, highluck, hudeqi, Hy (하이), IBeyondy, Iblis Lin, Igor Soarez, ilyazr, Ismael Juma, Ivan Vaskevych, Ivan Yurchenko, James Faulkner, Jamie Holmes, Jason Gustafson, Jeff Kim, jiangyuan, Jim Galasyn, Jinyong Choi, Joel Hamill, John Doe zh2725284321@gmail.com, John Roesler, John Yu, Johnny Hsu, Jorge Esteban Quilcate Otoya, Josep Prat, José Armando García Sancio, Jun Rao, Justine Olshan, Kalpesh Patel, Kamal Chandraprakash, Ken Huang, Kirk True, Kohei Nozaki, Krishna Agarwal, KrishVora01, Kuan-Po (Cooper) Tseng, Kvicii, Lee Dongjin, Leonardo Silva, Lianet Magrans, LiangliangSui, Linu Shibu, lixinyang, Lokesh Kumar, Loïc GREFFIER, Lucas Brutschy, Lucia Cerchie, Luke Chen, Manikumar Reddy, mannoopj, Manyanda Chitimbo, Mario Pareja, Matthew de Detrich, Matthias Berndt, Matthias J. Sax, Matthias Sax, Max Riedel, Mayank Shekhar Narula, Michael Edgar, Michael Westerby, Mickael Maison, Mike Lloyd, Minha, Jeong, Murali Basani, n.izhikov, Nick Telford, Nikhil Ramakrishnan, Nikolay, Octavian Ciubotaru, Okada Haruki, Omnia G.H Ibrahim, Ori Hoch, Owen Leung, Paolo Patierno, Philip Nee, Phuc-Hong-Tran, PoAn Yang, Proven Provenzano, Qichao Chu, Ramin Gharib, Ritika Reddy, Rittika Adhikari, Rohan, Ron Dagostino, runom, rykovsi, Sagar Rao, Said Boudjelda, sanepal, Sanskar Jhajharia, Satish Duggana, Sean Quah, Sebastian Marsching, Sebastien Viale, Sergio Troiano, Sid Yagnik, Stanislav Kozlovski, Stig Døssing, Sudesh Wasnik, TaiJuWu, TapDang, testn, TingIāu "Ting" Kì, vamossagar12, Vedarth Sharma, Victor van den Hoven, Vikas Balani, Viktor Somogyi-Vass, Vincent Rose, Walker Carlson, wernerdv, Yang Yu, Yash Mayya, yicheny, Yu-Chen Lai, yuz10, Zhifeng Chen, Zihao Lin, Ziming Deng, 谭九鼎

Apache Kafka 3.7.1 Release Announcement

28 June 2024 - Igor Soarez (@soarez)

We are proud to announce the release of Apache Kafka 3.7.1. This is a bugfix release. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.7.1 from any version 0.8.x through 3.6.x section in the documentation for the list of notable changes and detailed upgrade steps.

Summary

This was a community effort, so thank you to everyone who contributed to this release:
Adrian Preston, Anatoly Popov, Andras Katona, Andrew Schofield, Anna Sophie Blee-Goldman, Anton Liauchuk, Apoorv Mittal, Ayoub Omari, Bruno Cadonna, Calvin Liu, Cameron Redpath, Cheng-Kai Zhang, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe, David Arthur, David Jacot, Divij Vaidya, Dmitry Werner, Edoardo Comar, flashmouse, Florin Akermann, Gantigmaa Selenge, Gaurav Narula, Greg Harris, Igor Soarez, ilyazr, Ismael Juma, Jason Gustafson, Jeff Kim, jiangyuan, Joel Hamill, John Yu, Johnny Hsu, José Armando García Sancio, Josep Prat, Jun Rao, Justine Olshan, Kamal Chandraprakash, Ken Huang, Kuan-Po (Cooper) Tseng, Lokesh Kumar, Luke Chen, Manikumar Reddy, Mario Pareja, Matthias J. Sax, Mayank Shekhar Narula, Mickael Maison, Murali Basani, Omnia Ibrahim, Paolo Patierno, PoAn Yang, Sagar Rao, sanepal, Sean Quah, Sebastian Marsching, Stanislav Kozlovski, Vedarth Sharma, Walker Carlson, Yash Mayya

Apache Kafka 3.6.2 Release Announcement

4 April 2024 - Manikumar Reddy O. (@Manikumar)

We are proud to announce the release of Apache Kafka 3.6.2. This is a bugfix release. It contains 28 fixes. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.6.2 from any version 0.8.x through 3.5.x section in the documentation for the list of notable changes and detailed upgrade steps.

Summary

This was a community effort, so thank you to everyone who contributed to this release:
Akhilesh Chaganti, Andrew Schofield, Anton Liauchuk, Bob Barrett, Bruno Cadonna, Cheng-Kai, Zhang, Chia-Ping Tsai, Chris Egerton, Colin P. McCabe, Colin Patrick McCabe, David Arthur, David Jacot, David Mao, Divij Vaidya, Edoardo Comar, Emma Humber, Gaurav Narula, Greg Harris, hudeqi, Ismael Juma, Jason Gustafson, Jim Galasyn, Joel Hamill, Johnny Hsu, José Armando García Sancio, Justine Olshan, Luke Chen, Manikumar Reddy, Matthias J. Sax, Mayank Shekhar Narula, Mickael Maison, Mike Lloyd, Paolo Patierno, PoAn Yang, Ron Dagostino, Sagar Rao, Stanislav Kozlovski, Walker Carlson, Yash Mayya

Apache Kafka 3.7.0 Release Announcement

27 February 2024 - Stanislav Kozlovski (@BdKozlovski)

We are proud to announce the release of Apache Kafka 3.7.0. This release contains many new features and improvements. This blog post will highlight some of the more prominent features. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation for the list of notable changes and detailed upgrade steps.

In the last release, 3.6, the ability to migrate Kafka clusters from a ZooKeeper metadata system to a KRaft metadata system was ready for usage in production environments with one caveat -- JBOD was not yet available for KRaft clusters. In this release, we are shipping an early access release of JBOD in KRaft. (See KIP-858 and the release notes for details).

Additionally, client APIs released prior to Apache Kafka 2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. See KIP-896 for details and RPC versions that are now deprecated.

Java 11 support for the Kafka broker is also marked deprecated in 3.7, and is planned to be removed in Kafka 4.0. See KIP-1013 for more details. Note that clients can continue to use JDK >= 11 to connect to Kafka brokers.

Note: ZooKeeper is marked as deprecated since the 3.5.0 release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more information, please see the documentation for ZooKeeper Deprecation.

Kafka Broker, Controller, Producer, Consumer and Admin Client

Kafka Streams

  • KIP-925 Rack aware task assignment in Kafka Streams:
    In part one of KIP-925, the min_traffic assignment strategy for Kafka Streams was added. Part two finishes the KIP by introducing the second rack-aware assignment strategy: balanced_subtopology.
  • KIP-954 Expand default DSL store configuration to custom types:
    KIP-954 builds on KIP-591 and allows users to provide a default state store provider for their custom stores. As part of this change, a new interface has been provided along with default support for RocksDB and in-memory state stores.
  • KIP-962 Relax non-null key requirement in Kafka Streams:
    Kafka Streams treated records with null-keys as invalid input for joins and dropped them on the floor. KIP-962 relaxes this behavior for various left-joins and thus allows null-key records to be processed successfully.
  • KIP-988 Streams Standby Update Listener:
    Adds a new interface for handling cases where standby tasks have their a) state stores registered, b) load a batch of records and c) stop updates.
  • KIP-960 / KIP-968 IQ support for Versioned State Stores:
    Version state stores were added in Apache Kafka 3.5 release (KIP-889), but it was not possible so far to query the new stores. KIP-960 and KIP-968 close this gap by adding new query types for IQv2 namely VersionedKeyQuery and MultiVersionedKeyQuery, respectively. Both queries allow you to do lookups for a single key, to ask for the most recent value, a historic value, or a range of historic values for the provided key.
  • KIP-985 Add reverseRange and reverseAll query over kv-store in IQv2:
    IQv2 supports RangeQueries which allow you to query for a range of keys. However, the API did not provide any guarantee about the ordering of the returned result. With KIP-985 it’s now possible to request the result to be ordered (per partition) in either ascending or descending order, or leave the order unspecified.
  • KIP-992 Introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery:
    Finally, KIP-992 adds new timestamped-key and timestamped-range interactive queries for timestamped key-value state stores. This change improves the type safety of the IQv2 API. The existing RangeQuery now always returns only the value if issued against a timestamped key-value store.

Kafka Connect

Summary

Ready to get started with Apache Kafka 3.7.0? Check out all the details in the release notes and download Apache Kafka 3.7.0.

This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 146 contributors:
Abhijeet Kumar, Akhilesh Chaganti, Alieh, Alieh Saeedi, Almog Gavra, Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield, Anna Sophie Blee-Goldman, Anton Agestam, Apoorv Mittal, Arnout Engelen, Arpit Goyal, Artem Livshits, Ashwin Pankaj, ashwinpankaj, atu-sharm, bachmanity1, Bob Barrett, Bruno Cadonna, Calvin Liu, Cerchie, chern, Chris Egerton, Christo Lolov, Colin Patrick McCabe, Colt McNealy, Crispin Bernier, David Arthur, David Jacot, David Mao, Deqi Hu, Dimitar Dimitrov, Divij Vaidya, Dongnuo Lyu, Eaugene Thomas, Eduwer Camacaro, Eike Thaden, Federico Valeri, Florin Akermann, Gantigmaa Selenge, Gaurav Narula, gongzhongqiang, Greg Harris, Guozhang Wang, Gyeongwon, Do, Hailey Ni, Hanyu Zheng, Hao Li, Hector Geraldino, hudeqi, Ian McDonald, Iblis Lin, Igor Soarez, iit2009060, Ismael Juma, Jakub Scholz, James Cheng, Jason Gustafson, Jay Wang, Jeff Kim, Jim Galasyn, John Roesler, Jorge Esteban Quilcate Otoya, Josep Prat, José Armando García Sancio, Jotaniya Jeel, Jouni Tenhunen, Jun Rao, Justine Olshan, Kamal Chandraprakash, Kirk True, kpatelatwork, kumarpritam863, Laglangyue, Levani Kokhreidze, Lianet Magrans, Liu Zeyu, Lucas Brutschy, Lucia Cerchie, Luke Chen, maniekes, Manikumar Reddy, mannoopj, Maros Orsak, Matthew de Detrich, Matthias J. Sax, Max Riedel, Mayank Shekhar Narula, Mehari Beyene, Michael Westerby, Mickael Maison, Nick Telford, Nikhil Ramakrishnan, Nikolay, Okada Haruki, olalamichelle, Omnia G.H Ibrahim, Owen Leung, Paolo Patierno, Philip Nee, Phuc-Hong-Tran, Proven Provenzano, Purshotam Chauhan, Qichao Chu, Matthias J. Sax, Rajini Sivaram, Renaldo Baur Filho, Ritika Reddy, Robert Wagner, Rohan, Ron Dagostino, Roon, runom, Ruslan Krivoshein, rykovsi, Sagar Rao, Said Boudjelda, Satish Duggana, shuoer86, Stanislav Kozlovski, Taher Ghaleb, Tang Yunzi, TapDang, Taras Ledkov, tkuramoto33, Tyler Bertrand, vamossagar12, Vedarth Sharma, Viktor Somogyi-Vass, Vincent Jiang, Walker Carlson, Wuzhengyu97, Xavier Léauté, Xiaobing Fang, yangy0000, Ritika Reddy, Yanming Zhou, Yash Mayya, yuyli, zhaohaidao, Zihao Lin, Ziming Deng

Apache Kafka 3.5.2 Release Announcement

11 December 2023 - Luke Chen (@showuon1)

We are proud to announce the release of Apache Kafka 3.5.2. This is a bugfix release. It contains many bug fixes including upgrades the Snappy and Rocksdb dependencies. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.5.2 from any version 0.8.x through 3.4.x section in the documentation for the list of notable changes and detailed upgrade steps.

Summary

This was a community effort, so thank you to everyone who contributed to this release:
A. Sophie Blee-Goldman, Anna Sophie Blee-Goldman, atu-sharm, bachmanity1, Bill Bejeck, Calvin Liu, Chase Thomas, Chris Egerton, Christo Lolov, Colin P. McCabe, Colin Patrick McCabe, David Arthur, David Jacot, Divij Vaidya, Farooq Qaiser, Federico Valeri, flashmouse, Florin Akermann, Greg Harris, Guozhang Wang, Hao Li, hudeqi, Ismael Juma, Jason Gustafson, José Armando García Sancio, Josep Prat, Levani Kokhreidze, lixy, Lucas Brutschy, Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael Maison, Nick Telford, Okada Haruki, Omnia G.H Ibrahim, Philip Nee, Qichao Chu (@ex172000), Mickael Maison, Rajini Sivaram, Ritika Reddy, Robert Wagner, Rohan, Ron Dagostino, Sagar Rao, Said Boudjelda, sciclon2, Viktor Somogyi-Vass, Vincent Jiang, Xiaobing Fang, Yash Mayya

Apache Kafka 3.6.1 Release Announcement

7 December 2023 - Mickael Maison (@MickaelMaison)

We are proud to announce the release of Apache Kafka 3.6.1. This is a bugfix release. It contains 30 fixes including upgrades to the ZooKeeper and Netty dependencies. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.6.1 from any version 0.8.x through 3.5.x section in the documentation for the list of notable changes and detailed upgrade steps.

Summary

This was a community effort, so thank you to everyone who contributed to this release:
Anna Sophie Blee-Goldman, Arpit Goyal, atu-sharm, Bill Bejeck, Chris Egerton, Colin P. McCabe, David Arthur, David Jacot, Divij Vaidya, Federico Valeri, Greg Harris, Guozhang Wang, Hao Li, hudeqi, iit2009060, Ismael Juma, Jorge Esteban Quilcate Otoya, Josep Prat, Jotaniya Jeel, Justine Olshan, Kamal Chandraprakash, kumarpritam863, Levani Kokhreidze, Lucas Brutschy, Luke Chen, Manikumar Reddy, Matthias J. Sax, Mayank Shekhar Narula, Mickael Maison, Nick Telford, Philip Nee, Qichao Chu, Rajini Sivaram, Robert Wagner, Sagar Rao, Satish Duggana, Walker Carlson, Xiaobing Fang, Yash Mayya

Apache Kafka 3.6.0 Release Announcement

10 Oct 2023 - Satish Duggana (@SatishDuggana)

We are proud to announce the release of Apache Kafka 3.6.0. This release contains many new features and improvements. This blog post will highlight some of the more prominent features. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation for the list of notable changes and detailed upgrade steps.

The 3.6.0 release marked the first production ready release for the migration of Kafka clusters from a ZooKeeper metadata system to a KRaft metadata system. Users wanting to perform this migration are advised to first upgrade to 3.6.2 or 3.7.1 to receive important bug fixes.

See the ZooKeeper to KRaft migration operations documentation for details. Note that support for JBOD is still not available for KRaft clusters, therefore clusters utilizing JBOD can not be migrated. See KIP-858 for details regarding KRaft and JBOD.

Support for Delegation Tokens in KRaft (KAFKA-15219) was completed in 3.6, further reducing the gap of features between ZooKeeper-based Kafka clusters and KRaft. Migration of delegation tokens from ZooKeeper to KRaft is also included in 3.6.

Tiered Storage is an early access feature. It is currently only suitable for testing in non production environments. See the Early Access Release Notes for more details.

Note: ZooKeeper is marked as deprecated since 3.5.0 release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more information, please see the documentation for ZooKeeper Deprecation

Kafka Broker, Controller, Producer, Consumer and Admin Client

  • KIP-405: Kafka Tiered Storage (Early Access):
    Introduces Tiered Storage to Kafka. Note that this is an early access feature only advised for use in non-production environments (see the early access notes for more information). This feature provides a separation of computation and storage in the broker for pluggable storage tiering natively in Kafka Tiered Storage brings a seamless extension of storage to remote objects with minimal operational changes.
  • KIP-890: Transactions Server Side Defense (Part 1):
    Hanging transactions can negatively impact your read committed consumers and prevent compacted logs from being compacted. KIP-890 helps address hanging transactions by verifying partition additions. Part 2 of KIP-890 will optimize verification, which currently adds an extra hop.
    In 3.6.0, transaction verification will prevent hanging transactions on data partitions. In the next release, transactional offset commits will also be covered.
  • KIP-797: Accept duplicate listener on port for IPv4/IPv6:
    Until now, Kafka has not supported duplicate listeners on the same port. This works when using only a single IP stack, but presents an issue if you are working with both IPv4 and IPv6. With KIP-797, brokers can be configured with listeners that have the same port on different IP stacks. This update does not affect advertised listeners, which already have this feature.
  • KIP-863: Reduce CompletedFetch#parseRecord() memory copy:
    Reduces memory allocation and improves memory performance during record deserialization by using a ByteBuffer instead of byte[] for deserialization, which improves efficiency. Updated public interfaces include the Deserializer class, ByteBufferDeserializer class, and StringDeserializer class.
  • KIP-868: Metadata Transactions:
    Improves the overall durability of the KRaft layer by adding metadata transactions that consist of:
    • BeginTransaction
    • Number of records
    • EndTransaction or AbortTransaction
    KRaft uses record batches as a mechanism for atomicity. Typically, there was a limit to the fetch size on the Raft consensus layer, and the controller could generate a set of atomic records that exceeded this limit. This update introduces marker records that allow larger sets of atomic records to be sent to the Raft consensus layer in multiple batches. This bypasses the fetch limit.
  • KIP-902: Upgrade Zookeeper to 3.8.2:
    This upgrades the ZooKeeper version that is bundled with Kafka to version 3.8.2. The new version includes several updates and security improvements.
  • KIP-917: Additional custom metadata for remote log segment:
    It introduces having an optional custom metadata as part of remote log segment metadata. RemoteStorageManager returns the optional custom metadata when copyLogSegmentData() is invoked. It will be passed along with remote log segment metadata.
  • KIP-937: Improve Message Timestamp Validation:
    It improves data integrity and prevents potential pitfalls caused by inaccurate timestamp handling by adding more validation logic for message timestamps. While past timestamps are a normal occurrence in Kafka, future timestamps might represent an incorrectly formatted integer. KIP-937 rejects messages with future timestamps and provides a descriptive exemption.
  • KIP-938: Add more metrics for measuring KRaft performance:
    Adds new controller, loader, and snapshot emitter KRaft performance metrics.

Kafka Streams

  • KIP-923: Add A Grace Period to Stream Table Join:
    Adds a grace period to stream-table joins to improve table-side out-of-order data handling. The joined object has a new method called "withGracePeriod" that will cause the table side lookup to only happen after the grace period has passed.
  • KIP-941: Range queries to accept null lower and upper bounds:
    Previously, RangeQuery did not support null to specify “no upper/lower bound”. KIP-941 allows users to pass null into withRange(...) for lower/upper bounds to specify a full or half-open range:
    • `withRange(null, null)` == `withNoBounds()`
    • `withRange(lower, null)` == `withLowerBound(lower)`
    • `withRange(null, upper)` == `withUpperBound(upper)`

Kafka Connect

  • KIP-793: Allow sink connectors to be used with topic-mutating SMTs:
    Adds support for topic-mutating SMTs for async sink connectors. This is to address an incompatibility between sink connectors overriding the SinkTask::preCommit method and SMTs that mutate the topic field of a SinkRecord .
  • KIP-875: First-class offsets support in Kafka Connect:
    Provides first-class admin support for offsets in Kafka Connect. KIP-875 Part 1 added endpoints to get offsets and a new STOPPED state for connectors. The alter offsets and reset offsets endpoints have now been added.
    Action Description
    GET /connectors/{connector}/offsets Retrieve the offsets for a connector; the connector must exist
    PATCH /connectors/{connector}/offsets Alter the offsets for a connector; the connector must exist, and must be in the STOPPED state
    DELETE /connectors/{connector}/offsets Reset the offsets for a connector; the connector must exist, and must be in the STOPPED state
    PUT /connectors/{connector}/pause Pause the connector; the connector must exist
  • KIP-898: Modernize Connect plugin discovery:
    With KIP-898, Connect workers can now read from ServiceLoader manifests and module info directly during startup for more efficient plugin class discovery. Note that this update allows connector developers to add service declarations to their plugins.

Summary

Ready to get started with Apache Kafka 3.6.0? Check out all the details in the release notes and download Apache Kafka 3.6.0.

This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 139 contributors:
A. Sophie Blee-Goldman, Aaron Ai, Abhijeet Kumar, aindriu-aiven, Akhilesh Chaganti, Alexandre Dupriez, Alexandre Garnier, Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield, Andrew Grant, Aneel Kumar, Anton Agestam, Artem Livshits, atu-sharm, bachmanity1, Bill Bejeck, Bo Gao, Bruno Cadonna, Calvin Liu, Chaitanya Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190, Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur, David Jacot, David Mao, dengziming, Deqi Hu, Dimitar Dimitrov, Divij Vaidya, DL1231, Dániel Urbán, Erik van Oosten, ezio, Farooq Qaiser, Federico Valeri, flashmouse, Florin Akermann, Gabriel Oliveira, Gantigmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin, iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson, Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio, Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True, Lianet Magrans, LinShunKang, Liu Zeyu, lixy, Lucas Bradstreet, Lucas Brutschy, Lucent-Wong, Lucia Cerchie, Luke Chen, Manikumar Reddy, Manyanda Chitimbo, Maros Orsak, Matthew de Detrich, Matthias J. Sax, maulin-vasavada, Max Riedel, Mehari Beyene, Michal Cabak (@miccab), Mickael Maison, Milind Mantri, minjian.cai, mojh7, Nikolay, Okada Haruki, Omnia G H Ibrahim, Owen Leung, Philip Nee, prasanthV, Proven Provenzano, Purshotam Chauhan, Qichao Chu, Rajini Sivaram, Randall Hauch, Renaldo Baur Filho, Ritika Reddy, Rittika Adhikari, Rohan, Ron Dagostino, Sagar Rao, Said Boudjelda, Sambhav Jain, Satish Duggana, sciclon2, Shekhar Rajak, Sungyun Hur, Sushant Mahajan, Tanay Karmarkar, tison, Tom Bentley, vamossagar12, Victoria Xia, Vincent Jiang, vveicc, Walker Carlson, Yash Mayya, Yi-Sheng Lien, Ziming Deng, 蓝士钦

Apache Kafka 3.5.1 Release Announcement

21 July 2023 - Divij Vaidya (@DivijVaidya)

We are proud to announce the release of Apache Kafka 3.5.1. This is a security patch release. It upgrades the dependency, snappy-java, to a version which is not vulnerable to CVE-2023-34455. You can find more information about the CVE at Kafka CVE list. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.5.1 from any version 0.8.x through 3.4.x section in the documentation for the list of notable changes and detailed upgrade steps.

Kafka Broker, Controller, Producer, Consumer and Admin Client

  • Upgraded the dependency, snappy-java, to a version which is not vulnerable to CVE-2023-34455. You can find more information about the CVE at Kafka CVE list.
  • Fixed a regression introduced in 3.3.0, which caused security.protocol configuration values to be restricted to upper case only. After the fix, security.protocol values are case insensitive. See KAFKA-15053 for details.

Summary

This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 22 authors. Please report an unintended omission.

Alyssa Huang, Aman Singh, andymg3, Bo Gao, Calvin Liu, Chia-Ping Tsai, Chris Egerton, d00791190, Damon Xie, David Arthur, David Jacot, Divij Vaidya, DL1231, ezio, Manikumar Reddy, Manyanda Chitimbo, Mickael Maison, minjian.cai, Proven Provenzano, Sambhav Jain, vamossagar12, Yash Mayya

Apache Kafka 3.5.0 Release Announcement

15 June 2023 - Mickael Maison (@MickaelMaison)

We are proud to announce the release of Apache Kafka 3.5.0. This release contains many new features and improvements. This blog post will highlight some of the more prominent features. For a full list of changes, be sure to check the release notes.

See the Upgrading to 3.5.0 from any version 0.8.x through 3.4.x section in the documentation for the list of notable changes and detailed upgrade steps.

The ability to migrate Kafka clusters from ZK to KRaft mode with no downtime is still an early access feature. It is currently only suitable for testing in non production environments. See KIP-866 for more details.

Note: ZooKeeper is now marked deprecated in this release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)

Kafka Broker, Controller, Producer, Consumer and Admin Client

  • KIP-881: Rack-aware Partition Assignment for Kafka Consumers: Kafka 3.4.0 only contained the protocol changes for KIP-881. The built-in assignors have now been updated to support rack-awareness.
  • KIP-887: Add ConfigProvider to make use of environment variables: KIP-887 introduces a new ConfigProvider implementation, EnvVarConfigProvider, to retrieve configurations from environment variables.
  • KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for Kafka Brokers: KIP-900 updates the kafka-storage tool and adds a mechanism to configure SCRAM for inter broker authentication with KRaft.
  • KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR: KIP-903 addresses a limitation of the inter broker replication protocol which could lead to data loss in case of a broker failing while another broker had an unclean shutdown.

Kafka Streams

  • KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions: KIP-399 adds a method, handleSerializationException(), to the ProductionExceptionHandler interface to handle any serialization errors encountered while producing records.
  • KIP-889: Versioned State Stores: KIP-889 introduces versioned state stores to improve the accuracy of joins when out of order records are processed.
  • KIP-907: Add Boolean Serde to public interface: Kafka Streams includes built-in Serdes for most primitive types. KIP-907 adds a new one for booleans.

Kafka Connect

  • KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters: KIP-710 improves the dedicated mode of MirrorMaker. It now supports running multiple instances and handling automatic reconfigurations.
  • KIP-875: First-class offsets support in Kafka Connect: KIP-875 adds REST API endpoints for managing offset of Connectors. 3.5.0 only contains endpoints for listing offsets. Endpoints for updating and deleting offsets will come in a future release.
  • KIP-894: Use incrementalAlterConfig for syncing topic configurations: With KIP-894, MirrorMaker can use the IncrementalAlterConfig API when mirroring topic configuration between clusters.
  • KAFKA-14021: MirrorMaker 2 should implement KIP-618 APIs: KAFKA-14021 adds support for exactly-once semantics to MirrorSourceConnector.

Summary

This was a community effort, so thank you to everyone who contributed to this release, including all our users and our 103 authors: A. Sophie Blee-Goldman, Akhilesh Chaganti, Alex Sorokoumov, Alexandre Dupriez, Alyssa Huang, Anastasia Vela, Andreas Maechler, andymg3, Artem Livshits, atu-sharm, bachmanity1, Bill Bejeck, Brendan Ribera, Calvin Liu, Chaitanya Mukka, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Colin P. McCabe, csolidum, Daniel Scanteianu, David Arthur, David Jacot, David Karlsson, David Mao, Dejan Stojadinović, Divij Vaidya, dorwi, drgnchan, Dániel Urbán, Edoardo Comar, egyedt, emilnkrastev, Eric Haag, Farooq Qaiser, Federico Valeri, Gantigmaa Selenge, Greg Harris, Guozhang Wang, Hao Li, Hector Geraldino, Himani Arora, Hoki Min, hudeqi, iamazy, Iblis Lin, Ismael Juma, Ivan Yurchenko, Jakub Scholz, Jason Gustafson, Jeff Kim, Jim Galasyn, Jorge Esteban Quilcate Otoya, Josep Prat, José Armando García Sancio, Juan José Ramos, Junyang Liu, Justine Olshan, Kamal Chandraprakash, Kirk True, Kowshik Prakasam, littlehorse-eng, liuzc9, Lucas Brutschy, Lucia Cerchie, Luke Chen, Manikumar Reddy, Manyanda Chitimbo, Matthew Wong, Matthias J. Sax, Matthias Seiler, Michael Marshall, Mickael Maison, nicolasguyomar, Nikolay, Paolo Patierno, Philip Nee, Pierangelo Di Pilato, Proven Provenzano, Purshotam Chauhan, Qing, Rajini Sivaram, RivenSun, Robert Young, Rohan, Roman Schmitz, Ron Dagostino, Ruslan Krivoshein, Satish Duggana, Shay Elkin, Shekhar Rajak, Simon Woodman, Spacrocket, stejani-cflt, Terry, Tom Bentley, vamossagar12, Victoria Xia, Viktor Somogyi-Vass, Vladimir Korenev, Yash Mayya, Zheng-Xian Li

The contents of this website are © 2024 Apache Software Foundation under the terms of the Apache License v2. Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.
Security |  Donate |  Thanks |  Events |  License |  Privacy
Apache Feather