Kafka Snap Pack
Kalpana Malladi
Diane Miller (Deactivated)
Amritesh Singh
In this article
Overview
You can use this Snap Pack for accessing Apache Kafka and Confluent Kafka to produce and consume messages.
Supported Versions
The Kafka Snap Pack uses the following client libraries:
Apache-Kafka Version 3.5.2
Confluent-Kafka Version 7.5.2
Temporary Files
During execution, data processing on Snaplex nodes occurs principally in-memory as streaming and is unencrypted. When larger datasets are processed that exceed the available compute memory, Snap writes Pipeline data to local storage as unencrypted to optimize the performance. These temporary files are deleted when the Snap/Pipeline execution completes. You can configure the temporary data's location in the Global properties table of Snaplex's node properties, which can also help avoid Pipeline errors due to the unavailability of space. For more information, see Temporary Folder in Configuration Options.
Snap Pack History
Release | Snap Pack Version | Date | Type | Updates |
---|---|---|---|---|
November 2024 | main29029 | Stable | Updated and certified against the current SnapLogic Platform release. | |
August 2024 | 438patches28723 | Latest | Enhanced the Kafka MSK IAM Account to support debugging for the IAM configuration. When you configure IAM debugging (Global properties and Logging Level) in the Snaplex configuration, the account logs the IAM credential identity in the Snaplex log. | |
August 2024 | 438patches28225 | - | Enhanced the Kafka MSK IAM Account to support Cross Account IAM Access on Amazon MSK Cluster configured through Transit Gateway and multi VPC Peering. | |
August 2024 | main27765 | | Stable | Upgraded the |
August 2024 | 437patches27730 | Latest | Fixed an issue with the Kafka Consumer Snap where the Seek Type of Beginning or End (skip all existing messages) may not work correctly after a rebalance. | |
May 2024 | 437patches27404 | Latest | Fixed an issue with the Kafka SSL Account that displayed an error when the Kafka Consumer Snap could not commit offsets during the rebalancing process. This behavior avoids unstable partition allocation. Now, the Snap logs a warning, allowing the Kafka Consumer to continue processing instead of displaying an exception. | |
May 2024 | 437patches26651 | Latest | Added Kafka OAuth2 Account to the Kafka Snap Pack to enable the usage of OAuth within the Kafka SASL_SSL framework. | |
May 2024 | main26341 | Stable | Kafka Snap Pack: Enhanced the following fields to support expressions through pipeline parameters:
| |
February 2024 | main25112 | Stable | Updated and certified against the current SnapLogic Platform release. | |
November 2023 | main23721 |
| Stable | Updated and certified against the current SnapLogic Platform release. |
August 2023 | main22460 |
| Stable | Updated and certified against the current SnapLogic Platform release. |
May 2023 | 433patches21070 | Latest | Fixed an issue with the Kafka Consumer Snap that caused it to skip records in a partition when a pipeline failed without successfully processing any records from that partition. This was an edge case that was only possible with a specific configuration of the Snap and a specific ordering of events during termination of the pipeline. | |
May 2023 | main21015 | Stable | Upgraded with the latest SnapLogic Platform release. | |
February 2023 | 432patches20143 | Latest/Stable | Fixed an issue with the Kafka Producer Snap to avoid an error about delivery.timeout.ms . The Snap now adjusts the value of the Message Publish Timeout setting automatically and displays a warning in the Pipeline Execution Statistics. | |
February 2023 | main19844 | Stable | Upgraded with the latest SnapLogic Platform release. | |
November 2022 | 431patches19770 | Latest | Fixed a memory issue that occurred when the Kafka Producer Snap was used to produce many records in a long-running pipeline. | |
November 2022 | 431patches19211 | Latest | Enhanced the Kafka Consumer Snap with the new Max Poll Interval (ms) field, which allows you to specify the time limit between subsequent calls to poll. This field corresponds to the Kafka max.poll.interval.ms property. | |
November 2022 | main18944 | Stable | Upgraded with the latest SnapLogic Platform release. | |
August 2022 | main17386 | Stable |
| |
4.29 Patch | 429patches17109 | Latest |
| |
4.29 | main15993 | Stable | The Kafka Snap Pack now supports Azure Event Hubs streaming service. Learn more: Azure Event Hubs configuration in Azure portal for Kafka SSL Account. | |
4.28 Patch | 428patches14904 | Latest | Fixed an issue with Kafka Snaps where the Snaps failed with null pointer exceptions when logging the cluster information with the controller’s identity not known. | |
4.28 | main14627 | Stable | Upgraded with the latest SnapLogic Platform release. | |
4.27 | main12833 | | Stable | Upgraded with the latest SnapLogic Platform release. |
4.26 | main11181 | Stable | Upgraded with the latest SnapLogic Platform release. | |
4.25 Patch | 425patches10543 | Latest |
| |
4.25 | main9554 | Stable | Upgraded with the latest SnapLogic Platform release. | |
4.24 Patch | 424patches8805 | Latest | Fixed an issue in the Kafka Producer Snap by removing the validation of the account when the Snap initialized in a Pipeline execution. | |
4.24 | main8556 | Stable | Upgraded with the latest SnapLogic Platform release. | |
4.23 Patch | 423patches7900 | Latest |
|