/
Kafka Snap Pack

Kafka Snap Pack

In this article

Overview

You can use this Snap Pack for accessing Apache Kafka and Confluent Kafka to produce and consume messages.

Supported Versions

The Kafka Snap Pack uses the following client libraries:

  • Apache-Kafka Version 3.5.2

  • Confluent-Kafka Version 7.5.2

Temporary Files

During execution, data processing on Snaplex nodes occurs principally in-memory as streaming and is unencrypted. When larger datasets are processed that exceed the available compute memory, Snap writes Pipeline data to local storage as unencrypted to optimize the performance. These temporary files are deleted when the Snap/Pipeline execution completes. You can configure the temporary data's location in the Global properties table of Snaplex's node properties, which can also help avoid Pipeline errors due to the unavailability of space. For more information, see Temporary Folder in Configuration Options

Snap Pack History

 Click to view/expand
Release Snap Pack VersionDateType  Updates
February 2025main29887 StableUpdated and certified against the current SnapLogic Platform release.
November 2024main29029 StableUpdated and certified against the current SnapLogic Platform release.
August 2024438patches28723 Latest

Enhanced the Kafka MSK IAM Account to support debugging for the IAM configuration. When you configure IAM debugging (Global properties and Logging Level) in the Snaplex configuration, the account logs the IAM credential identity in the Snaplex log.

August 2024438patches28225 -

Enhanced the Kafka MSK IAM Account to support Cross Account IAM Access on Amazon MSK Cluster configured through Transit Gateway and multi VPC Peering.

August 2024

main27765

 

Stable

Upgraded the org.json.json library from v20090211 to v20240303, which is fully backward compatible.

August 2024437patches27730 Latest

Fixed an issue with the Kafka Consumer Snap where the Seek Type of Beginning or End (skip all existing messages) may not work correctly after a rebalance. 

May 2024437patches27404 Latest

Fixed an issue with the Kafka SSL Account that displayed an error when the Kafka Consumer Snap could not commit offsets during the rebalancing process. This behavior avoids unstable partition allocation. Now, the Snap logs a warning, allowing the Kafka Consumer to continue processing instead of displaying an exception.

May 2024437patches26651 Latest

Added Kafka OAuth2 Account to the Kafka Snap Pack to enable the usage of OAuth within the Kafka SASL_SSL framework.

May 2024main26341 Stable

Kafka Snap Pack: Enhanced the following fields to support expressions through pipeline parameters:

  • Truststore Password

  • Keystore File Password

  • SSL Key Password

  • SASL Password or Secret

  • Registry Password or Secret

February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023main23721

 

StableUpdated and certified against the current SnapLogic Platform release.

August 2023

main22460

 

Stable

Updated and certified against the current SnapLogic Platform release.

May 2023

433patches21070

 Latest

Fixed an issue with the Kafka Consumer Snap that caused it to skip records in a partition when a pipeline failed without successfully processing any records from that partition. This was an edge case that was only possible with a specific configuration of the Snap and a specific ordering of events during termination of the pipeline.

May 2023

main21015 

Stable

Upgraded with the latest SnapLogic Platform release.

February 2023432patches20143 Latest/StableFixed an issue with the Kafka Producer Snap to avoid an error about delivery.timeout.ms. The Snap now adjusts the value of the Message Publish Timeout setting automatically and displays a warning in the Pipeline Execution Statistics.
February 2023main19844 StableUpgraded with the latest SnapLogic Platform release.
November 2022431patches19770 LatestFixed a memory issue that occurred when the Kafka Producer Snap was used to produce many records in a long-running pipeline.
November 2022431patches19211 Latest

Enhanced the Kafka Consumer Snap with the new Max Poll Interval (ms) field, which allows you to specify the time limit between subsequent calls to poll. This field corresponds to the Kafka max.poll.interval.ms property.

November 2022main18944 StableUpgraded wit