In this article

Overview

You can use the Kafka account type to connect Confluent Kafka Snaps with data sources that use Kafka accounts.

Prerequisites

None.

Limitations

None.

Known Issues

None.

Account Settings

Parameter	Data Type	Description	Default Value	Example
Label	String	Required. Specify a unique label for the account.	N/A	Kafka Account
Bootstrap servers	Use this field set to configure a list of bootstrap servers. Click to add a new row in this table for configuring bootstrap servers. This field set contains the Bootstrap Server field.
Bootstrap Server	String/Expression	Specify the host:port pair to establish the initial connection to the Kafka cluster.	N/A	ec2-55-334-24-55.compute-1.amazonaws.com:9000
Schema registry URL	String/Expression	Specify the schema registry server URL.	N/A	http://ec2-55-334-24-55.compute-1.amazonaws.com:8000
Advanced Kafka Properties	Use this field set to specify any additional Kafka properties for connecting to the Kafka server that are not explicitly provided in the Kafka Snaps. These properties are passed directly to the server and have not been tested by SnapLogic, Inc. This field set contains the following fields: Key Value
Key	String/Expression	Specify the key for any Kafka parameters that are not explicitly supported by the Snaps.	N/A	max.request.size
Value	String//Expression	Specify the value for the corresponding key that are not explicitly supported by the Snaps.	N/A	500

Snap Pack History

Click to view/expand

Release	Snap Pack Version	Date	Type	Updates
November 2024	main29029	13 Nov 2024	Stable	Updated and certified against the current SnapLogic Platform release.
August 2024	438patches28723	07 Nov 2024	Latest	Enhanced the Kafka MSK IAM Account to support debugging for the IAM configuration. When you configure IAM debugging (Global properties and Logging Level) in the Snaplex configuration, the account logs the IAM credential identity in the Snaplex log.
August 2024	438patches28225	25 Sep 2024	-	Enhanced the Kafka MSK IAM Account to support Cross Account IAM Access on Amazon MSK Cluster configured through Transit Gateway and multi VPC Peering.
August 2024	main27765	21 Aug 2024	Stable	Upgraded the `org.json.json` library from v20090211 to v20240303, which is fully backward compatible.
August 2024	437patches27730	13 Aug 2024	Latest	Fixed an issue with the Kafka Consumer Snap where the Seek Type of Beginning or End (skip all existing messages) may not work correctly after a rebalance.
May 2024	437patches27404	26 Jul 2024	Latest	Fixed an issue with the Kafka SSL Account that displayed an error when the Kafka Consumer Snap could not commit offsets during the rebalancing process. This behavior avoids unstable partition allocation. Now, the Snap logs a warning, allowing the Kafka Consumer to continue processing instead of displaying an exception.
May 2024	437patches26651	12 Jun 2024	Latest	Added Kafka OAuth2 Account to the Kafka Snap Pack to enable the usage of OAuth within the Kafka SASL_SSL framework.
May 2024	main26341	08 May 2024	Stable	Kafka Snap Pack: Enhanced the following fields to support expressions through pipeline parameters: Truststore Password Keystore File Password SSL Key Password SASL Password or Secret Registry Password or Secret
February 2024	main25112	14 Feb 2024	Stable	Updated and certified against the current SnapLogic Platform release.
November 2023	main23721	08 Nov 2023	Stable	Updated and certified against the current SnapLogic Platform release.
August 2023	main22460	16 Aug 2023	Stable	Updated and certified against the current SnapLogic Platform release.
May 2023	433patches21070	12 May 2023	Latest	Fixed an issue with the Kafka Consumer Snap that caused it to skip records in a partition when a pipeline failed without successfully processing any records from that partition. This was an edge case that was only possible with a specific configuration of the Snap and a specific ordering of events during termination of the pipeline.
May 2023	main21015	10 May 2023	Stable	Upgraded with the latest SnapLogic Platform release.
February 2023	432patches20143	10 Mar 2023	Latest/Stable	Fixed an issue with the Kafka Producer Snap to avoid an error about `delivery.timeout.ms`. The Snap now adjusts the value of the Message Publish Timeout setting automatically and displays a warning in the Pipeline Execution Statistics.
February 2023	main19844	09 Feb 2023	Stable	Upgraded with the latest SnapLogic Platform release.
November 2022	431patches19770	12 Jan 2023	Latest	Fixed a memory issue that occurred when the Kafka Producer Snap was used to produce many records in a long-running pipeline.
November 2022	431patches19211	08 Dec 2022	Latest	Enhanced the Kafka Consumer Snap with the new Max Poll Interval (ms) field, which allows you to specify the time limit between subsequent calls to poll. This field corresponds to the Kafka max.poll.interval.ms property.
November 2022	main18944	10 Nov 2022	Stable	Upgraded with the latest SnapLogic Platform release.
August 2022	main17386	11 Aug 2022	Stable	Upgraded Kafka client libraries to Apache-Kafka Version 3.1.1 and Confluent-Kafka Version 7.1.1. Optimized the performance of the Snap with the support for LZ4 and ZSTD compression types of the messages produced. The Kafka account that connects to the Kafka Schema Registry uses TLS v1.2 protocol instead of the older version of TLS, which has security vulnerabilities.
4.29 Patch	429patches17109	21 Jul 2022	Latest	Enhanced the Kafka Producer Snap with LZ4 and ZSTD Compression Types for compressing the messages that optimize Snap's performance. Fixed an issue with the Kafka Producer Snap where the Snap wrote an output document even when there was an error. Fixed an issue with Kafka Account where the expression buttons were enabled, by default, for the Bootstrap Server and Advanced Property Value fields. Upgraded Kafka client libraries to Apache-Kafka Version 3.1.1 and Confluent-Kafka Version 7.1.1. Enhanced the Kafka account, which is configured to connect to the Kafka Schema Registry, to use TLS v1.2 protocol instead of TLS that has security vulnerabilities.
4.29	main15993	14 May 2022	Stable	The Kafka Snap Pack now supports Azure Event Hubs streaming service. Learn more: Azure Event Hubs configuration in Azure portal for Kafka SSL Account.
4.28 Patch	428patches14904	28 Feb 2022	Latest	Fixed an issue with Kafka Snaps where the Snaps failed with null pointer exceptions when logging the cluster information with the controller’s identity not known.
4.28	main14627	12 Feb 2022	Stable	Upgraded with the latest SnapLogic Platform release.
4.27	main12833	13 Nov 2021	Stable	Upgraded with the latest SnapLogic Platform release.
4.26	main11181	14 Aug 2021	Stable	Upgraded with the latest SnapLogic Platform release.
4.25 Patch	425patches10543	07 Jul 2021	Latest	Improved the handling of interrupted/aborted Kafka Snaps to ensure proper clean-up of metrics. Optimized the Kafka Producer Snap to initialize the Kafka API only if there is at least one input document. Fixed the issue of account passwords being included in the log messages output of Kafka Snaps. The password is now hidden in the logs for both Kafka Consumer and Kafka Producer Snaps. The Kafka Producer and Consumer Snaps create a default Client ID string to identify the specific Snap instance. Enhanced the Kafka Producer Snap to support Avro serialization format with a new target data type Avro under Key and Value Serializer. Enhanced the Kafka SSL Account with new fields (Registry Username or Key and Registry Password or Secret) for Schema Registry authentication. The two existing SASL properties (SASL Username and SASL Password) have been revised to SASL Username or Key and SASL Password or Secret respectively. Apache Kafka client library is upgraded from version 2.6.0 to 2.8.0. Confluent Kafka client libraries are updated from version 5.2.1 to 6.2.0.
4.25	main9554	08 May 2021	Stable	Upgraded with the latest SnapLogic Platform release.
4.24 Patch	424patches8805	08 Mar 2021	Latest	Fixed an issue in the Kafka Producer Snap by removing the validation of the account when the Snap initialized in a Pipeline execution.
4.24	main8556	13 Feb 2021	Stable	Upgraded with the latest SnapLogic Platform release.
4.23 Patch	423patches7900	11 Jan 2021	Latest	Removed the Confluent prefix from the Snaps and accounts in the Snap Pack. However, the Snap Pack name continues to be Confluent Kafka Snap Pack. Added the Wait For Full Count check box to the Kafka Consumer Snap to determine how a positive value for the Message Count field must be interpreted. Enabled (by default): The Snap continues polling for messages until the specified count is reached. Disabled: If the number of messages that are currently available is less than the specified count, then the Snap consumes the available messages and terminates. Known Issue The Wait For Full Count check box activates only when you provide a positive integer value in the Message Count field. However, it does not activate when you use an expression for Message Count even if the value evaluates to a positive number. Workaround: To activate this check box, temporarily replace the Message Count expression () with a positive integer, select the desired state for Wait For Full Count, and then restore the original value in the Message Count field. Added support for writing and reading record headers. The Kafka Producer Snap has a new Headers table to configure the Key, Value, and Serializer for each header to be written. The Kafka Consumer Snap reads any headers present on the records it consumes. It provides two new fields to configure how the header values should be deserialized – Default Header Deserializer and Header Deserializers, for any headers that require a deserializer other than the default. Added support for writing and reading each record’s timestamp. The Kafka Producer Snap has a new Timestamp field that can be configured to set each record’s timestamp, which is the number of milliseconds since the epoch (00:00:00 UTC on January 1, 1970). This can be set to an expression that evaluates to a long integer, or a string that can be parsed as a long integer, or a date. If you specify no expression, or the value is empty, then the timestamp is set to the current time. The Timestamp field is relevant only if the Kafka topic is configured with `message.timestamp.type` `= CreateTime` (which is the default). For more information, see the official Kafka documentation. The Kafka Consumer Snap has a new check box, Include Timestamp, which by default is disabled for backward compatibility. If enabled, the output for each record includes its timestamp in the metadata. The Kafka Producer Snap has a new check box, Output Records, to determine the format of each output document when configured with an output view. Disabled (by default): The Snap’s output includes only the basic metadata (topic, partition, offset) for each record, along with the original input document. Enabled: Each output document contains a more complete representation of the record produced, including its key, value, headers, and timestamp. The Kafka Consumer Snap has a new field, Output Mode, with two options: One output document per record (the default): Every record received from Kafka has a corresponding output document. One output document per batch: Preserves the batching of records as received from Kafka. Every poll that returns a non-empty set of records results in a single output document containing the list of records as `batch`, `batch_size` and `batch_index`. This mode is especially useful when Auto Commit is disabled and Acknowledge Mode is Wait after each batch of records, depending on the nature of the processing between the Kafka Consumer and the Kafka Acknowledge Snaps. Removed the Add 1 to Offsets check box from the Kafka Consumer Snap. Removed the Account tab from the Kafka Acknowledge Snap, because this Snap no longer needs an account.
4.23	main7430	14 Nov 2020	Stable	Fixed an intermittent issue of Confluent Kafka Consumer Snap stopping when Message Count is 0 while the records are available. The Snap will now stop only if a poll returns no records twice in a row, and has partitions assigned before and after each poll.
4.22	main6403	12 Sep 2020	Stable	Upgraded with the latest SnapLogic Platform release.
4.21 Patch	421patches6136	27 Jul 2020	Latest	Enhanced the Confluent Kafka Consumer Snap with an option to allow reprocessing of records that remain unacknowledged due to acknowledge timeouts. This ensures that all records from Kafka are processed.
4.21 Patch	421patches5862	08 Jun 2020	Latest	Fixes the Confluent Kafka Consumer Snap that aborts abruptly upon getting a `CommitFailedException` error. Upgraded the Kafka client library from version 2.2 to 2.5.
4.21	snapsmrc542	09 May 2020	Stable	Enhanced the Confluent Kafka Consumer Snap to significantly reduce the volume of commit requests for offsets when using the Record acknowledgement mode (Wait after each record), and thereby, improve performance and stability.
4.20 Patch	confluentkafka8747	28 Feb 2020	Latest	Updated the Confluent Kafka SSL Account settings to make the Truststore filepath and Truststore password properties optional.
4.20	snapsmrc535	08 Feb 2020	Stable	Upgraded with the latest SnapLogic Platform release.
4.19	snaprsmrc528	14 Nov 2019	Stable	Upgraded with the latest SnapLogic Platform release.
4.18 Patch	confluentkafka8111	23 Oct 2019	Latest	Fixed an issue with the Confluent Kafka Producer Snap wherein the Topic property doesn't evaluate expression against input documents.
4.18 Patch	confluentkafka8006	10 Oct 2019	Latest	Fixed an exception that occurs while consuming log-compacted topics, in the Confluent Kafka Consumer Snap.
4.18 Patch	confluentkafka7784	30 Aug 2019	Latest	Fixed the Consumer Snap to stop quickly and cleanly on CLOSE and STOP lifecycle events.
4.18 Patch	confluentkafka7732	15 Aug 2019	Latest	Added support for Kerberos-based authentication to enable connecting to Kerberos-enabled Kafka servers.
4.18	snapsmrc523	10 Aug 2019	Stable	Upgraded with the latest SnapLogic Platform release.
4.17 Patch	confluentkafka7537	12 Jul 2019	Latest	Updated the following Snaps: Confluent Kafka Consumer Snap Properties added: Max poll records, Acknowledge mode, Key deserializer, Value deserializer, Add 1 to offsets. Properties removed: Auto commit interval (msec), Standardize error output. Diagnostic fields added in the metadata object in the Snap's output: client_id: The unique ID assigned to a single Kafka consumer client instance. All documents written by a Consumer Snap during a single pipeline execution or validation will have the same value for this field. batch_size: The number of messages in the batch of messages that the current message came from. Note that “batch” isn’t Kafka terminology; we’re using it to mean the set of messages received from the broker in a single call to Kafka’s poll method. batch_index: The current batch number relative to a single consumer instance. This number starts at 0 for the first batch and is only incremented for batches which contain at least one message. record_index: The current record number relative to its batch. The first message in a batch has an index of 0; the last has an index of batch_size - 1. Confluent Kafka Acknowledge Snap: Added new property–Metadata path. Confluent Kafka Producer Snap: Added new properties–Key serializer and Value serializer.
4.17	ALL7402	11 Jun 2019	Latest	Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.
4.17	snapsmrc515	11 Jun 2019	Latest	Tested and certified all Snaps against CDH 6.1. Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.
4.16 Patch	confluentkafka7118	15 Apr 2019	Latest	Fixed an issue with the Confluent Kafka Producer Snap wherein the Snap fails to produce the upstream Snap's input schema for Partition Number property in the output preview.
4.16 Patch	confluentkafka6891	04 Mar 2019	Latest	Certified Confluent Platform version 5.x.
4.16	snapsmrc508	16 Feb 2019	Stable	Upgraded with the latest SnapLogic Platform release.
4.15 Patch	confluentkafka6446	26 Dec 2018	Latest	Fixed an issue with the Confluent Kafka Consumer Snap wherein some connections were not closed after a Pipeline is aborted.
4.15	snapsmrc500	15 Dec 2018	Stable	Upgraded with the latest SnapLogic Platform release.
4.14 Patch	confluentkafka5737	18 Sep 2018	Latest	Added a new property to the Confluent Kafka Producer Snap, Message publish timeout (milliseconds), that lets you specify a timeout value.
4.14 Patch	confluentkafka5663	13 Aug 2018	Latest	Fixed the Confluent Kafka Snaps that do not delete temp files after pipeline execution.
4.14	snapsmrc490	11 Aug 2018	Stable	Upgraded with the latest SnapLogic Platform release.
4.13 Patch	confluentkafka5614	09 Aug 2018	Latest	Fixed the Confluent Kafka Snap Pack that does not delete the temp files after executing pipelines.
4.13 Patch	confluentkafka5330	22 Jun 2018	Latest	Added support in Confluent Kafka Producer Snap the ability to write data to multiple partitions, in a round robin fashion.
4.13	snapsmrc486	12 May 2018	Stable	Upgraded with the latest SnapLogic Platform release.
4.12 Patch	confluentkafka5087	23 Apr 3018	Latest	Fixed an issue with the Consumer Snap that fails when the specified timeout value is less than the pipeline execution time.
4.12	snapsmrc480	17 Feb 2018	Stable	Enhanced the performance of the Confluent Kafka Producer Snap in writing incoming documents into the Confluent Kafka Cluster.
4.11 Patch	confluentkafka4626	02 Feb 2018	Latest	Fixed an issue with the Confluent Kafka Consumer Snap wherein the Consumer session is not closing properly when manually stopping a pipeline, which causes the subsequent pipeline execution to wait for the previous session to die and not consume messages.
4.11 Patch	confluentkafka4302	29 Nov 2017	Latest	Fixed an issue with the Confluent Kafka Consumer Snap that fetched 500 messages only, when message count set to 0 caused due to driver default value change.
4.11	snapsmrc465	11 Nov 2017	Stable	Updated the Confluent Kafka SSL Account with SCRAM_SASL support.
4.10	snapsmrc414	12 Aug 2017	Stable	Confluent v3.2 support for Confluent Kafka Snap Pack.
4.9 Patch	confluentkafka3156	22 Jun 2017	Latest	Fixed an issue regarding account validation failure due to unauthorized topics; Enhance the error message for empty topic suggestions
4.9	snapsmrc405	13 May 2017	Stable	Introduced the Confluent Kafka Acknowledge Snap in this release. Enhanced the Consumer and the Producer Snaps to use Kafka Connect API to enhance the performance.
4.8 Patch	confluentkafka2804	10 Apr 2017	Latest	Addressed an issue with Confluent Kafka Producer not including input document schema in Message Key and Value expression properties
4.8	snapsmrc398	11 Feb 2017	Stable	New Snap Pack for 4.8. It consists of Confluent Kafka Consumer and Confluent Kafka Producer.

Overview

Prerequisites

Limitations

Known Issues

See Also