Enhanced the Kafka MSK IAM Account to support debugging for the IAM configuration. When you configure IAM debugging (Global properties and Logging Level) in the Snaplex configuration, the account logs the IAM credential identity in the Snaplex log.

Enhanced the Kafka MSK IAM Account to support Cross Account IAM Access on Amazon MSK Cluster configured through Transit Gateway and multi VPC Peering.

August 2024

main27765

21 Aug 2024 

Stable

Upgraded the org.json.json library from v20090211 to v20240303, which is fully backward compatible.

Fixed an issue with the Kafka Consumer Snap where the Seek Type of Beginning or End (skip all existing messages) may not work correctly after a rebalance. 

Fixed an issue with the Kafka SSL Account that displayed an error when the Kafka Consumer Snap could not commit offsets during the rebalancing process. This behavior avoids unstable partition allocation. Now, the Snap logs a warning, allowing the Kafka Consumer to continue processing instead of displaying an exception.

Added Kafka OAuth2 Account to the Kafka Snap Pack to enable the usage of OAuth within the Kafka SASL_SSL framework.

Kafka Snap Pack: Enhanced the following fields to support expressions through pipeline parameters:

• Truststore Password

• Keystore File Password

• SSL Key Password

• SASL Password or Secret

• Registry Password or Secret

August 2023

main22460

Stable

Updated and certified against the current SnapLogic Platform release.

433patches21070

Fixed an issue with the Kafka Consumer Snap that caused it to skip records in a partition when a pipeline failed without successfully processing any records from that partition. This was an edge case that was only possible with a specific configuration of the Snap and a specific ordering of events during termination of the pipeline.

May 2023

Stable

Upgraded with the latest SnapLogic Platform release.

Enhanced the Kafka Consumer Snap with the new Max Poll Interval (ms) field, which allows you to specify the time limit between subsequent calls to poll. This field corresponds to the Kafka max.poll.interval.ms property.

• Upgraded Kafka client libraries to Apache-Kafka Version 3.1.1 and Confluent-Kafka Version 7.1.1.

• Optimized the performance of the Snap with the support for LZ4 and ZSTD compression types of the messages produced.

• The Kafka account that connects to the Kafka Schema Registry uses TLS v1.2 protocol instead of the older version of TLS, which has security vulnerabilities.

• Enhanced the Kafka Producer Snap with LZ4 and ZSTD Compression Types for compressing the messages that optimize Snap's performance.

• Fixed an issue with the Kafka Producer Snap where the Snap wrote an output document even when there was an error.

• Fixed an issue with Kafka Account where the expression buttons were enabled, by default, for the Bootstrap Server and Advanced Property Value fields.

• Upgraded Kafka client libraries to Apache-Kafka Version 3.1.1 and Confluent-Kafka Version 7.1.1.

• Enhanced the Kafka account, which is configured to connect to the Kafka Schema Registry, to use TLS v1.2 protocol instead of TLS that has security vulnerabilities.

Fixed an issue with Kafka Snaps where the Snaps failed with null pointer exceptions when logging the cluster information with the controller’s identity not known.

4.27

main12833

13 Nov 2021 

Stable

• Improved the handling of interrupted/aborted Kafka Snaps to ensure proper clean-up of metrics.

• Optimized the Kafka Producer Snap to initialize the Kafka API only if there is at least one input document.

• Fixed the issue of account passwords being included in the log messages output of Kafka  Snaps. The password is now hidden in the logs for both Kafka Consumer and Kafka Producer Snaps.

• The Kafka Producer and Consumer Snaps create a default Client ID string to identify the specific Snap instance.

• Enhanced the Kafka Producer Snap to support Avro serialization format with a new target data type Avro under Key and Value Serializer.

• Enhanced the Kafka SSL Account with new fields (Registry Username or Key and Registry Password or Secret) for Schema Registry authentication. The two existing SASL properties (SASL Username and SASL Password) have been revised to SASL Username or Key and SASL Password or Secret respectively.

• Apache Kafka client library is upgraded from version 2.6.0 to 2.8.0.

• Confluent Kafka client libraries are updated from version 5.2.1 to 6.2.0.

Fixed an issue in the Kafka Producer Snap by removing the validation of the account when the Snap initialized in a Pipeline execution.

• Removed the Confluent prefix from the Snaps and accounts in the Snap Pack. However, the Snap Pack name continues to be Confluent Kafka Snap Pack.
  
  

• Added the Wait For Full Count check box to the Kafka Consumer Snap to determine how a positive value for the Message Count field must be interpreted.

• Enabled (by default): The Snap continues polling for messages until the specified count is reached.

• Disabled: If the number of messages that are currently available is less than the specified count, then the Snap consumes the available messages and terminates.
  

  Known Issue

  The Wait For Full Count check box activates only when you provide a positive integer value in the Message Count field. However, it does not activate when you use an expression for Message Count even if the value evaluates to a positive number.

  Workaround: To activate this check box, temporarily replace the Message Count expression () with a positive integer, select the desired state for Wait For Full Count, and then restore the original value in the Message Count field.

• Added support for writing and reading record headers.

• The Kafka Producer Snap has a new Headers table to configure the Key, Value, and Serializer for each header to be written.

• The Kafka Consumer Snap reads any headers present on the records it consumes. It provides two new fields to configure how the header values should be deserialized – Default Header Deserializer and Header Deserializers, for any headers that require a deserializer other than the default.
  
  

• Added support for writing and reading each record’s timestamp.

• The Kafka Producer Snap has a new Timestamp field that can be configured to set each record’s timestamp, which is the number of milliseconds since the epoch (00:00:00 UTC on January 1, 1970). This can be set to an expression that evaluates to a long integer, or a string that can be parsed as a long integer, or a date. If you specify no expression, or the value is empty, then the timestamp is set to the current time.
  

  The Timestamp field is relevant only if the Kafka topic is configured with message.timestamp.type = CreateTime (which is the default). For more information, see the official Kafka documentation.

• The Kafka Consumer Snap has a new check box, Include Timestamp, which by default is disabled for backward compatibility. If enabled, the output for each record includes its timestamp in the metadata.
  
  

• The Kafka Producer Snap has a new check box, Output Records, to determine the format of each output document when configured with an output view.

• Disabled (by default): The Snap’s output includes only the basic metadata (topic, partition, offset) for each record, along with the original input document.

• Enabled: Each output document contains a more complete representation of the record produced, including its key, value, headers, and timestamp.
  
  

• The Kafka Consumer Snap has a new field, Output Mode, with two options:

• One output document per record (the default): Every record received from Kafka has a corresponding output document.

• One output document per batch: Preserves the batching of records as received from Kafka. Every poll that returns a non-empty set of records results in a single output document containing the list of records as batch, batch_size and batch_index. This mode is especially useful when Auto Commit is disabled and Acknowledge Mode is Wait after each batch of records, depending on the nature of the processing between the Kafka <