Kafka Acknowledge

In this article


You can use the Kafka Acknowledge Snap to notify the Kafka Consumer Snap to commit an offset at the specified metadata in each input document.

  • This Snap should be used only if the Auto commit field in the Consumer Snap is not selected (set to false).
  • This Snap no longer requires a Kafka account.


  • A Confluent Kafka server.
  • An upstream Snap (for example, a Kafka Consumer Snap) that sends metadata to the Kafka Acknowledge Snap in a Pipeline.

Support for Ultra Pipelines   

Works in Ultra Pipelines.



Limitations and Known Issues


Snap Input and Output

Input/OutputType of ViewNumber of ViewsExamples of Upstream and Downstream SnapsDescription
  • Min:1
  • Max:1
  • Mapper Snap
  • Copy Snap

Metadata from an upstream Kafka Consumer Snap. The input data schema is as follows:

"metadata": {
   "topic": "xyz",
   "partition": 2,
   "offset": 523,
   "consumer_group": "CopyGroup1",
   "client_id": "17a9bbc7-da8f-45f8-813e-1ebca9b80383",
   "tracker_index": 0,
   "batch_size": 500,
   "batch_index": 1,
   "record_index": 23,
   "auto_commit": false
  • Min:0
  • Max:1
  • Mapper Snap
  • JSON Formatter

Kafka messages that have been processed and acknowledged.
If the Auto Commit field is set to false in the input document and a notification is sent successfully to the corresponding Consumer Snap, the output schema looks is similar to the following:

 "status": "success",
 "original": {
 "metadata": {
 "consumer_group": "abc",
 "topic": "xyz",
 "partition": 123,
 "offset": 456,
 "auto_commit": false

If the Auto commit field is set to true in the input document, the output schema looks is similar to the following:

 "status": "Auto-commit is on",
 "original": {
 "metadata": {
 "consumer_group": "abc",
 "topic": "xyz",
 "partition": 123,
 "offset": 456,
 "auto_commit": true

Snap Settings

Parameter Name

Data Type

DescriptionDefault ValueExample
Specify a name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.
Kafka AcknowledgeKafka_Acknowledge
Metadata pathStringRequired. Specify the JSON path of the metadata within each input document. metadata$metadata
Snap ExecutionDrop-down list

Select one of the three modes in which the Snap executes:

  • Validate & Execute. Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only. Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled. Disables the Snap and all Snaps downstream from it.

Validate & ExecuteValidate & Execute




Acknowledging Messages

This example Pipeline demonstrates how we use the:

  • Kafka Producer Snap to produce and send messages to a Kafka topic, 
  • Kafka Consumer Snap to read messages from a topic, and
  • Kafka Acknowledge Snap to acknowledge the number of messages read (message count).

First, we use the Sequence Snap to enable the Pipeline to send documents in large numbers. We configure the Sequence Snap to send all the documents starting from 1 through 2500. Hence, we set the Initial value as 1 and Number of documents as 2500.

We configure the Kafka Producer Snap to send the documents to the Topic named SampleKafkaTopic and we set the Partition number to 0 to let the broker decide which partition to use.

We configure the Kafka Consumer Snap to read the messages from the Topic named SampleKafkaTopic and we set the Partition number to 0. The message count is set to 100, which means the Snap consumes 100 messages and sends those messages to the output view.

On successful execution of the Pipeline, we can view the consumed and acknowledged messages in the Pipeline Execution statistics. Note that the Message Count is set to 100 in the Consumer Snap; hence, the Acknowledge Snap acknowledges the same count.

Download the Pipeline.


Important Steps to Successfully Reuse Pipelines

  1. Download and import the Pipeline into SnapLogic.
  2. Configure Snap accounts as applicable.
  3. Provide Pipeline parameters as applicable.

  File Modified

File Example_Kafka_Acknowledge_Acknowledging_Messages.slp

Feb 08, 2021 by Kalpana Malladi

See Also

Snap Pack History

 Click to view/expand
Release Snap Pack VersionDateType  Updates





Upgraded with the latest SnapLogic Platform release.
4.26main11181 StableUpgraded with the latest SnapLogic Platform release.
4.25 Patch425patches10543 Latest
  • Improved the handling of interrupted/aborted Kafka Snaps to ensure proper clean-up of metrics.

  • Optimized the Kafka Producer Snap to initialize the Kafka API only if there is at least one input document.

  • Fixed the issue of account passwords being included in the log messages output of Kafka  Snaps. The password is now hidden in the logs for both Kafka Consumer and Kafka Producer Snaps.
  • The Kafka Producer and Consumer Snaps create a default Client ID string to identify the specific Snap instance.

  • Enhanced the Kafka Producer Snap to support Avro serialization format with a new target data type Avro under Key and Value Serializer.

  • Enhanced the Kafka SSL Account with new fields (Registry Username or Key and Registry Password or Secret) for Schema Registry authentication. The two existing SASL properties (SASL Username and SASL Password) have been revised to SASL Username or Key and SASL Password or Secret respectively.

  • Apache Kafka client library is upgraded from version 2.6.0 to 2.8.0.

  • Confluent Kafka client libraries are updated from version 5.2.1 to 6.2.0.

StableUpgraded with the latest SnapLogic Platform release.
4.24 Patch424patches8805 Latest

Fixed an issue in the Kafka Producer Snap by removing the validation of the account when the Snap initialized in a Pipeline execution.

StableUpgraded with the latest SnapLogic Platform release.
4.23 Patch423patches7900 Latest
  • Removed the Confluent prefix from the Snaps and accounts in the Snap Pack. However, the Snap Pack name continues to be Confluent Kafka Snap Pack.

  • Added the Wait For Full Count check box to the Kafka Consumer Snap to determine how a positive value for the Message Count field must be interpreted.

    • Enabled (by default): The Snap continues polling for messages until the specified count is reached.

    • Disabled: If the number of messages that are currently available is less than the specified count, then the Snap consumes the available messages and terminates.

      Known Issue

      The Wait For Full Count check box activates only when you provide a positive integer value in the Message Count field. However, it does not activate when you use an expression for Message Count even if the value evaluates to a positive number.

      Workaround: To activate this check box, temporarily replace the Message Count expression () with a positive integer, select the desired state for Wait For Full Count, and then restore the original value in the Message Count field.

  • Added support for writing and reading record headers.

    • The Kafka Producer Snap has a new Headers table to configure the Key, Value, and Serializer for each header to be written.

    • The Kafka Consumer Snap reads any headers present on the records it consumes. It provides two new fields to configure how the header values should be deserialized – Default Header Deserializer and Header Deserializers, for any headers that require a deserializer other than the default.

  • Added support for writing and reading each record’s timestamp.

    • The Kafka Producer Snap has a new Timestamp field that can be configured to set each record’s timestamp, which is the number of milliseconds since the epoch (00:00:00 UTC on January 1, 1970). This can be set to an expression that evaluates to a long integer, or a string that can be parsed as a long integer, or a date. If you specify no expression, or the value is empty, then the timestamp is set to the current time.

      The Timestamp field is relevant only if the Kafka topic is configured with message.timestamp.type = CreateTime (which is the default). For more information, see the official Kafka documentation.

    • The Kafka Consumer Snap has a new check box, Include Timestamp, which by default is disabled for backward compatibility. If enabled, the output for each record includes its timestamp in the metadata.

  • The Kafka Producer Snap has a new check box, Output Records, to determine the format of each output document when configured with an output view.

    • Disabled (by default): The Snap’s output includes only the basic metadata (topic, partition, offset) for each record, along with the original input document.

    • Enabled: Each output document contains a more complete representation of the record produced, including its key, value, headers, and timestamp.

  • The Kafka Consumer Snap has a new field, Output Mode, with two options:

    • One output document per record (the default): Every record received from Kafka has a corresponding output document.

    • One output document per batch: Preserves the batching of records as received from Kafka. Every poll that returns a non-empty set of records results in a single output document containing the list of records as batchbatch_size and batch_index. This mode is especially useful when Auto Commit is disabled and Acknowledge Mode is Wait after each batch of records, depending on the nature of the processing between the Kafka Consumer and the Kafka Acknowledge Snaps.

  • Removed the Add 1 to Offsets check box from the Kafka Consumer Snap.

  • Removed the Account tab from the Kafka Acknowledge Snap, because this Snap no longer needs an account.

Fixed an intermittent issue of Confluent Kafka Consumer Snap stopping when Message Count is 0 while the records are available. The Snap will now stop only if a poll returns no records twice in a row, and has partitions assigned before and after each poll.

StableUpgraded with the latest SnapLogic Platform release.
4.21 Patch421patches6136 Latest

Enhanced the Confluent Kafka Consumer Snap with an option to allow reprocessing of records that remain unacknowledged due to acknowledge timeouts. This ensures that all records from Kafka are processed.

4.21 Patch

421patches5862 Latest

Fixes the Confluent Kafka Consumer Snap that aborts abruptly upon getting a CommitFailedException error. Upgraded the Kafka client library from version 2.2 to 2.5.




Enhanced the Confluent Kafka Consumer Snap to significantly reduce the volume of commit requests for offsets when using the Record acknowledgement mode (Wait after each record), and thereby, improve performance and stability.

4.20 Patch

confluentkafka8747 Latest

Updated the Confluent Kafka SSL Account settings to make the Truststore filepath and Truststore password properties optional. 

StableUpgraded with the latest SnapLogic Platform release.
StableUpgraded with the latest SnapLogic Platform release.
4.18 Patchconfluentkafka8111 Latest

Fixed an issue with the Confluent Kafka Producer Snap wherein the Topic property doesn't evaluate expression against input documents.

4.18 Patchconfluentkafka8006 Latest

Fixed an exception that occurs while consuming log-compacted topics, in the Confluent Kafka Consumer Snap.

4.18 Patchconfluentkafka7784 Latest

Fixed the Consumer Snap to stop quickly and cleanly on CLOSE and STOP lifecycle events. 

4.18 Patch

confluentkafka7732 Latest

Added support for Kerberos-based authentication to enable connecting to Kerberos-enabled Kafka servers.

StableUpgraded with the latest SnapLogic Platform release.

4.17 Patch

confluentkafka7537 Latest

Updated the following Snaps:

  • Confluent Kafka Consumer Snap
    • Properties added: Max poll recordsAcknowledge mode, Key deserializer, Value deserializer, Add 1 to offsets.
    • Properties removed: Auto commit interval (msec), Standardize error output.
    • Diagnostic fields added in the metadata object in the Snap's output:
      • client_id: The unique ID assigned to a single Kafka consumer client instance. All documents written by a Consumer Snap during a single pipeline execution or validation will have the same value for this field.
      • batch_size: The number of messages in the batch of messages that the current message came from. Note that “batch” isn’t Kafka terminology; we’re using it to mean the set of messages received from the broker in a single call to Kafka’s poll method.
      • batch_index: The current batch number relative to a single consumer instance.  This number starts at 0 for the first batch and is only incremented for batches which contain at least one message.
      • record_index: The current record number relative to its batch.  The first message in a batch has an index of 0; the last has an index of batch_size - 1.
  • Confluent Kafka Acknowledge Snap: Added new property–Metadata path.
  • Confluent Kafka Producer Snap: Added new properties–Key serializer and Value serializer.

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

  • Tested and certified all Snaps against CDH 6.1.
  • Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.
4.16 Patchconfluentkafka7118 Latest

Fixed an issue with the Confluent Kafka Producer Snap wherein the Snap fails to produce the upstream Snap's input schema for Partition Number property in the output preview.

4.16 Patch

confluentkafka6891 Latest

Certified Confluent Platform version 5.x.

StableUpgraded with the latest SnapLogic Platform release.
4.15 Patchconfluentkafka6446 Latest

Fixed an issue with the Confluent Kafka Consumer Snap wherein some connections were not closed after a Pipeline is aborted.

StableUpgraded with the latest SnapLogic Platform release.
4.14 Patchconfluentkafka5737 Latest

Added a new property to the Confluent Kafka Producer Snap, Message publish timeout (milliseconds), that lets you specify a timeout value.

4.14 Patchconfluentkafka5663 Latest

Fixed the Confluent Kafka Snaps that do not delete temp files after pipeline execution.

StableUpgraded with the latest SnapLogic Platform release.
4.13 Patch confluentkafka5614 Latest

Fixed the Confluent Kafka Snap Pack that does not delete the temp files after executing pipelines.

4.13 Patch confluentkafka5330 Latest

Added support in Confluent Kafka Producer Snap the ability to write data to multiple partitions, in a round robin fashion.  



StableUpgraded with the latest SnapLogic Platform release.

4.12 Patch 

confluentkafka5087 Latest

Fixed an issue with the Consumer Snap that fails when the specified timeout value is less than the pipeline execution time. 




Enhanced the performance of the Confluent Kafka Producer Snap in writing incoming documents into the Confluent Kafka Cluster.

4.11 Patch

confluentkafka4626 Latest

Fixed an issue with the Confluent Kafka Consumer Snap wherein the Consumer session is not closing properly when manually stopping a pipeline, which causes the subsequent pipeline execution to wait for the previous session to die and not consume messages.

4.11 Patchconfluentkafka4302 Latest

Fixed an issue with the Confluent Kafka Consumer Snap that fetched 500 messages only, when message count set to 0 caused due to driver default value change.


Updated the Confluent Kafka SSL Account with SCRAM_SASL support.




Confluent v3.2 support for Confluent Kafka Snap Pack.

4.9 Patchconfluentkafka3156 Latest

Fixed an issue regarding account validation failure due to unauthorized topics; Enhance the error message for empty topic suggestions

  • Introduced the Confluent Kafka Acknowledge Snap in this release.
  • Enhanced the Consumer and the Producer Snaps to use Kafka Connect API to enhance the performance.
4.8 Patchconfluentkafka2804 Latest

Addressed an issue with Confluent Kafka Producer not including input document schema in Message Key and Value expression properties




New Snap Pack for 4.8. It consists of Confluent Kafka Consumer and Confluent Kafka Producer.