In this article

Overview

You can use this the Kafka Acknowledge Snap to notify the Kafka Consumer Snap to commit an offset at the specified metadata in each input document.
Image Modified

Prerequisites

• A Confluent Kafka server with a valid account.
• The Kafka Acknowledge Snap in a pipeline must receive the metadata from an

  An upstream Snap

  ,

  (for example, a Kafka Consumer Snap) that sends metadata to the Kafka Acknowledge Snap in a Pipeline.

Support for Ultra Pipelines   

Works in Ultra Pipelines.

Prerequisites

None.

Limitations

...

None.

and Known Issues

None.

Snap Input and Output

"metadata": {
   "topic": "xyz",
   "partition": 2,
   "offset": 523,
   "consumer_group": "CopyGroup1",
   "client_id": "17a9bbc7-da8f-45f8-813e-1ebca9b80383",
   "tracker_index": 0,
   "batch_size": 500,
   "batch_index": 1,
   "record_index": 23,
   "auto_commit": false
}

{
 "status": "success",
 "original": {
 "metadata": {
 "consumer_group": "abc",
 "topic": "xyz",
 "partition": 123,
 "offset": 456,
 "auto_commit": false
 }
 }
}

{
 "status": "Auto-commit is on",
 "original": {
 "metadata": {
 "consumer_group": "abc",
 "topic": "xyz",
 "partition": 123,
 "offset": 456,
 "auto_commit": true
 }
}

...

...

First, we use the Sequence Snap to enable the pipeline Pipeline to send the documents in large numbers. We configure the Sequence Snap to read 2500 documents with the initial value as 1 and hence the Snap numbers all the send all the documents starting from 1 through 2500. Hence, we set the Initial value as 1 and Number of documents as 2500.

We configure the Kafka Producer Snap to send the documents to the Topic , named SampleKafkaTopic and we set the Partition number to 0 to let the broker decide which partition to use.

We configure the Kafka Consumer Snap to read the messages from the Topic,  named SampleKafkaTopic and we set the Partition number to 0. The message count is set to 100, which means the Snap consumes 100 messages and sends those messages to the output view.

On successful execution of the Pipeline, we can view the consumed and acknowledged messages in the Pipeline Execution statistics as shown below. Note that the Message Count is set to 100 in the Consumer Snap; hence, hence the Acknowledge Snap acknowledges the same count.

...

...

See Also

• Apache Kafka
• Confluent - Schema Registry
• Getting Started with SnapLogic
• Snap Support for Ultra Pipelines
• SnapLogic Product Glossary