Read

This Snap fetches and consumes data from Kafka under specific topic or topics, and feed the data to the downstream pipeline.

• Expected input: The input should be a document. User can use parameters to specify the keys of topics, group id, message count or max waiting time in the input document.
• Expected output:  The output would be a binary output.  The data of the binary is the message from Kafka. The header of it contains the topic(under key “kafka_topic”), partition number(under key “kafka_partition”), offset(“kafka_offset”) and group id(“kafka_group_id”) of this message. If the “Pass Through” option is checked, the upstream document will be stored under key “original” of the header. 

[None]

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. See Configuring Kafka Accounts (Deprecated) for information on setting up this type of account.

Required. Topics this reader should subscribe to. Each reader can subscribe to one or more topics. Topics should be separated by comma.

This attribute can be from a text field, pipeline parameters, or upstream document.

Example: topic1,topic2

Default value: [None]

Required. The ID of the consumer group that this reader belongs to. All readers in a consumer group will share the message offset in the Kafka broker. They’ll consume messages cooperatively.

This attribute can be from text field, pipeline parameters or upstream document.

Default value: consumer_group_1

The number of message to read before consuming stop. If it is equal to 0, it'll wait for some time specified in Max Waiting Time and stop. If it is less than 0, it will wait indefinitely.

This attribute can be from text field, pipeline parameters or upstream document.

Default value: 0

Max milliseconds the reader should wait before it gets a message from Kafka. This is the timeout for each fetching request.

This attribute can be from text field, pipeline parameters or upstream document.

Default value: 100

Select this option if pass through the input document as the header of the binary output. This option only works when there is an input view.

Default value: Not selected

Commit offset of the message synchronously. If it is checked, the consumer will commit the offset after each message sending.

Default value: Not selected