Writer

This Snap publishes binary messages to Kafka brokers under a specific topic.

• Expected input:  The input is binary. The size of each binary should  be limited by the configuration of Kafka Broker. By default, it should be less than 1048576 Bytes.
• Expected output:  The output is a document that contains the response from Kafka brokers when receiving the  message. The document includes topic (“kafka_topic”), partition(“kafka_partition”), offset(“kafka_offset”), group id(“kafka_group_id”) and the message body in String (“kafka_msg”). 

[None]

Works in Ultra Task Pipelines.

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. See Configuring Kafka Accounts (Deprecated) for information on setting up this type of account.

Required. The topic that messages from this writer belong to. 

This attribute can be from pipeline parameters.

Example: topic1,topic2

Default value: [None]

Required. The size of data in bytes that the writer buffers before it sends to broker. The end of sending process will be triggered by both Batch size and Linger time depending on which match first. 

This attribute can be from pipeline parameters.

Default value: 16384

The time the writer should wait before it sends the data out. In this period, the writer will keep batching data. The end of sending process will be triggered by both Batch Size and Linger Time depending on who match first.

This attribute can be from pipeline parameters.

Default value: 0

The number of acknowledgments the writer requires the brokers to have received before considering a request complete.

For more details, see: http://kafka.apache.org/documentation.html 

Default value: 1 - Send with out replication acknowledgement

Option whether the writer should send data synchronously. By default, the writer sends data to Kafka broker asynchronously, so that it can batch data. If this option is checked, the writer will block after sending a message until feedback from broker is received. This option may slow down the sending process.

Default value: Not selected