Input

Document

• Min:1

• Max:1

• Mapper Snap

• Copy Snap

• Sequence Snap

A key/value pair for the Kafka message.

Output

Document

• Min:1

• Max:1

• Not required: The Consumer Snap is decoupled from the Producer Snap.

Kafka message records. Example,

[{"kafka_offset":"7139","kafka_partition":"0","kafka_topic":"PC_Test","original":{"msgID":"id1100","msgVal":"msg1100"}}]

Label

String

Required. The name for the Snap. You can modify the default name to be specific and meaningful, especially if you have more than one of the same Snaps in your pipeline.

Default Value: Kafka Producer
Example: Producer

Client ID

String/Expression

Specify the Client ID to be used in the Kafka message. If you do not specify a Client ID, a random string is used.

This Client ID is used for correlating the requests sent to the brokers with each client.

Default Value: N/A
Example: testclient001

Topic

String/Expression/Suggestion

Specify the topic to publish the message.
If the topic does not exist, a new topic and its partitions are created based on the configuration.

Default Value: N/A
Example: t1

Partition Number

String/Expression/Suggestion

Specify the partition number in which the messages should be published.
If you do not specify the partition number, the server publishes the messages to the next partition chosen, based on the configuration.

Default Value: N/A
Example: 0

Message Key

String/Expression

Enter the key for the message to be sent out.

Default Value: N/A
Example: customerID

Message Value

String/Expression

Required. Provide the value for the corresponding message key.
The messages are sent with a value and, optionally, a key.

Default Value: N/A
Example:

• JSON.stringify($)

• 1000

Timestamp

String/Expression

Specify a timestamp for the message in number, string, or date format.
A timestamp value is the number of milliseconds that have elapsed since midnight, January 1, 1970 UTC.

Default Value: CreateTime
Example: Date.parse($timestamp, "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")

Key Serializer

Dropdown list

Select the target data type for the Key Serializer of each record. The available options are:

• String 

• JSON (arbitrary complex data, schema-less)

• JSON_SR (requires schema registry)

• Avro (requires schema registry)

• JSON (arbitrary complex data, schema-less)

• Avro (arbitrary complex data, schema-centric)

• ByteArray (binary content, such as images)

• Int16 (16-bit integer)

• Int32 (32-bit integer)

• Int64 (64-bit integer)

• Float32 (32-bit floating-point number)

• Float64 (64-bit floating-point number)

• BigInteger (arbitrary precision integer)

• BigDecimal (arbitrary precision decimal number)

Default Value: String
Example: JSON

Value Serializer

Dropdown list

Choose the target data type for the Value Serializer of each record. The available options are:

• String

• JSON

• JSON_SR (requires schema registry)

• Avro (requires schema registry)

• ByteArray

• Int16 (16-bit integer)

• Int32 (32-bit integer)

• Int64 (64-bit integer)

• Float32 (32-bit floating-point number)

• Float64 (64-bit floating-point number)

• BigInteger (arbitrary precision integer)

• BigDecimal (arbitrary precision decimal number)

Default Value: String
Example: ByteArray

Headers

Use this field set to define an optional set of headers for each record. Click 

String/Expression

Specify a name to use for the Header.

You can add multiple headers with the same key in either of the following ways:

• Configure multiple rows in the Header table with the same key.

• Configure a row in the Header table with a value that evaluates to a list or an array.

Default Value: N/A
Example: City

String/Expression

Specify a value for the header.

Default Value: N/A
Example: SFO

Dropdown list

Select a Serializer for the header to convert the value from a specific data type such as string or Int32.

Default Value: String
Example: ByteArray

Acknowledgements

Dropdown list

Select the type of acknowledgment for the message, which determines the number of acknowledgments that the Producer requires the server to have received before considering a request to be complete.
The available options are:

• 0 - Send without any acknowledgment: The Producer does not wait for any acknowledgement by the server.

• 1 - Send with leader acknowledgment: The Producer waits for the replication leader to acknowledge the message received by the server.

• all - Send with full acknowledgment: The Producer waits for all replicas to acknowledge the message was received by the server.

Default Value: 1 - Send with leader acknowledgment
Example: all - Send with full acknowledgement

Batch Size (bytes)

Integer/Expression

Specify the number of bytes that are received before messages are sent to the server.

Default Value: 16384
Example: 15000

Linger Time (milliseconds)

Integer/Expression

Specify the linger time for the messages to be sent.

Linger time is the time in milliseconds, the Producer Snap should wait before it sends
out the data. During this period, the Producer will keep sending data in batches. The end of the sending process is triggered by both the Batch Size and Linger Time depending on which setting matches first.

Default Value: 0
Example: 1

Retries

Integer/Expression

Set the total number of times to attempt a message delivery in case of failure.

Default Value: 0
Example: 3

Preserve Message Order With Retry

Checkbox

Select this check box to resend the messages in the same order each time.

Default status: Deselected

Compression Type

Dropdown list

Choose the compression type before sending the messages. The available options are:

• NONE

• GZIP

• LZ4

• SNAPPY

• ZSTD

Default Value: None
Example: GZIP

Learn more:

Send Synchronously

Checkbox

Select this check box to send each message synchronously. The consumer blocks the message until metadata from the broker is received. 

Default status: Deselected

Output Records

Checkbox

Select this check box to include the record’s data and metadata in each output document, that is, key, value, headers, and timestamp.
If deselected, each document includes only the basic metadata (topic, partition, offset) for each record, including the original input document. 
A copy of each input document is stored in the output document under the key original.

Default status: Deselected

Message Publish Timeout (milliseconds)

Integer/Expression

Specify the timeout value in milliseconds to publish a message.
If the Snap fails to publish the message until the specified time, a timeout error appears.

Default Value: 60000
Example: 60000

Snap Execution

Dropdown list

Select one of the three modes in which the Snap executes. Available options are:

• Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

• Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

• Disabled: Disables the Snap and all Snaps that are downstream from it.

Default Value: Execute only
Example: Validate & Execute