On this Page
Table of Contents | ||||
---|---|---|---|---|
|
Snap Type: | Write | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Description: | This Snap sends messages to the Amazon Simple Queue Service (SQS) for consumption. The Snap supports standard queues as well as FIFO queues. Multiple messages can be sent using the Snap by batching as well as by client-side buffering. ETL Transformations & Data FlowThe SQS Producer Snap loads the messages
Input & Output
| ||||||||||||
Prerequisites: | Users of the Snap must have the following accesses to be able to use this Snap:
| ||||||||||||
Support, Limitations, and Known Issues | Works in Ultra Task PipelinesTasks. The Snap conforms to the limits within SQS. As such the following must be kept in mind when configuring the Snap:
For more information on message-related limits refer to Message-limits. For more information on queue-related limitations please refer to Queue-limits. The AWS extended library uses S3 only if the message size is more than 256 KB. However, you cannot enable the Client-side buffering option in Amazon SQS Snaps if you select the Enable Large Message Support a checkbox in your SQS account. | ||||||||||||
Configurations: | Account & AccessThis Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. See SQS Account for information on setting up this type of account. Views
| ||||||||||||
Troubleshooting: | None at this time. | ||||||||||||
Settings | |||||||||||||
Label* | Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. Default Value: SQS Producer | ||||||||||||
Region | Specify the AWS region where the application is running. This is a suggestible field and suggestions will be populated based on the Account settings. Default value: [None]
| ||||||||||||
Queue name | Specify the name of the target queue. This is a suggestible field and suggestions will be populated based on the region selected. Fore information on queue-related limitations please refer to Queue-limits. Default value: [None]
| ||||||||||||
Queue URL | Conditional. Specify the URL for the Queue, this can be found through the AWS console under the Details section. When this property is configured, the queue specified in Queue name property will be overridden. Default value: [None] | ||||||||||||
Maximum retries | Specify the maximum number of retries to be made in case of connection failures. Default value: 3
| ||||||||||||
Delay second | Specify the time duration in seconds for which the message is visible to consumers after being added to the queue. Default value: 0
| ||||||||||||
Message group ID | Specify a tag that indicates which message group this message belongs to. The field is used and required by FIFO queue. For more information refer to FIFO Queue Recommendations - Message Group Id. Default value: [None] | ||||||||||||
Deduplication ID | The token used for de-duplication of sent messages in FIFO queue. The property is used by FIFO queue only. For more information refer to FIFO Queue Recommendations - Deduplicate Id. Default value: [None] | ||||||||||||
Message attributes | Specify the table of the message attributes. Message attributes are structured metadata for the message to help the consumer identify how to deal with the message without having to read the message body. Each message can have up to 10 attributes. The value type is string. | ||||||||||||
Name | Specify the name for the message attribute. By default this field is expression enabled. Default value: [None] | ||||||||||||
Value | Specify the value for the message attribute. By default this field is expression enabled. Default value: [None] | ||||||||||||
Batch size | Specify the number of Documents the Snap should batch before processing. The maximum number is 10. This is ignored if client-side buffering is enabled Example: 5 Default value:10 | ||||||||||||
Client-side buffering | Select this checkbox to enable the Snap to automatically buffer a maximum 10 outbound requests (or less as configured in the Max buffer size property) and send them as a batch and allow message pre-fetching when consuming. The subsequent properties need to be configured if this property is enabled. Default value: Not selected
| ||||||||||||
Buffer configurations | The properties within this are the configurations for the client-side buffer. | ||||||||||||
Max buffer size | Specify the maximum number of messages that are batched together in a single request. The higher the setting, the fewer batches are required to carry out the same number of requests. The maximum batch size is 10 Default value:10 | ||||||||||||
Max buffer open time (ms) | Specify the maximum time in millisecond the buffer would wait before sending message batches. Default value: 200 | ||||||||||||
Max outbound batches in parallel | Specify the maximum number of concurrent batches for each type of outbound request. The greater the number, the greater the throughput that can be achieved (at the expense of consuming more threads). Default value: 5 | ||||||||||||
|
|
Examples
Basic Use Case
The following Pipeline describes how the Snap functions as a standalone Snap in a Pipeline:
The Snap is configured with the following parameters:
A preview of the output from executing this Snap is as follows:
Exported pipeline is available in the Downloads section below.
Typical Snap Configurations
Key configuration of the Snap lies in how the values are passed. Values can be passed:
- Without Expressions: Values are passed directly in the Snap:
- With Expressions:
- Using pipeline parameters: Values are passed as pipeline parameters:
Advanced Use Case
The following Pipeline demonstrates a business logic involving multiple ETL transformations, that shows how typically in an enterprise environment, the SQS messaging functionality is used. The Pipeline download is available in the Downloads section.
The Pipeline reads and moves documents from the Salesforce instance to the SQS Messaging and then to the Redshift Database.
The ETL transformations:
Extract: The Salesforce Read Snap extracts the Records to be sent to the SQS Messaging instance.
Transform: The JSON Formatter Snap transform the records into the JSON format.
Load: The SQS Producer Snap receives the records from the JSON input.
Extract: The SQS Consumer Snap reads the messages from the specified Queue.
Transform: The Binary to Document Snap converts the records into Document format to be moved onto the Redshift instance.
Load: The Redshift Bulk Load Snap loads the documents into the Redshift database.
Downloads
Multiexcerpt include macro | ||||
---|---|---|---|---|
|
Attachments | ||
---|---|---|
|
Insert excerpt | ||||||
---|---|---|---|---|---|---|
|