Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

On this Page

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

Snap Type:Write
Description:

This Snap sends messages to the Amazon Simple Queue Service (SQS) for consumption. The Snap supports standard queues as well as FIFO queues. Multiple messages can be sent using the Snap by batching as well as by client-side buffering. 


ETL Transformations & Data Flow

The SQS Producer Snap loads the messages

  • Extract Binary data (including header) from the upstream Snap containing the message to be sent to the queue.
  • Transform the input Binary into the message to be sent.
  • Load the data into the Producer Snap and send the message to the target queue.

Input & Output

  • InputThis Snap needs one upstream Snap with Binary output. e.g: DocumentToBinary or JsonFormatter
  • Output: The Snap can have one Document output, containing the ID of the message just sent
Prerequisites:

Users of the Snap must have the following accesses to be able to use this Snap:

  • Valid AWS account. 
  • Access Key and Secret Key.
  • IAM Role access
Support, Limitations, and Known Issues

Works in Ultra Task PipelinesTasks.

The Snap conforms to the limits within SQS. As such the following must be kept in mind when configuring the Snap:

  • The maximum allowed size for the message, including message attributes, is 256KB. For a workaround to overcome this please refer to Messages larger than 256KB.
  • The maximum number of inflight messages per queue are:
    • Standard queue - 120,000 inflight messages per queue. 
    • FIFO queue - 20,000 inflight messages per queue.
  • The queue name for a FIFO queue can be a maximum of 80 characters long, including the ".fifo" suffix.
  • Queue names are case-sensitive.

For more information on message-related limits refer to Message-limits. For more information on queue-related limitations please refer to Queue-limits.

The AWS extended library uses S3 only if the message size is more than 256 KB. However, you cannot enable the Client-side buffering option in Amazon SQS Snaps if you select the Enable Large Message Support a checkbox in your SQS account.

Configurations:

Account & Access

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. See SQS Account for information on setting up this type of account.

Views

InputThis Snap has exactly one document input view.
OutputThis Snap has at most one document output view.
ErrorThis Snap has at most one document error view and produces zero or more documents in the view.


Troubleshooting:None at this time.

Settings

Label*

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Default Value: SQS Producer

ExampleSQS Producer

Region

Specify the AWS region where the application is running. This is a suggestible field and suggestions will be populated based on the Account settings.

Default value: [None]
Example: us-east-1

Note
titleFIFO Queues - Regions

FIFO Queue is available only in US East (N. Virginia), US East (Ohio), US West (Oregon), and EU (Ireland) regions.


Queue name

Specify the name of the target queue. This is a suggestible field and suggestions will be populated based on the region selected. Fore information on queue-related limitations please refer to Queue-limits.

Default value: [None]
Example: snapqatest

Note
titleFIFO Queues - Queue name

When using FIFO queues, it is important that the queue name be suffixed with ".fifo". The maximum character limit for the queue name is 80 characters (including the ".fifo" suffix.


Queue URL

Conditional. Specify the URL for the Queue, this can be found through the AWS console under the Details section. When this property is configured, the queue specified in Queue name property will be overridden. 

Default value: [None]

Maximum retries

Specify the maximum number of retries to be made in case of connection failures. 

Default value: 3
Example: 2

Multiexcerpt include macro
nameretries
pageFile Reader

Delay second

Specify the time duration in seconds for which the message is visible to consumers after being added to the queue.

Default value: 0
Example: 1

Note

This property is not applicable to FIFO queues. 


Message group ID

Specify a tag that indicates which message group this message belongs to. The field is used and required by FIFO queue. For more information refer to FIFO Queue Recommendations - Message Group Id.

Default value: [None]
Example: 5

Deduplication ID

The token used for de-duplication of sent messages in FIFO queue. The property is used by FIFO queue only. For more information refer to FIFO Queue Recommendations - Deduplicate Id.

Default value: [None]
Example: 2

Message attributesSpecify the table of the message attributes. Message attributes are structured metadata for the message to help the consumer identify how to deal with the message without having to read the message body. Each message can have up to 10 attributes. The value type is string.
Name

Specify the name for the message attribute. By default this field is expression enabled.

Default value: [None]
Example: snap-runtime

Value

Specify the value for the message attribute.  By default this field is expression enabled. 

Default value: [None]
Example: snap.instanceId

Batch size

Specify the number of Documents the Snap should batch before processing. The maximum number is 10. This is ignored if client-side buffering is enabled

Example: 5

Default value:10

Client-side buffering

Select this checkbox to enable the Snap to automatically buffer a maximum 10 outbound requests (or less as configured in the Max buffer size property) and send them as a batch and allow message pre-fetching when consuming. The subsequent properties need to be configured if this property is enabled.

Default value: Not selected


Note

This property is not applicable to FIFO queues. 


Buffer configurationsThe properties within this are the configurations for the client-side buffer. 
Max buffer size

Specify the maximum number of messages that are batched together in a single request. The higher the setting, the fewer batches are required to carry out the same number of requests. The maximum batch size is 10

Default value:10
Example: 4

Max buffer open time (ms)

Specify the maximum time in millisecond the buffer would wait before sending message batches.  

Default value: 200
Example: 200

Max outbound batches in parallel

Specify the maximum number of concurrent batches for each type of outbound request. The greater the number, the greater the throughput that can be achieved (at the expense of consuming more threads).  

Default value: 5
Example: 3

Multiexcerpt include macro
nameSnap Execution
pageSOAP Execute

Multiexcerpt include macro
nameExecution_Detail_Write
pageSOAP Execute

Examples


Basic Use Case

The following Pipeline describes how the Snap functions as a standalone Snap in a Pipeline:

The Snap is configured with the following parameters:

A preview of the output from executing this Snap is as follows:

Exported pipeline is available in the Downloads section below.

Typical Snap Configurations

Key configuration of the Snap lies in how the values are passed. Values can be passed:

  • Without Expressions: Values are passed directly in the Snap:
    Image Removed
  • With Expressions:
    • Using pipeline parameters: Values are passed as pipeline parameters:

                    Image Removed   
Image Removed


Advanced Use Case


The following Pipeline demonstrates a business logic involving multiple ETL transformations, that shows how typically in an enterprise environment, the SQS messaging functionality is used. The Pipeline download is available in the Downloads section.

The Pipeline reads and moves documents from the Salesforce instance to the SQS Messaging and then to the Redshift Database.

Image Modified

Image Modified

The ETL transformations:

Extract: The Salesforce Read Snap extracts the Records to be sent to the SQS Messaging instance.

Transform: The JSON Formatter Snap transform the records into the JSON format.

Load: The SQS Producer Snap receives the records from the JSON input.


Extract: The SQS Consumer Snap reads the messages from the specified Queue.

Transform: The Binary to Document Snap converts the records into Document format to be moved onto the Redshift instance.

Load: The Redshift Bulk Load Snap loads the documents into the Redshift database.

Downloads

Multiexcerpt include macro
namedownload_instructions
pageOpenAPI


Attachments
patterns*.slp,*.zip
 

Insert excerpt
Amazon SQS Snap Pack
Amazon SQS Snap Pack
nopaneltrue