SQS Producer

On this Page

Snap Type:Write
Description:

This Snap sends messages to the Amazon Simple Queue Service (SQS) for consumption. The Snap supports standard queues as well as FIFO queues. Multiple messages can be sent using the Snap by batching as well as by client-side buffering. 


ETL Transformations & Data Flow

The SQS Producer Snap loads the messages

  • Extract Binary data (including header) from the upstream Snap containing the message to be sent to the queue.
  • Transform the input Binary into the message to be sent.
  • Load the data into the Producer Snap and send the message to the target queue.

Input & Output

  • InputThis Snap needs one upstream Snap with Binary output. e.g: DocumentToBinary or JsonFormatter
  • Output: The Snap can have one Document output, containing the ID of the message just sent
Prerequisites:

Users of the Snap must have the following accesses to be able to use this Snap:

  • Valid AWS account. 
  • Access Key and Secret Key.
  • IAM Role access
Support, Limitations, and Known Issues

Works in Ultra Tasks.

The Snap conforms to the limits within SQS. As such the following must be kept in mind when configuring the Snap:

  • The maximum allowed size for the message, including message attributes, is 256KB. For a workaround to overcome this please refer to Messages larger than 256KB.
  • The maximum number of inflight messages per queue are:
    • Standard queue - 120,000 inflight messages per queue. 
    • FIFO queue - 20,000 inflight messages per queue.
  • The queue name for a FIFO queue can be a maximum of 80 characters long, including the ".fifo" suffix.
  • Queue names are case-sensitive.

For more information on message-related limits refer to Message-limits. For more information on queue-related limitations please refer to Queue-limits.

The AWS extended library uses S3 only if the message size is more than 256 KB. However, you cannot enable the Client-side buffering option in Amazon SQS Snaps if you select the Enable Large Message Support a checkbox in your SQS account.

Configurations:

Account & Access

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. See SQS Account for information on setting up this type of account.

Views

InputThis Snap has exactly one document input view.
OutputThis Snap has at most one document output view.
ErrorThis Snap has at most one document error view and produces zero or more documents in the view.
Troubleshooting:None at this time.

Settings

Label*

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Default Value: SQS Producer



ExampleSQS Producer

Region

Specify the AWS region where the application is running. This is a suggestible field and suggestions will be populated based on the Account settings.

Default value: [None]
Example: us-east-1

FIFO Queues - Regions

FIFO Queue is available only in US East (N. Virginia), US East (Ohio), US West (Oregon), and EU (Ireland) regions.

Queue name

Specify the name of the target queue. This is a suggestible field and suggestions will be populated based on the region selected. Fore information on queue-related limitations please refer to Queue-limits.

Default value: [None]
Example: snapqatest

FIFO Queues - Queue name

When using FIFO queues, it is important that the queue name be suffixed with ".fifo". The maximum character limit for the queue name is 80 characters (including the ".fifo" suffix.

Queue URL

Conditional. Specify the URL for the Queue, this can be found through the AWS console under the Details section. When this property is configured, the queue specified in Queue name property will be overridden. 

Default value: [None]

Maximum retries

Specify the maximum number of retries to be made in case of connection failures. 

Default value: 3
Example: 2

  • If the value is larger than 0, the Snap first downloads the target file into a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and attempts to download the file again from the beginning. When the download is successful, the Snap streams the data from the temporary file to the downstream Pipeline. All temporary local files are deleted when they are no longer needed.
  • Ensure that the local drive has sufficient free disk space to store the temporary local file.

Delay second

Specify the time duration in seconds for which the message is visible to consumers after being added to the queue.

Default value: 0
Example: 1

This property is not applicable to FIFO queues. 

Message group ID

Appears when you select a FIFO queue.

Specify a tag that indicates which message group this message belongs to. The field is used and required by FIFO queue. For more information refer to FIFO Queue Recommendations - Message Group Id.

Default value: [None]
Example: 5

Deduplication ID

Appears when you select a FIFO queue.

The token used for de-duplication of sent messages in FIFO queue. The property is used by FIFO queue only. For more information refer to FIFO Queue Recommendations - Deduplicate Id.

Default value: [None]
Example: 2

Message attributesSpecify the table of the message attributes. Message attributes are structured metadata for the message to help the consumer identify how to deal with the message without having to read the message body. Each message can have up to 10 attributes. The value type is string.
Name

Specify the name for the message attribute. By default this field is expression enabled.

Default value: [None]
Example: snap-runtime

Value

Specify the value for the message attribute.  By default this field is expression enabled. 

Default value: [None]
Example: snap.instanceId

Batch size

Specify the number of Documents the Snap should batch before processing. The maximum number is 10. This is ignored if client-side buffering is enabled

Example: 5

Default value:10

Client-side buffering

Select this checkbox to enable the Snap to automatically buffer a maximum 10 outbound requests (or less as configured in the Max buffer size property) and send them as a batch and allow message pre-fetching when consuming. The subsequent properties need to be configured if this property is enabled.

Default value: Not selected


This property is not applicable to FIFO queues. 

Buffer configurations

Appears when you select the Client-side buffering checkbox.

The properties within this are the configurations for the client-side buffer. 

Max buffer size

Specify the maximum number of messages that are batched together in a single request. The higher the setting, the fewer batches are required to carry out the same number of requests. The maximum batch size is 10

Default value:10
Example: 4

Max buffer open time (ms)

Specify the maximum time in millisecond the buffer would wait before sending message batches.  

Default value: 200
Example: 200

Max outbound batches in parallel

Specify the maximum number of concurrent batches for each type of outbound request. The greater the number, the greater the throughput that can be achieved (at the expense of consuming more threads).  

Default value: 5
Example: 3

Snap execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Examples


Basic Use Case

The following Pipeline describes how the Snap functions as a standalone Snap in a Pipeline:

The Snap is configured with the following parameters:

A preview of the output from executing this Snap is as follows:

Exported pipeline is available in the Downloads section below.


Advanced Use Case


The following Pipeline demonstrates a business logic involving multiple ETL transformations, that shows how typically in an enterprise environment, the SQS messaging functionality is used. The Pipeline download is available in the Downloads section.

The Pipeline reads and moves documents from the Salesforce instance to the SQS Messaging and then to the Redshift Database.

The ETL transformations:

Extract: The Salesforce Read Snap extracts the Records to be sent to the SQS Messaging instance.

Transform: The JSON Formatter Snap transform the records into the JSON format.

Load: The SQS Producer Snap receives the records from the JSON input.


Extract: The SQS Consumer Snap reads the messages from the specified Queue.

Transform: The Binary to Document Snap converts the records into Document format to be moved onto the Redshift instance.

Load: The Redshift Bulk Load Snap loads the documents into the Redshift database.

Downloads

Important steps to successfully reuse Pipelines

  1. Download and import the pipeline into the SnapLogic application.
  2. Configure Snap accounts as applicable.
  3. Provide pipeline parameters as applicable.


 

Snap Pack History

 Click to view/expand
Release
Snap Pack Version 
Date
Type
Updates
November 2024main29029 StableUpdated and certified against the current SnapLogic Platform release.
August 2024main27765 StableUpdated and certified against the current SnapLogic Platform release.
May 2024main26341 StableUpdated and certified against the current SnapLogic Platform release.
February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023main23721 Stable

Updated and certified against the current SnapLogic Platform release.

August 2023434patches23363 Latest

Fixed the behavior change introduced in 4.32 GA related to Amazon SQS Accounts using the STS token. After the 4.32 GA, the account would automatically retry authentication on encountering 401 or 403 errors, and pipelines would not fail even after token expiration. The Amazon SQS Consumer Snap now behaves as expected and fails after the STS token expires.

August 2023main22460 StableUpdated and certified against the current SnapLogic Platform release.
May 2023main21015 StableUpgraded with the latest SnapLogic Platform release.
February 2023432patches19990 Latest

Added support for the SQS Account to reauthenticate when the authentication expires.

Behavior Change

With the 432patches19990, the Amazon SQS account reauthenticates when the authentication expires; the account automatically retires authentication on encountering 401 or 403 errors.

February 2023main19844 StableUpgraded with the latest SnapLogic Platform release.
November 2022main18944 Stable

Upgraded with the latest SnapLogic Platform release.

August 2022main17386 StableUpgraded with the latest SnapLogic Platform release.
4.29main15993 StableUpgraded with the latest SnapLogic Platform release.
4.28 Patch428patches14774 LatestFixed an issue in the SQS Consumer Snap where the Snap took longer time to process the messages when the Client-side buffering checkbox was selected.
4.28main14627 StableUpgraded with the latest SnapLogic Platform release.
4.27 Patch427patches14012 LatestFixed an issue in the SQS Consumer Snap where the Snap did not read message attributes when the Client-side buffering checkbox is selected.

4.27

main12833

 

Stable

Upgraded with the latest SnapLogic Platform release.
4.26main11181 StableUpgraded with the latest SnapLogic Platform release.
4.25 Patch425patches10994 Latest
  • Removed the duplicate strings in the request’s User Agent header sent by the Snaps in Amazon SQS Snap Pack.
  • Enhanced the Amazon SQS account to add Snap’s information to the User Agent header for Cross IAM account requests.
  • Enhanced the Amazon SQS Account with a new checkbox Enable Large Message Support that enables Amazon SQS client to send and receive messages up to 2GB through Amazon S3. 
4.25 Patch425patches9596 Latest

Added the Security token field to the Amazon SQS account.

4.25main9554
 
StableUpgraded with the latest SnapLogic Platform release.
4.24main8556
Stable

Enhances the Snap Pack to support Cross-Account IAM role that allows to access the queues residing in another AWS account.

4.23main7430
 
StableUpgraded with the latest SnapLogic Platform release.
4.22main6403
 
StableUpgraded with the latest SnapLogic Platform release.
4.21snapsmrc542

 

StableUpgraded with the latest SnapLogic Platform release.
4.20snapsmrc535
 
StableUpgraded with the latest SnapLogic Platform release.
4.19snaprsmrc528
 
StableUpgraded with the latest SnapLogic Platform release.
4.18 PatchMULTIPLE7778 Latest

Updated the AWS SDK library version to default to Signature Version 4 Signing process for API requests across all regions.

4.18snapsmrc523
 
StableUpgraded with the latest SnapLogic Platform release.
4.17ALL7402
 
Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17snapsmrc515
 
Latest

Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview checkbox.

4.16snapsmrc508
 
StableUpgraded with the latest SnapLogic Platform release.
4.15snapsmrc500
 
StableUpgraded with the latest SnapLogic Platform release.
4.14snapsmrc490
 
StableUpgraded with the latest SnapLogic Platform release.
4.13

snapsmrc486

 
StableUpgraded with the latest SnapLogic Platform release.
4.12

snapsmrc480

 
Stable

Introduced the Snap Pack in this release that includes SQS Producer, SQS Consumer, and the SQS Acknowledge Snaps.