SQS Consumer

On this Page

 

Snap type:Write
Description:

This Snap consumes the messages from the Amazon Simple Queue Service (SQS).

ETL Transformations & Data Flow

Consumes the messages from the SQS. The number of messages to be returned is configured in Message count. The maximum number of messages (up to 10) can be configured, however a particular message from the queue cannot be requested.

Input & Output

  • Input: This Snap can have one upstream Snap that has one Document output view (for example, the Mapper Snap).

  • Output: This Snap generates a Binary output that contains the Message body and a header. The header contains the queue URL, receipt handle, and message attributes

Modes

Prerequisites:

Valid AWS Account and SQS access.

Authentication such as IAM Role access, Access key, and Security Access key.

Limitations and Known Issues:

The AWS extended library uses S3 only when the message size is more than 256 KB. However, you cannot enable the Client-side buffering checkbox when you select the Enable Large Message Support checkbox in your SQS account.

Configurations:

Account & Access

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. See SQS Account for information on setting up this type of account.

Views

InputThis Snap has at most one document input view.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view and produces zero or more documents in the view.
Troubleshooting:The section describes typical issues you may encounter while using this Snap and instructions on how to work around them:

Settings

Label*


Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Default Value: SQS Consumer
ExampleSQS Consumer

Region

Specify the AWS region where the application is implemented.

Default value: None
Example: us-east-1, us-west-1

Queue name

Specify the name of the target Queue.

The FIFO Queue must end with a '.fifo'. suffix.

Default value: None

Queue URL

Specify the URL for the Queue. This can be found through the AWS console. 

When a new queue is created with a unique name, the Amazon SQS assigns each queue with an identifier called a queue URL that includes the queue name and other SQS components. This URL will override the queue specified in Queue name property.

Default value: None

Maximum retries

Specify the maximum number of retry attempts for a failed request.

Default value: 3

  • If the value is larger than 0, the Snap first downloads the target file into a temporary local file. If any error occurs during the download, the Snap waits for the time specified in the Retry interval and attempts to download the file again from the beginning. When the download is successful, the Snap streams the data from the temporary file to the downstream Pipeline. All temporary local files are deleted when they are no longer needed.
  • Ensure that the local drive has sufficient free disk space to store the temporary local file.

Wait timeout (second)

Specify the maximum time in seconds the Snap waits for each fetching. A zero wait timeout means that short polling is performed. Otherwise, long polling is performed. If the wait timeout is set to 0, the queue attribute, Receive Message Wait Time, is used.

Default value: 0

Message count*

Specify the number of messages to read before consuming stops. If this value is set to 0, the Snap reads all available messages and returns. If the count is less than 0, the Snap waits indefinitely.

Default value: -1

Visibility timeout

The SQS sets a visibility timeout to prevent other consumers from re-processing the same message from a queue. Visibility timeout is a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. Click here for more information.

Default value: 0

Auto acknowledge

Select this checkbox to automatically acknowledge the message after it is received by the Snap.

Default value: Selected

Client-side buffering

Select this checkbox to enable the Snap buffer up to 10 outbound requests and send them in batch, instead of sending each request separately. It allows pre-fetching of messages when consuming. The Amazon SQS Buffered Asynchronous Client currently does not support the FIFO queues. 

Default value: Not selected

Buffer configurations

The configurations for the client-side buffer.

Max buffer size

Specify the maximum number of messages that are batched together in a single request. The higher the setting, the fewer batches are required to carry out the same number of requests. The maximum batch size is 10.

Default value: 10

Max buffer open time (ms)

Specify the maximum buffer wait time in milliseconds before sending out the request.

Default value: 200 

Max outbound batches in parallel

Specify the maximum number of concurrent batches for each type of outbound request. The greater the number, greater the throughput that can be achieved (at the expense of consuming more threads).

Default value: 5

Max batches prefetching

Specify the number of receive batches the Snap pre-fetches for client-side buffering.

Default value: 10

Max inbound batches in parallel

Specify the maximum number of inbound batches that are processed in parallel. The higher the setting, the more messages can be received (at the expense of consuming more threads). If the property is set to 0, the pre-fetching is disabled.

Default value: 10

Snap execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

When you select the validate and execute option, the SQS Consumer reads 10 messages from the configured queue.

Basic Use Case


The following Pipeline describes how the Snap functions as a standalone Snap in a pipeline.

Advanced Use Case


The following describes a pipeline, with a broader business logic involving multiple ETL transformations, that shows how typically in an enterprise environment, the SQS messaging functionality is used. The pipeline download is available in the Downloads section.

The pipeline reads and moves documents from the Salesforce instance to the SQS Messaging and then to the Redshift Database.

The ETL transformations:

Extract: The Salesforce Read Snap extracts the Records to be sent to the SQS Messaging instance.

Transform: The JSON Formatter Snap transform the records into the JSON format.

Load: The SQS Producer Snap receives the records from the JSON input.


Extract: The SQS Consumer Snap reads the messages from the specified Queue.

Transform: The Binary to Document Snap converts the records into Document format to be moved onto the Redshift instance.

Load: The Redshift Bulk Load Snap loads the documents into the Redshift database.


Downloads

Important steps to successfully reuse Pipelines

  1. Download and import the pipeline into the SnapLogic application.
  2. Configure Snap accounts as applicable.
  3. Provide pipeline parameters as applicable.


  File Modified

File SQSConsumer_Basic_2018_01_29.slp

Jan 29, 2018 by Aparna Tayi

File Adv Use case of SF Read - SQS Snaps_2018_02_07.slp

Feb 08, 2018 by Aparna Tayi
 


Snap Pack History

 Click to view/expand
Release
Snap Pack Version 
Date
Type
Updates
May 2024main26341 StableUpdated and certified against the current SnapLogic Platform release.
February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023main23721 Stable

Updated and certified against the current SnapLogic Platform release.

August 2023434patches23363 Latest

Fixed the behavior change introduced in 4.32 GA related to Amazon SQS Accounts using the STS token. After the 4.32 GA, the account would automatically retry authentication on encountering 401 or 403 errors, and pipelines would not fail even after token expiration. The Amazon SQS Consumer Snap now behaves as expected and fails after the STS token expires.

August 2023main22460 StableUpdated and certified against the current SnapLogic Platform release.
May 2023main21015 StableUpgraded with the latest SnapLogic Platform release.
February 2023432patches19990 Latest

Added support for the SQS Account to reauthenticate when the authentication expires.

Behavior Change

With the 432patches19990, the Amazon SQS account reauthenticates when the authentication expires; the account automatically retires authentication on encountering 401 or 403 errors.

February 2023main19844 StableUpgraded with the latest SnapLogic Platform release.
November 2022main18944 Stable

Upgraded with the latest SnapLogic Platform release.

August 2022main17386 StableUpgraded with the latest SnapLogic Platform release.
4.29main15993 StableUpgraded with the latest SnapLogic Platform release.
4.28 Patch428patches14774 LatestFixed an issue in the SQS Consumer Snap where the Snap took longer time to process the messages when the Client-side buffering checkbox was selected.
4.28main14627 StableUpgraded with the latest SnapLogic Platform release.
4.27 Patch427patches14012 LatestFixed an issue in the SQS Consumer Snap where the Snap did not read message attributes when the Client-side buffering checkbox is selected.

4.27

main12833

 

Stable

Upgraded with the latest SnapLogic Platform release.
4.26main11181 StableUpgraded with the latest SnapLogic Platform release.
4.25 Patch425patches10994 Latest
  • Removed the duplicate strings in the request’s User Agent header sent by the Snaps in Amazon SQS Snap Pack.
  • Enhanced the Amazon SQS account to add Snap’s information to the User Agent header for Cross IAM account requests.
  • Enhanced the Amazon SQS Account with a new checkbox Enable Large Message Support that enables Amazon SQS client to send and receive messages up to 2GB through Amazon S3. 
4.25 Patch425patches9596 Latest

Added the Security token field to the Amazon SQS account.

4.25main9554
 
StableUpgraded with the latest SnapLogic Platform release.
4.24main8556
Stable

Enhances the Snap Pack to support Cross-Account IAM role that allows to access the queues residing in another AWS account.

4.23main7430
 
StableUpgraded with the latest SnapLogic Platform release.
4.22main6403
 
StableUpgraded with the latest SnapLogic Platform release.
4.21snapsmrc542

 

StableUpgraded with the latest SnapLogic Platform release.
4.20snapsmrc535
 
StableUpgraded with the latest SnapLogic Platform release.
4.19snaprsmrc528
 
StableUpgraded with the latest SnapLogic Platform release.
4.18 PatchMULTIPLE7778 Latest

Updated the AWS SDK library version to default to Signature Version 4 Signing process for API requests across all regions.

4.18snapsmrc523
 
StableUpgraded with the latest SnapLogic Platform release.
4.17ALL7402
 
Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17snapsmrc515
 
Latest

Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview checkbox.

4.16snapsmrc508