SQS Consumer
On this Page
Snap type: | Write | ||||||
---|---|---|---|---|---|---|---|
Description: | This Snap consumes the messages from the Amazon Simple Queue Service (SQS). ETL Transformations & Data FlowConsumes the messages from the SQS. The number of messages to be returned is configured in Message count. The maximum number of messages (up to 10) can be configured, however a particular message from the queue cannot be requested. Input & Output
Modes
| ||||||
Prerequisites: | Valid AWS Account and SQS access. Authentication such as IAM Role access, Access key, and Security Access key. | ||||||
Limitations and Known Issues: | The AWS extended library uses S3 only when the message size is more than 256 KB. However, you cannot enable the Client-side buffering checkbox when you select the Enable Large Message Support checkbox in your SQS account. | ||||||
Configurations: | Account & AccessThis Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. See SQS Account for information on setting up this type of account. Views
| ||||||
Troubleshooting: | The section describes typical issues you may encounter while using this Snap and instructions on how to work around them: | ||||||
Settings | |||||||
Label* | Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. Default Value: SQS Consumer | ||||||
Region | Specify the AWS region where the application is implemented. Default value: None | ||||||
Queue name | Specify the name of the target Queue. The FIFO Queue must end with a '.fifo'. suffix. Default value: None | ||||||
Queue URL | Specify the URL for the Queue. This can be found through the AWS console. When a new queue is created with a unique name, the Amazon SQS assigns each queue with an identifier called a queue URL that includes the queue name and other SQS components. This URL will override the queue specified in Queue name property. Default value: None | ||||||
Maximum retries | Specify the maximum number of retry attempts for a failed request. Default value: 3
| ||||||
Wait timeout (second) | Specify the maximum time in seconds the Snap waits for each fetching. A zero wait timeout means that short polling is performed. Otherwise, long polling is performed. If the wait timeout is set to 0, the queue attribute, Receive Message Wait Time, is used. Default value: 0 | ||||||
Message count* | Specify the number of messages to read before consuming stops. If this value is set to 0, the Snap reads all available messages and returns. If the count is less than 0, the Snap waits indefinitely. Default value: -1 | ||||||
Visibility timeout | The SQS sets a visibility timeout to prevent other consumers from re-processing the same message from a queue. Visibility timeout is a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. Click here for more information. Default value: 0 | ||||||
Auto acknowledge | Select this checkbox to automatically acknowledge the message after it is received by the Snap. Default value: Selected | ||||||
Client-side buffering | Select this checkbox to enable the Snap buffer up to 10 outbound requests and send them in batch, instead of sending each request separately. It allows pre-fetching of messages when consuming. The Amazon SQS Buffered Asynchronous Client currently does not support the FIFO queues. Default value: Not selected | ||||||
Buffer configurations | The configurations for the client-side buffer. | ||||||
Max buffer size | Specify the maximum number of messages that are batched together in a single request. The higher the setting, the fewer batches are required to carry out the same number of requests. The maximum batch size is 10. Default value: 10 | ||||||
Max buffer open time (ms) | Specify the maximum buffer wait time in milliseconds before sending out the request. Default value: 200 | ||||||
Max outbound batches in parallel | Specify the maximum number of concurrent batches for each type of outbound request. The greater the number, greater the throughput that can be achieved (at the expense of consuming more threads). Default value: 5 | ||||||
Max batches prefetching | Specify the number of receive batches the Snap pre-fetches for client-side buffering. Default value: 10 | ||||||
Max inbound batches in parallel | Specify the maximum number of inbound batches that are processed in parallel. The higher the setting, the more messages can be received (at the expense of consuming more threads). If the property is set to 0, the pre-fetching is disabled. Default value: 10 | ||||||
Snap execution | Select one of the three modes in which the Snap executes. Available options are:
When you select the validate and execute option, the SQS Consumer reads 10 messages from the configured queue. |
Basic Use Case
The following Pipeline describes how the Snap functions as a standalone Snap in a pipeline.
Advanced Use Case
The following describes a pipeline, with a broader business logic involving multiple ETL transformations, that shows how typically in an enterprise environment, the SQS messaging functionality is used. The pipeline download is available in the Downloads section.
The pipeline reads and moves documents from the Salesforce instance to the SQS Messaging and then to the Redshift Database.
The ETL transformations:
Extract: The Salesforce Read Snap extracts the Records to be sent to the SQS Messaging instance.
Transform: The JSON Formatter Snap transform the records into the JSON format.
Load: The SQS Producer Snap receives the records from the JSON input.
Extract: The SQS Consumer Snap reads the messages from the specified Queue.
Transform: The Binary to Document Snap converts the records into Document format to be moved onto the Redshift instance.
Load: The Redshift Bulk Load Snap loads the documents into the Redshift database.
Downloads
Important steps to successfully reuse Pipelines
- Download and import the pipeline into the SnapLogic application.
- Configure Snap accounts as applicable.
- Provide pipeline parameters as applicable.
Snap Pack History
Have feedback? Email documentation@snaplogic.com | Ask a question in the SnapLogic Community
© 2017-2024 SnapLogic, Inc.