REST Get
In this article
Overview
You can use the REST Get Snap to execute an HTTP Get method on a REST API service endpoint.
Prerequisites
None.
Supported Features
Works in Ultra Task Pipelines.
Limitations and Known Issues
The REST GET Snap Link HTTP response header maybe be malformed compared to other third-party applications like POSTMAN.
Snap Views
View Type | View Format | Number of Views | Examples of Upstream and Downstream Snaps | Description |
---|---|---|---|---|
Input | Document |
|
| Each input document offers details associated with the data required from the target RESTful server. |
Output | Document |
|
| Each output document contains data retrieved from the RESTful web server. If the Snap fails during the operation, it sends an error document to the error view containing the fields error, reason, resolution, and stacktrace. However, you must enable the Error view to view the error document. |
Error | Document |
| N/A | The error view contains error, reason, resolution and stack trace. For more information, see Handling Errors with an Error Pipeline |
Snap Settings
Field Name | Field Type | Description |
---|---|---|
Label* | String | Specify a name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline. Default Value: REST Get |
Service URL* | String/Expression | Specify the service endpoint URL of REST API. You can provide the URL in one of the following ways:
The Snap finds the value at the JSON path "$.widget.id" in the input data and replaces "%s" in the Service URL with the value. You can connect a File Reader and JSON Parser Snaps upstream of REST Put Snap and prepare the following JSON file to File Reader Snap: [ { "widget": { "id": "111", "name": "foo" } } { "widget": { "id": "555", "name": "bar" } } ] "https://www.example.com/data/v27.0/sobjects/Account" The Service URL for a REST snap has to be valid. If the Service URL contains any special characters, such as !, =, %21, $, and ^, the Snap throws an exception error. You can escape the special characters (using expression language) using one of the following methods:
We recommend you to use the former (encodeURIComponent) method to escape the special characters. Default Value: N/A |
Process array | Checkbox | Select to produce a stream of documents with one record in each document. Default Value: Deselected |
Trust all certificates | Checkbox | Select this checkbox to trust all certificates, such as self-signed certificates. Default Value: Deselected |
Follow redirects | Checkbox | Select this checkbox to enable the Snap to follow redirects.
Default Value: Selected |
Query parameters | Use this field set to add query parameters to your request. This field set comprises the following fields:
Default Value: N/A | |
Query parameter | String | The name of the query parameter. Default Value: N/A |
Query parameter value | String/Expression | The value that you want to assign to the parameter. This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on Pipeline Parameters, see Pipeline Properties. Default Value: N/A |
HTTP Header | Use this field set to create the HTTP header key-value pairs required for defining the headers for different types of input (JSON, PDF, DOCX, and so on). If you want to specify only content-type headers, you can configure the Multipart Content-Type property instead. This field set comprises the following fields:
| |
Key | String/Expression | The name of the HTTP header. This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on Pipeline Parameters, see Pipeline Properties. Default Value: N/A |
Value | String/Expression | The value that you want to assign to the HTTP header. This property is expression-enabled. For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. For information on Pipeline Parameters, see Pipeline Properties. Default Value: N/A |
Response entity type | String | This field allows you to select the response entity type in the output document. The available options include:
If TEXT or BINARY is selected, the Snap will not attempt to parse the entity content. If DEFAULT is selected, the Snap produces the expected result in most cases, but when failed to do so, users may set the Response entity type to TEXT or BINARY. Default Value: DEFAULT |
Read timeout | Integer | Specify the number of seconds for which the Snap waits before aborting the request due to a failure to read from the target service.
Default Value: 900 |
Connection timeout | Integer | Specify the number of seconds for which the Snap waits before aborting the request due to a failure to establish a connection to the target endpoint or service. Default Value: 30 |
Maximum request attempts | Integer | Specify the maximum number of attempts that the Snap must make to receive a response. If the attempts do not result in a response, the Snap terminates the request. Default Value: 5 |
Retry request interval | Integer | Specify the number of seconds for which the Snap waits between two successive requests. A retry happens only when the previous attempt resulted in an exception. Default Value: 3 |
Retry Policy | String | Select how you want to handle connection and error responses from the following options:
Default Value: Connection errors |
Normalize URL | Checkbox | Select this checkbox to normalize the Service URL. This enables the Snap to convert double slashes (//) in the URL path to single slash (/). Deselecting this check box reverts the Snap to 4.19 Snaplex behavior, wherein the URL paths were not normalized by default. In the 4.20 Release, due to the HTTP client upgrade the URL paths were normalized by default. Hence, there was a change in behavior in handling the URL paths in 4.20 release when compared to 4.19. This change in behavior should not impact the existing Pipelines, because most of the websites map URL paths with double or single slashes to the same endpoint. For example, https://snaplogic.com/company/diversity and https://snaplogic.com//company//diversity direct to the same endpoint. Therefore, we recommend you to select the Normalize URL checkbox. However, an exception to this is when you use non-standard URLs that differentiate the URL paths containing double slashes from those with single slashes and map them to different endpoints, in which case you must deselect Normalize URL check box. Default Value: Selected |
Has next | String/Expression | Enter the expression that must be evaluated on the output document to true or false, to indicate whether there are more records available. The Snap continues to read the next page until this expression evaluates to false in the new set of output documents. Has next is a property in the API response that indicates whether there are more records to be fetched. This property is used for the REST Get pagination feature, which retrieves 20 records at a time (referred to as a page), by default. To see this property in action, see Retrieving Contact Information from HubSpot or Retrieving Contact Information from HubSpot Using Upstream Values, below. This field is null-safe; if the evaluated value is null, or if there is no value provided in Has next field, the Snap stops the pagination. Default Value: N/A |
Next URL | String/Expression | Enter the expression that is evaluated on the output document to a URL that the Snap must use to get the next page. This property is used for the REST Get pagination feature. The Next URL has two components, which are structured as follows:
You can create the Next URL in either of the following ways:
For additional information, see Retrieving Contact Information from HubSpot or Retrieving Contact Information from HubSpot Using Upstream Values. Default Value: N/A |
Pagination interval(s) | Integer | The number of seconds for the Snap to wait before attempting to get the next page. Default Value: 0 |
Show all headers | Checkbox | The REST endpoint may return a response with multiple headers with the same header name.
If any of these objects has a key-value format, it is parsed into the map data. Default Value: Deselected |
Snap Execution | String | Select one of the three modes in which the Snap executes. Available options are:
Default Value: Validate & Execute |
Using REST Get to Process gzip Compressed Data
The response can provide gzipped data. In that case, the Snap can be configured to process it using the header parameter:
HTTP Header
Key | Value |
---|---|
Accept-Encoding | gzip |
Accept | */* |
The stream will then be uncompressed while parsing the data and making it available on the output view.
Troubleshooting
Error | Reason | Resolution |
---|---|---|
Failed to execute HTTP request | The Service URL must have a protocol, such as http://, https://. | Check the Snap properties. Specifically, check your entry in the Service URL field. |
<Service_URL>: Name or service not known. This basically means that the service URL you entered is not accessible. | ||
Failure: Validation errors: property_map.settings.serviceUrl.value: Could not compile expression: <Service_URL> | The snap settings are not correct. Encountered extraneous input ':' at line 1:5; expecting={<EOF>, '||', '&&', '^', '==', '!=', '>', '<', '>=', '<=', '+', '-', '*', '/', '%', '?', 'instanceof', 'in', 'matches', '[', '(', '.', PropertyRef}. This means that you have turned expressions on ( ) while entering string value into the Service URL field. | Fix the snap configuration and try executing again. Please check expression syntax. Ensure that your service URL is a proper expression; or turn off the expression control ( ).For more information on the expression language, see Understanding Expressions in SnapLogic and Using Expressions. |
URL Parse Exception - 403 | The Service URL path might be containing any of the following special characters: !, =, %, #, $, ^&()_¢äâêîôûñç¡¿ÉÙËǨ°¸ðø©¢¾A+²½µ®§÷¶þ | To escape the special characters, use the global function encodeURIComponent on any variables that might contain special characters so that they are encoded properly. |
Examples
Retrieving Contact Information from HubSpot
In this example, we use the Has next field to retrieve the list of users created most recently in HubSpot. You then separate user information from the other details in the API response.
The REST Get Snap
We use the REST Get Snap to retrieve user information from HubSpot, and configure the Snap as follows:
REST Get configuration details
The following configuration details are key to understanding this example:
Field | Value | Description |
---|---|---|
Service URL | https://api.hubapi.com/contacts/v1/lists/all/contacts/all?hapikey=demo | This is a demo API offered by HubSpot, which returns details of all contacts created in the demo account, in descending order. This API only scrolls back 30 days in time. |
Has next | $entity['has-more'] | The REST Get Snap lists 20 documents by default. If there are more contacts created during the 30 days preceding the call, this API returns true for the key has-more. |
Next URL | "https://api.hubapi.com/contacts/v1/lists/all/contacts/all?hapikey=demo&vidOffset=" + $entity['vid-offset'] | This is the URL that the Snap calls to retrieve the next set of 20 documents. In this URL, vid-offset is the ID of the last user whose details were retrieved in the preceding call. For example, if the ID of the last user in the API response is 12345, then that will be the value of the vid-offset parameter. |
When executed, this Snap lists out API response entities in sets of 20, until the last contact information is listed out, and the has-more
field changes to false:
The JSON Splitter Snap
The output from the REST Get Snap contains a lot of details; but we only want user details, which are available in the entity.contacts
array in the API response. We use the JSON Splitter Snap to isolate the information we need:
The output of the JSON Splitter Snap contains only the information we need:
Retrieving Contact Information from HubSpot Using Upstream Values
In this example, we use values from the upstream Snap in the Has next property of the REST Get Snap to retrieve the list of users created most recently in HubSpot.
The JSON Generator Snap
We enter the endpoint URL into the JSON Generator Snap:
When executed, the JSON Generator Snap gets the following output:
The REST Get Snap
We use the REST Get Snap to retrieve user information from HubSpot, and configure the Snap as follows:
REST Get configuration details
The following configuration details are critical to understand this example:
Field | Value | Description |
---|---|---|
Service URL | https://api.hubapi.com/contacts/v1/lists/all/contacts/all?hapikey=demo | This is a demo API offered by HubSpot, which lists out a few details of their most recently created users in descending order. This API only scrolls back 30 days in time. |
Has next | $entity['has-more'] | The Get all contacts API offers 20 documents by default. If there are more contacts created during the 30 days preceding the call, this API returns true for the key has-more. |
Next URL | $original.url+ "=" + $entity['vid-offset'] | This is the expression that uses the value received from the upstream JSON Generator Snap to create the URL that the Snap calls to retrieve the next set of documents. In this URL, vid-offset is the ID of the last user whose details were retrieved in the preceding call. For example, if the ID of the last user in the API response is 12345, then that will be the value of the vid-offset parameter. |
When executed, this Snap lists out API response entities in sets of 20, until the last contact information is listed out, and the has-more
field changes to false.
Connecting to the SnapLogic API
You can use the REST Get Snap to get information from the Pipeline Monitoring API. For example, to return the status of executed pipelines for the specified time within the org Snaplogic, the GET statement of GET https://elastic.snaplogic.com/api/1/rest/public/runtime/snaplogic?start=1430377200000&end=1430420399000 can be passed through the REST Get Snap as shown below:
Connecting to Rally with REST
See the Use Case example Pipeline: Connecting to Rally with REST.
Example #3: REST Snap Pack
The REST Get Snap in the above example pipeline gets the data for a JIRA issue created and updated in the preceding Snaps in the pipeline.
The following JSON data is the input document to the Snap:
Click this link to see the output provided by the Snap: Output Data
REST Get Pagination: Eloqua
For Eloqua, set:
- Has Next: ($total - ($page*$pageSize)) > 0
- Next URL: “/assets/campaigns?count=2&page=” + (resultdoc.page +1)
REST Get Pagination: Marketo
For Marketo, set:
- Has Next: resultdoc.moreResult
- Next URL: “/rest/v1/activities.json?nextPageToken=” + resultdoc.nextPageToken + ”&activityTypeIds=1&activityTypeIds=12”
REST Get handles multiple response cookies
In this Pipeline, we request for a service URL 'https://mockbin.com/bin/f580f547-310a-44da-bec2-e43890eebe26' using the Rest Get Snap, where the REST endpoint returns the HTTP response with multiple cookies under the same header. We connected the JSON Splitter Snap to view the cookies under the header 'set-cookie'.
We enter the JSON Path as $headers['set-cookie'] in the JSON Snap
The successful execution of the Pipeline displays the below output view in which we can see the two cookies, {JSESSIONID:first, Path:/} and {ORA_OD_OSI:second, Path:/} respectively under the header'set-cookie'.
Downloads
Important steps to successfully reuse Pipelines
- Download and import the Pipeline into SnapLogic.
- Configure Snap accounts as applicable.
- Provide Pipeline parameters as applicable.
Snap Pack History
See Also
- Video: REST API Demo
- SnapLogic Tips and Tricks: REST Snap Compression Capabilities
- REST GET and the SnapLogic Public APIs for Pipeline Executions
- Connecting SaaS providers with SnapLogic's OAuth-enabled REST Snaps
- SnapLogic Community: Passing account credentials to an Authorization Endpoint via the JSON body or as an Expression
- SnapLogic Community: Using the DocuSign eSignature REST API with JWT authentication and REST Get Snap