In this article

Overview

The Salesforce SOQL Snap is a Read-type Snap that retrieves object records from Salesforce using a Salesforce Object Query Language (SOQL) query. See Introduction to SOQL and SOSL for more information.

Image Added

Prerequisites

None.

Support for Ultra Pipelines

Works in Ultra Task PipelinesTasks.

Limitations

• This Snap does not respect the selection of the Match Data Type checkbox when the valued entered for Batch Size is greater than 50,000.

• For a Salesforce SOQL Snap using For a Salesforce SOQL Snap using the REST API, certain network failures while reading a REST response may result in a socket timeout exception that will not retire.
• When using Primary Key (PK) Chunking mode in this Snap, the output document schemas in preview mode and execution mode may differ. For more information, see the Note under PK Chunking.
• This Snap performs batch processing, that is, a batch of input documents are processed for each HTTP request sent to Salesforce. Values of all the expression-enabled fields must remain constant during the Snap execution or validation. Hence, all expression fields can support Pipeline parameters only when they are expression-enabled. The input data parameters are not supported for expression fields, for example, $serviceVersion.

Snap Views

• JSON Generator
• Mapper

• File Writer
• JSON Formatter

Snap Settings

Label*

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Default Value: Salesforce SOQL
Example: Salesforce SOQL

Service Version*

Salesforce API

SOQL Query*

"SELECT Name FROM Account"

"SELECT Id,Name,Phone FROM Account WHERE Id='" + $Id + "'and  
     Name='" + $name + "'"

Default value: [None]

Batch size

Object Type*

Polling Interval*

Specify the polling interval in seconds for the Bulk API query execution. At each polling interval, the Snap checks the status of the Bulk API query batch processing.

Polling Timeout*

Include Deleted records

Select this checkbox to include deleted records in the query result. This feature is supported in REST API version 29.0 or later and Bulk API version 39.0 or later.

Default Value: Not selected

Pass Through

Select this checkbox to pass the input document through to the output view under the key 'original'. This property is applicable for REST and Bulk APIs.

Default Value: Selected 

Ignore Empty Result

Select this checkbox to ignore empty result, so no document will be written to the output view when the operation does not produce any result.

If you deselect this checkbox and select Pass Through, the input document will be passed through to the output view.

If you do not select both the checkboxes, Ignore Empty Result and Pass Through, the Snap writes an empty output as below:

[{}]

Default value: Selected

Bulk Content Type

Match Data type

Select this checkbox to match the data types of the Bulk API results with the data types of the REST API. This property applies only when the content type is XML for Bulk API (it does not apply to JSON). If the Bulk content type is XML, Salesforce.com returns all values as strings. If Match data type is selected, the Snap attempts to convert string values to the corresponding data types if the original data type is one of the following: boolean, integer, double, currency, or percent. 

Default Value: Not selected

Select this checkbox to escape single quotes in the parameter values when the SOQL query property is expression-enabled. You must use this feature when the WHERE clause values are parameterized and contain single quotes. For example, "SELECT Name,Email FROM Contact WHERE Name='" + $Name + "'".
If you clear the selection, the SOQL query property is evaluated without escaping the single quotes.

Default Value: Selected

Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

Minimum value: 0

Default Value: 1
Example: 3

Use this field set to set advanced properties. By default, the advanced properties are not required and is an empty table property. Click Image Removed to add an advanced property to set a value other than the default value. This field set contains the following fields:

• Properties
• Values

Select either of the following properties:

• Use temp files in PK-Chunking 
• Number of threads in PK-Chunking
• Validate record count downloaded in Bulk API

Default Value: Use temp files in PK-Chunking
Example: Number of threads in PK-Chunking

Use temp files in PK-Chunking 

Enter true if Groundplex is located where the network connection to SFDC is relatively slow. This will have the Snap in PK-Chunking download CSV files into compressed temporary local files in order to prevent SFDC to close input streams prematurely. Ensure that the Goundplex node has sufficient free disk space. Compressed local files are less than 10% of the total CSV data size.

Setting this property to true may also be necessary if any of the downstream Snaps are relatively slow in processing the document stream. By default, the Snap in PK-Chunking downloads, parses and writes to the output view in streaming mode without using large memory or temporary files. In a normal network condition, this streaming mode is efficient and scalable. However, it has been observed that SFDC tends to close HTTP input streams prematurely if client apps pause reading the input stream momentarily to perform other processing.

Number of threads in PK-Chunking

In PK-Chunking, the Snap invokes multiple threads to download CSV files in parallel with one CSV file per each thread. The maximum number of threads is 8 by default, but a lower number may be more efficient if the network connection to SFDC is relatively slow or downstream Snaps are slow in processing documents.

This property applies to Bulks API with Batch size 10,000 or smaller (not including PK Chunking). If set to true, the Snap checks if the number of records downloaded is the same as the one processed in Salesforce.com.

Specify the value for property selected in the Properties field. The default values are:

• Use temp files in PK-Chunking: false
• Number of threads in PK-Chunking: 8
• Validate record count downloaded in Bulk API: false

Snap Execution

Known Issues

• None.

Snap Views

Snap Settings

"SELECT Name FROM Account"

"SELECT Id,Name,Phone FROM Account WHERE Id='" + $Id + "'and  
     Name='" + $name + "'"

[{}]

PK Chunking

PK chunking splits bulk queries on very large tables into chunks based on the record IDs, or primary keys, of the queried records. Each chunk is processed as a separate batch that counts toward your daily batch limit. PK chunking is supported for the following objects: Account, Campaign, CampaignMember, Case, Contact, Lead, LoginHistory, Opportunity, Task, User, and custom objects. 

PK chunking works by adding record ID boundaries to the query with a WHERE clause, limiting the query results to a smaller chunk of the total results. The remaining results are fetched with additional queries that contain successive boundaries. The number of records within the ID boundaries of each chunk is referred to as the chunk size. The first query retrieves records between a specified starting ID and the starting ID plus the chunk size, the next query retrieves the next chunk of records, and so on. Since Salesforce.com appends a WHERE clause to the query in the PK Chunking mode, if SOQL Query has LIMIT clause in it, the Snap will submit a regular bulk query job without PK Chunking. See PK Chunking Header for more details. 

If 100,000 or 250,000 is selected for Batch size, the value is used as a chunk size in the PK Chunking. Note the chunk size applies to the number of records in the queried table rather than the number of records in the query result. For example, if the queried table has 10 million records and 250,000 is selected for Batch size, 40 batches are created to execute the bulk query job. One additional control batch is also created and it does not process any record. The status of the submitted bulk job and its batches can be monitored by logging onto the your Salesforce.com account in a web browser and going to Setup > Administration Setup > Monitoring > Bulk Data Load Jobs. 

PK Chunking requires Service version 28.0 or later. Therefore, if 100,000 or 250,000 is selected for Batch size and Service version is older than 28.0, the Snap submits a regular bulk query job without PK Chunking.

CSV files are generated as a result of PK Chunking Bulk API request in Salesforce.com. The Snap downloads them into temporary files in local disk of the instance where the pipeline is executed. Therefore, users should make sure sufficient free disk space is available for all CSV files. These temporary files are removed when the CSV parsing is completed.

Example

Pipeline: Data from Salesforce.com and ServiceNow: This pipeline makes use of Salesforce SOQL Snap and illustrates how to work with data from Salesforce and ServiceNow. 

The following Salesforce SOQL Snap shows how the Snap is configured and how the object records are read using SOQL. The SOQL/query retrieves records from Account object that were last modified on a specified date:  

Successful execution of the Snap gives the following preview: