Versions Compared

Old Version 83

changes.mady.by.user Kalpana Malladi

Saved on

compared with

New Version 84

changes.mady.by.user Kalpana Malladi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

In this article

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

Overview

Salesforce Read is a Read-type Snap that provides the ability to retrieve all records for all fields of the Salesforce object from Salesforce by defining the Salesforce object name.

Prerequisites

None.

Support for Ultra Pipelines

Works in Ultra Task Pipelines.

Limitations

  • When using Primary Key (PK) Chunking mode in this Snap, the output document schemas in preview mode and execution mode may differ. For more information, see the Note under PK Chunking.

  • When you run more than 10 Salesforce Snaps simultaneously, the following error is displayed
    “Cannot get input stream from next records URL" or "INVALID_QUERY_LOCATOR"
    Workaround: You must ensure not to use more than 10 Salesforce Snaps simultaneously. because this might lead to opening of more than 10 query cursors in Salesforce. See Salesforce Knowledge Article for more information.

Known Issues

None.

Snap Views

Type FormatNumber of ViewsExamples of Upstream and Downstream SnapsDescription
InputDocument
Min: 0
Max: 1

Mapper

Copy

This Snap has at most one input view.
OutputDocument
Min: 1
Max: 2
File Writer

The snap allows you to add an optional second output view that exposes the schema of the target object as the output document.

ErrorDocument

The error view contains error, reason, resolution and stack trace. For more information, see Handling Errors with an Error Pipeline

Snap Settings

Field

Field Type

Description


Label*

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

Salesforce API*
 

String/Expression/Suggestion

The Salesforce API mode to use during the Pipeline execution.

Default Value: REST API 
Example: Bulk API


Note

We recommend you to set the Salesforce API mode to Bulk API if the table size of the Salesforce object referred to in the Object Type field is large (that is, 10,000 records or more) to prevent the time-out or connection error.


Service Version*

Dropdown list

Multiexcerpt include macro
nameME_Service_Version
pageSalesforce Batch Create

Batch Size*

Dropdown list

The number of records to process in each batch for downloading large query results. Each batch read requires an API call against Salesforce to retrieve the set of records.

  1. REST API:
    The batch sizes 200 and 2000 are valid in REST API.
    According to the SFDC REST API documentation:
    "There is no guarantee that the requested batch size is the actual batch size. Changes are made as necessary to maximize performance."
    In fact, if the size of each record in the query result is relatively large (for example, a few hundred output fields), SFDC may respond in a batch size of 200, And if the size of each record in the query result is relatively small, SFDC may respond in a batch size of 2000 regardless if the Batch size is set to 200 or 2000.
  2. Bulk API:
    Selecting the values 100,000 or 250,000 will make the Bulk API request as PK-chunking with a chunk size of 100,000 or 250,000. Other values 200 or 2000 will have the Snap submit a single regular Bulk API batch job to SFDC.
    According to the SFDC Bulk API documentation (Number of attempts to query):
    "15 attempts at 10 minutes each to process the batch. There is also a 2-minute limit on the time to process the query. If more than 15 attempts are made for the query, an error message of 'Tried more than fifteen times' is returned. If the query takes more than 2 minutes to process, a QUERY_TIMEOUT error is returned."
    Therefore, if the query is expected to take a long time to execute (for example, if the target SObject has more than several hundred thousand records and the number of output fields in a record is more than a few hundreds), it is highly recommended to set this property to 100,000 or 250,000 to avoid timeout errors.

Default Value: 2000 
Example: 2000

Note

The Snap validates using the Bulk API, even if you configure the Snap to use Bulk API with Primary Key (PK) Chunking.


Object Type*

String/Expression/Suggestion

This property enables you to define the name of the Salesforce object, such as Account.

Default Value:  Account
Example: Account

Output Fields

Use this field set to enter a list of field names for the SOQL SELECT statement. If empty, the Snap selects all fields.

Default Value: N/A
Example: Id
                Name
                ShippingAddress

Output Field Limit

String/Expression

The number of output fields to return from the Salesforce object.

Default Value:  N/A

Output Field Offset

String/Expression

Defines a starting field index for the output fields. This is where the result set should start. 

Default Value:  N/A

Note

If you enter an offset value that is greater than 1, the first field "Id" is always returned, but the following fields start from the offset position. This is because the first field is the only unique identifier.


Where Clause
 

String/Expression

Enter the WHERE clause for the SOQL SELECT statement. Do not include the word WHERE.

Default Value: N/A
Example: Id > '001i0000007FVjpAAG'

Note
titleUsing quotes in field names values
  • Do not use quotes for field names in the WHERE clause. Using quotes results in an error.
  • Use only single quotes for values in the WHERE clause as using double quotes results in an error.
  • The above rules do not apply when you are using SnapLogic expressions, you can use quotes for the field names and values as applicable. 


Order By ClauseString/Expression
Enter the ORDER BY clause that you want to use with your SOQL SELECT Query.


Note

PK Chunking does not support the ORDER BY clause.


Default Value: N/A
Example: LastName, FirstName

Limit ClauseString/Expression
Enter the LIMIT BY clause that you want to use with your SOQL SELECT Query.


Note

PK Chunking does not support the LIMIT clause.


Default Value N/A
Example: 2000

Polling Interval*

String

Define the polling interval in seconds for the Bulk API read execution. At each polling interval, the Snap checks the status of the Bulk API read batch processing.
Maximum value:  60 

Default Value:  5
Example:  5

Polling Timeout*

String

This property allows you to define the polling timeout in seconds for the Bulk API read batch execution. If the timeout occurs while waiting for the completion of the read batch execution, the Snap throws a Snap Execution Exception.

Default Value: 3000
Example: 300

Process Date/time

Checkbox

All date/time fields from Salesforce.com  are retrieved as string type.

Select this checkbox to allow the Snap to convert date/time fields to corresponding date/time types by accessing the metadata of the given SObject.

  • Salesforce datetime is converted to Joda DateTime,
  • Salesforce date to Joda LocalDate, 
  • Salesforce time to Joda LocalTime.

Deselect this checkbox to allow the Snap to send these date/time fields without any conversion.

Default Value: Deselected

Include Deleted Records

Checkbox

Select this checkbox to allow the Snap to include deleted records in the query. This feature is supported in REST API version 29.0 or later and Bulk API version 39.0 or later.

Default Value: Not selected

Pass Through

Checkbox

Select this checkbox to pass the input document to the output view under the key 'original'. This is applicable for REST and Bulk APIs.

Default Value: Selected

Ignore Empty Results

 

Checkbox

Select this checkbox to ignore empty results; no document will be written to the output view when the operation does not produce any result. If this property is not selected and Pass Through is selected, the input document will be passed through to the output view. This property is ignored in Bulk API mode.

Default Value: Selected

Bulk Content Type

 

Dropdown list

Select the content type for Bulk API: JSON or XML.  

Note

The numeric type field values will be read as numbers in JSON content type, and as strings in XML content type, in the output documents.

JSON content type for Bulk API is available in Salesforce API version 36.0.

In REST API, the number-type field values will always be read as numbers. If the Bulk API has been selected along with 100,000/ 250,000 as batch size value, the content-type will always be CSV regardless of the value set in this property.


Default Value:  XML
Example: JSON

Number Of RetriesString/Expression

Specify the maximum number of retry attempts in case of a network failure.

Note

  • If the value is larger than zero in PK-Chunking, the Snap overrides user settings and sets Use temp files in PK-Chunking property to true.

  • In Bulk API and PK-Chunking modes, the Snap downloads each batch of data into a temporary local file. When the download is complete, the Snap writes the output documents parsed from the temporary file to avoid the possibility of duplicated output documents in the downstream Pipeline. Ensure that the local drive has sufficient free disk space in the node where the Pipeline executes. All temporary local files are deleted when they are no longer needed.

Minimum value: 0

Default Value: 1
Example: 3

Retry Interval (seconds)String/Expression

Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

Minimum value: 0

Default Value: 1
Example: 3

Match Data Type

 

Checkbox

If checked, the data types of the Bulk API results are the same as in REST API. This property applies if the content type is not JSON in Bulk API. In Bulk API, Salesforce.com returns all values as strings if the Bulk content type is XML or Batch size is for PK Chunking. If this property is checked, the Snap attempts to convert strings values to the corresponding data types if the original data type is one of boolean, integer, double, currency, and percent.
 
This property is ignored in REST API or with JSON Bulk content type. In Bulk API (not with JSON Bulk content type), Salesforce.com does not return any value for null. Therefore, if the value is null, the key-value entry is not present in the output document, which is different from the REST API result.

Default Value: Not selected

Advanced Properties 

Use this field set to define additional advanced properties that you want to add to the Snap's settings. Additional advanced properties are not required by default, and the field-set represents an empty table property. Click  to add an advanced property. 

This field set contains the following fields:

  • Properties
  • Values

Properties

Dropdown list

You can use one or more of the following properties in this field:

PropertyDescriptionDefault ValueExample
Use temp files in PK-Chunking

Enter true if Groundplex is located where the network connection to SFDC is relatively slow. This will have the Snap in PK-Chunking download CSV files into compressed temporary local files in order to prevent SFDC to close input streams prematurely. Ensure that the Goundplex node has sufficient free disk space. Compressed local files are less than 10% of the total CSV data size.

Setting this property to true may also be necessary if any of the downstream Snaps are relatively slow in processing the document stream. By default, the Snap in PK-Chunking downloads, parses and writes to the output view in streaming mode without using large memory or temporary files. In a normal network condition, this streaming mode is efficient and scalable. However, it has been observed that SFDC tends to close HTTP input streams prematurely if client apps pause reading the input stream momentarily to perform other processing.

falsetrue

Number of threads in PK-Chunking

In PK-Chunking, the Snap invokes multiple threads to download CSV files in parallel with one CSV file per each thread. The maximum number of threads is 8 by default, but a lower number may be more efficient if the network connection to SFDC is relatively slow or downstream Snaps are slow in processing documents.

82
Validate record count downloaded in Bulk API

This is a boolean property that applies to Bulk API where the batch size 10,000 or less (not including PK Chunking). If set to true, the Snap checks if the number of records downloaded is the same as the one processed in Salesforce.com.

Note
titleMismatch in Record Count - Error View

If it is enabled (set to true), it is recommended that the Error View is enabled to allow the Snap to show record-count mismatch.


falsetrue
Disable automatic vertical slicing

The Salesforce SELECT query does not support wildcards in field lists, and its query length cannot exceed 20,000 characters. When Output fields, Output field limit, and Output field offset properties are empty, the Snap formats the entire list of column names in the SELECT statement. If the query exceeds 20,000 characters, the Snap divides the query into two or more queries, executes them separately, and merges the query results automatically. This Snap feature is called Vertical Slicing.

The Vertical Slicing feature may fail if columns used in the WHERE clause or ORDER BY clause change their values during query executions (for example, if the column is SystemModStamp). Since the query length threshold used in the Snap is smaller than 20,000 characters, disabling the automatic vertical slicing feature may resolve the issue if the actual query length is between the internal threshold and 20,000 characters.

falsetrue


ValuesString

The value that you want to associate with the property selected in the corresponding Properties field. The default values for the expected properties are:

PropertyValue

Use temp files in PK-Chunking

false
Number of threads in PK-Chunking8
Validate record count downloaded in Bulk APIfalse
Disable automatic vertical slicingfalse


Snap Execution

Dropdown list

Multiexcerpt include macro
nameSnap_Execution_Introduced
pageAnaplan Read

Default Value: Validate & Execute
Example: Execute only

Multiexcerpt include macro
nameTemporary Files
pageJoin

PK Chunking

PK chunking splits bulk queries on very large tables into chunks based on the record IDs, or primary keys, of the queried records. Each chunk is processed as a separate batch that counts toward your daily batch limit. PK chunking is supported for the following objects: Account, Campaign, CampaignMember, Case, Contact, Lead, LoginHistory, Opportunity, Task, User, and custom objects. 

PK chunking works by adding record ID boundaries to the query with a WHERE clause, limiting the query results to a smaller chunk of the total results. The remaining results are fetched with additional queries that contain successive boundaries. The number of records within the ID boundaries of each chunk is referred to as the chunk size. The first query retrieves records between a specified starting ID and the starting ID plus the chunk size, the next query retrieves the next chunk of records, and so on. Since Salesforce.com appends a WHERE clause to the query in the PK Chunking mode, if SOQL Query has LIMIT clause in it, the Snap will submit a regular bulk query job without PK Chunking.

For more details on the PK Chunking, refer to PK Chunking Header.

If 100,000 or 250,000 is selected for Batch size, the value is used as a chunk size in the PK Chunking. Please note the chunk size applies to the number of records in the queried table rather than the number of records in the query result. For example, if the queried table has 10 million records and 250,000 is selected for Batch size, 40 batches are created to execute the bulk query job. One additional control batch is also created and it does not process any record. The status of the submitted bulk job and its batches can be monitored by logging onto the your Salesforce.com account in a web browser and going to Setup > Administration Setup > Monitoring > Bulk Data Load Jobs. 

PK Chunking requires Service version 28.0 or later. Therefore, if 100,000 or 250,000 is selected for Batch size and Service version is older than 28.0, the Snap will submit a regular bulk query job without PK Chunking.

Note

The output document schemas in preview mode  (validation) and execution mode for this Snap may differ when using Primary Key (PK) Chunking mode. This is because the Snap intentionally generates the output preview using regular Bulk API instead of PK Chunking, to reduce the costs involved in the PK Chunking operation.
Salesforce recommends the use of PK Chunking if the target Salesforce object (table) is relatively large (for example, more than few 100,000 records).

Examples

Pipeline: Salesforce.com Data to a File: This Pipeline reads data using a Salesforce read and writes it to a file. 

Reading Records from an Object

The following Salesforce Read Snap shows how the Snap is configured and how the object records are read. The Snap reads records from the Account object, and retrieves values for the fields, Id, name, & type where the type field value is Analyst:  

Successful execution of the Snap gives the following preview:

 

Reading Records in Bulk 

The Salesforce Read Snap reads the records from the Standard Object, Account, and retrieves values for the 10 output fields (Output field limit) starting from the 3rd field (Output field offset). Additionally, we are passing the values dynamically for the Access token and the Instance URL fields in the Account settings of the Snap by defining the respective values in the pipeline parameters.  

1. The Salesforce Read Pipeline. 

2. The Key and Value parameters are assigned using the Edit Pipeline property on the designer. 

For this Pipeline, define the two Pipeline parameters:

  • access_Token
  • instance_URL

3. The Salesforce Read Snap reads the records from the Standard object, Account, to the extent of 10 output fields starting from the 3rd record(by defining the properties- Output field limit and Output field offset with the values 10 and 3 respectively).

4. Create a dynamic account and toggle (enable) the expressions for Access Token and Instance URL properties in order to pass the values dynamically.

Set Access token to _access_token and Instance URL to _instance_url. Note that the values are to be passed manually and are not suggestible. 

 

5. Successful execution of the Pipeline displays the below output preview:

Using Second Output View

This example Pipeline demonstrates how you can add an optional second output view that exposes the schema of the target object as the output document. To For this end, we configure the Pipeline using the Mapper and Salesforce Read Snaps.


First, we configure the Mapper Snap as follows. Upon validating the Snap, the target schema is populated in the Mapper Snap. Once this is available, we define the target path variables from the target schema.

Upon validation, the following output is generated in the Snap's preview.

Next, we configure the Salesforce Read Snap to read the specified records.

Upon validation, we can see the following outputs in both the output views of the Snap.

Output0 (default)

Output 1


Insert excerpt
Salesforce Snap Pack
Salesforce Snap Pack
nopaneltrue