In this article

Key New Features in SnapLogic

Platform:

4.25 Dot Releases

Platform

UI

Standard-mode Snaps

Data Automation

4.25 UAT Delta

Platform 

New Features

• API Management - Design-First: Introduced an API Management section in Manager where you can create an API based on an Open API Specification (OAS) 2.0. As an Org admin, you can also manage APIs in the new API Manager console. The API Manager - /apim space supports the following activities:
  • Create an API by uploading a Swagger 2.0 specification or by referencing it through a URL.
  • Create multiple versions of your API and manage their assets.
  • Add permissions to your APIs and versions. If you are an Org admin, you can set permissions at the API Manager - /apim space level.
  • Apply API policies at the API and version levels.
  • View details about APIs, Versions, and their Assets.
    

• Support for elastic scaling in Kubernetes: Introduced autoscaling for your Snaplex instances that are running in a Kubernetes environment. This implementation replaces the previous Kubernetes-based deployment of a Snaplex. Starting with the 4.25 release (May 2021), you can set up your Snaplex helm chart to initiate the scaling up and down of your node resources through Kubernetes orchestration. This implementation requires the Prometheus adapter and Grafana analytics.
  
  
• Anomaly detection and active Pipeline analysis: Introduced two alert notifications for tracking Task executions that take a longer time than usual. You can define a hard limit either in minutes or by the percentage based on the last 30-day historical executions.
  
  
  
• Enhanced memory and resource management to limit memory growth on Snaplex nodes. If the memory usage exceeds 95% of the configured maximum heap limit, the Pipelines that are causing the high memory utilization are terminated to prevent the Snaplex node from failing with an OutOfMemory error.

Enhancements to Existing Features

• Snaplex monitoring: Added support for granularity in the reporting of data spikes in your Pipeline executions in the widgets on the Snaplex Dashboard Wall. You can now select Pipelines within a 15-minute window (indicated by the vertical line) and correlate active Pipelines with high resource utilization.
  
  
  
• Additional notifications via Slack: Enhanced support for in-platform notifications. You can now add Slack recipients (channels and direct users) to Org-wide Snaplex notifications and Task notifications.
   
• Asset Palette for APIs: Support for searching for Pipelines that are used by APIs that are created in the new API Management console.
  
  
  
• Progressive loading of the Asset tree in Manager: Enhanced the search-and-browse experience in Manager. The assets are retrieved as needed, thus reducing latency.
  
  
• UI refresh of Task dialog windows: Redesigned dialog windows for Triggered, Ultra, and Scheduled Tasks, giving the displays the same appearance as that of the New Form UI. As an enhancement, you can now select the Task type when opening the dialog window from Designer.
  
  
• UI refresh of the Account dialog windows: Updated the UI and enhanced the support for creating accounts in Manager. Previously, in the 4.24 release and earlier, the option to create or modify an account using the New Form UI was available only in Designer.
  

Known Issues

• When you migrate a project more than once, the existing assets are overwritten. You might have to associate your Tasks with the correct Pipelines after the migration. Typically, even after multiple migrations, account references remain intact in the Pipeline, but we recommend that you verify all references before starting your Pipeline executions. This behavior is a result of the duplicate_flag parameter being disabled.
  
  
• When a start time of a Scheduled Task using the Snaplex-based scheduler is modified, the configured start time is not used to trigger the Pipeline execution.
  
  
• In some scenarios with the Snaplex-based Scheduler, multiple executions of a Scheduled Task occur when only one instance is set to execute.
  
  
• When calling a child Pipeline from a Groundplex to Cloudplex or vice versa, the Pipeline Execute Snap hangs, causing the parent Pipeline not to complete the execution.
  
  
• The New Form UI does not show the default value for a custom property when editing an existing instance of a Snap.
  
  
• In Manager, the Task Details page for some Triggered Tasks does not display the Download CA Certificate option.
  
  
• When creating iterative Scheduled Tasks in time zone regions that have Daylight Savings, we recommend that you select the Cron Job option. See best practices for details.

Behavior Change

Retired the legacy UI: The legacy UI for Snaps is retired. Starting in the 4.25 release (May 2021), the option to deselect the New Form UI in the User Settings dialog window is no longer available. All Snaps can be viewed only in the new UI.

Standard-mode Snaps

New Snaps

• NetSuite Call RESTlet: This Snap provides the functionality of the NetSuite RESTlet framework that supports the HTTP methods (GET, DELETE, POST, PUT) and MIME types. You can call RESTlet scripts through token-based authentication.

• SAP RFC Listener: This Snap facilitates Remote Function Calls (RFCs) between two SAP systems. The Snap simulates an RFC server and listens to the incoming RFC calls infinitely. Subsequently, the Snap generates output documents from the import parameters or triggers a child Pipeline. Business applications can use the RFC mechanism to communicate and exchange information (in predefined formats) with other systems.

• Twilio Snap Pack: This Snap Pack enables you to perform various communication functions programmatically, such as making phone calls and sending text messages using Twilio's Web Service APIs. This Snap Pack has the following Snaps:
  
  • Twilio Read Snap: Retrieves data resources from Twilio's REST APIs.
  • Twilio Create Snap: Creates data objects in Twilio.
  • Twilio Modify Snap: Modifies existing Twilio data objects.
  • Twilio Delete Snap: Deletes existing data objects from Twilio.

Updated Snap Packs

• Binary Snap Pack:
  • Enhanced the Binary Snap Pack by adding an account Two-Factor Auth Account that supports two layers of authentication for Secure File Transfer Protocol (SFTP). The two secure layers of verification to grant access include the Basic Auth (username/password) and the SSH Key (username/private key) methods.
  • Enhanced the S3 File Writer Snap with the following abilities:
    • Use an MD5 checksum that checks for data integrity and corruption while uploading the file.

    • Write S3 objects with Access Control List (ACL) details in a single API call.

      Note

      Behavior Change In the 4.23 release, the S3 File Writer Snap used to write an S3 file and then set ACL if the ACL property was not empty. If the file write was successful and the ACL setting failed, both the output document in the output view and the error document in the error view was displayed. This issue has been resolved in the 4.24 release by enabling the Snap to configure ACL while writing the file. Therefore, the Snap produces an error document in the error view if the ACL configuration fails, irrespective of whether the file write action passes or fails.

      
      

• Box Snap Pack:
  • Enhanced the Snaps in this Snap Pack with the Retry mechanism that includes the following settings:
    • Number of retries: Specifies the number of attempts the snap should make to perform the selected operation in case of connection failure or timeout.
    • Retry interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.

  • Enhanced the output view of the Box Directory Browser Snap to display the error message, Error retrieving contents of the folder with id:<Folder ID> instead of No folder exists with <Folder ID> when you enter an invalid folder ID.
    
    

• DynamoDB Snap Pack: Enhanced the DynamoDB Account to support the Expression Enabler for account properties and creation of dynamic accounts.
  
  
• Flow Snap Pack: Enhanced the HTTP Router Snap to support Ultra Pipelines.
  
  
• Google Directory Snap Pack: Enhanced the Google Directory Insert and Google Directory Update Snaps to support custom schema attributes. You can fetch these custom fields and their values in a user profile by adding them as custom schema attributes to upstream Snaps.
  
  
• Hadoop Snap Pack: Enhanced the HDFS Reader and HDFS Writer Snaps with the Retry mechanism that includes the following settings:
  • Number of Retries: Specifies the maximum number of retry attempts when the Snap fails to connect to the Hadoop server.
  • Retry Interval (seconds): Specifies the minimum number of seconds the Snap must wait before each retry attempt.
    
    
• Microsoft OneDrive Snap Pack: Enhanced the Update Share Permissions Snap with the Share Type and Expires fields that enable you to update the share permissions on your OneDrive items shared using the Invite or Link option. If the Share type is link, you can specify the expiry date and time for the URL that is used to share permissions.
  
  
• Microsoft Power BI Snap Pack: Enhanced the following Snaps in the Microsoft Power BI Snap Pack by supporting data flows with two options, Dataflow and Dataflow Storage Accounts.
  • Power BI Read
  • Power BI Post
  • Power BI Delete
  • Power BI Update
    
    

• Salesforce Snap Pack:

  • Enhanced the pass-through behavior in Bulk API mode for Salesforce Write Snaps to include the input document in the original key when the batch size is set to 1. When the batch size is greater than 1, the input document is not included in the original key. This enhancement impacts the following Snaps:

    • Salesforce Create
    • Salesforce Delete
    • Salesforce Update
    • Salesforce Upsert 
  • Enhanced the Salesforce Upsert Snap to support CSV payload using the new field Bulk Content Type in Bulk API mode. The Salesforce API has recently introduced support for CSV payloads. In the 4.24 release and earlier, this Snap supported only the XML data type; starting from the 4.25 release, you can select CSV as the Bulk Content Type drop-down list. The default content type is set to XML to support backward compatibility. This enhancement also addresses the null values issue in Bulk API mode.
  • Enhanced the Snaps in the Salesforce Snap Pack by updating the field labels to the title case. If you have existing Pipelines that use Salesforce Snaps that include the field names in error messages, you must update those Pipelines to use the new field labels.
    
    

• Snowflake Snap Pack:

  • Enhanced the Snowflake - Bulk Load Snap to enable data transformation using the new field Select Query before loading data into the Snowflake database. This option enables you to query the staged data files by either reordering the columns or loading a subset of table data from a staged file. It also enables you to transform data types. This Snap supports CSV and JSON file formats for data transformation.
    
    

• Transform Snap Pack: Enhanced the Group By N Snap with the following settings: 

  • Memory Sensitivity: The Snap's response to the memory changes. 
  • Group Size: The maximum number of input documents to be grouped into a single output document.
  • Min Group Size: The minimum number of input documents to be grouped into a single output document.
  • Flush Timeout: The number of seconds the Snap must wait before processing the records even if the incoming messages do not reach the specified Group Size.

Known Issues

• The Generic JDBC - Insert Snap in the JDBC Snap Pack might fail due to a connection issue. We recommend that you roll back to the main8614 version.

• The Avro Parser Snap in the Transform Snap Pack might fail due to ClassCastException error. We recommend that you upgrade to the 425patches9815 version. 

Prior 4.24 Snap Dot Releases that are Merged into 4.25 GA

Data Automation

SnapLogic's data automation solution speeds up the identification and integration of new data sources, and the migration of data from legacy systems. The solution can automatically detect duplicate, erroneous, or missing data, and identify structures and formats that do not match the data model. Data automation can accelerate the loading and transformation of your data into the data warehouse, speeding up the data-to-decisions process.

New Features in ELT Snap Pack

• Starting with the 4.25 release, SnapLogic has now certified the ELT Snap Pack to work with Snowflake hosted on Google Cloud Platform (GCP) as the target database, in addition to the other flavors of Snowflake hosted on AWS and Microsoft Azure. 
• Introduced the ELT Execute Snap to enable you to run DML, DDL, and DCL SQL queries in Snowflake in Snowflake, Redshift, and Azure Synapse.
• Introduced the ELT SCD2 Snap to support Type 2 Slowly Changing Dimensions (SCD2) updates to the target databases—Snowflake, Redshift, and Azure Synapse.
• Enhanced the ELT Database Account to introduce:
  • Support for Google Cloud Storage as a storage location (source) in addition to AWS S3 and Azure Data Lake Storage (ADLS) when your target database is Snowflake.
  • Automatic download of the JDBC driver required for the selected Database Type using the new Download JDBC Driver Automatically check box.

Supported JDBC JAR Versions

ELT Snaps automatically use a corresponding JDBC JAR file to connect to your target database and to perform the load and transform operations. 

Updated ELT Snaps

• Enhanced the ELT Load Snap to prevent changes to existing tables during Pipeline validation. If you set the Load Action as Drop and Create table, and the target table does not exist, the Snap creates a new (empty) target table based on the schema specified in its settings.
• Enhanced the ELT Window Functions Snap to support Covariance, Correlation, and Linear Regression Functions on Snowflake, Redshift, and Azure Synapse databases. The Snap uses function-specific query re-writes to support these functions on Redshift and Azure Synapse databases.
• Enhanced the ELT Merge Into and ELT Insert Select Snaps to support up to one output view, and added the Get Preview Data check box to these Snaps. You can now connect downstream ELT Snaps to these Snaps.

Updated eXtreme Snap Packs

• Introduced the SCD2 - Spark SQL 2.x Snap to support Type 2 Slowly Changing Dimensions (SCD2) updates to the target databases in the eXtreme mode.
• Upgraded the /wiki/spaces/SD/pages/410189849to support Spark 3.0.1 on the following cloud platform versions. Ensure that you update your Snaplex to 4.25 GA before running your existing eXtreme Pipelines.
  
  • Amazon EMR 6.2.0 (Hadoop distribution: Amazon)
  • Azure Databricks 7.5

Known Issues

• When ELT and Spark SQL 2.x Snap account credentials—such as user names, passwords, client secrets, auth codes and tokens, secret keys, and keystores—are auto-filled using the Google Chrome browser, the accounts, and hence the Pipelines fail. This is because the browser overwrites the field values with its own encrypted values that the SnapLogic Platform cannot read. SnapLogic recommends that you do not auto-save your Snap account credentials in the Chrome browser.
  Ensure that you delete any credentials that the browser has already saved for elastic.snaplogic.com, and then perform ONE of the following actions:
  • Option 1: Click  that appears in the address bar after you submit your login credentials at elastic.snaplogic.com, and then click Never. 
  • Option 2: Disable the Offer to save Passwords option at chrome://settings/passwords while working with your SnapLogic Pipelines. If you disable this option, your Chrome browser will not remember your passwords on any other website.

• For Redshift and Azure Synapse databases, the ELT Load Snap displays JSON values for fields with datatypes Date and Time when validating the Pipeline.

  Snowflake	Redshift	Azure Synapse
  ”time_data” : “09:23:56”,	”time_data” : {“time”: “09:23:56”},	”time_data” : {“time”: “09:23:56”},
  ”date_data” : “2020-12-20”,	”date_data” : “2020-12-20”,	”date_data” : {“date”: “2020-12-20”},

  However, the ELT Load Snap outputs the correct date and time values when executing the Pipeline. 

  Info
  To prevent these values from showing up in JSON format in the previews and ELT Transform Snap, you can cast the time or date values to a VARCHAR type in your custom query as follows:  CAST (<time_column_type_name> AS VARCHAR(<size_based_on_precision_required>)). Example: CAST (time_data AS VARCHAR(20)).

  
  

• ELT Transform Snap displays incorrect data types (string instead of the actual data type) for column names populated in its Input schema section.

New! Introducing Fast Data Loader (FDL) 

Ideal for business/data analysts, FDL is a data ingestion tool that simplifies the tasks of replicating and synchronizing data from the following source endpoints:

• Coupa

• SQL Server

• Oracle

• Salesforce

• ServiceNow

• Weather (pre-configured endpoint)

• Covid (pre-configured endpoint)
  
  You can integrate your data source with the following destination data warehouses:

• Redshift
• SAP DWC
• Snowflake

To gain your complimentary access to FDL, sign-up on the registration page. For more information, see the FDL user guide. For support questions, contact fdl-support@snaplogic.com. 

Prior 4.24 eXtreme Dot Releases that are Merged into 4.25 GA