In this article

Key New Features in SnapLogic

Studio:

• Introduced runtime details for the Execution overview when you click on a Task or Pipeline execution.

• Added support for search in the table of executions on the Execution overview.

• Introduced new icons for Pipeline execution status that show executions with errors and those that were completed with warnings.

Platform:

• Added the ability to view child Pipeline execution status from the parent Pipeline in Dashboard or the Studio Execution overview.

• Added Flow as an asset type when migrating Projects and Project Spaces.

• Improved Snaplex network logging.

• Updated Snaplex Helm chart.

API Management:

• Added the ability to publish a Proxy to the Developer Portal.

• Added conditional mapping rules between a Proxy endpoint and an upstream API.

• Added support for Open API Specification (OAS) 3.0 in creating and publishing an API. Generated specifications still use OAS 2.0.

• Added the ability to enable the Try It Out feature for all APIs in the Developer Portal.

• Added the ability to specify custom subscription key parameters.

• Replaced Token authorization with Bearer authorization for new API requests in the Developer Portal.

Snaps:

• Introducing new SAP S/4HANA Cloud Snap Pack.

• Added OAuth2 Account support for Coupa.

• Added on-premises support for Microsoft Dynamics 365 For Sales and a new Dynamics 365 NTML Account type for this Snap Pack.

• Added REST In-memory OAuth2 Account type for REST Snap Pack.

Data Automation:

• ELT Snap Pack

  • Added the following Snaps:

    • ELT Router

    • ELT Cast Function

    • ELT String Function

  • Enhanced the ELT SCD2 Snap to improve its flexibility and efficiency.

• FDL

  • Added Google Sheets as a source endpoint.

SnapLogic Flows:

• Renamed the What’s New page to the Release Updates page.

• Added Flows as an asset type when exporting/migrating projects in SnapLogic IIP.

• Added new data collection types for the Marketo, Salesforce, and Jira endpoints.

4.29 UAT Delta

4.29 Dot Releases

Snaps

SnapLogic Studio (Preview)

New Features

Execution Overview Includes Pipeline and Task Execution Details: You can now view details and execution statistics for your Pipelines and Tasks. When you click on a Pipeline or Task, SnapLogic Studio displays a fly-out panel with information about the invocation type, the name of the Snaplex, execution times and duration, as well as information about Snap statistics, Pipeline logs, and Pipeline parameters. The Pipeline and Task execution details enable you to survey Pipeline and Task executions in your Org and then drill deeper to understand individual Pipeline execution details.

Open

New Execution Status Icons: New icons indicate Completed with Errors and Completed with Warnings statuses of your Pipeline executions when you select Route Error Data to Error View in Views > Error setting in a Snap.

Open

Likewise, when Snaps encounter warnings or errors, the Snap displays an icon for each state respectively to indicate the status after Pipeline execution.

Search by Execution Status: You can now filter results by several attributes, including Pipeline execution status (for example, Started) and the person who ran the execution. You can also add and remove columns and rearrange them. This functionality enables you to quickly find Pipelines in specific states, as shown in the following screenshot:

Platform 

Enhancements

• Child Pipeline Execution States: Added the capability in both the SnapLogic Studio Execution overview and Dashboard to view a child Pipeline’s status by hovering over the parent Pipeline. You can quickly identify which child Pipeline execution failed in Pipelines with the Pipeline Execute Snap. The Pipeline execution displays the icon when you select when you select Route Error Data to Error View in Views > Error setting in the Pipeline Execute Snap and either the Snap or child Pipelines encounter errors.

• Include Flows when Exporting and Migrating: Added Flows as an asset type in Manager. While exporting or migrating a project, you can now also select Flows as a separate asset type.

• Logging and Monitoring: Improved the logging and monitoring of events and errors related to network conditions around Snaplex instances, enabling SnapLogic Support more information to troubleshoot.

• Migration to shared Directory: The Project Migration Public API can migrate assets to the shared directory of the project space.

• Updated Snaplex Helm chart: Version 3 of the Snaplex Helm chart includes the following enhancements:

  • Support for Liveness, Readiness, and Startup Probe.

  • Support for setting SL_FILE_ULIMIT and SL_PROCESS_ULIMIT.

  • Default Kubernetes Service type changed from NodePort to LoadBalancer.

Known Issues

In Studio, when an Ultra Pipeline has a Snap that has an execution status of Completed with errors, the Snap Statistics tab displays the error in red and yellow, instead of only yellow as the Dashboard version of the Snap Statistics tab does.

API Management

New Features

• Proxy Mapping Rules: You can now create Proxy Mapping Rules to map your Proxy endpoints to the right target depending on the conditions. Learn more about Proxy Mapping Rules.

• Proxy Publication: You can also publish Proxies in the API Manager. In the Developer Portal, they appear like other APIs. Learn more about publishing a Proxy.

• Dynamic Search: Dynamic Search is now available in the API Manager.

• CORS Org-wide Setting for “Try It Out”: Org Administrators can now enable the “Try It Out” feature for all APIs in the Developer Portal using a default CORS Restriction Policy. However, if a CORS Restriction Policy exists at the API level or API version level, it has precedence over the default Org-wide policy. Learn more about setting up the Developer Portal.

• Auto-Approve Subscriptions: When you publish an API or Proxy, you can now choose to auto-approve requests for subscriptions to the API. Learn more about publishing an API version or Proxy.

• Custom Subscription Key Parameters: When you publish an API or Proxy, you can specify the header or the query parameter that the API consumer must set to the subscription secret key. Learn more about publishing an API version or Proxy.

• Fast Secret Copy: API consumers can also copy the application's secret key by clicking the key icon () on the application's card, instead of having to edit the application entry to copy the secret key.

Enhancements

• Open API Specification (OAS) 3.0: The API Manager now accepts OAS 3.0 specifications for APIs; however, API consumers can download only OAS 2.0 specifications.

• Bearer Authorization Header: When calling an API with an approved subscription, you must set the Authorization header as Authorization: Bearer <client secret>, instead of using Token.

• Featured APIs: In the Grid View of the Developer Portal, featured APIs are now easier to find with the existence of the Featured pill at the top of the API card. Featured APIs appear at the top of the API list. The API's tags are also displayed as pills at the bottom of the card.

• New Documentation Article: Using Long-running Triggered Tasks with an Enterprise Scheduler describes how to use a script to prevent timeout or lost connection errors when executing long-running tasks from an enterprise scheduler. The script starts Pipeline execution and polls continuously for completion status. This keeps the connection alive and prevents the scheduler from reporting an error.

Known Issues

• API Dashboard:

  • The API Dashboard does not collect metrics for APIs that do not generate a response entity.

  • When the API Management feature is enabled for the Org, the API Dashboard does not automatically start collecting data. You must restart the Snaplex nodes and allow the Dashboard to start.

• CORS Setting Delay: In the Portal Manager » Settings tab, changes to CORS Setting » Enable Try Me might not take effect for up to two hours due to caching. This occurs only in JCC nodes with Snaplex Version 4.28 and older.

• No Notification About Modified Key Settings: If an API is republished with modified Key Settings, existing subscribers are not notified of the change.

Snaps

New Snaps

SAP S/4HANA Cloud: SAP S/4HANA Cloud is an enterprise resource planning (ERP) system with built-in intelligent technologies, such as AI, machine learning, and advanced analytics. It helps companies adopt new business models, manage business change, and organize internal and external resources. This Snap Pack offers the following Snaps:

• SAP S/4HANA Cloud Read: Retrieves data from SAP S/4HANA Cloud applications. 

• SAP S/4HANA Cloud Create: Creates data objects in SAP S/4HANA Cloud applications.

• SAP S/4HANA Cloud Update: Updates data objects in SAP S/4HANA Cloud applications.

• SAP S/4HANA Cloud Delete: Removes data objects from SAP S/4HANA Cloud applications.

Enhancements

• Azure Active Directory

• Azure SQL:

  • Enhanced the Azure SQL Account and Azure SQL Active Directory Account with the Disable connection pooling checkbox that allows you to manage session states sharing.

• Binary:

  • Enhanced the File Writer Snap with the Prevent URL encoding checkbox to manage the URL encoding.

  • Enhanced the following Snaps and Accounts that use SnapLogic File System (SLFS) to display Lint warnings in Pipelines when using incorrect file paths to read and write files. We recommend that you use the File Upload icon to specify the file path.

    • File Writer

    • File Reader

    • File Delete

    • HDFS Reader

    • HDFS Writer

    • HDFS ZipFile Reader

    • HDFS ZipFile Writer

    • S3 File Writer

    • S3 File Reader

    • ORC Reader

    • ORC Write

    • ZipFile Read

    • ZipFile Write

    • Google Service Account

    • Two-Factor Auth Account

• Box:

  • Enhanced the Box Write Snap with the Write empty file checkbox that allows you to write an empty file when the incoming binary document has empty data.

• Coupa:

  • Added a new account type Coupa OAuth2 that enables OAuth2 authentication for all Coupa Snaps.

  • Enhanced the Coupa Create and Coupa Update Snaps with the minimum number of output views to zero.

• Google Sheets:

  • Enhanced the Spreadsheets Browser, Worksheet Reader, and Worksheet Writer Snaps with the Include Shared Drives checkbox to support working with the Google Shared Drives.

  • Enhanced the Worksheet Reader and Worksheet Writer Snaps with the Display Spreadsheet ID in Suggestions checkbox. You can select this checkbox to display both spreadsheet ID and the spreadsheet name in the suggestions. This helps you to choose among the spreadsheets having the same name.

  • Enhanced the Worksheet Writer Snap with the Write Mode dropdown list to perform the desired write option for the given worksheet—that is either create a new worksheet or append or edit an existing worksheet. You can write new data to the worksheet by specifying a cell reference as the starting point or append data to the worksheet at the last row in the worksheet containing content.

  • Enhanced the Spreadsheet Browser Snap with the Fetch Mode dropdown list to fetch the spreadsheets and worksheets. You can either fetch spreadsheets and worksheets within the specified Spreadsheet or only spreadsheets.

• Hadoop:

  • Added the AWS S3 Dynamic account for Parquet Reader and Parquet Writer Snaps.

  • Enhanced the AWS S3 Account for Hadoop account to include the S3 Region field that allows cross-region or proxied cross-region access to S3 buckets in the Parquet Reader and Parquet Writer Snaps.

• JDBC:

  • Enhanced the Generic Database Account in the JDBC Snap pack to include Netsuite as an option for the Database name field to connect to the NetSuite database. While configuring the account, if you select Netsuite or Autodetect (if the Snap detects the target database as NetSuite) then the Limit rows field in the Generic JDBC - Select Snap is ignored.

• Kafka:

  • Kafka Snap Pack now supports Azure Event Hubs streaming service. Learn more: Azure Event Hubs configuration in Azure portal for Kafka SSL Account.

• MS Dynamics 365 for Sales:

  • MS Dynamics 365 For Sales Snaps now supports on-premise installations of MS Dynamics 365.

  • Added a new account type Dynamics 365 NTLM Account that enables to connect to MS Dynamics 365 Snaps through NTLM authentication.

  • Updates in the MS Dynamics 365 for Sales Create and MS Dynamics 365 for Sales Upsert Snaps:

    • Improved performance by streamlining the process of creating new entities.

    • Fixed an issue where the Content-Length header was incorrectly set when Unicode characters were present in the request.

• OpenAPI:

  • Enhanced the Base URL field in the OpenAPI Snap to support suggestions associated with OpenAPI Specifications.

  • Added Dynamic OAuth2 account that enables OAuth authentication for OpenAPI Snap.

• REST:

  • Added the REST In-memory OAuth2 Account type with Client Credentials Grant Type that supports and stores the access token in memory for reuse until it expires. When Snap encounters 401 or 403 errors, this account generates a new token after refreshing the cache.

• Slack:

  • Enhanced the Slack - Send Message Snap with User Emails option for the Target Type dropdown list to send messages on the Slack app using the user’s email.

• Snowflake:

  • Updated the Snowflake - Bulk Upsert Snap for the following:

    • The Snap displayed an incorrect resolution when the length of the value exceeded the value defined in a column.

    • The Snap failed with a NullPointer Exception when no value was provided for Error Limit field and Error percentage limit fields.

    • The Snap displayed an incorrect error message when S3 details were not provided.

    • Added the On Error dropdown list, where you can select an action to perform when the Snap encounters errors in a file.

  • The following enhancements were done for the Snowflake - Bulk Load Snap:

    • The Snap optimizes AWS account configuration by skipping account validation if the S3 Storage Integration field value is provided.

    • The Snap overrides the storage integration specified in the account settings with the storage integration specified in the Snap settings.

  • Enhanced the Snowflake - Unload Snap to skip AWS account validation if S3 Storage Integration property is provided.

• SQL Server:

  • Enhanced the SQL Server - Bulk Load Snap with the Batch Size field to process records in a batch.

  • Enhanced the SQL Server - Select and SQL Server - Lookup Snaps with Option for Unique Identifier Column that allows you to convert the case to lower or upper while displaying the Unique Identifier output (display) value.

• Teradata:

  • Enhanced the Teradata FastExport Snap with Character Set dropdown list to support encoding of data when you export data from the Teradata Database.

• Transform:

  • Fixed an issue with CSV Formatter and CSV Generator Snaps that displayed an incorrect error when one or more header values in the CSV file were missing. This error is now routed to the error view.

  • Fixed an issue with the CSV Parser Snap, where the Snap stopped indefinitely while processing certain inputs.

  • Fixed an issue with the CSV Formatter Snap, where the escape character selected was not used.

  • Updated the CSV library used in the CSV Generator, CSV Parser, and CSV Formatter Snaps. This library update enforces values for each column in a parsed CSV header.

  • Enhanced the XML Formatter Snap with the Output Character Set field. You can specify or select the character set you want for your binary output. For example, UTF-8 or ISO860.

• Workday:

  • This Snap Pack supports the latest version of Workday APIs (certified to be compatible with version 37.0).

  • Enhanced the Workday Cancel Snap with Number of Retries and Retry Interval (seconds) fields to support the retry mechanism.

Deprecated

• The Twitter, Facebook, Foursquare Snap Packs are deprecated and are no longer available. Contact support@snaplogic.com, if your existing Pipelines use Snaps from the deprecated Snap Pack.

• The Coupa Account will be deprecated after 4.29 GA, as Coupa will discontinue the support of the API-based authentication in their upcoming September 2022 Release (R34).

Known Issues

• Worksheet Reader: When you provide a spreadsheet ID to fetch data, and if a spreadsheet with that ID already exists as a spreadsheet name, the Snap fails to fetch the data, instead it fetches the data of the spreadsheet whose name matches with the spreadsheet ID.
  Suggestion: If you want to fetch the data of the specified spreadsheet ID, ensure that you specify the spreadsheet name of that spreadsheet ID or both (spreadsheet name with spreadsheet ID).

• Worksheet Writer: When you provide a spreadsheet ID to append or insert data, and if a spreadsheet with the same ID already exists as the spreadsheet name, the Snap fails to append or insert the data for the specified spreadsheet ID and creates a new worksheet for the spreadsheet name
  Suggestion: Ensure that you specify a unique spreadsheet name or spreadsheet name with spreadsheet ID.

Behavior Change

• The CSV Formatter Snap now allows an empty value for the Escape Character field if no escape character is used in the input CSV data. Earlier, this Snap used a '\' (forward slash) in the input when the Escape Character field was left empty causing an invalid data error.

• The expected output for the MongoDB - Update Snap is modified due to the upgrade of the Spring Core framework version and the matchedDocuments definition is introduced. Earlier, the updateOfExisting would be true if matchedDocuments was greater than 0. Now, updateOfExisting is true if updatedDocuments is greater than 0.

• The Snowflake Execute and Snowflake Select Snaps now write the output value as-is for FLOAT or DOUBLE datatype columns if these columns have the value as NaN (Not a Number). Earlier, the Snaps displayed an exception error when the FLOAT or DOUBLE datatype column has the value as NaN. This behavior is not backward compatible.

• The Salesforce Subscriber Snap now stops and reports the error immediately when it encounters the daily limit exceeded error. Earlier, Snap reported an error and stopped only at the next resubscribe attempt when it encountered the daily limit exceeded error.

• The Dynamics 365 For Sales Update and Dynamics 365 For Sales Upsert Snaps now write the error to the error view and do not write to the output view when the Snaps encountered an error.

Data Automation

SnapLogic’s Data Automation solution for cloud data platforms accelerates the movement of data from legacy systems, applications, and many other sources into the cloud data warehouse (CDW). This flexible solution delivers traditional ETL integrations to achieve complex transformations of data in-flight, as well as ELT capabilities to transform data in-place, such that data does not move out of the CDW. Coupled with comprehensive API management capabilities, SnapLogic’s Data Automation solution drives full end-to-end automation of data processes, accelerates the loading/transformation of data into the CDW, and ultimately streamlines business processes for quicker data-to-decision delivery.

New ELT Snaps

• Introduced a new Snap ELT Router to enable routing input SQL queries into multiple output views based on the given conditional expressions.

• Introduced ELT Cast Function Snap to convert a data type of a column in the input SQL string into other supported data types.

• Introduced ELT String Function Snap to support the various string functions supported by the different databases.

Enhancements

• Enhanced the following Snaps to display the final SQL query in their output preview upon Pipeline validation.

  • ELT Load

  • ELT Merge Into

  • ELT Insert-Select

• Enhanced the ELT Database Account to support OAuth2-based authentication on the target Snowflake database.

• Enhanced the ELT Select, ELT Insert Select, ELT SCD2, ELT Merge Into, and ELT Load Snaps to display suggestions on the Schema Name field based on the Default Database Name provided in the Snap Account configuration when the Database Name is not specified in the respective Snap.

  • Improved usability of the suggestions features for these Snaps by making them case-insensitive. For example, typing default in the Schema Name field displays both default and DEFAULT, if they co-exist. You do not need to type DEFAULT to invoke and select the schema name DEFAULT from the suggestions list.

• Enhanced the ELT SCD2 Snap to address different feature requests and issues raised by multiple customers. These changes provide more flexibility in configuring your SCD2 operations using this Snap.

  • Removed Check for nulls and duplicates in source field and added two dropdown lists - Null Value Behavior and Invalid Row Handling.

  • Refer to the ELT SCD2 scenarios to learn more.

  • Made the following items in the Meaning field of the Target Table Temporal Fields fieldset mandatory while making the Invalid historical rows flag optional.

    • Current row

    • Historical row

• Enhanced the ELT Aggregate and ELT Window Functions Snaps to support the following functions across all supported CDWs:

  • KURTOSIS

  • MODE

  • SKEW

• Enhanced the ELT Aggregate Snap to support the following GROUP BY features across all supported CDWs:

  • Group by Cube

  • Group by Grouping Sets

  • Group by Rollup

  • Automatic GROUP BY for all input columns.

• Fixed an issue with ELT Merge Into Snap where the Snap erroneously modified the target table column name when the column name contained the target table name.

• Fixed an issue in ELT SCD2 Snap where the Snap causes incorrect results with Snowflake targets, when:

  • The Historical Row End Date value is provided.

  • Nulls and Invalid rows are recognized, but one or more start dates in the source are null.

• Fixed the issue in ELT Transform Snap where the Output Schema of the Snap does not populate all the column names from its Input Schema.

Known Issues

• [This issue is fixed in the July 2022 Release]: In the case of BigQuery, the ELT Load Snap fails to load data from files in S3 or Redshift when you specify more than one individual file in the File List fieldset. This is due to the API restrictions in Google BigQuery for loading separate files/tables from S3 and Redshift. Alternatively, you can use the File Name Pattern field to load data from multiple files in S3 or Redshift to your BigQuery instance.

• If you have dropped one or more columns from your target table outside of the ELT Load Snap and then use this Snap to add these column(s) back and load data into the table, the Snap adds the new columns to the target table, but not the corresponding data from the source file.

• When running without Sub-Query Pushdown Optimization (SPDO), ELT Pipelines that contain an ELT Aggregate Snap configured with one ore more GROUP BY ROLLUP fields do not verify the column data types while inserting the Snap output values in the target table. This may lead to incorrect data written to the target table. However, as long as SPDO is on, the same Pipeline runs without this issue.

• In the case of Azure Synapse, if you are configuring the Start_Date column as a Target Table Natural Key in the ELT SCD2 Snap, the Snap fails in each of the following scenarios:

  • The source table/file contains one or more null values.

  • The target table is empty.

  • The End date of current row or End date of historical row has a static value, for example: '2021-01-01'.

• [This issue is fixed in the June 2022 Release] When you are enclosing the column names within backticks (`<Column_Name>`) for creating a new target table in a DLP instance using the ELT Load Snap—typically to ensure that the column names are used verbatim—you may encounter the error: ERROR processing query/statement: no viable alternative at input 'FROM'(line X, pos XX). To prevent this error, edit the header in your source file to exclude any special characters, thereby avoiding the use of backticks in the target table column names.

• [This issue is fixed in the July 2022 Release]: The ELT Select Snap does not honor the LIMIT clause included to query a BigQuery instance, during Pipeline validation. However, this issue does not occur during Pipeline runtime.

• [This issue is fixed in the July 2022 Release]: In the case of Azure Synapse, BigQuery, DLP, and Redshift CDWs, the SKEW function available in the General Aggregate Functions List of the ELT Aggregate Snap returns null instead of the actual division-by-zero exception for certain ranges of values (for example, for a series of exactly same values).

Fast Data Loader

New Feature

Added Google Sheets as a source endpoint in Fast Data Loader (FDL). Google Sheets enables you to read data from the spreadsheet and write the data to the specified destination endpoint such as SAP-DWC or Snowflake.

Flows

Enhancements

• The What’s New page has been renamed to the Release Updates page.

• You can now select Flows assets when exporting or migrating projects in IIP.

• Added two new data collection types for the Marketo endpoint:

  • Get Program by Name (source). Use this data collection type to find a Marketo program.

  • Clone Program (target). Use this data collection type to clone an existing Marketo program.

• Added five new data delivery types for the Salesforce target endpoint:

  • Delete Account: Use this data delivery type to delete an account.

  • Delete Contact: Use this data delivery type to delete a contact.

  • Delete Lead: Use this data delivery type to delete a lead.

  • Delete Opportunity: Use this data delivery type to delete an opportunity

  • Delete another object: Use this data delivery type to delete another object.

• Added the Transition an Issue data collection type for the Jira target endpoint. Use this delivery type to transition the status of a Jira issue.

• Added the Dynamics 365 NTLM Account type for the Microsoft Dynamics 365 endpoint.

• Added the REST In-memory OAuth2 Account type for the REST source endpoint.

Fixed Issues

• Fixed an issue that prevented users from renaming Flows.

• Fixed an issue that prevented users from loading a Flow from the Projects tab.