Highlights

Snap Updates in June

The SnapLogic® May 2023 Release includes the following:

API Management

• Support for SOAP endpoints in Proxies

AutoSync

• New endpoints, Google BigQuery, Amazon S3 (for CSV and JSON files), Marketo, and CSV files

• Synchronize with SCD2 between Salesforce and Snowflake

• For a Salesforce source, AutoSync automatically handles changes to the column precision, scale, or character limit by propagating the changes to the target.

Platform

• The new Pipeline Cache feature (private beta) enables you to leverage in-memory lookup

• New APIM Public APIs allow you to publish, unpublish, deprecate, or retire a set of API versions

• Support for CyberArk Conjur in Secrets Manager

Snaps

• Introduced the GraphQL Client Snap in the API Suite Snap Pack, which enables you to query a GraphQL endpoint and retrieve data

• Added the Dynamics 365 Finance and SCM Create, Update, and Delete Snaps to the Microsoft Dynamics 365 Finance and Supply Chain Management Snap Pack

Studio

• Improved accessibility

Documentation

• New article about pipeline dependency on the control plane

• Enhancements in Data Automation articles

• Improved documentation of Snaps developed by the Professional Services team

• New APIM video about managing your Developer Portal

May UAT Delta

About Upgrading to this Release

Snap Updates in June

Upcoming

Enhancements

• The Dynamics 365 for Sales Snaps include a new Enable continue on error checkbox to allow the Snap to continue executing the remaining batch requests, even if the previous request does not execute. (433patches21467)

• The Dynamics365 for Sales Update Snap performs the update operation as expected when the Batch size is more than 1, and the Snap successfully executes when the batch size is 1. (433patches21467)

• The pagination feature implemented for the ABFS protocol in the Hadoop Directory Browser Snap enables the Snap to return all the output documents as expected. (433patches21494)

• The PGP Encrypt Snap now includes an Encryption key ID field, which allows you to specify the key ID for encrypting the data. If you do not specify an encryption key ID, the Snap uses the first key in the public key (master key). (433patches21482)

• The following Transform Snaps include new fields to improve memory management: Aggregate, Group By Fields, Group By N, Join, Sort, Unique. (433patches21461)

• The SQL Server Snap Pack is upgraded to the Microsoft JDBC Driver 12.2.0.jre11 for SQL Server. Existing pipelines that do not use the default driver (bundled with the SQL Server Snap Pack) are not impacted. If you want to use NTLM or other authentications with the latest default driver (where the DLL file is mandatory), then you must copy the DLL file to the Snaplex path. (433patches21386)

• The SQL Server Snap Pack now incorporates an identifying connection property allowing the SQL Server instance to identify the source of connections for auditing purposes. (433patches21535)

Behavior Changes

• Starting from version 433patches21482, the PGP Encrypt Snap will no longer support encryption with an expired key. To ensure continued support for encryption, we highly recommend extending the expiration of your PGP key.

Breaking Changes

• Pipelines that use the default Microsoft JDBC Driver in the SQL server accounts might be impacted after the upgrade because the latest driver is backward-incompatible. To prevent an impact to your accounts, reconfigure the URL properties field in account settings as follows:
  trustServerCertificate = true

• With the 4.33 GA release, we no longer support some algorithms for SFTP connection negotiation because we’ve updated the library used to connect to SFTP sources and to improve security. We are working on a fix to support the removed algorithms automatically. With Snap Pack version 433patches21645, you can modify the global properties. Refer to the Configuration Settings for Snaps for details about how to revert to the previous settings.

Known Issues

• The SFTP protocol is now incompatible with some deprecated algorithms. This affects mostly the Binary Snap Pack, but might also apply to the Email Snap Pack. (433patches21482)

• SFTP connections may experience connectivity issues due to the removal of some deprecated algorithms. To add these connections back, refer to Configuration Settings for Snaps for details.

Dot Releases - Snaps

API Management

New Features

• Support for SOAP web services in Proxies. You can now reference a SOAP service when you create a Proxy endpoint.

  

• Refer to the Platform section for Lifecycle Management Public APIs. You can now use Public API to publish, unpublish, deprecate, and retire APIs.

Enhancements

• Added the Trust All Certificates checkbox in the Create proxy endpoint dialog. Similar to the setting in the REST Snaps, you can choose to allow your Proxy to trust self-signed, unsigned, or expired certificates in the endpoints.

• API policy violations are now delivered in JSON. When a client executes an API that results in an authorization or authentication error, the error is returned in the JSON format.

• Logging added to the Callout Authenticator API policy. You can now use the Snaplex logs to troubleshoot why the authentication failed when your Callout Authenticator policy rejects a client request.

• Improved error output for the JSON Validator API policy. The API caller can easily determine which part of the JSON failed validation.

• Wildcard endpoint support for dynamic base paths and error handling in Proxy endpoint mapping. When the base path is a path parameter, the Proxy uses this endpoint and routes through the mappings.

• Improved error handling in the Developer Portal API specification editor. Specification violations now appear in the editor.

• Added the Time Unit field in the Client Throttle API policy. You can now define a Request Limit value in requests per second.

• API Manager now supports asynchronous API migration with Public APIs.

Known Issues

• Previously selected HTTP methods are not cleared when you reopen the Create proxy endpoint dialog and select the REST to SOAP checkbox.
  Workaround: Refresh the browser.

Fixed Issues

• Fixed an issue where the Try it functionality did not work for an API Proxy published on the Developer Portal.

AutoSync

New Features

• AutoSync now supports the following endpoints:

  • Google BigQuery as a target. Learn more.

  • Marketo as a source. Learn more.

  • CSV or JSON files stored in Amazon S3 as a source. Learn more.

  • CSV file upload from a local or network drive as a source. Learn more.

• The ability to change load type for synchronization. The types available depend on what AutoSync supports for the source and endpoint. AutoSync supports loading from Salesforce to Snowflake using SCD2.
  To change the load type after creating a data pipeline, edit it. From the Auto Synchronize tab, select the Load type. Learn more.
  

  

• For a Salesforce source, AutoSync automatically handles changes to the column precision, scale, or character limit by propagating the changes to the target. Learn more.

Enhancements

• Improved synchronization scheduling. As the last step in the wizard, choose to:

  • Continue without scheduling.

  • Pick a preset schedule and set the start time.

  • Create a custom schedule, which allows you to set an end date.

Known Issues

• If you remove or rename a source table or object after using it in a data pipeline, its name will still be visible in the source configuration and you will not be able to de-select it.

• For Amazon S3 data sources, you must add a forward slash if you copy the URL from the Amazon S3 console. For example, change s3://my_bucket to s3:///my_bucket.

• The first row of CSV files must define the names for the column headings in the target table, with no empty or null values.

• AutoSync cannot upload a CSV file to the cloud data warehouse if the file name includes single or double quotes or a special character, such as #, $, %, or &.

• Google BigQuery does not support special characters for names. Data pipelines where the source file, table, or object names include special characters will fail. Whitespace, dash, and underscore in the names are not a problem.

• Sometimes, AutoSync cannot clean up the Google Cloud Storage staging area after loading from Salesforce to Google BigQuery. If this occurs, you should manually remove the files to retrieve the storage.

• In the IIP AutoSync Manager screen, you can create Accounts to use with AutoSync. For Google BigQuery, the Account Type dropdown allows you to select Google Service Account or Google Service Account JSON, which are not supported by AutoSync.

• Sometimes, when you create or edit the synchronization schedule, you can select a start time in the past. If you do this for a data pipeline that is scheduled to run once, it will not run unless you start it manually.

• Data pipelines that load the Marketo Program Manager object can time out and fail.

• To use the SCD2 load type for Snowflake, you must modify Snowflake configurations created before the May 2023 release. Because AutoSync automatically sets the timezone to UTC for SCD2 operations, do the following:

  • For an Account created in the IIP, add a Uri property parameter with the name TIMEZONE and a value of UTC.

  • For credentials saved in AutoSync, delete them and create a new configuration.

Fixed Issues

On May 30, 2023 fixed an issue with SCD2 load type for Salesforce to Snowflake where the effective begin and end column values were strings. They are now timestamps

Data Automation

Fixed Issues

31 May 2023: Fixed a null pointer exception so no 5XX errors can occur if you download non-existent query details from the Pipeline Execution Statistics of an ELT (write-type) Snap.

Platform

New Features

• Pipeline Cache is a subscription feature (currently available in private beta) that enables you to cache data in memory for reference in another pipeline. This information can be a dataset, and Expression Language functions can retrieve the data from this dataset. Looking up values based on database rows/columns is a costly operation if the table is queried repeatedly to use the same data as reference fields. An in-memory data store allows your pipelines to look up the data as key/value pairs in other Snaps at runtime. If you have data in an external system that you need to look up (such as an employee object or an identifier field), then you can use Pipeline Cache to optimize the mapping to match the references in this type of operation.

• Secrets Management introduces support for CyberArk Conjur. Learn more.

• New Public APIs are available for APIM.

  • Publish a set of API versions

  • Unpublish a set of API versions

  • Deprecate a set of API versions

  • Retire a set of API versions

Enhancements

• When creating a new project from a Git repository, you can also create a new branch for the new project. Learn more.

• Support for HashiCorp KV Secrets Engine Version 1 is available, in addition to KV Search Engine Version 2.

Fixed Issues

• Fixed an issue where Orgs could not be provisioned.

• The subscription feature Secrets Management -CyberArk is now displayed correctly on the Manager > Subscriptions page.

Known Issues

• Bearer Tokens are not included when an Org admin migrates a Triggered Task to another Org.
  Workaround: Manually change the Bearer Token for the migrated task in the Triggered Task dialog.

• When you delete a user or group from a Project space, their access might not be removed from the shared folder. Users or groups added to a Project space are not always correctly propagated to the respective shared folder, so shared folder permissions might not match the Project space permissions for some users or groups.
  Workaround: Manually remove the user or group and make sure permissions for the remaining users and groups at the Project space and shared folder level match.

Behavior Changes

• The March 2023 release added support for enabling/disabling Pipeline Validation at the Project level. This means that validation settings for a Pipeline set at the Project level could override the Org-level setting. As an Org admin, you can communicate the preferred validation settings on Projects through User Notifications.

• The com.snaplogic.jsonpath.ReadVisitor.THROW_ON_BRANCH feature flag has been removed, and the platform no longer produces an error when a value is missing when using jsonpath. Previously, this would result in a null value (since the field referenced in the expression did not exist) and a lint warning. In this release, the behavior results in an empty array [], and the UI does not display a lint warning.

Snaps

New Snaps

• Added the Dynamics 365 Finance and SCM Create, Update, and Delete Snaps to the Microsoft Dynamics 365 Finance and SCM Snap Pack.

• Introduced the GraphQL Client Snap in the API Suite Snap Pack, which enables you to query a GraphQL endpoint and retrieve data. This Snap features a built-in query builder to help you efficiently build queries.

• Introduced the SAP Concur Snap Pack. This Snap Pack contains the following Snaps and Account:

  • SAP Concur Read: Retrieves the data resources from the SAP Concur endpoint.

  • SAP Concur Write: Creates and updates the data resources from the SAP Concur endpoint.

  • SAP Concur Delete: Deletes data resources from the SAP Concur endpoint.

  • SAP Concur OAuth2 Account: Connect SAP Concur Snaps with data sources that use the SAP Concur OAuth2 account.

Enhancements

• The Bouncy Castle library version is upgraded from bcpg-jdk150n[1.69] to bcpg-jdk180n[1.73] in all our Snap Packs. This upgrade brings in the latest security features to enhance the performance of the SnapLogic platform.

• The Generic Database Account now supports the SSH Tunneling connection. You can now encrypt the network connection between your client and the database server, ensuring a highly secure connection.

• The Hive Snap Pack is Cloudera-certified for Cloudera Data Warehouse (CDW). You can use the Hive Execute Snap to work with CDW clusters through a Generic Hive Database account.

• The Marketo Bulk Extract Snap works successfully in the non-lineage path in an Ultra task.

• The Key passphrase field in the Private Key Account of the Binary Snap Pack now supports expressions, allowing dynamic evaluation using pipeline parameters when the expression button is enabled.

• Snowflake

  • SnapLogic is specified as a partner tag in all requests directing to Snowflake, making it easier for Snowflake to identify the requests coming from SnapLogic.

  • The default JDBC JAR for the Snowflake Snap Pack is upgraded to version 3.13.28 to support the GEOMETRY data type.

Fixed Issues

• Fixed an issue with the Encrypt Field Snap, where the Snap failed to support an RSA public key to encrypt a message or field. Now the Snap supports the RSA public key to encrypt the message.

Behavior Changes

• Previously, in a Project, you could use Snaps from a private Snap Pack uploaded to the Project, the Project shared folder, or the global shared folder. Now, if a Snap Pack is uploaded to the Project containing different Snaps than those uploaded to a shared folder, only the Snaps uploaded to the Project are visible. If you have existing pipelines using Snap Packs that are no longer available because of this change, you must fix them by uploading the required Snaps to the Project.

• As of 432patches19873 Snap Pack version, any LDAP entry attribute in the LDAP Search and LDAP Lookup Snaps containing binary data will be displayed as a byte array (Base64 encoded) instead of a string in the output.

Studio

Enhancements

• The UI now uses accessible colors. Some icons are larger and more readable. For example, note the status icons on the summary cards in the Execution overview:
  

  

Changes in Behavior

• For the Execution Overview:

  • The Executed by column was renamed to Owner.

  • Invoker is a new optional column that identifies who ran a Triggered Task or manually ran an AutoSync data pipeline. It contains the user ID and their IP address. To add the Invoker column, click in the Search bar and click the Invoker pill:
    

    

The following screenshot shows the Owner and Invoker columns. The last row in the example has a value in the Invoker column for an AutoSync data pipeline that was run manually:

Documentation Enhancements

New Articles

The new page Groundplexes and Pipeline Dependency on the Control Plane provides information for designing and running pipelines that reduce dependency on the control plane.

Snap Documentation Enhancements

The following application configuration pages for the Exchange Online and SharePoint Online Snap Packs are enhanced with the key steps, workflow, and frequently asked questions:

• Application Configuration in Exchange Online

• Application Configuration in SharePoint Online

Data Automation Enhancements

• Enhanced the following Snap documents in the ELT Snap Pack to improve usability and updated the content:

  • The Snap Pack landing page

  • ELT Load

  • ELT SCD2

  • ELT Merge Into

  • Detailed prerequisites for successful validation of the ELT DLP Account.

• Revised and updated the existing lists of Limitations and Known Issues across the Snaps and the Accounts in the ELT Snap Pack.

Professional Services Snaps

Enhanced the following PS Snap documents by adding missing information, improving clarity in field descriptions, and implementing the latest UI changes.

• Adobe Experience Snap Pack

• Amazon ElasticCache Snap Pack

• AWS EC2 Utilities Snap Pack

New Videos

• API Management IV - Manage the SnapLogic APIM Developer Portal: To continue with the previous API Management videos, this video shows you how to configure and use a Developer Portal where all the published videos can be published for the end user.