Highlights

The SnapLogic® July 2023 Release includes the following:

SnapGPT (Early Access)

You can now build your pipelines using simple text prompts.

Snap Updates in July

API Management

• Generate specifications for your API Proxy

• New Response policies for your API Proxy

• Support for Git Integration

AutoSync

• New source endpoints: Gainsight PX, Microsoft Dynamics 365 for Sales, and ServiceNow

• New destination endpoints: Azure Synapse and Amazon Redshift

• Connect to Oracle and SQL Server on-premises data sources

• Share data pipelines and transfer ownership

• Org admins can now retrieve deleted data pipelines

• For Snowflake and Google BigQuery, create error tables that capture which records failed to load

• SCD2 support for some new sources and destinations

• Enable notifications for when a data pipeline runs, completes, or completes with errors.

Platform

• Third-party tool integration for observability using OpenTelemetry Services

• Enhanced options for scaling across Snaplex nodes for Ultra Pipeline Tasks

• Project selection when copying the Assets

Snaps

• Introduced the Google Analytics 4 Snap Pack:

  • Create Real-time Report Snap

  • Create Report Snap

  • Google Service Account

• Introduced the following new Snaps:

  • PGP Sign

  • SAP SuccessFactors Read Advanced

Studio

• Renamed to Monitor to better reflect its functionality

Dot Releases - Snaps

SnapGPT

26 Jul 2023: Launched SnapGPT. SnapGPT is an AI-driven platform that lets you generate your pipelines using simple text prompts. Learn more.

API Management

New Features

• New API Policies for Proxies:

  • Response Mapper: Transforms the response body. The policy controls provide expression/target path mappings and a Pass Through option for unaffected mappings.

  • Response Transformer: Transforms the response headers of the API. You can use this policy to add, remove, or modify response headers.

• Support for Git Integration: You can now create APIs from Git repositories and treat your API Versions as Git assets. When your Org is integrated with Git, you can access the same set of functionalities for APIM version assets, and the Version is treated as the Project folder. This support does not extend to API Proxies.

• Generate a Specification of your API Proxy. When you create a Proxy from a specification and edit it or modify the endpoints, you can regenerate the spec from the Proxy Details tab. This enables you to create the Proxy with a specification and generate new specifications to keep in sync with your modifications.

Enhancements

• Support for restoring deleted API Proxies. When you delete a proxy or its endpoints, the assets are sent to the Recycle Bin in Manager. Before this release, the delete operation was irreversible for API Proxies.

• Email sent for deprecated APIs: On the Developer Portal, when you deprecate an API, an email is sent to subscribers to that API with a generic deprecation message. You can also indicate newer versions of the API that are available on the Developer Portal.

• Ultra Time Stamps collected in Public API. When you run the Public API, the response map includes new fields that indicate the time the request was made and the wait time before the request is processed. Learn more.

Known Issues

• In the Create API dialog, if you use Azure Git to create the API assets, neither SnapLogic Assets nor the GitHub repository display correctly in the Asset table, even though the dialog closes, until the underlying Git checkout operation is completed. Workaround: Navigate back to the API version and verify the Git operation is complete.

• You cannot copy Proxy files to another API Proxy or Proxy endpoint. Workaround: Download the Proxy file and add it manually into the target Proxy or endpoint.

AutoSync

New Features

• AutoSync now supports the following endpoints:

  • Gainsight PX as a source.

  • Microsoft Dynamics 365 as a source.

  • Azure Synapse as a target.

  • ServiceNow as a source.

  • Amazon Redshift as a target.

• Support for connections to on-premises Oracle and SQL Server. You can configure one Oracle or SQL Server table as a source per data pipeline. On-premises connectivity requires:

  • A customer-managed Snaplex (Groundplex). Learn more.

  • For this release, the source Accounts must be created in the IIP. Known Issues provides the details.

• Share your data pipelines with colleagues or groups. Allow them to view, view and run, or edit. You can transfer ownership of a data pipeline that you created. After the transfer, you no longer own it, so cannot transfer it again. Learn more.

• Enable notifications for when a data pipeline runs, completes, or completes with errors. Learn more.

• Deleted data pipelines now stay in the trash for 30 days. Org admins can retrieve or permanently delete any data pipelines in the trash. Learn more.

• For Snowflake and Google BigQuery, track records that failed to load correctly in an error table. For new data pipelines, the option is available when you configure credentials. Edit existing data pipelines to enable this option in the endpoint configuration. Learn more about error tables in How AutoSync handles errors.

• SCD2 support for Salesforce or ServiceNow sources to Snowflake or Amazon Redshift destinations.

Enhancements

• Navigate between the IIP, Monitor (formerly Studio), and AutoSync with the new application menu and find links to the documentation and community:

  

• The summary card and flyout panel now show the elapsed time of the last run:

  

• Improved interaction for CSV upload and new card view:

  

• To aid troubleshooting, error messages that show in the data pipeline details panel now contain a link that opens the relevant execution details:

  

Known Issues

• For Azure Synapse, AutoSync only supports Blob storage in this release. If you try to specify ADLS Gen 2 storage, the data pipeline fails.

• AutoSync uses bulk load for data pipelines with a Salesforce source and the Full and Incremental load types. This causes AutoSync to attempt to chunk the data. When a few records, or none, are changed, synchronization can take longer than expected.

• For the Amazon Redshift endpoint, the Redshift cluster and the S3 staging location should be in the same Amazon region. AutoSync does not currently detect whether the regions match. To avoid issues, make sure Redshift and the S3 bucket are in the same region.

• Oracle columns with a number type can have a maximum value of 9.99..9 x 10^125. This is larger than the number types supported for most CDWs. For example, Google BigQuery supports number(38). If a source number column has greater precision than the destination can accept, an error occurs when AutoSync attempts to load the data.

• To use Oracle and SQL Server on-premises sources with AutoSync, you must provide a customer-managed Snaplex (Groundplex). You cannot use AutoSync in an Org with Enhanced Account Encryption. If all of your Orgs are currently configured for Enhanced Account Encryption, a new Org must be provisioned for AutoSync where you can set up your customer-managed Snaplex to connect to your on-premises sources.

• For Oracle and SQL Server on-premises sources, you must create an Account for the on-premises source endpoint in the IIP before creating your data pipeline. You can create the Account from AutoSync Manager or from the Manager Accounts tab. When validating the source Account, select a self-managed Snaplex from the global shared folder that can connect to both the on-premises source and the CDW:
  

  

  When creating the data pipeline in AutoSync, select the same self-managed Snaplex:
  

  

Platform

New Features

• New Public APIs are available:

  • Export APIM policies

  • Import APIM policies

  • Change the Snaplex of a task

• Git Integration now supports the .gitignore file, where you can specify which assets to ignore when checking into your repository. Learn more.

New Features in the Restricted Snaplex Version

• Introducing the initial release of our third-party Observability tools integration feature in private beta. SnapLogic platform uses OpenTelemetry to support telemetry data integration with third-party Observability tools. This beta feature release includes implementing the service to enable the monitoring of your Pipeline execution runtime logs in Datadog. Learn more.

• In Manager, you can configure notifications to receive an email or a Slack message when a node in a Snaplex fails or crashes. When you set a notification rule, a Snaplex node crash option is available under a Snaplex. To create, update, or manage notifications, refer to Manage Notifications. You can also view the node fail/crash information on the Manager > Alerts tab. A red triangle indicates the status of a new alert, and if the alert is older than an hour, the status is indicated by a gray triangle. To view alerts, refer to Viewing Alert Information.

• For Ultra Tasks, you can now configure the Auto scaling of instances from the specific options to provide flexibility and the ability to respond to peak loads without manual intervention. Learn more.

Enhancements

• The Activity Log now displays Asset creation by import for pipelines and other assets. When you import a Project and select Create a new project, the Activity Log event description displays the Asset Create by Operation label. The description also indicates that the Project import operation created the Assets. Tasks are not included.

  

• The APIM Public API Migrate an API version accepts a new request body parameter async_mode for asynchronous migration.

• The APIM Public API Retrieve API Management logs returns more detailed time information for Ultra Tasks.

• The Public API Retrieve activities in Org accepts the new value org_concurrent_call_limit for the query parameter filter_event_type. This returns events where the maximum number of concurrent pipeline executions for the Org was met or exceeded.

• In the Git Integration dialog for Azure Repos, the Client ID field is deleted. The App ID field now takes the value from the Azure Repos setting of the same name. Learn more.

• When an Org admin deletes a user from an Org, the user’s assets are first reassigned to the Org admin. After the owner transfer completes, the user is deleted. Previously, the delete user operation would hang if the user owned a large number of assets. Learn more.

• In Manager, you can now select any Project under a specific Project Space when you copy the Assets. Learn more.

• New application menu to navigate between the IIP Designer, IIP Manager (Classic Manager), Monitor (formerly Studio), and AutoSync:

  

Known Issues

• Node crash events are sometimes incorrectly reported when a Snaplex is upgraded or restarted.

• When you delete a user or group from a Project space, their access might not be removed from the shared folder. Users or groups added to a Project space are not always correctly propagated to the respective shared folder, so shared folder permissions might not match the Project space permissions for some users or groups.
  Workaround:

  • If the user or group does not exist in the Project space but shows up in the respective shared folder, add them back in Project space permissions with Propagate permissions selected. Then, remove them and check the shared folder again to verify that user or group is removed.

  • If a user or group exists in the Project space but not in the respective shared folder, remove them in Project space permissions and add them back with Propagate permissions selected. Then, check the shared folder again to verify if that user or group shows up.

  • If a user or group exists in both the Project space and shared folder but permissions, don’t match, change the permissions to match.

• If you stop a node and it is not able to reach the JCC, an error message is logged. You can find it in the Dashboard by viewing the More details tab for the node. The message is incorrectly repeated 1000 times.

Snaps

New Snaps

• Introduced the Google Analytics 4 Snap Pack, a cloud-based analytics platform that provides you the ability to gain deeper insights into user behavior, track conversions, and make data-driven decisions. This Snap Pack includes the following Snaps and account. (433patches21872):

  • Create Realtime Report Snap: Enables you to create a customized report of your Google Analytics real-time event data.

  • Create Report Snap: Enables you to create a customized report of your Google Analytics event data.

  • Google Service Account: Enables you to authenticate and authorize applications or services to access Google APIs.

• Introduced the following new Snaps:

  • PGP Sign Snap: This Snap from the Binary Snap Pack enables you to use PGP to sign a file digitally without encrypting the file. (433patches21870)

  • SuccessFactors Read Advanced Snap: This Snap from the SAP SuccessFactors Snap Pack enables you to read objects recursively for objects containing links to traverse through the tree structure. (main137)

Upcoming

Enhancements

• The Aggregate, Group By Fields, Group By N, Join, Sort, Unique Snaps from the Transform Snap Pack now include new fields to improve memory management. (433patches21461)

• The Amazon S3 Snaps automatically detect the Maximum session duration value for the Cross-Account IAM role (1 through 12 hours). The Snaps round down the value to the nearest hour. So, if the Snap administrator sets the Maximum session duration at 3 hours and 45 minutes, the Snaps read it as 3 hours. The Snaps also refresh the session before it expires. However, the automatic session refresh does not support the case of very large file upload or download that takes longer than the maximum session duration. (433patches21816)

• The Decrypt Field and Encrypt Field Snaps from the Transform Snap Pack now support CTR (Counter mode) for the AES (Advanced Encryption Standard) block cipher algorithm. (433patches21779)

• The Decrypt Field Snap now supports the decryption of various encrypted fields by providing a valid decryption key. (433patches21586)

• The Deduplicate Snap now manages memory efficiently and eliminates out-of-memory crashes using the following fields: (433patches21572):

  • Minimum memory (MB)

  • Minimum free disk space (MB)

• The Dynamics 365 for Sales Snaps include a new Enable continue on error checkbox to allow the Snap to continue executing the remaining batch requests, even if the previous request does not execute. (433patches21467)

• The Dynamics365 for Sales Update Snap performs the update operation as expected when the Batch size is more than 1, and the Snap successfully executes when the batch size is 1. (433patches21467)

• The pagination feature implemented for the ABFS protocol in the Hadoop Directory Browser Snap enables the Snap to return all the output documents as expected. (433patches21494)

• The LDAP Search and LDAP Lookup Snaps now support a string representation for binary attributes using the Include string representation for binary attributes checkbox. (433patches21488)

• The NetSuite Snap Pack now supports Customer-Subsidiary and Vendor-Subsidiary relationship Object types.
  Note: NetSuite's application requires the Oneworld extension for utilizing these subsidiary relationships, and Snap integration should use WSDL 2019_1 or later versions.
  (433patches21878)

• The PGP Encrypt Snap now includes an Encryption key ID field, which allows you to specify the key ID for encrypting the data. If you do not specify an encryption key ID, the Snap uses the first key in the public key (master key). (433patches21482)

• The SQL Server Snap Pack is upgraded to the Microsoft JDBC Driver 12.2.0.jre11 for SQL Server. Existing pipelines that do not use the default driver (bundled with the SQL Server Snap Pack) are not impacted. If you want to use NTLM or other authentications with the latest default driver (where the DLL file is mandatory), then you must copy the DLL file to the Snaplex path. Learn more about the SQL Server driver upgrade: FAQ for Upgrading to SQL Server JDBC Driver 12.2.0. (433patches21386)

Fixed Issues

• Fixed an issue with the Azure Synapse SQL Bulk Load Snap that caused a 404 error when the External Location was set to ADLS Gen2 and the Azure Folder field was not specified in the account settings. (433patches21718)

• Fixed an issue with the HTTP Client Snap where the Snap now conceals the additional authentication headers in the debug information for Secured Headers Account. (433patches21860)

• Fixed an issue with the SQL Server Insert Snap that was causing performance degradation between GA release 4.32 and 4.33 if the network connection between the Groundplex and the database server was slow. (433patches21808)

Deprecated

The TDE Formatter and Tableau Write Snaps from the Tableau Snap Pack are deprecated. Contact support@snaplogic.com if your existing pipelines use Snaps from the deprecated Snap Pack. (433patches21835)

Behavior Changes

Starting from version 433patches21482, the PGP Encrypt Snap will no longer support encryption with an expired key. To ensure continued support for encryption, we highly recommend extending the expiration of your PGP key.

Known Issues

• The Send Email Snap may fail because of a 429 error response from the Graph API endpoint.

• The SFTP protocol is now incompatible with some deprecated algorithms. This affects mostly the Binary Snap Pack but might also apply to the Email Snap Pack. (433patches21482)

• SFTP connections might experience connectivity issues because of the removal of some deprecated algorithms. To add these connections back, refer to Configuration Settings for Snaps for details.

Custom Snaps

Enhanced the following Custom Snap documents by adding missing information, improving clarity in field descriptions, and implementing the latest UI changes:

• AS2 Snap Pack

• EDI Parser Snap Pack

• Marketing Cloud Snap Pack

• Google Drive Snap Pack

Studio

To better reflect its functionality, Studio has been renamed to Monitor. It has a new fixed navigation pane:

Enhancements

• Application menu to navigate between the Monitor, Designer, and AutoSync:

  

Documentation

Video

• API Management V - Create and Publish a Proxy API - Having learned how to create, publish, and manage APIs in the APIM-1, APIM-2, APIM-3, and APIM-4 videos, in this video, we learn how to create a Proxy API and publish it on the Developer Portal.

• Automate Data Transformations with AutoPrep - In this video, we learn how the AutoPrep Snap helps to flatten a composite structure, rename a field, and change the data type of a column.

Enhancements

• Enhanced the Application Configuration in Azure Portal for OAuth2 Account to use in OneDrive to improve usability and readability.

• Added the following information to the existing Account and Snap documents in the ELT Snap Pack:

  • Multiple example pipelines/configurations.

  • Troubleshooting items.