The Amazon SNS Account now includes an Enable large message support checkbox to support publishing messages larger than 256 KB. When you select this checkbox, the Snap displays S3 related properties that enables you to store the messages in the S3 bucket.
The Pipeline Execution Statistics of the Join Snap now has a status message that displays the parameters - Free disk space, Available memory, and Average document size.
The internal sort buffer size is reduced to a minimum of 10MB when the available memory in the node becomes lower than 500MB to avoid the out-of-memory crash.
The internal sort buffer size is restored to its original size when the available memory becomes larger than 2GB.
We have improved the readability of the error message for the out of disk space on node error. The updated error message now provides clearer information and guidance for users, as shown below: Reason: Insufficient free disk space available to stage sort data into temporary files. Resolution: Increase the amount of free disk space and try again.
The PostgreSQL - Bulk Load and PostgreSQL - Insert Snaps no longer fail with the message ERROR: type modifier is not allowed for type 'bytea' when creating a new table if Create table if not present is selected and the target table does not exist. This issue occurred when metadata from the second input view document contained columns of the bytea data type.
Fixed an issue with the Snowflake - Bulk Load Snap that resulted in lowercase (or mixed case) column names when creating a new table under specific conditions. The new Create table with uppercase column names checkbox addresses this issue.
Fixed an issue with the Kafka Producer Snap to avoid an error about delivery.timeout.ms. The Snap now automatically adjusts the value of the Message Publish Timeout setting and displays a warning in the Pipeline Execution Statistics.
Fixed an issue that occurred with the JSON Splitter Snap when used in an Ultra pipeline. The request was acknowledged before it was processed by the downstream Snaps, which caused a 400 Bad Request response.
As of March 29, 2023, a new AutoSync release is available in Studio. No changes were made to AutoSync V1.0 for the March release. Customers who purchased or are evaluating AutoSync V1.0 will still have access to it until they migrate their data pipelines to the new version. To evaluate the new AutoSync, contact your CSM.
This is what you need to know:
The AutoSync Studio release has a new user-friendly interface:
This release supports the synchronization of data from Salesforce to Snowflake. More endpoints are coming soon.
Both the new release and AutoSync V1.0 are available simultaneously for Orgs where they are enabled:
Access V1.0 from the pulldown menu on the IIP Designer tab:
Access the new AutoSync from the Studio navigation menu:
Org admins who enable the new AutoSync for new users must also give them access to the Intelligent Integration Platform (IIP).
SnapLogic provisions Snaplexes dedicated to AutoSync. For V1.0, the Snaplex name is AutoSyncPlex and for AutoSync in Studio, the Snaplex name is AutoSync. Only run AutoSync data pipelines, not pipelines created in the IIP, on these Snaplexes.
Orgs with AutoSync Snaplexes provisioned before March 29, 2023 need to upgrade to main-16624 - 4.32 Patch 4.
On some screens, including laptops, browsers with enhanced zoom, and screens using a non-standard resolution:
The text on AutoSync dashboard cards might overlap.
In the Studio Execution overview, all five status summary cards might not display. If this occurs reduce the zoom percentage to view them all:
In the Studio Execution overview, if you click a summary card to filter the execution list, child data pipeline executions do not show the associated parent. To find the parent, click on the execution. The last component of the Path is the parent pipeline name. For example, mydatapipeline:
If Auto synchronize is disabled for a data pipeline, you cannot enable it from the data pipeline details panel:
Although the toggle appears to be on, the Next run value remains blank in the panel. Instead, use the Auto synchronize toggle on the card to enable synchronization.
When you edit a data pipeline, the screen with Sources and Destinations to choose from might be blank. Refresh the page to continue. Learn how to edit a data pipeline.
The AutoSync dashboard auto-refreshes periodically. However, it does not correctly update the Next run value on data pipeline cards.
New data pipeline wizard
A data pipeline unexpectedly runs twice if the synchronization schedule start time is in the past and you click Save and run at the end of the wizard. To avoid running it twice, either make sure the schedule time is in the future, or click Save instead of Save and run.
Changing the data pipeline name at the last step of the wizard causes an error. To avoid the error, change the name before the last step or edit it later.
Credentials and accounts
In IIP Manager, Org admins can add Accounts to a user group so that AutoSync users can share them. The dropdown list contains Account types that are not supported by AutoSync. For example, in the following screen, you should only select Salesforce or Snowflake, because those endpoints are supported in this release:
For Accounts created in the IIP, AutoSync only supports the following Account types. In AutoSync, only the supported types display in the dropdown.
Snowflake S3 Database and Snowflake S3 OAuth2
Salesforce.com and Salesforce OAuth2 Account types
AutoSync does not validate that an Amazon S3 account for staging data to Snowflake has write permission. Use an Amazon S3 account with write permission or the data pipeline will fail.
Include deleted records checkbox for Salesforce
AutoSync might incorrectly report new records on the data pipeline card and in the details panel. This can happen when:
A data pipeline is edited to enable tracking of deleted records in Salesforce, with Include deleted records selected.
Records were recently deleted from Salesforce before the data pipeline was edited.
On the next run, AutoSync might report the deleted records as if they were new records added to Snowflake. However, in the target table, these records are correctly marked as deleted. Learn more about how AutoSync tracks deleted records for Salesforce.
SQL Threat Detection API policy. To protect APIs against SQL commands injected in the header, path, query, and payload of incoming HTTP requests, apply this API policy.
Support for the OAS 3.0 specification. You can generate a specification in the Version Details tab and choose the JSON or YAML version in addition to the OAS version.
YAML support is now added to the following workflows:
The Developer Portal features a redesigned navigation menu.
In the Subscription Manager > User Access tab, the table for tracking the users of your Developer Portal now includes a column for user type. Portal indicates Developer Portal users who are not members of the SnapLogic Org. Org indicates users who are already members of the Org. When you add a new user to the Developer Portal, you receive a message if the user already exists.
You can now permanently delete APIs and API versions from the Recycle Bin in Manager. Previously, deleting these assets from the Recycle Bin resulted in an error.
All policy violation error messages are returned in JSON format for the following API policies:
Before this release, some policies returned policy violation error messages in plain text.
Starting with the February release, we introduced an Ultra enhancement that will significantly speed up the creation, restart, and disabling of Ultra Tasks by reducing scheduling times. The Ultra Task is decentralized by moving it out of the Control Plane and into the production Snaplexes. Extensive internal and private beta testing for a few months shows startup times reduced from several minutes down to several seconds.
This change automatically applies when you upgrade your Org Snaplex to the 4.32 GA. This enhancement will have little to no impact on the resource use of Snaplex nodes.
Sensitive Account information is protected when you change an encrypted field to use an expression.
Label changes for the Copy Project Assets dialog correctly describe fields for the Target Org and Target Project or Folder:
Enable and disable Pipeline Validation at the project level when you create or edit a project. Previously, this was an Org-level setting.
The Task Details page allows you to enable/disable a task through a toggle switch. Previously, you could only disable or enable a task in the assets table in Manager.
The Project Import wizard now supports duplicate asset handling. When you upload a project ZIP with assets containing the same name as existing assets, you can choose to duplicate the asset, make a copy of the project, or select which assets to overwrite.
In line with the support for Git, the new icon for Git operations replaces the icon displayed prior to the March 2023 release.
Fixed an issue where the Insights tab View By Snaplex field displayed the Snaplex environment instead of the Snaplex name.
The Snaps mentioned in this section are deployed on the Latest distribution type. To access the updated Snap Packs, your Org admin will need to select the mentioned Snap Pack version in Manager > Snap Packs.
Added support for SQS accounts to reauthenticate when authentication expires and the IAM Role checkbox is selected. (432patches19990)
The Kerberos Account that is available for a subset of Snaps in the Hadoop Snap Pack now supports a configuration that enables you to read from and write to the Hadoop Distributed File System (HDFS) managed by multiple Hadoop clusters. You can specify the location of the Hadoop configuration files in the Hadoop config directory field. The value in this field overrides the value that is set on the Snaplex system property for configuring a single cluster. (432patches20139)
The JDBC driver class for Redshift accounts is bundled with the com.amazon.redshift.jdbc42.Driver as the default driver. This upgrade is backward compatible. The existing pipelines will continue to work as expected and the new pipelines will use the Redshift Driver as the default driver. SnapLogic will support providing fixes for the issues you might encounter with accounts that use the PostgreSQL driver only until November 2023. After November 2023, SnapLogic will not provide support for the issues with the PostgreSQL driver. Therefore, we recommend migrating from the PostgreSQL JDBC driver to the Redshift JDBC driver. Learn more about migrating from the PostgreSQL JDBC Driver to the Amazon Redshift Driver. (432patches20101)
The Instance type option in the Redshift Bulk Load Snap enables you to use the Amazon EC2 R6a instance. This property appears only when the parallelism value is greater than one. (432patches20101)
The Redshift Snaps that earlier supported only the Redshift Cluster now also support Redshift Serverless. With Redshift Serverless, you can avoid setting up and managing data warehouse infrastructure when you run or scale analytics (432patches20035).
You can choose to NOT assign these permissions in your Snap accounts. (432patches20035)
Intermittent connectivity issues no longer occur when using some Snaps in the Azure SQL Snap Pack. These issues caused the following message to display: The connection is broken and recovery is not possible. The connection is marked by the client driver as unrecoverable. No attempt was made to restore the connection. (432patches20049)
In the 4.27 (427patches13615) release and later, pipelines that truncated (or deleted and re-created) a table and then used the BigQuery Write or BigQuery Bulk Load (Streaming) Snaps were allowed to retry the load operation. This lead to potential data loss because Google BigQuery uses an eventually consistent backend where one server might respond that it’s okay to write, but other servers still truncate the table. In the 4.32 (432patches20298) release and later, we disabled retries on truncated tables in these two Snaps to prevent a potential loss of data. We recommend that you use the Google BigQuery Bulk Load (Cloud Storage) Snap instead.
Fixed an issue with the JSON Splitter Snap for some use cases where it was not backward compatible with the 4.31 GA version. These cases involved specific uses of either the Include scalar parents feature or the Include Paths feature.
In the Redshift Select Snap, when you configure Output fields and deselect the Match data types checkbox, the display of the label name for the timestamptz data type in the output preview varies with the Redshift JDBC and the PostgreSQL JDBC drivers. For the Redshift JDBC driver, the Snap prefixes Redshift to the Timestamp label. For the PostgreSQL JDBC driver, Snap displays the labels as configured in the Snap settings. This does not impact the performance of the Snap. Note: The behavior of the Snap remains the same when you select the Match data types checkbox, regardless of using the PostgreSQL or the Redshift driver—the label names are displayed as configured in the Snap settings.
The new ability to visually correlate metrics assists when you monitor and troubleshoot pipeline execution and system behavior. For example, to determine why memory use spiked, you might want to check the number of pipelines that were running at the same time. Do that on the Studio Metrics page by hovering over a chart to view data point values. The value indicator tracks the same point in time for all charts: