4.30 UAT Delta

SnapLogic Studio (Preview)

Enhancements

Execution Time for Requests

FeedMaster node details include the mean execution time for the last 1000 processed requests for active Ultra Tasks:

Open

Behavior Change

The Snaplex type label for Customer managed was changed to Self managed. From the System overview you can click a Snaplex to view its details, as shown below:

Open

Patterns Catalog

The following Patterns were added to the Studio Pattern Catalog:

• Convert Salesforce Lead to Account, Contact, and Opportunity (Optional)

• Create Box Folders for ServiceNow Service Requests

• Create Microsoft Dynamics CRM Opportunity and Workday Invoice if Sale Occurs

• Create Request Ticket in ServiceNow and Send Email

• Create Request Ticket in ServiceNow When Employee Added in Workday

• Creating a ServiceNow Incident From a JIRA Issue

• Creating Users Within SnapLogic (Excel + REST)

• Creating Users Within SnapLogic (JSON Generator + REST)

• Create Users Within ServiceNow Through Workday Approval Process

• CSV to Workday Tenant

Known Issues

TBD

Platform

New Features

GitHub Enterprise Server

The SnapLogic Git Integration supports GitHub Enterprise Server (GHES). From the SnapLogic user interface, you can use GHES to track SnapLogic Projects, Files, Pipelines, Tasks, and Accounts. Similar to the Git Integration for GitHub, the GHES Integration enables you to manage your Project Assets across development, test, and production Orgs, to achieve continuous integration and development (CI/CD).

Using GHES Git Integration requires the following:

1. A subscription to the Git Integration for GHES.

2. Configuration of GHES.

3. Configuration of the network to support communication between SnapLogic and GHES.

4. Configuration of SnapLogic Org settings.

When the configuration is finished, each user logs in to SnapLogic to authorize and authenticate with their GHES account. After a Project is associated with a repository, the controls to check out, pull, add, delete, and commit are available for Project Assets. Learn more about the Git Integration and Enabling the Git Integration for GHES.

Secrets Management - HashiCorp Vault

Many organizations must comply with internal or external regulations for controlling access to sensitive information such as account credentials The SnapLogic Secrets Management feature allows organizations to use a third-party secrets manager to store credentials. Instead of entering credentials directly in SnapLogic Accounts and relying on SnapLogic to encrypt them, the Accounts contain only the information necessary to access the secrets manager. During validation and execution, Pipelines obtain the credentials directly from the secrets manager. With Secrets Management, secrets are never stored by the SnapLogic control plane or by Snaplex nodes. With the August 2022 Release, SnapLogic supports the HashiCorp Vault Cloud, Enterprise, and Open Source editions.

The high-level tasks required to use Secrets Management include the following:

1. Use of self-managed Snaplexes (Groundplexes) and a subscription to Secrets Management.

2. An administrator of the secrets manager must configure the storage for endpoint credentials, create authentication roles and access permissions, and generate secrets.

3. A Snaplex administrator must configure the Groundplex nodes with the token and the information required to communicate with the secrets manager.

4. In SnapLogic, a Pipeline designer or Org admin must configure the dynamic accounts to authenticate with the secrets manager.

Learn more about https://docs.snaplogic.com/cicd/secrets-mgmt/secrets-mgmt-hashicorp.html .

Enhancements

Propagate Project Space Permissions

When adding user or group permissions for a Project Space, Org admins can specify that those permissions should apply to all Projects. After you add propagated permissions, the user or group displays only in the Permissions dialog for the shared folder, not for the individual projects. Although they don’t show in the Permissions dialog for individual projects, they have the permission granted at the Project Space level.

Known Issues

TBD

API Management

New Features

• New widget in the API Dashboard. In the API tab of the Dashboard, you can add the new Top Consumer by Requests widget to display metrics about consumers of your APIs. Learn more about API Insights.

• SSO authentication for the Developer Portal. If a user logs in to SnapLogic through an SSO provider and the user navigates from the Portal Manager to the Org’s Developer Portal, the user is taken directly to the Developer Portal landing page without having to log in again. Learn more about setting up SSO in SnapLogic.

Snaps

New Snaps

Amazon S3: Introduces the S3 Snap Pack that allows you to manage objects in S3. This Snap Pack contains the following Snaps:

• S3 Browser: Lists the attributes of S3 objects in the specified bucket matching the prefix.

• S3 Copy: Sends a copy request to the S3 service to copy an S3 object from a source bucket to a target bucket.

• S3 Delete: Removes an object from the specified bucket.

• S3 Download: Downloads S3 objects from the S3 bucket.

• S3 Upload: Uploads binary data to S3 objects.

• S3 Presigned: Generates a presigned URL in the output document to access an S3 object.

• S3 Account: Enables you to access resources on the S3 bucket.

Databricks: A new Snap Pack for Databricks Lakehouse Platform (Databricks or DLP) introduces the following Snaps:

• Databricks - Select: Retrieves information from the target Databricks table.

• Databricks - Insert: Inserts new rows of data in the target Databricks table.

• Databricks - Delete: Deletes data from a target Databricks table.

• Databricks - Bulk Load: Loads millions of rows of data in the target table with a single operation.

• Databricks - Unload: Unloads data from a target Databricks table with a single operation.

• Databricks - Merge Into: Updates millions of existing rows and inserts new rows in a target Databricks table with a single operation.

• Databricks - Multi Execute: Runs multiple SQL statements on the target Databricks instance.

Google BigQuery: Introduces the following Snaps and accounts to manage datasets in Google BigQuery (GBQ):

• BigQuery Dataset Create: Creates datasets in the GBQ.

• BigQuery Dataset Delete: Removes datasets from the GBQ.

• BigQuery Dataset List: Lists datasets from the GBQ.

• BigQuery Table Delete: Removes tables from the GBQ.

• BigQuery Table List: Lists tables that are read from the GBQ dataset.

• Google Service Account JSON: Accesses the resources on the GBQ using a JSON Key.

Google Sheets: Introduces the following Snaps to subscribe to and unsubscribe from Google spreadsheets.

• Google Sheet Subscribe: Subscribes to the specific spreadsheet for notifications of the changes in the Google Sheet.

• Google Sheet Unsubscribe: Unsubscribes from the specific spreadsheet.

Transform Snap Pack: Introducing a data preparation application within a Snap that offers a new user experience.

• AutoPrep Snap: Enables you to profile a data set before forwarding the data for further processing. You can flatten structured data, include and exclude data fields, and change data types through menu-like displays, making Pipeline building easier.

Enhancements

• All Snaps with the JSON type output preview include the Render whitespace checkbox, which enables or disables the rendering of whitespace in the output. The Snap renders blank spaces as a dot “.” and tab spaces as an underscore “_”.
  
  

• Dynamic Account types across all the Snap Packs support Secrets Management, a SnapLogic add-on that allows you to store endpoint credentials in a third-party secrets manager. Orgs using Secrets Management provide the information necessary to retrieve the secrets in expression-enabled dynamic Account fields. During validation and execution, Pipelines obtain the credentials directly from the Secrets Manager.
  
  

• Azure Active Directory

  • The Azure Active Directory Search Entries Snap includes a Display Properties field where you can specify the properties to display in the output for the user or group. For the Snap to correctly return the attributes in the output, you must specify the attribute name as described in User profile attributes in Azure Active Directory B2C. Learn more about Properties for a user and Properties for a group.
    
    

• Binary

  • The File Operation Snap supports moving data from a local node to an Azure blob through the AZ Copy utility.

  • The Azure Storage Account includes the Request Size (MB) field to set the buffer limit before writing to Azure storage to enhance the performance.

  • The SSH Auth Account supports dynamic values for the following fields that enable you to use Pipeline parameters:

    • Username

    • Private key

    • Key passphrase
      
      

• Dynamo DB

  • The DynamoDB Account includes the AWS Security Token field to pass the security token when making calls using temporary credentials.
    
    

• Email

  • The Email Sender Snap supports the file:/// protocol in the Attachments field.
    
    

• Google BigQuery

  • The Location field in the BigQuery Execute Snap lists all the locations in the suggestions.
    
    

• Google Directory

  • The original payload is passed to the error view in the Google Directory Insert Snap.

  • The original payload is displayed under the original tag in the Google Directory Update Snap.
    
    

• Google Sheets

  • The Allow empty rows checkbox in the Worksheet Reader Snap allows you to output empty rows. Select this checkbox to include empty rows between other rows that contain data.

  • The Worksheet Writer Snap supports expressions for the Starting Cell Reference field. This Snap also alerts you about the potential overwriting of existing data for the Starting Cell Reference field when writing data in the same range of cells.
    
    

• Hadoop

  • Extended the AWS S3 Dynamic Account support to ORC Reader and ORC Writer Snaps to support the AWS Security Token Service (STS) using temporary credentials.
    
    

• JDBC

  • Enhanced the Generic Database account to support dynamic values for Account properties, Advanced properties, and URL properties.
    
    

• Kafka

  • Upgraded Kafka client libraries to Apache-Kafka Version 3.1.1 and Confluent-Kafka Version 7.1.1.

  • Optimized the performance of the Snap with the support for LZ4 and ZSTD compression types of the messages produced.

  • The Kafka account that connects to the Kafka Schema Registry uses TLS v1.2 protocol instead of the older version of TLS, because it has security vulnerabilities.
    
    

• MS Dynamics 365 for Sales

  • The Alternate Keys fieldset in the Dynamics 365 For Sales Update and Dynamics 365 For Sales Upsert Snaps now enable you to refer to entities by unique or nonunique combinations of columns.
    
    

• OpenAPI

  • The OpenAPI Snap supports external files referenced in the OpenAPI specification field and includes the following new fields:

    • The Pass through checkbox allows you to pass input data to the output under the original key.

    • The Trust all certificates checkbox allows you to use self-signed certificates.

    • The Enable URL Encoding checkbox allows automatic URL encoding of the request path.
      
      

• Oracle

  • Enhanced the Oracle - Merge and Oracle - Update Snaps with the Session parameters fieldset to set the parameters for National Language Support (NLS).

  • Enhanced the Oracle - Stored Procedure Snap with a process that caches the fetched procedure metadata. Hence, the Snap processes the metadata calls only for the first few documents and does not process for subsequent documents, thereby improving the Snap's performance.
    
    

• Postgres SQL

  • Enhanced the PostgreSQL Account and PostgreSQL Dynamic Account with SSH Tunneling configurations to encrypt the network connection between the client and the PostgreSQL Database server, thereby ensuring the secure network connection.
    
    

• RabbitMQ

  • The RabbitMQ Consumer and RabbitMQ Producer Snaps can consume and produce messages, respectively, from the quorum queue by setting the Argument key field as x-queue-type and the Argument value as quorum to consume or publish messages from/to the quorum queue.
    
    

• Redshift

  • The Redshift accounts support:

    • Expression enabler to pass values from Pipeline parameters.

    • Security Token for S3 bucket external staging.
      
      

• REST

  • You can set cookie policy specifications using the Cookie Policy dropdown list in REST Snaps .

  • Support for auto-retries where the REST Snaps automatically retry when they encounter the 429 HTTP status code - Too Many Requests error.

  • The REST AWS Sig v4 Account supports:

    • Passing AWS Security Token Service (STS) temporary credentials in the Security Token field.

    • Expression values for Access-key ID and Secret Key fields.

  • Improved the performance of REST Snaps when the Snaps encounter Unauthorized (401) or Forbidden (403) errors even when the user credentials are available in the organization and the account type is OAuth2.
    
    

• Salesforce

  • Updates in Salesforce Subscriber Snap:

    • CometD version upgrade from 5.0.9 to 7.0.6 (latest).

    • The simplified logging provides useful diagnostic information without logging any sensitive data.

    • The Snap asynchronously makes callbacks or listens to the CometD API to identify and respond to specific events.

    • The Snap processes the most recent message so that it can send that replay ID if there is a need to resubscribe. If the server responds with an invalid replay ID, the Snap automatically sets the Replay ID value to -2 to get all available messages.

  • You can select the Null Setting with Bulk API checkbox in the Salesforce Update, Salesforce Create, and Salesforce Upsert Snaps to format null values in the input document.
    
    

• SharePoint Online

  • Added the SharePoint Online Application Account type that supports Client Credentials flow.
    
    

• Snowflake

  • The following Snowflake Accounts support Key Pair Authentication:

    • Snowflake Azure Database Account

    • Snowflake Google Storage Database Account

    • Snowflake S3 Database Account

    • Snowflake S3 Dynamic Account
      
      

• Transform

  • The Hide whitespace option in the CSV Generator and JSON Generator Snaps allows you to hide the rendering of whitespace as symbols (dot or underscore) in the output that you may have in the CSV or JSON input documents.

  • The Render whitespace checkbox in the Mapper Snap enables or disables the rendering of whitespace in the input document. When a value in the Expression field has blank spaces (leading, trailing, or spaces in the middle of a string), the spaces are rendered as symbols (dot “.” or underscore “_”) in the output on selecting this checkbox.

  • The Excel Parser Snap includes the Custom Locale dropdown list that allows selecting a user-defined locale for formatting numbers.

  • The Selected fields in the Pivot Snap allow you to define fields to be unpivoted so that the remaining fields are automatically pivoted.

  • The XML Generator Snap includes examples on how to escape single (') and double quotes (“) when used with elements or attributes.
    
    

• The Snap Catalog has a new category for Custom Snaps developed by the SnapLogic Professional Services team. To subscribe to Custom Snaps, or if you have questions about Custom Snaps, contact your SnapLogic Customer Success Manager (CSM).

Deprecated Snaps

The Microsoft SOAP-based API (Organization Service) and the WS-Trust Security Protocol are being deprecated. Therefore, the Microsoft Dynamics CRM Snap Pack is no longer being updated and will be removed from the Snap Catalog in a future release. If you are having any issues with your existing Dynamics CRM Pipelines, contact support@snaplogic.com.

Known Issues

• Recent changes in Kafka to enable idempotent writes by default may result in authorization errors when using the Kafka Snaps if your Kafka cluster version is lower than 2.8.

  To fix this issue, we recommend you take one of the following actions:

  • Upgrade your Kafka cluster to at least version 2.8.

  • Use Kafka’s ACL management to ensure that IdempotentWrite is allowed for the cluster.

• When you add an input view to the Databricks - Delete Snap, ensure that you configure the Batch size as 1 in the Snap’s account configuration. For any other batch size, the Snap fails with the exception: Multi-batch parameter values are not supported for this query type.

• [This issue is fixed in the July 2022 Release]: The Databricks - Unload Snap fails with the error: External source/target type is invalid when you attempt unloading data from your DLP instance to a DBFS location.

Data Automation

ELT Snap Pack

The following enhancements are deployed to the SnapLogic production environment as a stable version of the ELT Snap Pack in this release (4.30 GA). The latest versions have been deployed in either the June 2022 or July 2022 releases.

Enhancements

• The ELT Load Snap can infer the schema from the source files located in S3 buckets and create, overwrite, and append the target table in your Redshift instance with the source data. It can infer the table schema from the data available in AVRO, CSV, JSON, ORC, and PARQUET files. Learn more at Automatic Schema Inference with ELT Load Snap.

• The ELT Snap Pack supports the latest JDBC drivers across CDWs—Azure Synapse, BigQuery, DLP, Redshift, and Snowflake. Learn more in Configuring ELT Database Accounts or the respective Account page for the specific versions.

• The Value List field in the ELT Pivot Snap provides dynamic suggestions.

• The ELT Snowflake Account supports Key Pair Authentication on the cloud environment hosting your Snowflake instance.

• You can configure S3 Bucket, Azure Storage, and DeltaLake Storage Gen2 mounts in the ELT DLP Account.

• The end Snap SQL query in the ELT Insert Select Snap’s preview output displays the CREATE TABLE... or the DELETE/DROP TABLE statements to be run before the query that inserts/loads data into a new table in the Snowflake target CDW.

• The ELT Load Snap now seamlessly reads from a CSV file in the S3 mount point on DBFS (DLP target instance) without causing the SQL exception—[Simba][SparkJDBCDriver](500051) ERROR processing query/statement.

• The ELT Insert Select, ELT Merge Into, ELT Load, and the ELT SCD2 Snaps run successfully even when the specified target table does not exist. The Snaps now create a new target table if it does not exist during Pipeline validation.

• The ELT SCD2 Snap:

  • On Pipeline validation, displays the final SQL query to be executed on the target CDW in its output preview.

  • Can replace an existing target table with a new table and load the SCD2 entries from the source table and files into it. Use the new Overwrite existing table option in the Target Table action field to perform this action.

• Field names across the ELT Aggregate, ELT Cast Function, ELT String Function, and ELT Transform Snaps are normalized—to maintain consistency.

Known Issues

• Refer to the Known Issues sections in May 2022, 4.29 Release Notes and July 2022 Release Notes for the complete list.