November 2024 Release Notes

Highlights

The SnapLogic November 2024 Release includes the following highlights:

Admin Manager

  • Configure Asset catalog (formerly Integration catalog) visibility

  • Manage Enhanced Account Encryption and Trusted environments for account migration

  • Configure the Git Integration

AgentCreator (formerly GenAI App Builder)

Provides the following capabilities to create agents:

  • Advanced response configuration: Enables consistent output for all LLM providers

  • Stop sequence support: Allows reduced token usage on output when the expected sequences hit

  • Tool calling: Included in the Flow Snap Pack with the PipeLoop Snap and in the LLM Utilities Snap Pack with the Message Appender Snap

  • Multimodal input support available in all LLM provider Snap Packs

AutoSync

  • New SQL Server Bulk Load endpoint

Monitor

  • Integration catalog renamed to Asset catalog

  • Ability to edit custom Asset catalog metadata from Task details

Platform

  • New API for managing Asset catalog custom metadata

Snaps

 


Dot Releases: Snaps

 

Snap Pack

Date

Snap Pack Version

Updates

 

Snap Pack

Date

Snap Pack Version

Updates

JDBC

Nov 28, 2024

439patches29008

  • Upgraded the JOOQ library of the JDBC Snap Pack from v3.9.1 to v3.17.x. Pipelines using the Generic - JDBC Snaps might be impacted by the jOOQ upgrade because the latest jOOQ version is not fully backward-compatible. Learn more.

  • Fixed an issue with Generic JDBC - Execute and Select Snaps when connecting to the Oracle database where the data was displayed in an unexpected structure and without the actual row address value when querying table columns of ROWID type. Now, the Snaps handle the ROWID type columns correctly to prevent the display of the nested output when ROWID is combined in the query with other selected columns.

Breaking change:

With the 439patches29008 Snap Pack version, if any of your existing pipelines with the Generic JDBC Execute and Select Snaps when connecting to the Oracle endpoint use the encoded ROWID(by manually handling the Base64 encoded data), your pipelines might break. To prevent your pipelines from failing, you must update your Snap Pack version to the previous version.

Behavior change:

Earlier, the ROWID columns were displayed in binary (Base64 encoded data) form in the output. With the 439patches29008, the Generic JDBC Execute and Select Snaps that integrate with the Oracle endpoint display the ROWID columns in string form in the output (which can be used for other operations downstream).

API Suite

Nov 22, 2024

439patches29124

  • Fixed an issue in the GraphQL Client Snap where improper extensions caused a validation error; however, it resulted in an empty error view when the Route Error Data to Error View was enabled.

  • Fixed an issue with the GraphQL Client Snap that did not load the Query Builder properly.

Transform

Nov 18, 2024

439patches29078

Fixed an issue with the CSV Parser Snap that introduced unexpected characters into the records and output data because of incorrect handling of the delimiter.

Salesforce

Nov 15, 2024

439patches29014

Fixed an issue with the Salesforce Lookup, SOQL, and SOSL Snaps, where errors were not routed to the error pipeline when Route Error Data to Error View was enabled, and a referenced expression variable was missing in the upstream or pipeline parameters.

Oracle

Nov 14, 2024

439patches29008

Fixed an issue with Oracle Execute, Select, and Stored Procedure Snaps where the data was displayed in an unexpected structure and without the actual row address value when querying table columns of ROWID type. Now, the Snaps handle the ROWID type columns correctly to prevent the display of the nested output when ROWID is combined in the query with other selected columns.

Breaking change:

With the 439patches29008Snap Pack version if any of your existing Oracle pipelines use the encoded ROWID(by manually handling the Base64 encoded data), your pipelines might break. To prevent your pipelines from failing, you must update your Snap Pack version to the latest version.

AutoSync

Nov 19, 2024

439patches29077

438patches29076

Fixed an issue loading to Databricks. To consume this fix:

  • For a version 4.39 AutoSync Snaplex or Groundplex, upgrade data pipelines with Databricks as a target to use the AutoSync 439patches29077 Snap Pack.

  • For a version 4.38 AutoSync Snaplex or Groundplex, upgrade data pipelines with Databricks as a target to use the AutoSync 438patches29076 Snap Pack.

Upgrading to this Release

Admin Manager

New Features

  • System Configuration now includes the Git Integration page for selecting a Git provider. This supports continuous integration and continuous development (CI/CD) across different environments. Learn more.

    image-20241101-072520.png
  • In the Security section, configure Enhanced Account Encryption (EAE) from the Account encryption page. With a subscription to EAE and use of a Groundplex, sensitive account data is encrypted with a public key in the SnapLogic control plane and decrypted with a private key on the Groundplex. Learn more.

    account-encryption.png

     

  • In the Application settings section, configure Asset catalog visibility. The Asset catalog is available to enterprise SnapLogic customers by request. Learn more.

AgentCreator

AgentCreator represents the next major revision of the GenAI App Builder. It offers advanced features for enterprises to create agents that have the ability to reason and use tools to perform tasks autonomously. The tool can be any SnapLogic connector, pipeline, or API. It also includes comprehensive observability and validation capabilities that allow enterprises to understand and evaluate agent actions.

New Features

Enhancements

Known Issues

Limitation

  • The OpenAI documentation states that cancelled is a supported value for the Filter field in OpenAI List Vector Store Files. However, using this value can result in a status code 400 error from the endpoint. To avoid the 400 status code, use only the following supported values: completed, in_progress, and failed.

API Management

Fixed Issues

  • On the APIM Metrics/API Dashboard, Alerts no longer display the error, unable to instantiate policy.

AutoSync

New Features

  • The new SQL Server Bulk Load endpoint provides better performance for data pipelines that synchronize large datasets. This endpoint requires:

    • The data pipeline must run on a Groundplex configured to use the Microsoft SQL Server Bulk Copy Utility, bcp. Before using this endpoint in an AutoSync data pipeline, you must install the utility on each Groundplex node.

    • The 438patches28952 AutoSync Snap Pack. Refer to AutoSync settings for information on selecting a Snap Pack for a data pipeline.

Fixed Issues

  • Fixed an issue loading to Databricks. To consume this fix:

    • For a version 4.39 AutoSync Snaplex or Groundplex, upgrade data pipelines with Databricks as a target to use the AutoSync 439patches29077 Snap Pack.

    • For a version 4.38 AutoSync Snaplex or Groundplex, upgrade data pipelines with Databricks as a target to use the AutoSync 438patches29076 Snap Pack.

Documentation Enhancements

New Videos

  • Learn how to create an APIM API version using assets in the Git Repository

  • Compare the interface for node metrics in Dashboard with the new one in Monitor: Dashboard to Monitor - Node Metrics

Monitor

Enhancements

  • The Integration catalog, available to enterprise SnapLogic customers by request, is now the Asset catalog. It includes the following enhancements:

    • With the new API described in the Platform section, you can add custom metadata columns to the table of tasks and add, update, or delete existing custom metadata values from a task.

    • With the new configuration described in the Admin Manager section, you can allow all users view-only access to the Asset catalog or restrict it to admins.

    • The new Pipelines tab displays standard pipeline metadata:

    • Task details are easier to read, with important information moved to the header and execution statistics split out into a new tab:

    • The Custom tab on the Task details pane includes the ability to edit custom metadata values:

Known Issues

  • The Asset catalog Edit metadata functionality in the Custom tab of task details has an issue. When editing, if you click Clear to remove an existing value and then click Cancel, the value is deleted instead of being retained as expected.
    Workaround: Re-enter the cleared value and click Save.

  • The average number of documents processed for pipelines that contain a BigQuery Execute Snap is being incorrectly reported as zero. This impacts both the Asset catalog and the Execution overview.

  • In the Asset catalog Pipelines tab, when you filter by the Path column, a search might not return all pipelines that match.

  • The Asset catalog Execution statistics tab for task and pipeline details currently displays no data.

Fixed Issues

  • The Node status and Node type filters are no longer missing from the System overview node view.

  • The documentation Graph descriptions page clarifies which graphs represent total values and which show the average percentage of the value.

Platform

New Features

Enhancements

Fixed Issues

  • The runtime pusher (the mechanism by which the pipelines are initialized before they process data) was improved for better node performance.

  • Pipelines were failing because the Snaplex JCC ran out of the connections, resulting in the error: Timeout waiting for connection from pool.

  • Pipeline executions were taking longer due to the slow preparation of the pipelines.

  • On the APIM Metrics/API Dashboard, alerts were displaying the error: unable to instantiate policy.

RFC Compliance Enforcement

Snaps

Breaking Changes

New Snaps/Accounts

  • Snowflake: Added the Snowflake - Snowpipe Streaming Snap that enables data insertion into Snowflake using the Snowpipe Streaming API. It allows for continuous data ingestion into Snowflake tables as soon as data becomes available.

Enhancements

  • API Suite:

    • Enhanced the GraphQL Client Snap with the Include extensions checkbox to support extensions that enable you to access query content and extension data in a structure similar to the GraphQL response. The non-standard fields in the GraphQL response are now ignored to avoid a possible error.

    • The GraphQL Client Snap now includes an improved error-handling mechanism that emulates how errors are reported in the raw GraphQL response. This enhancement provides additional context and pagination to the error output, which was previously limited to one page. Learn more.

  • Amazon S3:

    • Enhanced the S3 Download Snap with an Enable Staging checkbox that enables you to download an S3 object into a local temporary file. This enhancement addresses the cases where some downstream Snaps take longer to process large volumes of data, potentially causing a connection reset error and pipeline failure.

  • Flow:

    • Added the PipeLoop Snap that enables you to loop on pipeline executions based on a stop condition or iteration limit. The execution of this Snap enables the parent pipeline to control and manage iterative processes using the child pipeline for scenarios where repeated processing is needed until a specific stop condition is met.

  • Kafka:

    • Enhanced the Kafka MSK IAM Account to support Cross-Account IAM Access on Amazon MSK Cluster configured through the Transit Gateway and multi VPC Peering.

    • Enhanced the Kafka MSK IAM Account to support debugging for the IAM configuration. When you enable the IAM debugging (Global properties and Logging Level) in the Snaplex configuration, the account logs the IAM credential identity in the Snaplex log. Learn more.

  • MongoDB:

    • Enhanced the MongoDB Replica and Mongo ReplicaSet Dynamic Accounts to define read preference options when querying data. The default option is Primary, so you cannot allocate the read load to the secondary node. Note that the Secondary preferred mode is not supported for the MongoDB Execute Snap.

  • RabbitMQ:

    • Removed the Route connection errors checkbox from the RabbitMQ Producer and RabbitMQ Consumer Snaps. As a result, any existing RabbitMQ pipelines that are configured with Route connection errors checkbox enabled will stop functioning as before.

      Suggested workaround

      To run your pipelines successfully, you must manually Route the errors to the error view under the Views tab.

    • Added 5672, the default RabbitMQ server value, for the Port field in the RabbitMQ Account.

  • Tableau:

    • Enhanced the Tableau Account and Tableau REST Account with Personal Access Token (PAT) authentication for flexibility and security of Tableau integrations. This enhancement ensures compatibility with MFA (Multi-Factor Authentication) or SSO (Single Sign-on) environments. PAT authentication is the default authentication type for the Tableau REST Account.

Deprecated Snaps

The Netezza Snap Pack is deprecated.

SnapGPT

Enhancements

  • Improved the user experience in RAG menu options. When prompting SnapGPT to generate a pipeline, only one pipeline is generated, and related pipeline patterns are suggested.