Highlights

The SnapLogic November 2024 Release includes the following highlights:

Admin Manager

• Configure Asset catalog (formerly Integration catalog) visibility

• Manage Enhanced Account Encryption and Trusted environments for account migration

• Configure the Git Integration

AgentCreator (formerly GenAI App Builder)

Provides the following capabilities to create agents:

• Advanced response configuration: Enables consistent output for all LLM providers

• Stop sequence support: Allows reduced token usage on output when the expected sequences hit

• Tool calling: Included in the Flow Snap Pack with the PipeLoop Snap and in the LLM Utilities Snap Pack with the Message Appender Snap

• Multimodal input support available in all LLM provider Snap Packs

AutoSync

• New SQL Server Bulk Load endpoint

Monitor

• Integration catalog renamed to Asset catalog

• Ability to edit custom Asset catalog metadata from Task details

Platform

• New API for managing Asset catalog custom metadata

Snaps

• Added the Snowflake - Snowpipe Streaming Snap

• The Kafka MSK IAM Account now supports Cross-Account IAM Access via Transit Gateway and multi-VPC Peering

• Added Personal Access Token (PAT) authentication for the Tableau Account and Tableau REST Account to improve flexibility and security of Tableau integrations.

Dot Releases: Snaps

Upgrading to this Release

Admin Manager

New Features

• System Configuration now includes the Git Integration page for selecting a Git provider. This supports continuous integration and continuous development (CI/CD) across different environments. Learn more.

  Open

  

• In the Security section, configure Enhanced Account Encryption (EAE) from the Account encryption page. With a subscription to EAE and use of a Groundplex, sensitive account data is encrypted with a public key in the SnapLogic control plane and decrypted with a private key on the Groundplex. Learn more.

  Open

  

   

• In the Application settings section, configure Asset catalog visibility. The Asset catalog is available to enterprise SnapLogic customers by request. Learn more.

AgentCreator

AgentCreator represents the next major revision of the GenAI App Builder. It offers advanced features for enterprises to create agents that have the ability to reason and use tools to perform tasks autonomously. The tool can be any SnapLogic connector, pipeline, or API. It also includes comprehensive observability and validation capabilities that allow enterprises to understand and evaluate agent actions.

New Features

• Added the following Snaps to the Azure OpenAI Snap Pack :

  • Add Message: Adds messages to an existing thread.

  • Create Thread: Creates a thread by making an HTTP request to the Assistant API.

  • Create and Run Thread: Creates a thread and retrieves the result on a specified assistant.

  • Run Thread: Executes the specified thread and retrieves the result on a specified assistant.

  • Azure OpenAI Tool Calling: Provides external tools for the model to call, supplying internal data and information for the model's responses.

  • Azure OpenAI Function Result Generator: Formats the results generated by user-run functions into a unified structure.

  • Azure OpenAI Function Generator: Generates a tool definition that can be used for tool calling in the chat completions endpoint.

  • Azure OpenAI Multimodal Content Generator: Generates multimodal content for the user prompt with Azure OpenAI.

• Flow:

  • Added the PipeLoop Snap that enables you to loop on pipeline executions based on a stop condition or iteration limit. The execution of this Snap enables the parent pipeline to control and manage iterative processes using the child pipeline for scenarios where repeated processing is required until a specific stop condition is met.

• Added the following Snaps to the Amazon Bedrock LLM Snap Pack:

  • Amazon Bedrock Converse API Tool Calling: Provides external tools for the model to call, supplying internal data and information for the model's responses.

  • Amazon Bedrock Function Result Generator: Formats the results generated by user-run functions into a unified structure.

  • Amazon Bedrock Function Generator: Generates a tool definition in the output to inform the model about the tools that you want to call.

  • Amazon Bedrock Multimodal Content Generator Snap: Generates multimodal content for the user prompt with Amazon Bedrock.

• Google GenAI LLM:

  • Added the Google GenAI Multimodal Content Generator Snap that generates multimodal content for the user prompt with Google GenAI.

• LLM Utilities:

  • Added the Message Appender Snap to the LLM Utilities Snap Pack that enables you to collect all the results from the Result Generator Snaps and append them to the messages or contents list in the output of the Tool Calling Snap.

• Added the following Snaps to the OpenAI LLM Snap Pack:

  • OpenAI Add Vector Store File: Adds an existing file from OpenAI storage into the specified vector store using the vector store and file IDs.

  • OpenAI List Files: Retrieves and lists all files associated with the OpenAI account.

  • OpenAI List Vector Store Files: Retrieves and lists all vector store files based on the specified vector store ID.

  • OpenAI Remove Vector Store File: Removes a file from the specified vector store using the file ID.

  • OpenAI Upload File: Uploads a file to the specified vector store using the vector store and file IDs.

  • OpenAI Tool Calling: Provides external tools for the model to call, supplying internal data and information for the model's responses.

  • OpenAI Function Result Generator: Formats the results generated by user-run functions into a unified structure.

  • OpenAI Function Generator: Generates tool definitions and informs the model about the tools that you can call.

  • OpenAI Multimodal Content Generator: Generates multimodal content for the user prompt with OpenAI.

Enhancements

• Enhanced the following Snaps with Advanced response configurations. These configurations generate precise customization of response behavior, enabling finely tuned outputs that align with specific content requirements and user intent:

  • Amazon Bedrock Converse API

  • Anthropic Claude on AWS Chat Completions

  • Anthropic Claude on AWS Messages

  • Create and Run Thread

  • Run Thread

  • Azure OpenAI Chat Completions

  • Google Gemini Generate

  • Create and Run Thread

  • Run Thread

  • OpenAI Chat Completions

• Enhanced the Amazon Bedrock Account and Unstructured API Account with the Read timeout (seconds) field that enables the Snap to wait for a response from Amazon Bedrock and the Unstructured API.

• Enhanced the OpenAI API Key Account with the Project ID field that enables you to connect to the OpenAI APIs, ensuring that the correct organization and project are being accessed for API usage.

• Enhanced the following Snaps with the Stop sequences field to support stop sequences as a model parameter. You can specify the sequence of tokens or text to indicate to the model when to stop generating text:

  • Anthropic Claude on AWS Chat Completions

  • Anthropic Claude on AWS Messages

  • Azure OpenAI Chat Completions

  • Google Gemini Generate

  • OpenAI Chat Completions

• Enhanced the following Snaps with the support of multimodal inputs:

  • Amazon Bedrock Converse API

  • Anthropic Claude on AWS Messages

  • Azure OpenAI Chat Completions

  • Add Message

  • Create and Run Thread

  • Run Thread

  • OpenAI Chat Completions

  • Google Gemini Generate

• Enhanced the following Snaps to support incoming content from their respective Multimodal Content Generator Snaps:

  • Amazon Bedrock Prompt Generator

  • Azure OpenAI Prompt Generator

  • Google GenAI Prompt Generator

  • OpenAI Prompt Generator

Known Issues

• OpenAI Remove Vector Store File Snap can indicate a successful deletion in the output; however, the file might still appear in the output of the OpenAI List Vector Store Files Snap. This is a known bug from OpenAI. Learn more.

Limitation

• The OpenAI documentation states that cancelled is a supported value for the Filter field in OpenAI List Vector Store Files. However, using this value can result in a status code 400 error from the endpoint. To avoid the 400 status code, use only the following supported values: completed, in_progress, and failed.

API Management

Fixed Issues

• On the APIM Metrics/API Dashboard, Alerts no longer display the error, unable to instantiate policy.

AutoSync

New Features

• The new SQL Server Bulk Load endpoint provides better performance for data pipelines that synchronize large datasets. This endpoint requires:

  • The data pipeline must run on a Groundplex configured to use the Microsoft SQL Server Bulk Copy Utility, bcp. Before using this endpoint in an AutoSync data pipeline, you must install the utility on each Groundplex node.

  • The 438patches28952 AutoSync Snap Pack. Refer to AutoSync settings for information on selecting a Snap Pack for a data pipeline.

Fixed Issues

• Fixed an issue loading to Databricks. To consume this fix:

  • For a version 4.39 AutoSync Snaplex or Groundplex, upgrade data pipelines with Databricks as a target to use the AutoSync 439patches29077 Snap Pack.

  • For a version 4.38 AutoSync Snaplex or Groundplex, upgrade data pipelines with Databricks as a target to use the AutoSync 438patches29076 Snap Pack.

Documentation Enhancements

New Videos

• Learn how to create an APIM API version using assets in the Git Repository

• Compare the interface for node metrics in Dashboard with the new one in Monitor: Dashboard to Monitor - Node Metrics

Monitor

Enhancements

• The Integration catalog, available to enterprise SnapLogic customers by request, is now the Asset catalog. It includes the following enhancements:

  • With the new API described in the Platform section, you can add custom metadata columns to the table of tasks and add, update, or delete existing custom metadata values from a task.

  • With the new configuration described in the Admin Manager section, you can allow all users view-only access to the Asset catalog or restrict it to admins.

  • The new Pipelines tab displays standard pipeline metadata:

  • Task details are easier to read, with important information moved to the header and execution statistics split out into a new tab:

  • The Custom tab on the Task details pane includes the ability to edit custom metadata values:

Known Issues

• The Asset catalog Edit metadata functionality in the Custom tab of task details has an issue. When editing, if you click Clear to remove an existing value and then click Cancel, the value is deleted instead of being retained as expected.
  Workaround: Re-enter the cleared value and click Save.

• The average number of documents processed for pipelines that contain a BigQuery Execute Snap is being incorrectly reported as zero. This impacts both the Asset catalog and the Execution overview.

• In the Asset catalog Pipelines tab, when you filter by the Path column, a search might not return all pipelines that match.

• The Asset catalog Execution statistics tab for task and pipeline details currently displays no data.

Fixed Issues

• The Node status and Node type filters are no longer missing from the System overview node view.

• The documentation Graph descriptions page clarifies which graphs represent total values and which show the average percentage of the value.

Platform

New Features

• Public API for managing Asset catalog custom metadata for tasks. The catalog/custom-metadata endpoint adds, updates, or deletes custom metadata for a task. The catalog/custom-metadata/columns endpoint creates or deletes custom columns.

Enhancements

• The Third-party Observability documentation now provides detailed instructions for configuring OpenTelemetry and a Groundplex deployed on Docker to provide logging information to New Relic.

Fixed Issues

• The runtime pusher (the mechanism by which the pipelines are initialized before they process data) was improved for better node performance.

• Pipelines were failing because the Snaplex JCC ran out of the connections, resulting in the error: Timeout waiting for connection from pool.

• Pipeline executions were taking longer due to the slow preparation of the pipelines.

• On the APIM Metrics/API Dashboard, alerts were displaying the error: unable to instantiate policy.

RFC Compliance Enforcement

Snaps

Breaking Changes

New Snaps/Accounts

• Snowflake: Added the Snowflake - Snowpipe Streaming Snap that enables data insertion into Snowflake using the Snowpipe Streaming API. It allows for continuous data ingestion into Snowflake tables as soon as data becomes available.

Enhancements

• API Suite:

  • Enhanced the GraphQL Client Snap with the Include extensions checkbox to support extensions that enable you to access query content and extension data in a structure similar to the GraphQL response. The non-standard fields in the GraphQL response are now ignored to avoid a possible error.

  • The GraphQL Client Snap now includes an improved error-handling mechanism that emulates how errors are reported in the raw GraphQL response. This enhancement provides additional context and pagination to the error output, which was previously limited to one page. Learn more.

• Amazon S3:

  • Enhanced the S3 Download Snap with an Enable Staging checkbox that enables you to download an S3 object into a local temporary file. This enhancement addresses the cases where some downstream Snaps take longer to process large volumes of data, potentially causing a connection reset error and pipeline failure.

• Flow:

  • Added the PipeLoop Snap that enables you to loop on pipeline executions based on a stop condition or iteration limit. The execution of this Snap enables the parent pipeline to control and manage iterative processes using the child pipeline for scenarios where repeated processing is needed until a specific stop condition is met.

• Kafka:

  • Enhanced the Kafka MSK IAM Account to support Cross-Account IAM Access on Amazon MSK Cluster configured through the Transit Gateway and multi VPC Peering.

  • Enhanced the Kafka MSK IAM Account to support debugging for the IAM configuration. When you enable the IAM debugging (Global properties and Logging Level) in the Snaplex configuration, the account logs the IAM credential identity in the Snaplex log. Learn more.

• MongoDB:

  • Enhanced the MongoDB Replica and Mongo ReplicaSet Dynamic Accounts to define read preference options when querying data. The default option is Primary, so you cannot allocate the read load to the secondary node. Note that the Secondary preferred mode is not supported for the MongoDB Execute Snap.

• RabbitMQ:

  • Removed the Route connection errors checkbox from the RabbitMQ Producer and RabbitMQ Consumer Snaps. This change should not impact your existing pipelines because the Route Error Data to Error View is now enabled by default.

  • Added 5672, the default RabbitMQ server value, for the Port field in the RabbitMQ Account.

• Tableau:

  • Enhanced the Tableau Account and Tableau REST Account with Personal Access Token (PAT) authentication for flexibility and security of Tableau integrations. This enhancement ensures compatibility with MFA (Multi-Factor Authentication) or SSO (Single Sign-on) environments. PAT authentication is the default authentication type for the Tableau REST Account.

Deprecated Snaps

The Netezza Snap Pack is deprecated.

SnapGPT

Enhancements

• Improved the user experience in RAG menu options. When prompting SnapGPT to generate a pipeline, only one pipeline is generated, and related pipeline patterns are suggested.