November 2024 Release Notes
Highlights
The SnapLogic November 2024 Release includes the following highlights:
Configure Asset catalog (formerly Integration catalog) visibility
Manage Enhanced Account Encryption and Trusted environments for account migration
Configure the Git Integration
AgentCreator (formerly GenAI App Builder)
Provides the following capabilities to create agents:
Advanced response configuration: Enables consistent output for all LLM providers
Stop sequence support: Allows reduced token usage on output when the expected sequences hit
Tool calling: Included in the Flow Snap Pack with the PipeLoop Snap and in the LLM Utilities Snap Pack with the Message Appender Snap
Multimodal input support available in all LLM provider Snap Packs
New SQL Server Bulk Load endpoint
Integration catalog renamed to Asset catalog
Ability to edit custom Asset catalog metadata from Task details
New API for managing Asset catalog custom metadata
Added the Snowflake - Snowpipe Streaming Snap
The Kafka MSK IAM Account now supports Cross-Account IAM Access via Transit Gateway and multi-VPC Peering
Added Personal Access Token (PAT) authentication for the Tableau Account and Tableau REST Account to improve flexibility and security of Tableau integrations.
Dot Releases: Snaps
Snap Pack | Date | Snap Pack Version | Updates |
---|---|---|---|
Nov 28, 2024 | 439patches29008 |
Breaking change: With the Behavior change: Earlier, the ROWID columns were displayed in binary (Base64 encoded data) form in the output. With the | |
Nov 22, 2024 | 439patches29124 |
| |
Nov 18, 2024 | 439patches29078 | Fixed an issue with the CSV Parser Snap that introduced unexpected characters into the records and output data because of incorrect handling of the delimiter. | |
Nov 15, 2024 | 439patches29014 | Fixed an issue with the Salesforce Lookup, SOQL, and SOSL Snaps, where errors were not routed to the error pipeline when Route Error Data to Error View was enabled, and a referenced expression variable was missing in the upstream or pipeline parameters. | |
Nov 14, 2024 | 439patches29008 | Fixed an issue with Oracle Execute, Select, and Stored Procedure Snaps where the data was displayed in an unexpected structure and without the actual row address value when querying table columns of Breaking change: With the | |
AutoSync | Nov 19, 2024 | 439patches29077 438patches29076 | Fixed an issue loading to Databricks. To consume this fix:
|
Upgrading to this Release
Admin Manager
New Features
System Configuration now includes the Git Integration page for selecting a Git provider. This supports continuous integration and continuous development (CI/CD) across different environments. Learn more.
In the Security section, configure Enhanced Account Encryption (EAE) from the Account encryption page. With a subscription to EAE and use of a Groundplex, sensitive account data is encrypted with a public key in the SnapLogic control plane and decrypted with a private key on the Groundplex. Learn more.
In the Application settings section, configure Asset catalog visibility. The Asset catalog is available to enterprise SnapLogic customers by request. Learn more.
AgentCreator
AgentCreator represents the next major revision of the GenAI App Builder. It offers advanced features for enterprises to create agents that have the ability to reason and use tools to perform tasks autonomously. The tool can be any SnapLogic connector, pipeline, or API. It also includes comprehensive observability and validation capabilities that allow enterprises to understand and evaluate agent actions.
New Features
Added the following Snaps to the Azure OpenAI Snap Pack :
Add Message: Adds messages to an existing thread.
Create Thread: Creates a thread by making an HTTP request to the Assistant API.
Create and Run Thread: Creates a thread and retrieves the result on a specified assistant.
Run Thread: Executes the specified thread and retrieves the result on a specified assistant.
Azure OpenAI Tool Calling: Provides external tools for the model to call, supplying internal data and information for the model's responses.
Azure OpenAI Function Result Generator: Formats the results generated by user-run functions into a unified structure.
Azure OpenAI Function Generator: Generates a tool definition that can be used for tool calling in the chat completions endpoint.
Azure OpenAI Multimodal Content Generator: Generates multimodal content for the user prompt with Azure OpenAI.
Flow:
Added the PipeLoop Snap that enables you to loop on pipeline executions based on a stop condition or iteration limit. The execution of this Snap enables the parent pipeline to control and manage iterative processes using the child pipeline for scenarios where repeated processing is required until a specific stop condition is met.
Added the following Snaps to the Amazon Bedrock LLM Snap Pack:
Amazon Bedrock Converse API Tool Calling: Provides external tools for the model to call, supplying internal data and information for the model's responses.
Amazon Bedrock Function Result Generator: Formats the results generated by user-run functions into a unified structure.
Amazon Bedrock Function Generator: Generates a tool definition in the output to inform the model about the tools that you want to call.
Amazon Bedrock Multimodal Content Generator Snap: Generates multimodal content for the user prompt with Amazon Bedrock.
Added the Google GenAI Multimodal Content Generator Snap that generates multimodal content for the user prompt with Google GenAI.
Added the Message Appender Snap to the LLM Utilities Snap Pack that enables you to collect all the results from the Result Generator Snaps and append them to the
messages
orcontents
list in the output of the Tool Calling Snap.
Added the following Snaps to the OpenAI LLM Snap Pack:
OpenAI Add Vector Store File: Adds an existing file from OpenAI storage into the specified vector store using the vector store and file IDs.
OpenAI List Files: Retrieves and lists all files associated with the OpenAI account.
OpenAI List Vector Store Files: Retrieves and lists all vector store files based on the specified vector store ID.
OpenAI Remove Vector Store File: Removes a file from the specified vector store using the file ID.
OpenAI Upload File: Uploads a file to the specified vector store using the vector store and file IDs.
OpenAI Tool Calling: Provides external tools for the model to call, supplying internal data and information for the model's responses.
OpenAI Function Result Generator: Formats the results generated by user-run functions into a unified structure.
OpenAI Function Generator: Generates tool definitions and informs the model about the tools that you can call.
OpenAI Multimodal Content Generator: Generates multimodal content for the user prompt with OpenAI.
Enhancements
Enhanced the following Snaps with Advanced response configurations. These configurations generate precise customization of response behavior, enabling finely tuned outputs that align with specific content requirements and user intent:
Enhanced the Amazon Bedrock Account and Unstructured API Account with the Read timeout (seconds) field that enables the Snap to wait for a response from Amazon Bedrock and the Unstructured API.
Enhanced the OpenAI API Key Account with the Project ID field that enables you to connect to the OpenAI APIs, ensuring that the correct organization and project are being accessed for API usage.
Enhanced the following Snaps with the Stop sequences field to support stop sequences as a model parameter. You can specify the sequence of tokens or text to indicate to the model when to stop generating text:
Enhanced the following Snaps with the support of multimodal inputs:
Enhanced the following Snaps to support incoming content from their respective Multimodal Content Generator Snaps:
Known Issues
OpenAI Remove Vector Store File Snap can indicate a successful deletion in the output; however, the file might still appear in the output of the OpenAI List Vector Store Files Snap. This is a known bug from OpenAI. Learn more.
Limitation
The OpenAI documentation states that cancelled is a supported value for the Filter field in OpenAI List Vector Store Files. However, using this value can result in a
status code 400
error from the endpoint. To avoid the 400 status code, use only the following supported values: completed, in_progress, and failed.
API Management
Fixed Issues
On the APIM Metrics/API Dashboard, Alerts no longer display the error,
unable to instantiate policy
.
AutoSync
New Features
The new SQL Server Bulk Load endpoint provides better performance for data pipelines that synchronize large datasets. This endpoint requires:
The data pipeline must run on a Groundplex configured to use the Microsoft SQL Server Bulk Copy Utility, bcp. Before using this endpoint in an AutoSync data pipeline, you must install the utility on each Groundplex node.
The
438patches28952
AutoSync Snap Pack. Refer to AutoSync settings for information on selecting a Snap Pack for a data pipeline.
Fixed Issues
Fixed an issue loading to Databricks. To consume this fix:
For a version 4.39 AutoSync Snaplex or Groundplex, upgrade data pipelines with Databricks as a target to use the AutoSync 439patches29077 Snap Pack.
For a version 4.38 AutoSync Snaplex or Groundplex, upgrade data pipelines with Databricks as a target to use the AutoSync 438patches29076 Snap Pack.
Documentation Enhancements
New Videos
Learn how to create an APIM API version using assets in the Git Repository
Compare the interface for node metrics in Dashboard with the new one in Monitor: Dashboard to Monitor - Node Metrics
Monitor
Enhancements
The Integration catalog, available to enterprise SnapLogic customers by request, is now the Asset catalog. It includes the following enhancements:
With the new API described in the Platform section, you can add custom metadata columns to the table of tasks and add, update, or delete existing custom metadata values from a task.
With the new configuration described in the Admin Manager section, you can allow all users view-only access to the Asset catalog or restrict it to admins.
The new Pipelines tab displays standard pipeline metadata:
Task details are easier to read, with important information moved to the header and execution statistics split out into a new tab:
The Custom tab on the Task details pane includes the ability to edit custom metadata values:
Known Issues
The Asset catalog Edit metadata functionality in the Custom tab of task details has an issue. When editing, if you click Clear to remove an existing value and then click Cancel, the value is deleted instead of being retained as expected.
Workaround: Re-enter the cleared value and click Save.The average number of documents processed for pipelines that contain a BigQuery Execute Snap is being incorrectly reported as zero. This impacts both the Asset catalog and the Execution overview.
In the Asset catalog Pipelines tab, when you filter by the Path column, a search might not return all pipelines that match.
The Asset catalog Execution statistics tab for task and pipeline details currently displays no data.
Fixed Issues
The Node status and Node type filters are no longer missing from the System overview node view.
The documentation Graph descriptions page clarifies which graphs represent total values and which show the average percentage of the value.
Platform
New Features
Public API for managing Asset catalog custom metadata for tasks. The
catalog/custom-metadata
endpoint adds, updates, or deletes custom metadata for a task. Thecatalog/custom-metadata/columns
endpoint creates or deletes custom columns.
Enhancements
The Third-party Observability documentation now provides detailed instructions for configuring OpenTelemetry and a Groundplex deployed on Docker to provide logging information to New Relic.
Fixed Issues
The runtime pusher (the mechanism by which the pipelines are initialized before they process data) was improved for better node performance.
Pipelines were failing because the Snaplex JCC ran out of the connections, resulting in the error:
Timeout waiting for connection from pool
.Pipeline executions were taking longer due to the slow preparation of the pipelines.
On the APIM Metrics/API Dashboard, alerts were displaying the error:
unable to instantiate policy
.
RFC Compliance Enforcement
Snaps
Breaking Changes
New Snaps/Accounts
Snowflake: Added the Snowflake - Snowpipe Streaming Snap that enables data insertion into Snowflake using the Snowpipe Streaming API. It allows for continuous data ingestion into Snowflake tables as soon as data becomes available.
Enhancements
Enhanced the GraphQL Client Snap with the Include extensions checkbox to support extensions that enable you to access query content and extension data in a structure similar to the GraphQL response. The non-standard fields in the GraphQL response are now ignored to avoid a possible error.
The GraphQL Client Snap now includes an improved error-handling mechanism that emulates how errors are reported in the raw GraphQL response. This enhancement provides additional context and pagination to the error output, which was previously limited to one page. Learn more.
Enhanced the S3 Download Snap with an Enable Staging checkbox that enables you to download an S3 object into a local temporary file. This enhancement addresses the cases where some downstream Snaps take longer to process large volumes of data, potentially causing a connection reset error and pipeline failure.
Flow:
Added the PipeLoop Snap that enables you to loop on pipeline executions based on a stop condition or iteration limit. The execution of this Snap enables the parent pipeline to control and manage iterative processes using the child pipeline for scenarios where repeated processing is needed until a specific stop condition is met.
Enhanced the Kafka MSK IAM Account to support Cross-Account IAM Access on Amazon MSK Cluster configured through the Transit Gateway and multi VPC Peering.
Enhanced the Kafka MSK IAM Account to support debugging for the IAM configuration. When you enable the IAM debugging (Global properties and Logging Level) in the Snaplex configuration, the account logs the IAM credential identity in the Snaplex log. Learn more.
Enhanced the MongoDB Replica and Mongo ReplicaSet Dynamic Accounts to define read preference options when querying data. The default option is Primary, so you cannot allocate the read load to the secondary node. Note that the Secondary preferred mode is not supported for the MongoDB Execute Snap.
Removed the Route connection errors checkbox from the RabbitMQ Producer and RabbitMQ Consumer Snaps. As a result, any existing RabbitMQ pipelines that are configured with Route connection errors checkbox enabled will stop functioning as before.
Suggested workaround
To run your pipelines successfully, you must manually Route the errors to the error view under the Views tab.
Added 5672, the default RabbitMQ server value, for the Port field in the RabbitMQ Account.
Enhanced the Tableau Account and Tableau REST Account with Personal Access Token (PAT) authentication for flexibility and security of Tableau integrations. This enhancement ensures compatibility with MFA (Multi-Factor Authentication) or SSO (Single Sign-on) environments. PAT authentication is the default authentication type for the Tableau REST Account.
Deprecated Snaps
The Netezza Snap Pack is deprecated.
SnapGPT
Enhancements
Improved the user experience in RAG menu options. When prompting SnapGPT to generate a pipeline, only one pipeline is generated, and related pipeline patterns are suggested.
Have feedback? Email documentation@snaplogic.com | Ask a question in the SnapLogic Community
© 2017-2024 SnapLogic, Inc.