In-Memory Lookup

In this article

Snap type:

Transform

Description:

This Snap joins left and right input data streams. It uses the right input data as an in-memory lookup table.

  • Expected upstream Snaps: Any Snap with a document output view, such as CSV Parser, JSON Parser, Mapper.
  • Expected downstream Snaps: Any Snap with a document input view, such as CSV Formatter, JSON Formatter, Mapper.
  • Expected input:   
    • Input data streams may be unsorted.
    • The right input document stream is loaded in memory as a lookup table while the left input document stream is not stored in the Snap.
    • The JOIN operation starts when the right input document stream ends.
    • All input document data should be of a flat map data type.
  • Expected output: Each left input document is joined with the right input data if a match is found. If not, the left input document is written to the output view without join. If the Single document output property is true, only one document is written to the output view for each left input document regardless of the number of matches found in the in-memory lookup table.
Prerequisites:

Enough free memory should be available to load all right input data to the in-memory lookup table.

Support and limitations:

Limited support in Ultra Tasks when the Single document output field is selected. Only one document is written to the output view for each input document, which is a prerequisite for Ultra Pipelines.

Account: 

Accounts are not used with this Snap.

Views:
InputThis Snap has exactly two document input views. Users may want to edit the right input view label in the 'Views' section of the Snap since the right input view label is used as a prefix during the JOIN operation if the same column name exists in the left input data.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view and produces zero or more documents in the view.

Settings

Label

Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Join paths

Required. Field names to use for left and right sides of join. Each row in the table defines a relationship between a left field and a right field. The Snap supports flat map data only for all input documents, and a structured JSON path like '$customer.address' is not supported.

Left path

Required. Field name in the left input data. It can be selected from the suggested list. This property does not support expressions.

Example: customer_id, $customer_id, _leftFieldName (for pipeline parameter with the expression button enabled)

Default value:  None, the expression toggle enabled

Right path

Required. Field name in the right input data. The Right path suggestion is not available yet, except pipeline parameters. This property does not support expressions.

Example: customer_id, $customer_id, _rightFieldName (for pipeline parameter with the expression button enabled)

Default value: None, the expression button enabled

Single document outputIf selected, only one document is always written to the output view for each input document. If more than one row in the lookup table matches with the left input data, the first one in the list is joined with the left input data. If there is no match, the left input data is written to the output view. Leave this property selected if the pipeline is executed in Ultra Tasks mode.
If unselected, each of the matching rows is joined with the left input data. Therefore, the number of output documents may be larger than the input document counts.
Minimum memory (MB)

If the available memory is less than this value when building the in-memory lookup table from the right input documents, the Snap stops to fetch the next right input document until more memory is available. This feature is disabled if this value is 0.

This Snap loads all right input documents into the in-memory lookup table before it starts to perform the JOIN operation. Therefore, if the input data in the right input view exceeds the available memory, it may cause an out-of-memory failure. This field helps reduce the possibility of out-of-memory failures.

Default value: 200 MB
Example: 500 MB

Out-of-memory timeout (minutes)

If the Snap pauses longer than this value while waiting for more memory to become available, it throws an exception to prevent the system from running out of memory.

Default value: 30 minutes
Example: 10 minutes

Snap Execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default Value: Execute only
Example: Validate & Execute

Examples


Views Page with Inputs Renamed

In-Memory Lookup overview image


Example Pipeline - Views Page with Inputs Renamed


Views page with inputs renamed:

Left input data:

 Right input data:

Output data with Single document output selected:

Output data with Single document output not selected:


 Click to view/expand
ReleaseSnap Pack VersionDateType Updates
August 2024438patches28073 Latest

Fixed an issue with the JSON Generator and XML Generator Snaps that caused unexpected output displaying '__at__' and '__h__' instead of '@' and '-' respectively because the Snap could not update them to their original values after the Velocity library upgrade.

August 2024438patches27959 Latest

Fixed an issue with the Sort where the Snap could not sort files larger than 52 MB. This fix applies to Join Snap also.

August 2024main27765 StableUpgraded the org.json.json library from v20090211 to v20240303, which is fully backward compatible.
May 2024437patches26643 Latest
  • Fixed an issue with the Sort Snap that displayed an error when estimating the size of the input document provided by the upstream S3 Browser Snap.
  • Fixed an issue with the Parquet Formatter Snap that was unable to route errors to the error view.
May 2024437patches26453 Latest
  • Added expression support to the Skip lines field in the CSV Parser Snap to enable passing pipeline parameters and upstream values. 

  • Fixed an issue with the XML Parser Snap that caused an error when using the Splitter option in the Snap settings. 

May 2024main26341 Stable
  • Added Parquet Parser and Parquet Formatter Snaps to the Transform Snap Pack:
    • Parquet Parser: Reads the binary Parquet data and writes document data to the output.
    • Parquet Formatter: Reads the document data and writes it to the output in binary Parquet format.
  • Enhanced the JSON Splitter Snap to capture metadata and lineage information from the input document.

February 2024436patches25564 Latest

Fixed an issue with the JSON Formatter Snap that generated incorrect schema.

February 2024436patches25292 Latest

Fixed an out-of-memory error issue with the Aggregate Snap. This Snap no longer performs the presort for the input documents.

If the input documents are unsorted and GROUP-BY fields are used, you must use the Sort Snap upstream of the Aggregate Snap to presort the input document stream and set the Sorted stream field Ascending or Descending to prevent the out-of-memory error. However, if the total size of input documents is expected to be relatively small compared to the available memory, then Sort Snap is not required upstream.

Learn more about presorting unsorted input documents to be processed by the Aggregate Snap.

February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023435patches24802 LatestFixed an issue with the Excel Parser Snap that caused a null pointer exception when the input data was an Excel file that did not contain a StylesTable.
November 2023435patches24481 Latest

Fixed an issue with the Aggregate Snap where the Snap was unable to produce the desired number of output documents when the input was unsorted and the GROUP-BY fields field set was used.

November 2023435patches24094 Latest

Fixed a deserialization issue for a unique function in the Aggregate Snap.

November 2023main23721 StableUpdated and certified against the current SnapLogic Platform release.
August 2023434patches23076 LatestFixed an issue with the Binary to Document Snap where an empty input document with Ignore Empty Stream selected caused the Snap to stop executing.
August 2023434patches23034 Latest
  • Fixed an issue with the Transform Snap Pack that caused an error when the input file was a binary JSON file that contained a string value of more than 20,000,000 characters.
  • Fixed a memory issue with the Aggregate Snap that occurred when using GROUP-BY fields.

August 2023434patches22705 Latest

Fixed an issue with the JSON Splitter Snap that caused the pipeline to terminate with excessive memory usage on the Snaplex node after the 4.33 GA upgrade. The Snap now consumes less memory.

August 2023main22460 StableUpdated and certified against the current SnapLogic Platform release.
May 2023433patches22431 Latest
  • Fixed an issue with the Excel Multi Sheet Formatter Snap that caused it to produce binary output data when there was no input document and Ignore empty stream was selected.
  • Introduced the following new Snaps:
    • GeoJSON Parser: Parses geospatial data from binary data input and outputs the contents as a GeoJSON document downstream.

    • WKT Parser: Parses geospatial data from binary data input and outputs the contents as a WKT (Well Known Text) document downstream.

May 2023433patches21779 Latest

The Decrypt Field and Encrypt Field Snaps now support CTR (Counter mode) for the AES (Advanced Encryption Standard) block cipher algorithm.

May 2023433patches21586 Latest

The Decrypt Field Snap now supports the decryption of various encrypted fields on providing a valid decryption key.

May 2023433patches21461 Latest

The following Transform Snaps include new fields to improve memory management: Aggregate, Group By Fields, Group By N, Join, Sort, Unique.

May 2023433patches21336 Latest

Fixed an issue with the AutoPrep Snap where dates could potentially be rendered in a currency format because currency format options were displayed for the DOB column.

May 2023433patches21196 Latest

Enhanced the In-Memory Lookup Snap with the following new fields to improve memory management and help reduce the possibility of out-of-memory failures:

  • Minimum memory (MB)

  • Out-of-memory timeout (minutes)

These new fields replace the Maximum memory % field.

May 2023main21015 StableUpgraded with the latest SnapLogic Platform release.
February 2023432patches20535 Latest

Fixed an issue with the Encrypt Field Snap, where the Snap failed to support an RSA public key to encrypt a message or field. Now the Snap supports the RSA public key to encrypt the message.

February 2023432patches20446 Latest

The Join Snap is enhanced with the following:

  • The Pipeline Execution Statistics of the Join Snap now has a status message that displays the parameters - Free disk space, Available memory, and Average document size.

  • The internal sort buffer size is reduced to a minimum of 10MB when the available memory in the node becomes lower than 500MB to avoid the out-of-memory crash.

  • The internal sort buffer size is restored to its original size when the available memory becomes larger than 2GB.

  • We have improved the readability of the error message for the out of disk space on node error. The updated error message now provides clearer information and guidance for users, as shown below:
    Reason: Insufficient free disk space available to stage sort data into temporary files.
    Resolution:  Increase the amount of free disk space and try again.

February 2023

432patches20250

 Latest
  • Fixed an issue with the JSON Splitter Snap that was causing errors when using multiple repeated dots in the JSON Path.
  • The Sort Snap includes the following improvements:

    • The Maximum memory % field is revised to Maximum memory.

    • The Maximum memory unit (new dropdown list) enables you to choose a unit, percentage (%), or MB for better memory control.

February 2023432patches20151 Stable/Latest

Fixed an issue that occurred with the JSON Splitter Snap when used in an Ultra pipeline. The request was acknowledged before it was processed by the downstream Snaps, which caused a 400 Bad Request response.

February 2023432patches20062 Stable/LatestFixed the behavior of the JSON Splitter Snap for some use cases where its behavior was not backward compatible with the 4.31 GA version. These cases involved certain uses of either the Include scalar parents feature or the Include Paths feature.
February 2023432patches19974 Stable/Latest

Fixed the "Json Splitter expects a list" error by restoring the JSON Splitter Snap's previous behavior of handling the case where the document element referenced by the JSON Path to Split field is an object instead of a list or array.

Review your pipelines where this error occurred to check your assumptions about the input to the JSON Splitter and whether the value referenced by the JSON Path to Split field will always be a list. If the input is provided by an XML-based or SOAP-based Snap like the Workday or NetSuite Snaps, a result set or child collection that’s an array when there's more than one result or child will be an object when there's only one result or child. In these cases, we recommend using a Mapper Snap and the sl.ensureArray() function to ensure that the value being split by the JSON Splitter is always an array (even for the single element cases).

February 2023432patches19918 Stable/Latest
  • Fixed an issue with the CSV Formatter Snap where the Unicode character delimiters using [0-9a-f] did not work.

  • Fixed an issue with the JSON Splitter Snap that was generating null values for empty input data.

February 2023main19844 StableUpgraded with the latest SnapLogic Platform release.
November 2022431patches19441 Stable

The Encrypt Field Snap supports decryption of encrypted output in Snowflake Snaps.

November 2022431patches19385 Latest

The Transform Join Snap now doesn’t fail with the Null Pointer Exception when you configure the Sorted streams field with Ascending.

November 2022431patches19359 LatestThe JSON Splitter Snap includes memory improvements and a new Exclude List from Output Documents checkbox. This checkbox enables you to prevent the list that is split from getting included in output documents, and this also improves memory usage.
November 2022main18944 Stable
  • The Excel Formatter and Excel Multi Sheet Formatter Snaps now include a Convert formula strings to formulas checkbox.
  • The Mapper Snap now has a Sorted checkbox in the Input Schema and Target Schema panels, which allows you to sort the input and target schemas. When unchecked, the Snap unsorts the input and the target schema.

October 2022430patches18800 LatestThe Sort and Join Snaps now have improved memory management, allowing used memory to be released when the Snap stops processing.
October 2022430patches18610 Latest

The CSV Formatter and CSV Parser Snaps now support shorter values of Unicode characters.

 
October 2022430patches18454 Latest
  • The AutoPrep feature now includes the following new transformation options that enable you to:
    • Change the field data type from Data type menu.
    • Format dates and date Strings.
    • Rename a field.
    • Mask sensitive data using an MD5, SHA-1, SHA-256, or SHA-512 algorithm.
  • The data in the Preview data pane format is easier to read and the buttons have been changed to improve usability.

The CSV Parser Snap now parses data with empty values in the columns when using a multi-character delimiter.

September 2022430patches18119 Latest

The Transcoder Snap used in a low-latency feed Ultra Pipeline now acknowledges the requests correctly.

September 2022430patches17802 Latest

The Avro Parser Snap now displays the decimal number correctly in the output view if the column’s logical type is defined as a decimal.

September 2022430patches17737 Stable/LatestAutoPrep now enables you to handle empty or null values.
September 2022430patches17643 LatestThe CSV Parser and CSV Formatter Snaps now support either \ or \\ for a single backslash delimiter which were failing earlier.
September 2022430patches17589 Latest

The CSV Formatter Snap does not hang when running in specific situations involving multibyte characters in a long field. If you notice the CSV Formatter Snap is hung, we recommend that you update to the 430patches17589 version and restart your Snaplex.

August 2022main17386 Stable
  • New Snap Application: The Auto Prep Snap provides a data preparation application where you can flatten structured data, include and exclude data fields, and change data types before forwarding the data for further processing.

  • The Hide whitespace option in the CSV Generator and JSON Generator Snaps allows you to hide the rendering of whitespace as symbols (dot or underscore) in the output that you may have in the CSV or JSON input documents.

  • The Render whitespace checkbox in the Mapper Snap enables or disables the rendering of whitespace in the input document. When a value in the Expression field has blank spaces (leading, trailing, or spaces in the middle of a string), the spaces are rendered as symbols (dot “.” or underscore “_”) in the output on selecting this checkbox.

  • The Excel Parser Snap includes the Custom Locale dropdown list that allows selecting a user-defined locale for formatting numbers..

  • The Selected fields in the Pivot Snap allow you to define fields to be unpivoted so that the remaining fields are automatically pivoted.

  • The XML Generator Snap includes examples on how to escape single (') and double quotes (“) when used with elements or attributes.

4.29 Patch429patches16990 Latest
  • Fixed an issue with the Aggregate Snap where the Snap failed to validate (after first successful validation) while using a field that may contain a date for MIN and MAX functions. The Snap now supports DATE-type fields.
  • Enhanced the Pivot Snap with the Treat selected fields as static checkbox that enables the Snap to treat the selected fields as static to preserve the structure of the selected fields while all other fields are pivoted.

4.29 Patch429patches16923 Latest
  • Enhanced the CSV Formatter and CSV Parser Snaps to support multiple characters or strings as delimiters.
  • Fixed an issue with the Join Snap where the Snap displayed an incorrect error if the Left path or Right path fields were expression-enabled or if you have specified properties other than the field name and irrespective of whether the Sorted streams field is Unsorted or not. For example, "$first + '2' . Now, the Snap runs properly if the Left path or Right path was expression enabled and you have specified properties other than the field name and the Sorted streams field is Sorted or Unsorted. The Snap now displays a proper error that is more informative, in case there is a problem while executing this Snap.

  • Fixed an issue with CSV Parser Snap where the Snap failed when more than six characters are used as delimiters. Now, the Snap executes properly when you use more than six characters as delimiters.

4.29 Patch429patches16521 Latest
  • Removed the default value for the Root element field in the XML Formatter Snap.
  • Fixed an issue with the Transcoder Snap where the Input character-set field was not displaying the suggestions properly.
4.29 Patch429patches16026 LatestEnhanced the Excel Parser Snap with the Custom Locale dropdown list that allows you to select a user-defined locale to format numbers as per the selected locale.
4.29main15993 Stable
  • Fixed an issue with CSV Formatter and CSV Generator Snaps that displayed an incorrect error when one or more header values in the CSV file were missing. This error is now routed to the error view.

  • Fixed an issue with the CSV Parser Snap, where the Snap stopped indefinitely while processing certain inputs.

  • Fixed an issue with the CSV Formatter Snap, where the escape character selected was not used.

  • Updated the CSV library used in the CSV Generator, CSV Parser, and CSV Formatter Snaps. This library update enforces values for each column in a parsed CSV header.

  • Enhanced the XML Formatter Snap with the Output Character Set field. You can select the character set you want for your binary output. For example, UTF-8 or ISO860.

4.28 patch428patches14370 Latest

Fixed an issue with the XML Generator Snap, where the Snap failed with an invalid UTF-8 sequence error when running on the Windows Snaplex.

4.28main14627 StableUpgraded with the latest SnapLogic Platform release.
4.27 Patch427patches12966 LatestEnhanced the Avro Formatter Snap to display meaningful error message while processing invalid and null values from the input.
4.27main12989Stable and Latest

Fixed an issue in the Group by Fields Snap that caused the Snap to abort with an error when executed with zero input documents. 

4.27main12833Stable
  • Added dynamic memory management to the Group By Fields Snap, which can activate with the following settings: 

    • Memory Sensitivity: The Snap's response to the memory changes. 

    • Min Part Size: The minimum part size that you want the Snap to split larger groups into multiple parts. 

  • Enhanced the JSON FormatterJSON Generator, and JSON Parser Snaps with the Support Type Extensions checkbox. On selecting this checkbox, the Snap supports SnapLogic-specific syntax for representing non-standard data types such as byte arrays and date objects in JSON.
  • Added the following checkboxes in the Excel Multi Sheet Formatter Snap:

    • Convert numeric string to number: Sets the behavior of the Snap when a value in the input document is a numeric string. This checkbox supports the conversion of numeric values to string values.

    • Translate date and time types: Sets the behavior of the Snap when a value in the input document is a date and datetime type. This checkbox supports translating date and time types to Excel date numeric type with date format style.

4.26 Patch426patches12086

LatestFixed an issue with the Join Snap, where it exhausted the memory while buffering millions of objects.
4.26 Patch426patches11780Latest

Fixed an issue in the XML Formatter Snap, where the Map input to first repeating element in XSD checkbox is selected, while no XSD is specified for mapping the input.

4.26 Patch426patches11725 LatestFixed an issue with the Join Snap where the upstream document flow of the right view is blocked by the left view, which hung the Join Snap.
4.26main11181
 
Stable
  • Enhanced the JSON Splitter Snap with a new field Show Null Values for Include Paths that enables the Snap to show key-value entries of the null values for the objects added to the Include Paths field in the output document.
  • Enhanced the Join Snap with a new field Available Memory Threshold (%) that enables the Snap to keep all the Right-input view documents with the same join-path values in memory until the join operation is done for the specific join-path values.
4.25 Patch425patches10663
 
Latest

Fixed an issue in the CSV Formatter Snap, where even if the Ignore empty stream checkbox is not enabled, the Snap did not produce an empty binary stream output in case there is no input document.

4.25 Patch425patches10152
 
Latest
  • Replaced the Strict XSD output field with Map input to first repeating element in XSD field in the XML Formatter Snap. If selected, the Snap ignores the root element from the XSD file.

  • Enhanced the CSV Parser Snap with a new checkbox Preserve Surrounding Spaces that enables you to preserve the surrounding spaces for the values that are non-quoted.
4.25 Patch 425patches9815
 
Latest

Fixed a ClassCastException error in the Avro Parser Snap and handling of the map, fixed, enum, and bytes data types in the Avro Formatter and Avro Parser Snaps.

4.25 Patch425patches9749
 
Latest

Enhanced XML Parser Snap to recognize input headers when defining inbound schema. 

4.25 Patch425patches9638-Latest

Reverts the Join Snap to the 4.24 release behavior. This is in response to an issue encountered in the Join Snap in the 4.25 release version (main9554), which can result in incorrect outputs from all Join Types. 425patches9638 is the default version for both stable and latest Transform Snap Pack versions for orgs that are on the 4.25 release version. No action is required by customers to receive this update and no impact is anticipated.

4.25main9554
 
Stable

Enhanced the Group By N Snap with the following settings: 

  • Memory Sensitivity: The Snap's response to the memory changes. 
  • Group Size: The maximum number of input documents to be grouped into a single output document.
  • Min Group Size: The minimum number of input documents to be grouped into a single output document.
4.24 Patch424patches8938
 
Latest
  • Fixed the timestamp issue in the JSON Formatter Snap that changed the time zone offset to include colon by default after upgrading to 4.24.

  • Fixed the null pointer exception at runtime in the Fixed Width Parser Snap by setting the Trim column data field to false for empty columns.
  • Enhances the Group By N Snap to process the records efficiently by adding the Flush Timeout field that enables the Snap to flush a partial group of records if the time specified in this field passes with no new input.
4.24main8556
 
StableUpgraded with the latest SnapLogic Platform release.
4.23 Patch423patches7958
  
Latest

Fixes an issue in the JSON Splitter Snap by logging an error when the matcher does not find a pattern.

4.23 Patch423patches7898
 
Latest
  • Fixes an issue in the In-Memory Lookup Snap to correctly handle the Join path in the format like $['join path'].

  • Fixes an issue in the XSLT Snap, wherein null binary header values are now converted to blank strings when injecting them as parameters in the stylesheet.
4.23 Patch423patches7792
 
Latest

Fixes an issue in the XML Formatter Snap when it fails to convert input JSON data, with JSON property having a special character as its prefix, to XML format by sorting the elements.

4.23 Patch423patches7753
Latest

Fixes an issue with the JSON Splitter Snap's behavior in Ultra Pipelines that prevents processed requests to be acknowledged and removed from the FeedMaster queue, resulting in retries of requests that are already processed successfully.

4.23main7430
 
Stable

Enhances the JSON Formatter Snap to render groups output from upstream (Group by) Snaps with one document per group and a new line per group element. You can now select the Format each document and JSON lines check boxes simultaneously.

4.22 Patch422patches6395
Latest

Fixes the JSON Splitter Snap data corruption issue by copying the data in JSON Splitter Snap before sending it to other downstream Snaps.

4.22 Patch422patches6505
Latest

Fixes the XML Generator Snap issue reflecting empty tags and extra space by removing the extra space in the XML output.

4.22main6403
 
LatestUpgraded with the latest SnapLogic Platform release.
4.21 Patch421patches5901
Latest

Enhances the JSON Generator Snap to include pass-through functionality where the Snap embeds the upstream input document under the original field of the output document along with other records.

4.21 Patch421patches5848
 
Latest
  • Fixes the Sort Snap that fails while performing sorting, displaying a NoClassDefFoundError.  
  • Fixes the Excel Formatter Snap that fails to create an empty worksheet when you do not select the Ignore Empty Stream checkbox.
4.21snapsmrc542-Stable

Adds support in the Mapper Snap to display schemas with complex nesting. For example, if Snaps downstream from the

4.20 Patchtransform8792-Latest

Resolves the NoClassDefFoundError in the Join Snap on Windows Snaplex instances.

4.20 Patchtransform8788-Latest

Resolves the NullPointerException in the Join Snap on Windows Snaplex instances.

4.20 Patchtransform8760-Latest
  • Fixes an issue with the CSV Parser Snap wherein the Snap fails when it is configured in the new Form UI if the Contains headers and Validate headers fields are not selected but empty rows exist in the Column names field.
  • Fixes the JSON Formatter Snap to filter the input data based on the value specified in the Content field in the Snap settings when the JSON lines field is also selected. Previously, the Snap was writing the entire input data to the output file.

The JSON Formatter Snap output now includes only those fields from the input file that are specified in the Content field under Settings. 

If your Pipelines use the JSON Formatter Snap with the JSON lines field selected, they may fail to execute correctly if the Content field mentions a specific object or field but the downstream Snap is expecting the entire data. Hence, for backward compatibility, you must review the entries in the Content field based on the desired output, as follows: 

  • Enter to include all the fields in the output.
    OR
  • Enter or select the specific fields to restrict the output to. 

The behavior of the Snap when the JSON lines field is not selected is correct and remains unchanged.  

The Example: Filtering Output Data Using the JSON Formatter Snap's Content Setting further illustrates the corrected vs the old behavior.

  • Fixes an issue in the XML Generator Snap due to which the custom XML data in the Edit XML field got ignored when inbound schema and root XML element were also provided, and instead, the output was generated from upstream data. For Pipelines with a standalone XML Generator Snap, a "Premature end of file" error was displayed. 

XML Generator Snap behavior might break existing Pipelines

The XML Generator Snap now gives precedence to any custom XML data that is provided over data coming from upstream Snaps, to generate the output. 

Existing Pipelines using the XML Generator Snap may fail in the following scenarios. Use the resolution provided to update the Snap settings based on the XML data you want to pass to downstream Snaps.

Breaking Scenario

Resolution

Downstream Snaps expect XML output from upstream Snaps, but custom XML data exists in the XML Generator Snap.

  • To use custom XML data, ensure that the Inbound schema and XML root element are specified and custom XML data is entered using the Edit XML option.
  • To use data from upstream Snaps, ensure that the Inbound schema and XML root element fields are blank and no custom XML data exists in the XML editor.


In the XML Generator Snap settings, the XML root element and Inbound schema are specified but no custom XML data is provided. This will generate a validation error. 
In the XML Generator Snap settings, custom XML data is provided and Validate XML checkbox is selected, but XML root element and Inbound schema are not specified. This will generate a validation error.
4.20 Patchtransform8738-Latest
  • Fixes the error that occurs while transforming an XML document into a string or while using the Excel Formatter Snap by retaining only one version of the Saxon library – net.sf.Saxon: Saxon-HE:9.6.0-10 across the Transform Snap Pack.
  • Fixes null handling in the Aggregate Snap whereby changing the order of entries in the Aggregate fields fieldset results in different outputs when the input documents contain one or more null values.
  • Fixes the issue with the Excel Parser Snap whereby the Snap does not parse the date and time correctly for custom date formats.
4.20snapsmrc535-StableUpgraded with the latest SnapLogic Platform release.
4.19 Patchtransform8280-Latest

Fixed an issue with the Excel Parser Snap wherein the Snap incorrectly outputs Unformatted General Number format. 

4.19snaprsmrc528-Stable

The output of the AVG function in the Aggregate Snap now rounds up all numeric values that have more than 16 digits.

4.18 Patch transform8199-Latest

Fixed an issue with the Excel Multi Sheet Formatter Snap wherein the Snap fails to create sheets in the expected order.

4.18 Patchtransform7994-Latest

Added a field, Round dates, to the Excel Parser Snap which enables you to round numeric excel data values to the closest second.

4.18 Patchtransform7780-Latest
  • Fixed an issue with the Excel Parser Snap by upgrading Apache POI to version 3.14, wherein the Snap is unable to parse an excel file with custom namespaces.
  • Fixed an issue with the Excel Parser Snap wherein the Snap converts real numbers to two decimal places when formatted for currency. A new property Cell formatting now supports unformatted outputs.
4.18 Patchtransform7741-Latest

Fixed an issue with the Sort and Join Snaps wherein the platform removes all temp files at the end of Pipeline execution.

4.18 Patchtransform7711-Latest

Fixed an issue with the XML Parser Snap wherein a class cast exception occurs when the Snap is configured with a Splitter and Namespace Context.

4.18snapsmrc523-Stable
  • Enhanced the sort feature of the Sort Snap to support specifying sort order at the field level. Added two new fields, Sort Path and Sort Order, to the Sort paths property; renamed the Sort order property to Sort order (Global).
  • Enhanced the Group Size property of the Group by N Snap to be expression enabled and removed the upper bound threshold (of 10000).
4.17 Patch Transform7431-Latest

Added a new field, Ignore empty stream, to the Avro Formatter Snap that writes an empty array in the output in case of no input documents.

4.17 PatchTransform7417-Latest

Added a new field, Format as canonical XML, to the XSLT Snap that enables canonical XML formatting.

4.17 PatchALL7402-Latest

Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17 snapsmrc515-Stable
  • Added the Derive schema from the sample size menu option to the JSON Formatter Snap, whereby you select the sampling size of the schema from the data source.
  • Added the capability to select either document (previously supported) or binary data (new) for your input and output views to the Mapper Snap.
  • Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during the preview check box.
4.16 Patchtransform7093-Latest

Fixed an issue with the XSLT Snap failure by enhancing the Saxon version.

4.16 Patchtransform6962-Latest
  • Added a new property, Escape special characters, to the XML Generator Snap that enables you to escape XML special characters in template variable values.
  • Added a new property, Header size error policy, to the CSV Formatter and CSV Parser Snaps. The property enables you to handle header size errors.
4.16 Patchtransform6869-Latest

Fixed an issue with the XML Parser Snap wherein XSD with annotations were incorrectly interpreted.

4.16snapsmrc508-StableUpgraded with the latest SnapLogic Platform release.
4.15 Patchtransform6736-Latest

Fixed an issue wherein the XML Generator Snap was unable to escape some of the special characters.

4.15 Patchtransform6680-Latest

Fixed an issue with the Excel Parser Snap wherein headers were not parsed properly and header columns were also maintained in an incorrect order.

4.15 Patchtransform6402-Latest

Fixed an issue with an object type in the Fixed Width Formatter Snap.

4.15 Patchtransform6386-Latest

Fixed an issue with the Fixed Width Parser Snap wherein Turkish characters caused incorrect parsing of data on Windows plex.

4.15 Patchtransform6321-Latest

Fixed an issue with the XML Parser Snap wherein the Snap failed to get data types from XSD.

4.15 Patchtransform6265-Latest

Fixed an issue with the XML Parser wherein XML validation was failing if the XSD contained xsd:include.

4.15snapsmrc500-Stable

Added a new property, Binary header properties, to the Document to binary Snap that enables you to specify the properties to add to the binary document's header.

4.14 Patchtransform6098-Latest

Fixed an issue wherein the XML Parser Snap was not maintaining the datatype mentioned in the XSD file.

4.14 Patchtransform6080-Latest

Fixed an issue with the Avro Formatter Snap wherein the Snap was failing for a complex JSON input.

4.14 PatchMULTIPLE5732-Latest

Fixed an issue with S3 file reads getting aborted intermittently because of incomplete consumption of input stream.

4.14 Patchtransform5684-Latest

Fixed the JSON Parser Snap that causes the File Reader Snap to fail to read S3 file intermittently with an AbortException error.

4.14 snapsmrc490-StableUpgraded with the latest SnapLogic Platform release.
4.13 Patchtransform5411-Latest

Fixed an issue in the CSV Formatter Snap where the output showed extra values that were not provided in the input.

4.13snapsmrc486-StableUpgraded with the latest SnapLogic Platform release.
4.12 Patch transform5005-Latest

Fixed an issue with the Excel Parser Snap that drops columns when the value is null at the end of the row.

4.12 Patchtransform4913-Latest

Fixed an issue in the Excel Formatter Snap, wherein opening an output stream prematurely causes the Box Write Snap to fail. Excel Formatter now awaits the first input document, before opening an output stream.

4.12 Patchtransform4747-Latest

Fixed an issue that caused the Join Snap to go out of memory.

4.12snapsmrc480-Stable
  • Improved performance for both Aggregate and Join Snaps.

  • Added Select All and Deselect All buttons to the Mapper table

4.11 Patchtransform4558-Latest

Enforced UTF-8 character encoding for the Fixed Width Formatter Snap.

4.11 Patchtransform4343-Latest

Enhanced enum labels on the Binary to Document and the Document to Binary Snaps for the Encode or Decode property with DOCUMENT and NONE options.

4.11 Patch transform4361-Latest

Fixed a sorting issue with the Join and Sort Snaps where the end of the file was not detected correctly.

4.11 Patchtransform4281-Latest

Added support on the Kryo serialization for UUID and other types to the Sort, Join and In-Memory Lookup Snaps.

4.11snapsmrc465--
  • Updated the JSON Formatter Snap with Binary Header and Content properties to allow the Snap to pass through binary header data.
  • Enhanced the In-memory Lookup Snap for Performance Optimization.
  • Enhanced the Sort Snap for Performance Optimization.
  • Updated the Excel Parser with will Insert null columns property to insert 'null' on missing columns at end.
4.10 Patchtransform4058--

Addressed an issue with the Excel Parser Snap that failed with out of memory when using large input Data (eg. 191 MB).

4.10 Patchtransform3956--

Conditional Snap: fixed an issue with the "Null-safe access" Snap Setting not being respected for return values.

4.10 snapsmrc414--
  • CSV Parser Snap updated with Ignore empty stream property to support passing the empty data.
  • XML Formatter Snap updated with Max schema levels property to support the outbound schema XSD containing import statements.
  • Addressed spelling errors in messages across the Snap Pack.
4.9.0 Patchtransform3343--

Join Snap - All input documents from all input views should be consumed before the end of Snap execution.

4.9.0 Patchtransform3281--

Made all four output views in Diff snap as mandatory.

4.9.0 Patchtransform3264--

Made all four output views in Diff snap as mandatory.

4.9.0 Patchtransform3220--

CSV Parser Snap - A new Snap property Ignore empty data with true default is added

4.9.0 Patchtransform3019--

Addressed an issue with the transform2989 build.

4.9.0 Patchtransform2989--

Addressed an issue with Excel Parser not displaying the most recent cached value for vlookups containing missing external references.

4.9.0
--

Introduced Encrypt Field and Decrypt Field Snaps.

4.8.0 Patch transform2956--

[CSVParser] Fixed an issue where an empty Quote Character config field was defaulting to the unicode quote character U+0000 (null). This caused issues if the input CSV had U+0000 characters in it.

4.8.0 Patchtransform2848--
  • Addressed an issue with Excel Multi sheet Formatter creating unreadable data on the output view when there is no input document.
  • Addressed an issue with CSV Formatter Snap failing on empty input data.
  • Addressed an issue with an upstream Script Snap throwing a NoClassDefFoundError in the Sort Snap.
4.8.0 Patchtransform2768--

Addressed an issue with CSV Parser causing a spike in CPU usage.

4.8.0 Patchtransform2736--

Addressed an issue with Excel Formatter dropping the first record when Ignore empty stream is selected.

4.8.0
--
  • Updated the CSV Formatter Snap with Newline property. This lets you select newline characters as a line break.
  • CSV Formatter Snap: Snap-aware error handling policy enabled for Spark mode. This ensures the error handling specified on the Snap is used.
  • Mapper: Snap-aware error handling policy enabled for Spark mode. This ensures the error handling specified on the Snap is used.
4.7.0 Patchtransform2549--

Addressed an issue with Excel Formatter altering decimal numbers to text.