Structure

On this Page

Overview


This Snap modifies the structure of an incoming document. It can reuse the source document's data or create new data structures (list/map). When the Snap reuses source data, all untouched values will be preserved after all the operations are executed; so a source path can be moved multiple times. However, a target path cannot be the target of a move, otherwise the pipeline will fail. In addition, if a source path is deleted, then the path will not exist after it is executed. Paths can also be updated and the original value will not be deleted.

If the source and target structures are the same, you do not need to use this Snap.

There are five cases for update: 

Update value - the value is replaced
        Example:


 Source Data Pass Through Source Path Operation Target Path Target Data
{
    first_name: "Joe",
    last_name: "Smith"
}
 Yes $first_name update $last_name
{
    first_name: "Joe",
    last_name: "Joe"
}

Update map - key in map is updated

    Example:

 Source Data Pass Through Source Path Operation Target Path Target Data
{
    first_name: "Joe",
    address: {
        street: "12 2nd Ave."
    }
}
 Yes

 $first_name

 update

 $address

{
    first_name: "Joe",
    address: {
        first_name: "Joe",
        street: "12 2nd Ave."
    }
}

Update list - value is appended to list

    Example:

 Source Data Pass Through Source Path Operation Target Path Target Data
{
    first_name: "Joe",
    names: [
        "John", "Sally"
    ]
}
 Yes

 $first_name

 update

 $names

{
    first_name: "Joe",
    names: [
        "John", "Sally", "Joe"
    ]
}

Update list of maps - value is added to each map of the list
Note: If a list does not have all the same type scalar, map, or list, then the pipeline will fail.
    Example:

 Source Data Pass Through Source Path Operation Target Path Target Data
{
    first_name: "Joe",
    customers: [
        {
            last_name: "Smith",
            phone_num: "111-222-3333"
        },
        {
            last_name: "Smith",
            phone_num: "123-456-7890"
        }   
    ]
}
 Yes

 $first_name

 update

 $customers

{
    first_name: "Joe",
    customers: [
        {
            first_name: "Joe",
            last_name: "Smith",
            phone_num: "111-222-3333"
        },
        {
            first_name: "Joe",
            last_name: "Smith",
            phone_num: "123-456-7890"
        }   
    ]
}

Update list of lists - value is appended to the end of each list
Note: If a list does not have all the same type scalar, map, or list, then the pipeline will fail.

    Example:

 Source Data Pass Through Source Path Operation Target Path Target Data
{
    first_name: "Joe",
    lists_of_lists: [
        [ "Sue", "Sally"],
        [ "John", "Sam"]
    ]
}
 Yes

 $first_name

 update

 $list_of_lists

{
    first_name: "Joe",
    lists_of_lists: [
        [ "Sue", "Sally", "Joe"],
        [ "John", "Sam", "Joe"]
    ]
}


Snap Views

InputThis Snap has exactly one document input view.
OutputThis Snap has exactly one document output view.
Error

This Snap has at most one document error view and produces zero or more documents in the view.


Support

Works in Ultra Pipelines.

Snap Settings

Label


Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Pass Through
 


Required. Determine if data should be passed through or not. If set to yes, then the input data and its structure will be used. However, the output data structure can be changed by setting pass through to [None] - new map or [None] - new list. For example, if the input data is a key/value (map) structure, then the output structure can be changed to a list by setting pass through to [None] - new list and using target paths of list indices ($[0], $[1], $[2], etc).
Options available include:

  • Yes
  • [None] - new map
  • [None] - new list

Default value: Yes


Mapping Table

Required. Source path, operation, and target path to use in the Snap. Source path is the JSONPath to move/delete. Operation is the type of operation to execute (move/delete). Target path is the JSONPath to write a value to in a move operation.

Example: 

Source Path    | Operation | Target Path

$first_name       |    move       | $name.first

$phone_num     |    delete      |  

$names[2]        |    delete      | 

$names[*].first |  move         |   $first_names

 

Default value:  none


Snap Execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Default Value: Execute only
Example: Validate & Execute

Example Output


 Source Data Pass ThroughNull-safe access Source Path Operation Target Path Target Data
{
  "first_name" : "John",
  "last_name": "Smith"  
}
 YesDeselected

 $first_name

move

 $fname

[
{
"last_name":"Smith",
"fname":"John"
}
]
{
  "first_name" : "John",
  "last_name": "Smith"  
}
 [None] - new mapDeselected

 $first_name

move 

 $fname

[
{
"fname":"John"
}
]
{
  "first_name" : "John",
  "last_name": "Smith"  
}
 [None] - new listDeselected

 $first_name

move 

$[0].fname 

[
[
{
"fname":
"John"
}
]
]    
{
  "first_name" : "John",
  "last_name": "Smith"  
}
YesDeselected

$mid_name

move

 $middle

 Pipeline fails
{
  "first_name" : "John",
  "last_name": "Smith"  
}
 YesSelected

 $mid_name

move 

$middle 

[
{
"first_name":
"John"
"last_name":
"Smith"
"middle":
null
}
]
{
  "first_name" : "John",
  "last_name": "Smith"  
}
 YesDeselected

 $mid_name

 delete
 Pipeline fails
[  
   {  
      "person":{  
         "first":"John",
         "last":"Smith"
      },
      "id":2
   }
]
 YesDeselected

$person.first 

move 

$first 

[
  {
    "person":
  {
    "last":"Smith"
  }
  "id":2,
  "first":"John"
  }
]
{
  "first_name" : "John",
  "last_name": "Smith"  
}
 YesSelected$first_name
$first_name

move
move

$name1
$name2 
[
  {
    "last_name":"Smith",
    "name1":"John",
    "name2":"John"
  }
]
{
  "first_name" : "John",
  "mid_name" : "Alan",
  "last_name": "Smith" 
}
YesDeselected$mid_namedelete
[
  {
    "first_name":"John",
    "last_name":"Smith"
  }
]

Examples


See Also


 Click to view/expand
ReleaseSnap Pack VersionDateType Updates
November 2024439patches29078 Latest

Fixed an issue with the CSV Parser Snap that introduced unexpected characters into the records and output data because of incorrect handling of the delimiter.

November 2024main29029 StableUpdated and certified against the current SnapLogic Platform release.
August 2024438patches28073 Latest

Fixed an issue with the JSON Generator and XML Generator Snaps that caused unexpected output displaying '__at__' and '__h__' instead of '@' and '-' respectively because the Snap could not update them to their original values after the Velocity library upgrade.

August 2024438patches27959 Latest

Fixed an issue with the Sort where the Snap could not sort files larger than 52 MB. This fix applies to Join Snap also.

August 2024main27765 StableUpgraded the org.json.json library from v20090211 to v20240303, which is fully backward compatible.
May 2024437patches26643 Latest
  • Fixed an issue with the Sort Snap that displayed an error when estimating the size of the input document provided by the upstream S3 Browser Snap.
  • Fixed an issue with the Parquet Formatter Snap that was unable to route errors to the error view.
May 2024437patches26453 Latest
  • Added expression support to the Skip lines field in the CSV Parser Snap to enable passing pipeline parameters and upstream values. 

  • Fixed an issue with the XML Parser Snap that caused an error when using the Splitter option in the Snap settings. 

May 2024main26341 Stable
  • Added Parquet Parser and Parquet Formatter Snaps to the Transform Snap Pack:
    • Parquet Parser: Reads the binary Parquet data and writes document data to the output.
    • Parquet Formatter: Reads the document data and writes it to the output in binary Parquet format.
  • Enhanced the JSON Splitter Snap to capture metadata and lineage information from the input document.

February 2024436patches25564 Latest

Fixed an issue with the JSON Formatter Snap that generated incorrect schema.

February 2024436patches25292 Latest

Fixed an out-of-memory error issue with the Aggregate Snap. This Snap no longer performs the presort for the input documents.

If the input documents are unsorted and GROUP-BY fields are used, you must use the Sort Snap upstream of the Aggregate Snap to presort the input document stream and set the Sorted stream field Ascending or Descending to prevent the out-of-memory error. However, if the total size of input documents is expected to be relatively small compared to the available memory, then Sort Snap is not required upstream.

Learn more about presorting unsorted input documents to be processed by the Aggregate Snap.

February 2024main25112 StableUpdated and certified against the current SnapLogic Platform release.
November 2023435patches24802 LatestFixed an issue with the Excel Parser Snap that caused a null pointer exception when the input data was an Excel file that did not contain a StylesTable.
November 2023435patches24481 Latest

Fixed an issue with the Aggregate Snap where the Snap was unable to produce the desired number of output documents when the input was unsorted and the GROUP-BY fields field set was used.

November 2023435patches24094 Latest

Fixed a deserialization issue for a unique function in the Aggregate Snap.

November 2023main23721 StableUpdated and certified against the current SnapLogic Platform release.
August 2023434patches23076 LatestFixed an issue with the Binary to Document Snap where an empty input document with Ignore Empty Stream selected caused the Snap to stop executing.
August 2023434patches23034 Latest
  • Fixed an issue with the Transform Snap Pack that caused an error when the input file was a binary JSON file that contained a string value of more than 20,000,000 characters.
  • Fixed a memory issue with the Aggregate Snap that occurred when using GROUP-BY fields.

August 2023434patches22705 Latest

Fixed an issue with the JSON Splitter Snap that caused the pipeline to terminate with excessive memory usage on the Snaplex node after the 4.33 GA upgrade. The Snap now consumes less memory.

August 2023main22460 StableUpdated and certified against the current SnapLogic Platform release.
May 2023433patches22431 Latest
  • Fixed an issue with the Excel Multi Sheet Formatter Snap that caused it to produce binary output data when there was no input document and Ignore empty stream was selected.
  • Introduced the following new Snaps:
    • GeoJSON Parser: Parses geospatial data from binary data input and outputs the contents as a GeoJSON document downstream.

    • WKT Parser: Parses geospatial data from binary data input and outputs the contents as a WKT (Well Known Text) document downstream.

May 2023433patches21779 Latest

The Decrypt Field and Encrypt Field Snaps now support CTR (Counter mode) for the AES (Advanced Encryption Standard) block cipher algorithm.

May 2023433patches21586 Latest

The Decrypt Field Snap now supports the decryption of various encrypted fields on providing a valid decryption key.

May 2023433patches21461 Latest

The following Transform Snaps include new fields to improve memory management: Aggregate, Group By Fields, Group By N, Join, Sort, Unique.

May 2023433patches21336 Latest

Fixed an issue with the AutoPrep Snap where dates could potentially be rendered in a currency format because currency format options were displayed for the DOB column.

May 2023433patches21196 Latest

Enhanced the In-Memory Lookup Snap with the following new fields to improve memory management and help reduce the possibility of out-of-memory failures:

  • Minimum memory (MB)

  • Out-of-memory timeout (minutes)

These new fields replace the Maximum memory % field.

May 2023main21015 StableUpgraded with the latest SnapLogic Platform release.
February 2023432patches20535 Latest

Fixed an issue with the Encrypt Field Snap, where the Snap failed to support an RSA public key to encrypt a message or field. Now the Snap supports the RSA public key to encrypt the message.

February 2023432patches20446 Latest

The Join Snap is enhanced with the following:

  • The Pipeline Execution Statistics of the Join Snap now has a status message that displays the parameters - Free disk space, Available memory, and Average document size.

  • The internal sort buffer size is reduced to a minimum of 10MB when the available memory in the node becomes lower than 500MB to avoid the out-of-memory crash.

  • The internal sort buffer size is restored to its original size when the available memory becomes larger than 2GB.

  • We have improved the readability of the error message for the out of disk space on node error. The updated error message now provides clearer information and guidance for users, as shown below:
    Reason: Insufficient free disk space available to stage sort data into temporary files.
    Resolution:  Increase the amount of free disk space and try again.

February 2023

432patches20250

 Latest
  • Fixed an issue with the JSON Splitter Snap that was causing errors when using multiple repeated dots in the JSON Path.
  • The Sort Snap includes the following improvements:

    • The Maximum memory % field is revised to Maximum memory.

    • The Maximum memory unit (new dropdown list) enables you to choose a unit, percentage (%), or MB for better memory control.

February 2023432patches20151 Stable/Latest

Fixed an issue that occurred with the JSON Splitter Snap when used in an Ultra pipeline. The request was acknowledged before it was processed by the downstream Snaps, which caused a 400 Bad Request response.

February 2023432patches20062 Stable/LatestFixed the behavior of the JSON Splitter Snap for some use cases where its behavior was not backward compatible with the 4.31 GA version. These cases involved certain uses of either the Include scalar parents feature or the Include Paths feature.
February 2023432patches19974 Stable/Latest

Fixed the "Json Splitter expects a list" error by restoring the JSON Splitter Snap's previous behavior of handling the case where the document element referenced by the JSON Path to Split field is an object instead of a list or array.

Review your pipelines where this error occurred to check your assumptions about the input to the JSON Splitter and whether the value referenced by the JSON Path to Split field will always be a list. I