Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Updated Works with Ultra Pipelines condition

In this article

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

Snap type:

Transform

Description:

This Snap joins left and right input data streams. It uses the right input data as an in-memory lookup table.

  • Expected upstream Snaps: Any Snap with a document output view, such as CSV Parser, JSON Parser, Mapper.
  • Expected downstream Snaps: Any Snap with a document input view, such as CSV Formatter, JSON Formatter, Mapper.
  • Expected input:   
    • Input data streams may be unsorted.
    • The right input document stream is loaded in memory as a lookup table while the left input document stream is not stored in the Snap.
    • The JOIN operation starts when the right input document stream ends.
    • All input document data should be of a flat map data type.
  • Expected output: Each left input document is joined with the right input data if a match is found. If not, the left input document is written to the output view without join. If the Single document output property is true, only one document is written to the output view for each left input document regardless of the number of matches found in the in-memory lookup table.
Prerequisites:

Enough free memory should be available to load all right input data to the in-memory lookup table.

Support and limitations:

Works Limited support in Ultra Task Pipelines when the Single document output field is selectedOnly one document is written to the output view for each input document, which is a prerequisite for Ultra Pipelines.

Account: 

Accounts are not used with this Snap.

Views:


InputThis Snap has exactly two document input views. Users may want to edit the right input view label in the 'Views' section of the Snap since the right input view label is used as a prefix during the JOIN operation if the same column name exists in the left input data.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view and produces zero or more documents in the view.


Settings

Label

Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Join paths

Required. Field names to use for left and right sides of join. Each row in the table defines a relationship between a left field and a right field. The Snap supports flat map data only for all input documents, and a structured JSON path like '$customer.address' is not supported.

Left path

Required. Field name in the left input data. It can be selected from the suggested list. This property does not support expressions.

Example: customer_id, $customer_id, _leftFieldName (for pipeline parameter with the expression button enabled)

Default value:  None, the expression toggle enabled

Right path

Required. Field name in the right input data. The Right path suggestion is not available yet, except pipeline parameters. This property does not support expressions.

Example: customer_id, $customer_id, _rightFieldName (for pipeline parameter with the expression button enabled)

Default value: None, the expression button enabled

Single document outputIf selected, only one document is always written to the output view for each input document. If more than one row in the lookup table matches with the left input data, the first one in the list is joined with the left input data. If there is no match, the left input data is written to the output view. Leave this property selected if the pipeline is executed in Ultra Task Pipelines mode.
If unselected, each of the matching rows is joined with the left input data. Therefore, the number of output documents may be larger than the input document counts.
Maximum memory %

Enter the percentage of maximum used memory compared to the maximum heap space. Since all right input documents are loaded into the in-memory lookup table, it is possible for the Snap to cause an out-of-memory failure or a poor performance. In order to help users avoid such situations, the Snap checks the percentage of total memory used in the JVM every 1000 right input documents. If the percentage exceed the threshold set by this property, it performs a garbage collection and checks the memory status again. If it finds the threshold is still exceeded,  it throws an exception. 100% means that the Snap does not check the memory status.

Example: 85

Default: 90

Multiexcerpt include macro
nameSnap Execution
pageAnaplan Read

Multiexcerpt include macro
nameSnap_Execution_Introduced
pageAnaplan Read

Examples


Views Page with Inputs Renamed

Views page with inputs renamed:

Left input data:

 Right input data:

Output data with Single document output selected:

Image RemovedImage Added

Output data with Single document output not selected:

Image RemovedImage Added

Excerpt Include
Transform Snap Pack
Transform Snap Pack
nopaneltrue