Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

On this Page

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

Snap type:

Transform


Description:

This Snap eliminates duplicate documents in a document stream, such as duplicate rows in a CSV file. To understand the functionality of the Unique Snap, it is important to understand that SnapLogic is streaming data, processing "one document at a time" in each Snap. 

Note

The Unique Snap does NOT sort the documents before making that comparison. 

If the data being passed was in the following order:


 
It would recognize that the data was in pairs and the result would be just the 10 de-duplicated rows: 

 
If the data however were in a different order:  

The result would be exactly the same as the input, as the duplicate rows are not adjacent in the flow and therefore, are not identified.
 
If you wanted to eliminate the duplicates, it is possible to use a Sort Snap in the flow before the Unique, in which case the duplicates would be adjacent, and hence de-duplicated. 

  • Expected upstream Snaps:  Any Reader followed by any file Parser.
  • Expected downstream Snaps:  Any file Formatter followed by a Writer.
  • Expected input:  Document input, likely with duplicate data.
  • Expected output:  Unique document data.
Prerequisites:

[None]


Support and limitations:Works Does not work in Ultra Pipelines.
Account: 

Accounts are not used with this Snap.


Views:


InputThis Snap has exactly one document input view.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view and produces zero or more documents in the view.


Settings

Label


Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Multiexcerpt include macro
nameSnap Execution
pageAnaplan Read

Multiexcerpt include macro
nameSnap_Execution_Introduced
pageAnaplan Read

Multiexcerpt include macro
nameTemporary Files
pageJoin

Example


A simple pipeline for this Snap would include:

File Reader + CSV Parser + Unique + CSV Formatter + File Writer

Insert excerpt
Transform Snap Pack
Transform Snap Pack
nopaneltrue