On this Page
Table of Contents | ||||
---|---|---|---|---|
|
Snap type: | Transform | |||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Description: | This Snap eliminates duplicate documents in a document stream, such as duplicate rows in a CSV file. To understand the functionality of the Unique Snap, it is important to understand that SnapLogic is streaming data, processing "one document at a time" in each Snap.
If the data being passed was in the following order: The result would be exactly the same as the input, as the duplicate rows are not adjacent in the flow and therefore, are not identified.
| |||||||||||||
Prerequisites: | [None] | |||||||||||||
Support and limitations: | Ultra pipelines: Does not work Works in Ultra Pipelines.Spark mode: Not supported in /wiki/spaces/SD/pages/1437917 mode. | |||||||||||||
Account: | Accounts are not used with this Snap. | |||||||||||||
Views: |
| |||||||||||||
Settings | ||||||||||||||
Label | Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. | |||||||||||||
|
|
Example
A simple pipeline for this Snap would include:
File Reader + CSV Parser + Unique + CSV Formatter + File Writer
Insert excerpt | ||||||
---|---|---|---|---|---|---|
|