CSV Parser

In this article

Overview

You can use this Snap to read CSV binary data from its input view, parse it, and then write it to its output view as CSV document data.

Snap Type

CSV Parser is a PARSE-type Snap that reads the binary data from the input view, parses, and writes to the output view.

Prerequisites

None.

Support for Ultra Pipelines

Works in Ultra Pipelines. 

Limitations and Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Binary

 

  • Min: 1

  • Max: 2

  • File Reader

This Snap has at most two binary input views, where it gets the CSV binary data to be parsed.

If there are two input views, it gets the CSV binary data to be parsed from the first input view and the CSV metadata from the second input view. The metadata should be a CSV format with two lines of CSV data: the first line is the CSV header, the second, data types. Supported data types are 'string', 'integer', 'float' and 'boolean'. If 'string' is a default data type, empty data type fields are considered to be 'string' type. An example of CSV metadata is:

Last name,First name,age,commute_km,isDriving

string, ,integer,float,boolean

Output

Document

  • Min: 1

  • Max: 1

Document

This Snap has exactly one document output view, where it provides the CSV document data stream.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab are:

  • Stop Pipeline Execution: Stops the current pipeline execution when the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the rest of the records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk (*): Indicates a mandatory field.

  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon (): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon (): Indicates that you can add fields in the fieldset.

  • Remove icon (): Indicates that you can remove fields from the fieldset.

Field Name

Field Type

Description

Field Name

Field Type

Description

Label*

Default Value: CSV Parser
Example: CSV_Parser

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

 

Quote character

Default Value: “
Example: “

Character

Specify the character to be used for a quote. As of 4.3.2, this property can be an expression, which is evaluated with the values from the Pipeline parameters.

Only a single character is allowed as a quote character.

Delimiter*

Default value: , 
Examples:

  • \t

  • \u0001

String/Expression

Specify the string or the character to be used as a delimiter in formatting the delimited data. Any combination of characters may be used, adhering to the following guidelines.

The input must be submitted with any control characters escaped. For example, \t (tab), \n (new line), or \\ (single backslash) must be escaped accordingly. Unicode characters should be specified using the Unicode escape sequence \uXXXX, where each X represents a hexadecimal digit (0-9, a-f) with all four hexadecimal digits defined.

When using a single backslash (\) as a delimiter, it does not need to be escaped (\\). However, if you are using a multi-character delimiter that contains one or more backslashes (\), you must escape all backslashes (\\).

Escape character

Default value: \
Example: \

Character

Specify the escape character that is to be used when parsing rows. Only single characters are supported. As of 4.3.2, this property can be an expression, which is evaluated with the values from the pipeline parameters. Leave this property empty if no escape character is used in the input CSV data.

Skip lines*

Default value: 0
Example: 5

Integer/Expression

Specify the number of lines that are to be skipped in the input data before the Snap starts parsing it. This example explains how to skip lines.

Contains header

Default value: Selected
Example: Deselected

Checkbox

Select the checkbox to specify if the input data contains the CSV header or not.

Column names 

Use this fieldset to specify the column header names, which is a composite table property.

Header

String

Specify the list of headers that are to be used as a CSV header in case you deselect the Contains header property. 
Example: 

  • Last Name 

  • First Name 

  • Street 

  • City 

  • State 

Validate headers

Default value: Deselected
Example: Selected

Checkbox

Select the checkbox to specify if the headers from the input data should be validated against the Column names table property or not. If this option is selected, the Snap throws an exception when they do not match exactly.

Header size error policy

Default value: Both
Example: Trim record to fit header

Dropdown list

Select an option to define how to handle errors for records that do not match the header columns in the CSV file. This error condition occurs if the input document has fewer or additional columns that do not match with the header columns. The available options are:

  • Trim record to fit header: The Snap trims the records to match the header columns and sends them to the output view if the values in the CSV file are more than the header columns. If the values are less than the header columns, the Snap sends the values as-is to the output view with blank spaces.

  • Fail if record is larger than header: The Snap sends the document to the output view if the values in the CSV file match with the header columns. Else, the Snap writes the output to the error view when the values in the CSV file are either more or less than the header columns. 

  • Both: The Snap sends the trimmed records to the output view and also sends those records to the error view whose values are either more or less than the header columns.

Character set

Default value: Auto BOM detect.
Example: UTF-8

Dropdown list

Select an option to specify the character set in which input CSV data is encoded. The available options are:

  • Auto BOM detect: The Snap attempts to detect BOM (Byte Order Mark) in the input CSV data. If no BOM is found, the java runtime’s default character set is used.

  • UTF-8

  • UTF-16LE

  • UTF-16BE

  • ISO-LATIN-1:  This character set is also called ISO-8859-1 and generally intended for Western European languages.

Ignore empty data

Default value: Selected
Example: Deselected

Checkbox

Select this checkbox to send the document to error view if the input is empty.

If you do not select this checkbox, then it produces an empty output document when the input CSV data is empty (both an empty binary stream and a binary stream with CSV headers only). This feature may be useful if the downstream Snaps should be executed whether the input CSV data is empty or not.

Preserve Surrounding Spaces

Default value: Deselected
Example: Selected

Checkbox

Select this checkbox to preserve the surrounding spaces for the values that are non-quoted. 

  • If you enable the expression icon when the checkbox is selected, then the value of this setting is set to true.

  • If you enable the expression icon when the checkbox is not selected, then the value of this setting is set to false.

For example, if you are using data with a delimiter as follows:

NAME|AGE|GENDER
AA| 12| F
BB| 23| M
If you deselect this checkbox, then surrounding spaces are removed before 12, 23, F, and M.

Snap Execution

Default Value: Validate & Execute
Example: Execute only

Dropdown list

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Examples

Skip lines from input data

The following example pipeline demonstrates how to skip lines from the input data (CSV) and parse the remaining data.

ex-csv-parser-pipeline-overview.png

Configure the File Reader Snap to pass CSV file as input.

ex-file-reader-config.png

 

 

 

 

Configure the CSV Parser Snap to skip two lines (value from the upstream Snap via expression enabler) while parsing the CSV data.

On validation, the CSV Parser Snap skips two lines from the input and parses the remaining data.

 

Pipeline: Split a CSV File

Using the CSV Parser Snap Schema Capability

One of the features in the CSV Parser which customers sometimes request is the ability to define the fields (and their data types) for incoming CSV files. This is made easy by adding a second input view to the CSV Parser Snap, and providing the definition of the fields, and their data types in the flow.

For example, if you have input data in the CSV file as follows, with no header line:

Then, you can create a definition of the CSV data in another CSV file as follows:

Note the data types are optional, and defined on the second line of the input file.  The parser supports the use of 'string', 'integer', 'float' and 'boolean' types. String is the default data type, any empty data type fields are considered to be strings.

The configuration of the pipeline for this use is as follows:

where the Read Snaps are File Readers.

The CSV Parser is configured as follows:

Views is configured as follows:

The resulting data in the SnapLogic pipeline data flow looks like this:

  File Modified

File Example--CSV Parser-Skip-lines.slp

Jun 13, 2024 by Kalpana Malladi

Snap Pack History

Related Links

 

 

Â