Avro Parser

In this article

Overview

You can use this Snap to deserialize Avro-formatted data into SnapLogic document objects.

Snap Type

Avro Formatter Snap is a PARSE-type Snap that parses Avro-formatted data into SnapLogic document objects.

Prerequisites

None.

Support for Ultra Pipelines

Works in Ultra Pipelines. 

Limitations

None.

Known Issues

The Snap might fail due to the ClassCastException error. SnapLogic recommends you to upgrade to the 425patches9815 version. 

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

 

  • Min: 1

  • Max: 1

Binary

Any Snap that provides a serialized stream which contains data and its schema according to Avro specification.

Output

Document

 

  • Min: 1

  • Max: 1

  • Mapper

  • Binary

Any document processing Snap document that processes the documents created through the deserialization process.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution: Stops the current Pipeline execution when the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the rest of the records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk (*): Indicates a mandatory field.

  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon (): Indicates whether the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( ): Indicates that you can add fields in the field set.

  • Remove icon (): Indicates that you can remove fields from the field set.

Field Name

Type

Description

Field Name

Type

Description

Label*

Default Value: Avro Parser
Example: Avro_Parser

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

 

Snap Execution

Default value: Validate & Execute
Example: Disabled

 

Dropdown list

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.

  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Temporary Files

During execution, data processing on Snaplex nodes occurs principally in-memory as streaming and is unencrypted. When larger datasets are processed that exceeds the available compute memory, the Snap writes Pipeline data to local storage as unencrypted to optimize the performance. These temporary files are deleted when the Snap/Pipeline execution completes. You can configure the temporary data's location in the Global properties table of the Snaplex's node properties, which can also help avoid Pipeline errors due to the unavailability of space. For more information, see Temporary Folder in Configuration Options. 

Examples

Deserializing Avro data

This example pipeline demonstrates how to read an Avro file and convert it to a structured data format that can be used in downstream applications.

Download this pipeline.

 

  1. Configure the File Reader Snap to read the avro2-data (3).avro file. On validation, the Snap displays the data in binary format.

 

 

2. Configure the Avro Parser Snap to parse the data. On validation, the Snap displays the deserialized data.

 

 

 

 

Downloads

  File Modified

File Example_Avro_Parser.slp

May 01, 2023 by Amritesh Singh

Snap Pack History

Related Links