Versions Compared

Old Version 14

changes.mady.by.user Lakshmi Manda

Saved on

compared with

New Version 15

changes.mady.by.user Kalpana Malladi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Overview

You can use this Snap to read both read the input document data and write the data in the binary (Parquet) format to the output.

...

Field Name

Field Type

Description

Label*

 

Default Value: Parquet Formatter
Example: Transform Parquet Formatter

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

 

Edit Schema

 

 

 

 

 

 

 

Button

Specify a valid Parquet schema that describes the data.  

The following is an example of a schema using all the primitive and some examples of logical types:

Code Block
message document {
  # Primitive Types
  optional int64 32_num;
  optional int64 64_num;
  optional boolean truth;
  optional binary message;
  optional float pi;
  optional double e;
  optional int96 96_num;
  optional fixed_len_byte_array (1) one_byte;
  # Logical Types
  optional binary snowman (UTF8);
  optional int32 8_num (INT_8);
  optional int32 16_num (INT_16);
  optional int32 u8_num (UINT_8);
  optional int32 u16_num (UINT_16);
  optional int32 u32_num (UINT_32);
  optional int64 u64_num (UINT_64);
  optional int32 dec_num (DECIMAL(5,2));
  optional int32 jan7 (DATE);
  optional int32 noon (TIME_MILLIS);
  optional int64 jan7_epoch (TIMESTAMP_MILLIS);
  optional binary embedded (JSON);
}
Note

"Generate template" does not support nested structures like MAP and LIST type.

Compression 

 

 

 

Default Value: NONE

Example: SNAPPY

Dropdown list

Choose the type of compression to use when writing the file. The following are the available options are:

  • NONE

  • SNAPPY

  • GZIP

Many compression algorithms require both Java and system libraries, and the algorithms fail if the latter is not installed. If you see unexpected errors, ask your system administrator to verify if all the required system libraries are installed because they are typically not installed by default. The system libraries have names such as liblzo2.so.2 or libsnappy.so.1 and can be located in the /usr/lib/x86_64-linux-gnu directory.

 

Decimal rounding mode

 

Default Value: Half up

Example: Up

Dropdown list

Select Choose the required rounding method for decimal values when they exceed the required number of decimal places. The following are the available options:

  • Half up

  • Half down

  • Half even

  • Up

  • Down

  • Ceiling

  • Floor

  • Truncate

Snap execution

Default Value: Validate & Execute
Example: Execute only

Dropdown list

Select one of the following three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.

  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

...

Code Block
{
   "schema": "message document {\n  optional binary AUTOSYNC_PRIMARYKEY (STRING);\n  optional binary AUTOSYNC_VALUEHASH (STRING);\n  optional binary AUTOSYNC_CURRENTRECORDFLAG (STRING);\n  optional int64 AUTOSYNC_EFFECTIVEBEGINTIME (TIMESTAMP(MILLIS,true));\n  optional int64 AUTOSYNC_EFFECTIVEENDTIME (TIMESTAMP(MILLIS,true));\n  optional double ID1;\n  optional binary ID2 (STRING);\n  optional binary ID3 (STRING);\n  optional binary ID4 (STRING);\n  optional binary ID5 (STRING);\n  optional binary ID6 (STRING);\n  optional binary ID7 (STRING);\n  optional binary ID8;\n  optional double ID9;\n  optional double ID10;\n  optional double ID11;\n  optional double ID12;\n  optional double ID13;\n  optional double ID14;\n  optional int32 ID15 (DATE);\n  optional int64 ID16 (TIMESTAMP(MILLIS,true));\n  optional int64 ID17 (TIMESTAMP(MILLIS,true));\n  optional int64 ID18 (TIMESTAMP(MILLIS,true));\n  optional double ID100;\n}\n"
}

Examples

Transform

...

document data into Parquet

...

format and vice-versa

This example demonstrates how to convert the input document data to parquet and parquet data back to document output.

...