Versions Compared

Old Version 13

changes.mady.by.user Chandna Mitra

Saved on

compared with

New Version Current

changes.mady.by.user Mohammed Iqbal

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

On this Page

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

Required for any Compression Type in Spark mode.

Snap type:

Format


Description:

This Snap formats the incoming document from the upstream Snaps to Hadoop sequence file format, the native binary data format to persist intermediate data between different stages of MapReduce jobs. 

Note
To enable Snappy compression for sequence file, a cluster level setting needs to be set. Unlike Parquet or ORC writer the Sequence Formatter Snap will not have "Snappy" listed in the "compression" options.
  • Expected upstream Snaps: The upstream Snap for Sequence Formatter should output map/table/key-value formatted data. Valid data types include String, Integer, Number and Boolean.
  • Expected downstream Snaps: The Sequence Formatter Snap outputs binary data, so the downstream Snap must be a data store output Snap like (File Writer, HDFS Writer, etc.).


Prerequisites:

[None]


Support and limitations:Works in Ultra Pipelines.
Account: 

Accounts are not used with this Snap.


Views:


InputThis Snap has at most one document input view.
OutputThis Snap has at most one binary output view.
Error

This Snap has at most one document error view and produces zero or more documents in the view. 

In Spark mode:

  • the errors will be routed to error view documents if the error policy is defined as CONTINUE;
  • the execution will stop on first error if the error policy is defined as FAIL;
    • the execution will simply ignore the error if the error policy is defined as IGNORE.


    Settings

    Label


    		Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

    Key



    Required. JSON path for the key.

    Example: $input_column_name

    Default value: [None]


    Value


    Required. JSON path for the value.

    Example: $input_column_name

    Default value:  [None]


    Compression type


    For Spark mode, when compression is required, this value has to be populated. In standard mode, it is not required.

    Sequence file compression type. The options available include:

    • Record: Only values are compressed.
    • Block: Both keys and values are compressed.
    • [None]: Records are uncompressed when none is selected.

    Default value: [None]


    Compression codec


    Fully qualified compression codec class name

    Example: org.apache.hadoop.io.compress.GzipCodec

    Multiexcerpt include macro
    nameSnap Execution
    pageAnaplan Read


    Multiexcerpt include macro
    nameSnap_Execution_Introduced
    pageAnaplan Read

    Troubleshooting

    Insert excerpt
    Hadoop Directory Browser
    Hadoop Directory Browser
    nopaneltrue

    Insert excerpt
    Hadoop Snap Pack
    Hadoop Snap Pack
    nopaneltrue