Write

This Snap converts documents into the ORC format and writes the data to HDFS, S3, or the local file system.

• Expected upstream Snaps: Any Snap with a document output view.
• Expected downstream Snaps: [None]
• Expected input: A document.
• Expected output: [None]

[None]

• Ultra pipelines: Works in Ultra Pipelines.
• Spark: Not supported in Spark pipelines.
• All expression Snap properties (when '=' button is pressed) can be evaluated from pipeline parameters only, not from input documents. Input documents are data to be formatted and written to the target files.

The ORC Writer works with the following accounts:

• AWS S3
• Azure Data Lake Gen2 OAuth2
• Azure Storage

Required. The name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

Default Value: ORC Writer

Required. The path to a directory from which you want the ORC Reader Snap to read data. All files within the directory must be ORC formatted.

Basic directory URI structure

• HDFS: hdfs://<hostname>:<port>/
• S3: s3:///<S3 bucket name>/<file-path>
• ABFS(S): abfs(s):///filesystem/<path>/
• ABFS(S): abfs(s)://filesystem@accountname.endpoint/<path>

The Directory property is not used in the pipeline execution or preview, and used only in the Suggest operation. When you press the Suggest icon, the Snap displays a list of subdirectories under the given directory. It generates the list by applying the value of the Filter property.

Example: 

• hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/

• webhdfs://cdh-qa-2.fullsail.Snaplogic.com:50070/user/ec2-user/csv/
• s3://test-s3-drea/8867_output.json

• _dirname 

• file:///home/snaplogic/file.orc
• abfs:///filesystem2/dir1
• abfs://filesystem2@snaplogicaccount.dfs.core.windows.net/dir1

Default value: hdfs://<hostname>:<port>/

Required for standard mode. Filename or a relative path to a file under the directory given in the Directory property. It should not start with a URL separator "/". The File property can be a JavaScript expression which will be evaluated with values from the input view document. When you press the Suggest icon, it will display a list of regular files under the directory in the Directory property. It generates the list by applying the value of the Filter property.

Use Hive tables if your input documents contains complex data types, such as maps and arrays.

Example: 

• sample.orc
• tmp/another.orc
• _filename

Default value:  [None]

Required. Select an action to take when the specified file already exists in the directory. Please note the Append file action is supported for SFTP, FTP, and FTPS protocols only.

Default value: [None]

Set the user and desired permissions.

Default value: [None]

This setting is used to assist in setting the schema along with the database and table setting.  If the data being written has a Hive schema, then the Snap can be configured to read the schema instead of manually entering it. Set the value to a Hive Metastore URL where the schema is defined.

Default value: [None]

The Hive Metastore database where the schema is defined. See the Hive Metastore URL setting for more information.

Paths where the column values appear in the document. This property is required if the Hive Metastore URL property is empty.

Examples:

• Column Name: Fun
• Column Path: $column_from_input_data
• Column Type: string

Default value: [None]