File Operation

In this article

Overview

You can use this Snap to perform a file operation (move/copy/rename) for a file from a source to a target in the same server. The supported file protocols are: local file (file:///), FTP, SFTP, S3, WASB, and WASBS. The File Operation Snap performs extract and load operations on the input file or folder.

Snap Type

File Operation Snap is a WRITE-type Snap that writes files to the target.

Prerequisites

  • The provided account must have 'write' access to the specified directory and file in order to perform the file operation successfully.

  • IAM Roles for Amazon EC2 

Support for Ultra Pipelines

Works in Ultra Task Pipelines.

Limitations

  • The Snap can move, rename, or copy a file within the same file server, but not across file servers.

  • The Snap can move or copy S3 files across buckets within the same region, but not across regions.

Known Issues

  • This Snap does not support using the ABFS protocol with a Windows-based Snaplex.

  • When you use special characters, such asðø©¢¾A²½µ®÷¶þ~ for Source and Target directory and filenames, this Snap fails and results in the following error as the special characters are not supported.
    Error: Illegal character in fragment at index 71:abfs://bigdataqa@bigdataqassl.dfs.core.windows.net/simplechar/owner!@#$^&()_¢äâêîôûñç¡¿ÉÙËǨ°¸ðø©¢¾A²½µ®§÷¶þ~.json

  • The format: abfs(s)://filesystem@accountname.endpoint/<path> for file path for Source and Target fields does not work as this URL syntax is not supported. The file protocol must begin with abfs(s):///, else the container, account name, and endpoint are interpolated into the URL at runtime, which results in the following error. 
    Error: Unsupported protocol or URL syntax error in abfs(s)://filesystem@accountname.endpoint/<path>,
    Workaround: Use the supported file protocol and correct URL syntax.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input

Document

Min: 0

Max: 1

  • Mapper

  • File Writer

Any document with key-value pairs to evaluate expression properties in the File Operator Snap can be used, where each input document will cause one complete execution of the Snap.

Output

Document

Min: 0

Max: 1

  • CSV Formatter

  • JSON Formatter

  • Mapper

A typical output from this Snap is a document with column headers, such as Source, Target, and Status (Moved or Copied).



Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab. The available options are:

  • Stop Pipeline Execution: Stops the current pipeline execution when the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the rest of the records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk (*): Indicates a mandatory field.

  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon (  ): Indicates whether the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon (  ): Indicates that you can add fields in the fieldset.

  • Remove icon (): Indicates that you can remove fields from the fieldset.

 

Field Name

Field Type

Description

Field Name

Field Type

Description

Label*

 

Default ValueFile Operation
ExampleFile Operation

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

Source*

Default Value: [None]
Example:

  • file:///tmp/test.csv (if you're using Linux)

  • ftp://ftp.snaplogic.com/home/mrtest/source.csv (if your file is on an FTP server)

  • sftp://ftp.snaplogic.com/home/mrtest/source.csv (if you are using Secure FTP)

  • s3:///test_bucket/folder1/test.json (if you are using AWS without specifying any region)

  • s3:///mybucket@eu-west-1/folder1/test.json (if you're using an AWS account that is region-specific)

  • wasb:///test_container/testFolder/sample.json (if you are using Windows Azure Storage Blob)

  • wasbs:///test_container/testFolder/sample.json (if you are using Secure Windows Azure Storage Blob)

  • _source (A key/value pair with "source" key should be defined as a Pipeline parameter.)

  • $source(A key/value pair with "source" key should be defined the input document.)

  • file:///Z:/somedirectory/somefile.csv (if you have a Groundplex on Windows)

  • file:////somedirectory/somfile.csv (if you're using the Universal Naming Convention

  • abfs(s):///filesystem/<path>/

  • abfs(s)://filesystem@accountname.endpoint/<path>

String/Expression

Specify the URL for the source, where the binary data is read from. This Snap also supports S3 Virtual Private Cloud (VPC) endpoint. For example, s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com

Example of Source as an expression: "s3:///mybucket/out_" + Date.now() + ".csv"
the evaluated filename will be: s3:///mybucket/out_2013-11-13T00:22:31.880Z.csv

This property should have the syntax: scheme://[hostname:port]/[path to source]

You can also copy or move the file from your local system to Azure blob container for better performance.

When using expressions to build a file name, ensure that the resulting file name does not contain characters that are not supported by the target platform

Amazon AWS S3 SDK Limitation - File operations

  • When performing operations on S3 folders ensure that a trailing forward slash ( / ) is added; otherwise, the Snap considers it a file instead of a folder and an error message will be displayed.

  • Ensure the source does not contain '?' character because it is not fully supported and when present, the Snap might fail.

Target*

Default Value: [None]
Example:

  • file:///tmp/test.csv

  • ftp://ftp.snaplogic.com/home/mrtest/source.csv

  • ftp://ftp.snaplogic.com/home/mrtest/

  • sftp://ftp.snaplogic.com/home/mrtest/source.csv

  • sftp://ftp.snaplogic.com/home/mrtest/

  • s3:///test_bucket/folder1/test.json

  • s3://test_bucket/folder1/

  • wasb:///test_container/testFolder/sample.json

  • wasbs:///test_container/testFolder/sample.json

  • _target (A key/value pair with "target" key should be defined as a Pipeline parameter.)

  • $target (A key/value pair with "target" key should be defined the input document.)

  • abfs(s):///filesystem/<path>/

  • abfs(s)://filesystem@accountname.endpoint/<path>

String/Expression

This property specifies the URL of the destination where the selected file operation must be performed. This Snap also supports S3 Virtual Private Cloud (VPC) endpoint. For example, s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com

Amazon AWS S3 SDK Limitation - File operations

  • When performing operations on S3 folders ensure that a trailing forward slash ( / ) is added; otherwise, the Snap considers it a file instead of a folder and an error message will be displayed.

  • Ensure the file name, folder name, or the file path does not contain '?' character because it is not supported. The Snap fails with an error if you include the '?' character.



File Operation

Default ValueMove
Example:

  • Copy

  • _operation (A key/value pair with "operation" key should be defined as a Pipeline parameter.)

  • $operation(A key/value pair with "operation" key should be defined the input document.)

String/Expression

Enter or select the operation you want the Snap to perform on the file. Available options are: 

  • Move - Moves the source file to the target location.

  • Copy - Creates a copy of the source file in the target location.

  • Rename - Renames a file in the source location. Rename is handled the same as Move. The source URL and target URL are the same, with the exception of the file name. 

This Snap supports Azure Data Lake Storage (ADLS) Gen 2 Storage and ADLS Gen 2 Storage (ABFS) protocol for moving and copying files in the Azure Blob File System (ABFS).

Rename is handled the same as Move. The source URL and target URL are the same, with the exception of the file name. 

Error if exists

Default ValueSelected

Checkbox

If enabled, the Snap displays an error when the target exists. If disabled, the Snap replaces or overwrites the target with the source.

Advanced Properties


Use this field set to customize or control the Snap's validation and execution mechanism. 

Properties

Default Value: SAS URI
Example: Simple file operation



Dropdown list

The available options are:

  • SAS URI:

 

If the SAS URI value is provided in the Snap settings, then the settings provided in the account (if any account is attached) are ignored.

  • Simple file operation: Select and set this property to true to allow the Snap carry out the move or copy operation without checking if the source and target are directory or regular file or does not exist. This feature is useful when the account does not have the permissions to list the source and/or target directories.
    Note: This feature is currently supported only for FTP file operations.

If Simple file operation is set to true, 

  • The selection for Error if exists property is ignored.

  • None of the logic described in Snap Behavior for Key Operations applies.

  • The exact behavior of the Snap varies depending on the File Server implementation and its administration settings.

  • Az copy absolute path: Specify the directory and wildcard for Azure blob storage for Copy, Move, and Rename operations. For example,  "\t" + blob_demo

You can also copy or move the file from your local system to Azure blob container for better performance.

This field is applicable only if you are using Azure Storage account for this Snap.

  • You must install the AzCopy utility in the Snaplex for fetching the path. If the path is null, native Azure Storage SDK is used for all operations. If you specify '/' in this field, the Snap searches for AzCopy in the path or in the specified path and runs the AzCopy command. You can use https as a valid URL, as AzCopy supports httpsfile system. Learn more about AzCopy command.

  • Azure storage SDK does not support file system for https. So, when you do not provide AzCopy absolute path and Source/Target URL is https, the Snap displays an error. For this to work along with the container URL of the Azure blob, you must provide SAS URI under the Advanced properties field set for proper authentication.

Values

Default Value: None

String/Expression

Specify a value for the above property.

Snap Execution

Dropdown list

Snap Behavior for Key Operations

State Tables

Following is a collection of state tables to depict the Snap's behavior for different configurations of the Source, Target and Error if exists properties. These combinations are applied to move, and copy operations. Use the following legend to understand what the values mean:

Source

Target

Error if exists

Source

Target

Error if exists

--

--

x

State does not affect outcome

x

State does not affect outcome

00

Does not exist

00

Does not exist

0

Not selected

01

Regular file

01

Regular file

1

Selected

10

Directory

10

Empty directory

--

--

--

--

11

Non-empty directory

--

--

FTP, SFTP, and File Protocols

Move Operation

S. No.

Source

Target

Error if exists

Snap's Behavior

Status

Description

1

00

x

x

Error

Source not found.

2

01

00

x

Moved

--

3

01

01

0

Moved

Target file is overwritten.

4

01

01

1

Error

Target already exists.

5

01

10

0

Moved

Empty target directory is replaced with a new file of the same name.

6

01

10

1

Error

Target already exists.

7

01

11

0

Error

Target directory not empty.

8

01

11

1

Error

Target already exists.

9

10

00

x

Moved

--

The source directory is renamed to the given target path if the target’s parent directory is not a subdirectory of the source.

10

10

01

0

Moved

Target file is removed and source directory is moved to the target directory.

11

10

01

1

Error

Target already exists.

12

10

10

0

Moved

Empty target directory is replaced with source directory.

13

10

10

1

Error

Target already exists.

14

10

11

0

Error

Target directory not empty.

15

10

11

1

Error

Target already exists.



Copy Operation

S. No.

Source

Target

Error if exists

Snap's Behavior

Status

Description

1

00

x

x

Error

Source not found.

2

01

00

x

Copied

Target file is created.

3

01

01

0

Copied

Target file is overwritten.

4

01

01

1

Error

Target already exists.

5

01

10

0

Copied

Empty target directory is replaced with a new file of the same name.

For some SFTP file servers, the Snap may produce an error with the following Reason: "Cannot open output stream for <target path>" if the Target property value ends with '/'.

6

01

10

1

Error

Target already exists.

7

01

11

0

Error

Target directory not empty.

8

01

11

1

Error

Target already exists.

9

10

00

x

Copied

Empty target directory created.

10

10

01

0

Copied

Target file replaced with an empty directory of the same name.

11

10

01

1

Error

Target already exists.

12

10

10

0

No operation

Empty target directory remains the same.

13

10

10

1

Error

Target already exists.

14

10

11

0

Error

Target directory not empty.

15

10

11

1

Error

Target already exists.



S3, WASB, and WASBS Protocols

Move Operation

S. No.

Source

Target

Error if exists

Snap's Behavior

Status

Description

1

00

x

x

Error

Source not found.

2

01

00

x

Moved

--

3

01

01

0

Moved 

Target file is overwritten.

4

01

01

1

Error

Target already exists.

5

01

10

x

Moved

Source file is moved to the target directory.

6

01

11

0

Moved

Source file is moved to the target directory.

7

01

11

1

Moved or Error

Error if a file with the same name already exists in the target directory.

8

10

00

x

Moved

All files in the source directory are moved to the target directory.

9

10

01

0

Moved

A new target directory with a name same as the target file is created and all files in the source directory are moved to the target directory. As a result, a regular file and a directory with the same name will coexist in the target. 

10

 10

01

1

Error

Target already exists.

11

10

10

x

Moved

All files in the source directory are moved to the target directory.

12

10

11

0

Moved

All files in the source directory are moved to the target directory.

13

10

11

1

Moved or Error

Error if a file with the same name already exists in the target directory.



Copy Operation

S. No.

Source

Target

Error if exists

Snap's Behavior

Status

Description

1

00

x

x

Error

Source not found.

2

01

00

x

Copied

Target file is created.

3

01

01

0

Copied

Target file is overwritten.

4

01

01

1

Error

Target already exists.

5

01

10

x

Copied

Source file is copied to the target directory.

6

01

11

0

Copied

Source file is copied to the target directory.

7

01

11

1

Copied or Error

Error if the file of the same name exists in the target directory already.

8

10

00

x

Copied

All files in the source directory are copied to the target directory.

9

10

01

0

Copied

A new target directory with a name same as the target file is created and all files in the source directory are copied to the target directory. As a result, a regular file and a directory with the same name will coexist in the target.

10

10

01

1

Error

Target already exists.

11

10

10

x

Copied

All files in the source directory are copied to the target directory.

12

10

11

0

Copied

All files in the source directory are copied to the target directory.

13

10

11

1

Copied or Error

Error if the file of the same name exists in the target directory already.



Examples

Moving File from Local System to Azure Blob Container

This example Pipeline demonstrates how to move files from your local system to the Azure blob for improved performance.

  1. Configure the File Reader Snap that reads airlines.csv file from the local machine.

  2. Next, we configure the File Writer Snap to write the airlines.csv file.

     

  3. Next, we configure the File Operation Snap to move the file from Source (local directory) to Target (Azure Blob container).

    • SAS URI: https://blobbatchingdev.blob.core.windows.net/?sv=2020-08-04&ss=bfqt&srt=sco&sp=rwdlacupitfx&se=2022-05-20T02:51:39Z&st=2022-05-19T18:51:39Z&spr=https&sig=ey9YraJOEsokwBz4GsvT4p%2FCb4eoAuMIVwrf3zymeeI%3D

    • AzCopy absolute path: /Users/vkshirsagar/Downloads/azcopy_darwin_amd64_10.12.2/azcopy


Copying and Moving of File from One Folder to Another Folder

This example Pipeline demonstrates copying and moving of files with ABFS protocol from one folder to another folder.

First, we configure the JSON Generator Snap to pass the following sample data. After validation we see the following output.


Next, we format the data using the JSON Formatter Snap and then, use the HDFS Writer Snap to write the sample file (that uses ABFS protocol and ADLS Gen 2) to the target location.

Next, we configure two File Operation Snaps to copy and move files from one folder to another folder. We use the first File Operation Snap to copy the sample file from abfss:///bigdataqa/Samplefile/ to abfss:///bigdataqa/samplefile_diff/).

We add the second File Operation Snap to move the same sample file (uni_code.json) to another file (uni_code_move.json).

The target location of the copied and moved file can be viewed in the output preview of the Snap.

Finally, we connect the HDFS Reader Snap to read the target file.


Basic Use Case

The following Pipeline describes how the Snap functions as a standalone Snap in a Pipeline:

Download the Pipeline here or from the Downloads section below.

In the Pipeline shown above, a copy of the file Emp_Data_03032020.json is created in the specified target location. The Snap's configuration is as below:

A preview of the output from executing this Pipeline:

The exported Pipeline is available in the Downloads section below. More examples of operations are also available in the Downloads section - variousCasesOfFileOperations.pdf.

Typical Snap Configurations

Key configuration of the Snap lies in how the values are passed. The values can be passed:

  • Without Expressions: Values are passed in the Snap directly:

         

  • With Expressions:

    • Using Pipeline parameters: Values are passed as Pipeline parameters:

                      



Downloads