Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

In this article

...

You can use this Snap to perform a file operation (move/copy/rename) for a file from a source to a target in the same server. The supported file protocols are: local file (file:///), FTP, SFTP, S3, WASB, and WASBS.

...

Info

ETL Transformations & Data Flow

The File Operation Snap performs extract and load operations on the input file or folder.

Multiexcerpt include macro
nameMigrating from Binary to Amazon S3
templateData[]
pageFile Reader
addpanelfalse

...

Snap Type

File Operation Snap is a WRITE-type Snap that writes files to the target.

...

Field Name

Field Type

Description

Label*

Default ValueFile Operation
ExampleFile Operation

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

Source*

Default Value: [None]
Example:

  • file:///tmp/test.csv (if you're using Linux)

  • ftp://ftp.snaplogic.com/home/mrtest/source.csv (if your file is on an FTP server)

  • sftp://ftp.snaplogic.com/home/mrtest/source.csv (if you are using Secure FTP)

  • s3:///test_bucket/folder1/test.json (if you are using AWS without specifying any region)

  • s3:///mybucket@eu-west-1/folder1/test.json (if you're using an AWS account that is region-specific)

  • wasb:///test_container/testFolder/sample.json (if you are using Windows Azure Storage Blob)

  • wasbs:///test_container/testFolder/sample.json (if you are using Secure Windows Azure Storage Blob)

  • _source (A key/value pair with "source" key should be defined as a Pipeline parameter.)

  • $source(A key/value pair with "source" key should be defined the input document.)

  • file:///Z:/somedirectory/somefile.csv (if you have a Groundplex on Windows)

  • file:////somedirectory/somfile.csv (if you're using the Universal Naming Convention

  • abfs(s):///filesystem/<path>/

  • abfs(s)://filesystem@accountname.endpoint/<path>

String/Expression

Specify the URL for the source, where the binary data is read from. This Snap also supports S3 Virtual Private Cloud (VPC) endpoint. For example, s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com

Example of Source as an expression: "s3:///mybucket/out_" + Date.now() + ".csv"
the evaluated filename will be: s3:///mybucket/out_2013-11-13T00:22:31.880Z.csv

This property should have the syntax: scheme://[hostname:port]/[path to source]

You can also copy or move the file from your local system to Azure blob container for better performance.

Note

When using expressions to build a file name, ensure that the resulting file name does not contain characters that are not supported by the target platform

Warning

Amazon AWS S3 SDK Limitation - File operations

  • When performing operations on S3 folders ensure that a trailing forward slash ( / ) is added; otherwise, the Snap considers it a file instead of a folder and an error message will be displayed.

  • Ensure the source does not contain '?' character because it is not fully supported and when present, the Snap might fail.

Target*

Default Value: [None]
Example:

  • file:///tmp/test.csv

  • ftp://ftp.snaplogic.com/home/mrtest/source.csv

  • ftp://ftp.snaplogic.com/home/mrtest/

  • sftp://ftp.snaplogic.com/home/mrtest/source.csv

  • sftp://ftp.snaplogic.com/home/mrtest/

  • s3:///test_bucket/folder1/test.json

  • s3://test_bucket/folder1/

  • wasb:///test_container/testFolder/sample.json

  • wasbs:///test_container/testFolder/sample.json

  • _target (A key/value pair with "target" key should be defined as a Pipeline parameter.)

  • $target (A key/value pair with "target" key should be defined the input document.)

  • abfs(s):///filesystem/<path>/

  • abfs(s)://filesystem@accountname.endpoint/<path>

String/Expression

This property specifies the URL of the destination where the selected file operation must be performed. This Snap also supports S3 Virtual Private Cloud (VPC) endpoint. For example, s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com

Warning

Amazon AWS S3 SDK Limitation - File operations

  • When performing operations on S3 folders ensure that a trailing forward slash ( / ) is added; otherwise, the Snap considers it a file instead of a folder and an error message will be displayed.

  • Ensure the file name, folder name, or the file path does not contain '?' character because it is not supported. The Snap fails with an error if you include the '?' character.


File Operation

Default ValueMove
Example:

  • Copy

  • _operation (A key/value pair with "operation" key should be defined as a Pipeline parameter.)

  • $operation(A key/value pair with "operation" key should be defined the input document.)

String/Expression

Enter or select the operation you want the Snap to perform on the file. Available options are: 

  • Move - Moves the source file to the target location.

  • Copy - Creates a copy of the source file in the target location.

  • Rename - Renames a file in the source location. Rename is handled the same as Move. The source URL and target URL are the same, with the exception of the file name. 

Info

This Snap supports Azure Data Lake Storage (ADLS) Gen 2 Storage and ADLS Gen 2 Storage (ABFS) protocol for moving and copying files in the Azure Blob File System (ABFS).

Note

Rename is handled the same as Move. The source URL and target URL are the same, with the exception of the file name. 

Error if exists

Default ValueSelected

Checkbox

If enabled, the Snap displays an error when the target exists. If disabled, the Snap replaces or overwrites the target with the source.

Advanced Properties

Use this field set to customize or control the Snap's validation and execution mechanism. 

Properties

Default Value: SAS URI
Example: Simple file operation


Dropdown list

The available options are:

  • SAS URI:

 

Multiexcerpt include macro
nameSASURI_Description
pageFile Reader

Info

If the SAS URI value is provided in the Snap settings, then the settings provided in the account (if any account is attached) are ignored.

  • Simple file operation: Select and set this property to true to allow the Snap carry out the move or copy operation without checking if the source and target are directory or regular file or does not exist. This feature is useful when the account does not have the permissions to list the source and/or target directories.
    Note: This feature is currently supported only for FTP file operations.

Info

If Simple file operation is set to true, 

  • The selection for Error if exists property is ignored.

  • None of the logic described in Snap Behavior for Key Operations applies.

  • The exact behavior of the Snap varies depending on the File Server implementation and its administration settings.

  • Az copy absolute path: Specify the directory and wildcard for Azure blob storage for Copy, Move, and Rename operations. For example,  "\t" + blob_demo

You can also copy or move the file from your local system to Azure blob container for better performance.

Note

This field is applicable only if you are using Azure Storage account for this Snap.

Info
  • You must install the AzCopy utility in the Snaplex for fetching the path. If the path is null, native Azure Storage SDK is used for all operations. If you specify '/' in this field, the Snap searches for AzCopy in the path or in the specified path and runs the AzCopy command. You can use https as a valid URL, as AzCopy supports httpsfile system. Learn more about AzCopy command.

  • Azure storage SDK does not support file system for https. So, when you do not provide AzCopy absolute path and Source/Target URL is https, the Snap displays an error. For this to work along with the container URL of the Azure blob, you must provide SAS URI under the Advanced properties field set for proper authentication.

Values

Default Value:None

String/Expression

Specify a value for the above property.

Snap Execution

Dropdown list

Multiexcerpt include macro
nameExecution_Detail_Write
pageSOAP Execute

...

  1. Configure the File Reader Snap that reads airlines.csv file from the local machine.

  2. Next, we configure the File Writer Snap to write the airlines.csv file.

  3. Next, we configure the File Operation Snap to move the file from Source (local directory) to Target (Azure Blob container).

    • SAS URI: https://blobbatchingdev.blob.core.windows.net/?sv=2020-08-04&ss=bfqt&srt=sco&sp=rwdlacupitfx&se=2022-05-20T02:51:39Z&st=2022-05-19T18:51:39Z&spr=https&sig=ey9YraJOEsokwBz4GsvT4p%2FCb4eoAuMIVwrf3zymeeI%3D

    • AzCopy absolute path: /Users/vkshirsagar/Downloads/azcopy_darwin_amd64_10.12.2/azcopy

...

The exported Pipeline is available in the Downloads section below. More examples of operations are also available in the Downloads section - variousCasesOfFileOperations.pdf.

Typical Snap Configurations

...