In this article
...
You can use this Snap to perform a file operation (move/copy/rename) for a file from a source to a target in the same server. The supported file protocols are: local file (file:///), FTP, SFTP, S3, WASB, and WASBS.
...
Info |
---|
The File Operation Snap performs extract and load operations on the input file or folder. Multiexcerpt include macro |
---|
name | Migrating from Binary to Amazon S3 |
---|
templateData | [] |
---|
page | File Reader |
---|
addpanel | false |
---|
|
...
Snap Type
File Operation Snap is a WRITE-type Snap that writes files to the target.
...
Field Name | Field Type | Description |
---|
Label* Default Value: File Operation Example: File Operation | String | Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline. |
Source* Default Value: [None] Example: file:///tmp/test.csv (if you're using Linux) ftp://ftp.snaplogic.com/home/mrtest/source.csv (if your file is on an FTP server) sftp://ftp.snaplogic.com/home/mrtest/source.csv (if you are using Secure FTP) s3:///test_bucket/folder1/test.json (if you are using AWS without specifying any region) s3:///mybucket@eu-west-1/folder1/test.json (if you're using an AWS account that is region-specific) wasb:///test_container/testFolder/sample.json (if you are using Windows Azure Storage Blob) wasbs:///test_container/testFolder/sample.json (if you are using Secure Windows Azure Storage Blob) _source (A key/value pair with "source" key should be defined as a Pipeline parameter.) $source(A key/value pair with "source" key should be defined the input document.) file:///Z:/somedirectory/somefile.csv (if you have a Groundplex on Windows) file:////somedirectory/somfile.csv (if you're using the Universal Naming Convention abfs(s):///filesystem/<path>/ abfs(s)://filesystem@accountname.endpoint/<path>
| String/Expression | Specify the URL for the source, where the binary data is read from. This Snap also supports S3 Virtual Private Cloud (VPC) endpoint. For example, s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com Example of Source as an expression: " s3:///mybucket/out_" + Date.now() + ".csv" the evaluated filename will be: s3:///mybucket/out_2013-11-13T00:22:31.880Z.csv This property should have the syntax: scheme://[hostname:port]/[path to source] You can also copy or move the file from your local system to Azure blob container for better performance. Note |
---|
When using expressions to build a file name, ensure that the resulting file name does not contain characters that are not supported by the target platform. |
Warning |
---|
Amazon AWS S3 SDK Limitation - File operationsWhen performing operations on S3 folders ensure that a trailing forward slash ( / ) is added; otherwise, the Snap considers it a file instead of a folder and an error message will be displayed. Ensure the source does not contain '?' character because it is not fully supported and when present, the Snap might fail.
|
|
Target* Default Value: [None] Example: file:///tmp/test.csv ftp://ftp.snaplogic.com/home/mrtest/source.csv ftp://ftp.snaplogic.com/home/mrtest/ sftp://ftp.snaplogic.com/home/mrtest/source.csv sftp://ftp.snaplogic.com/home/mrtest/ s3:///test_bucket/folder1/test.json s3://test_bucket/folder1/ wasb:///test_container/testFolder/sample.json wasbs:///test_container/testFolder/sample.json _target (A key/value pair with "target" key should be defined as a Pipeline parameter.) $target (A key/value pair with "target" key should be defined the input document.) abfs(s):///filesystem/<path>/ abfs(s)://filesystem@accountname.endpoint/<path>
| String/Expression | This property specifies the URL of the destination where the selected file operation must be performed. This Snap also supports S3 Virtual Private Cloud (VPC) endpoint. For example, s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com Warning |
---|
Amazon AWS S3 SDK Limitation - File operationsWhen performing operations on S3 folders ensure that a trailing forward slash ( / ) is added; otherwise, the Snap considers it a file instead of a folder and an error message will be displayed. Ensure the file name, folder name, or the file path does not contain '?' character because it is not supported. The Snap fails with an error if you include the '?' character.
|
|
File Operation Default Value: Move Example: | String/Expression | Enter or select the operation you want the Snap to perform on the file. Available options are: Move - Moves the source file to the target location. Copy - Creates a copy of the source file in the target location. Rename - Renames a file in the source location. Rename is handled the same as Move. The source URL and target URL are the same, with the exception of the file name.
Info |
---|
This Snap supports Azure Data Lake Storage (ADLS) Gen 2 Storage and ADLS Gen 2 Storage (ABFS) protocol for moving and copying files in the Azure Blob File System (ABFS). |
Note |
---|
Rename is handled the same as Move. The source URL and target URL are the same, with the exception of the file name. |
|
Error if exists Default Value: Selected | Checkbox | If enabled, the Snap displays an error when the target exists. If disabled, the Snap replaces or overwrites the target with the source. |
Advanced Properties
| Use this field set to customize or control the Snap's validation and execution mechanism. |
Properties Default Value: SAS URI Example: Simple file operation
| Dropdown list | The available options are: Multiexcerpt include macro |
---|
name | SASURI_Description |
---|
page | File Reader |
---|
|
Info |
---|
If the SAS URI value is provided in the Snap settings, then the settings provided in the account (if any account is attached) are ignored. |
Info |
---|
If Simple file operation is set to true, The selection for Error if exists property is ignored. None of the logic described in Snap Behavior for Key Operations applies. The exact behavior of the Snap varies depending on the File Server implementation and its administration settings.
|
Az copy absolute path: Specify the directory and wildcard for Azure blob storage for Copy, Move, and Rename operations. For example, "\t" + blob_demo
You can also copy or move the file from your local system to Azure blob container for better performance. Note |
---|
This field is applicable only if you are using Azure Storage account for this Snap. |
Info |
---|
You must install the AzCopy utility in the Snaplex for fetching the path. If the path is null, native Azure Storage SDK is used for all operations. If you specify '/' in this field, the Snap searches for AzCopy in the path or in the specified path and runs the AzCopy command. You can use https as a valid URL, as AzCopy supports https file system. Learn more about AzCopy command. Azure storage SDK does not support file system for https . So, when you do not provide AzCopy absolute path and Source/Target URL is https , the Snap displays an error. For this to work along with the container URL of the Azure blob, you must provide SAS URI under the Advanced properties field set for proper authentication.
|
|
Values Default Value:None | String/Expression | Specify a value for the above property. |
Snap Execution | Dropdown list | Multiexcerpt include macro |
---|
name | Execution_Detail_Write |
---|
page | SOAP Execute |
---|
|
|
...
Configure the File Reader Snap that reads airlines.csv file from the local machine.
Next, we configure the File Writer Snap to write the airlines.csv file.
Next, we configure the File Operation Snap to move the file from Source (local directory) to Target (Azure Blob container).
SAS URI: https://blobbatchingdev.blob.core.windows.net/?sv=2020-08-04&ss=bfqt&srt=sco&sp=rwdlacupitfx&se=2022-05-20T02:51:39Z&st=2022-05-19T18:51:39Z&spr=https&sig=ey9YraJOEsokwBz4GsvT4p%2FCb4eoAuMIVwrf3zymeeI%3D
AzCopy absolute path: /Users/vkshirsagar/Downloads/azcopy_darwin_amd64_10.12.2/azcopy
...
The exported Pipeline is available in the Downloads section below. More examples of operations are also available in the Downloads section - variousCasesOfFileOperations.pdf.
Typical Snap Configurations
...