In this article

Overview

You can use this Snap to read a binary data stream from the input view and write it to a specified file destination. Possible file destinations include: SLDB, HTTP, S3, FTP, SFTP, FTPS, or HDFS. If File permissions for the file are provided, the Snap set those permissions to the file. This is a Write Snap type.

Prerequisites

Support for Ultra Pipelines

Works in Ultra Pipelines.

Limitations

Files uploaded to SLFS have a 100-MB per file limit. This limit does not apply when writing to external storage. 

Known Issues

• This Snap does not create an output file when using the input from SAS Generator Snap configured with only the DELETE SAS permission. This is not the case when the target file exists.
• This Snap does not fail and turns green after execution despite providing an expired SAS URI. As a workaround, select Validate after write to fail the Snap in case of invalid credentials.

Snap Input and Output

Any Snap with a binary output view can be connected upstream, such as CSV Formatter, JSON Formatter, XML Formatter, or File Reader.

The output view for this Snap is optional. If an output view is open and the file write action is successful, the output view provides a document with information on the filename, result, and original data. An example is:

{
        "filename": "ftp://ftp.Snaplogic.com/home/qatest/user_manual.pdf",
        "result": "overwritten",
        "original": {
            "content-type" : "application/json"
        }
    }

The value of the "result" field can be "overwritten", "created", "ignored", or "appended". The value "ignored" indicates that the Snap did not overwrite the existing file because the value of the File action property is "IGNORE".

Account

This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. This Snap supports several account types, as listed in the table below, or no account. See Configuring Binary Accounts for information on setting up accounts that work with this Snap.

Account types supported by each protocol are as follows:

Azure Storage

Google Storage

Azure Data lake 

Required settings for account types are as follows:

Username, Password

Snap Settings

Label*

Default Value: File Writer
Example: File Writer

File name*

Specify the URI of the destination file to which the the data (binary input from the upstream input view) is written. It may start with one of the following protocols:

• http:
• https:
• s3:
• sftp:
• ftp:
• ftps:
• hdfs:
• sldb:
• smb:

• file: (only for use with a Groundplex)

• wasb:

• wasbs:

• gs:

• adl: 

You can also access the fields in a binary header when specifying a file name. For example, if you have a File Reader Snap upstream of a File Writer Snap, you can access the "content-location" header field to get the original path of the file. You can then use a new file name based on the old one, for instance, to make a backup file:
$['content-location'].match('/([^/]+)$')[1] + '.backup'

For http: and https: protocols, the Snap uses http PUT method only. This property should have the following syntax:

[protocol]://[host][:port]/[path]

Default Value: [None]

Examples: 

• s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>

  For region names and their details, see AWS Regions and Endpoints.
  Example: s3:///mybucket@s3.eu-west-1.amazonaws.com/test.json

• sftp://ftp.snaplogic.com:22/dir/filename
• smb://smb.Snaplogic.com:445/test_files/csv/input.csv
• A key/value pair with "filename" key should be defined as a Pipeline parameter: _filename
• If the Snap is executed in the Windows Groundplex and needs to access D: drive: file:///D:/testFolder/
• To write 'sample.csv' file into the 'testDir' folder in the 'Snaplogic' container: wasb:///snaplogic/testDir/sampl/csv 
• To read 'test.csv' file in the 'csv/' folder of the 'mybucket' bucket): gs:///mybucket/csv/test.csv
• To read the file from a location of the storage: adl://storename/folder/filename

File action*

Select this checkbox to write an empty file when the incoming binary document has empty data. If there is no incoming document at the input view of the Snap, no file is written regardless of the value of the property.

Default Value: Not selected.

Write header file

The binary data stream in the input view may contain header information about the binary data in the form of a document with key-value-pair map data.
Select this checkbox to enable the Snap to write a header file whose name is generated by appending ".header" to the value of the File name property. The same header information is also included in the output view data, as shown in the "Expected output" section above, under the key "original". Note that if the header has no keys other than Content-Type or Content-Encoding, the .header file will not be written

Default Value: Not selected 

Validate after write

Select this checkbox to enable the Snap to check if the file exists after the completion of the file write. This may delay a few more seconds for the validation.

Default Value: Not selected 

Specify the maximum number of retry attempts when the Snap fails to write. If the value is larger than 0, the Snap first stores the input data in a temporary local file before writing to the target file.

Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

Minimum value: 1

Default Value: 1
Example: 3

File permissions for various users

Use this field set to provide any combination of permissions to the available users. 

Choose one of the three standard user types. The available options are:

• owner
• group
• others

Each row can have only one user type and each user type should appear only once. Select one from the suggested list. Specify at most one row per user type.

Default Value: N/A
Example: owner

Specify the privilege to provide to the user on the file. The available permissions are:

• read
• write
• execute
• read+write
• read+execute
• write+execute
• read+write+execute

Default Value: N/A
Example: read+write+execute

Create directory if not present  

Select this checkbox to enable the Snap to create a new directory if the specified directory path does not exist. This field is not supported for HTTP, HTTPS, SLDB and SMB file protocols.

Default value: Not selected

Flush interval in kilobytes during the file upload. The Snap can flush a given size of data output stream written to the target file server. If the value is zero, the Snap flushes in maximum frequency after each byte block is written. Larger the value is, the less frequent flushes the Snap performs. Leave the property at default -1 for no flush during the upload. This property may help if the file upload experiences an intermittent failure. However, more frequent flushes will result in a slower file upload.

Default value: -1
Example: 100

Snap Execution

Examples

• For SFTP files, if you attempt to write a file into a directory where you do not have a write access right, the write operation will fail with "access denied" error. When you get an SFTP account credential, it is also important to know where your home directory is, for example, sftp://ftp.snaplogic.com/home/mrtest for username "mrtest"
• HDFS Example

For HDFS file access, please use a SnapLogic on-premises Groundplex and make sure that its instance is within the Hadoop cluster and SSH authentication has already been established. You can access HDFS files in the same way as other file protocols in File Reader and File Writer Snaps. There is no need to use any account in the Snap.

An example for HDFS is: 

hdfs://<hostname>:<port number>/<path to folder>/<filename>

hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/sample.csv

Troubleshooting

In this article

Overview

You can use the File Writer Snap to read a binary data stream from the input view and write it to a specified file destination. Possible file destinations include: SLDB, HTTP, S3, FTP, SFTP, FTPS, or HDFS. If you provide file permissions for the file, the Snap sets those permissions to the file.

Image Added

Snap Type

File Writer Snap is a Write-type Snap that writes data to the SnapLogic database or external target database.

Prerequisites

Support for Ultra Pipelines

Works in Ultra Pipelines.

Limitations

• Files uploaded to SLFS have a 100-MB per file limit. This limit does not apply when writing to external storage. 
• Do not use SLDB as a file system or storage. File Assets are intended only for specialized files that a pipeline uses to reference certain data, such as accounts, expressions, or JAR files. Use a Cloud storage provider to store production data. File Assets should not be used as a file source or as a destination in production pipelines. When you configure the Writer Snaps, set the file path to a cloud provider or external file system.
  

Known Issues

• This Snap does not create an output file when using the input from SAS Generator Snap configured with only the DELETE SAS permission. This is not the case when the target file exists.
• This Snap does not fail and turns green after execution despite providing an expired SAS URI. As a workaround, select Validate after write to fail the Snap in case of invalid credentials.
• This Snap Pack does not natively support SHA1-based algorithms to connect to SFTP endpoints. With the August 2023 GA release, you can now leverage the properties specified in the Configuration settings for Snaps to add support for ones that are disabled on your Snaplex. 

Snap Views

{
        "filename": "ftp://ftp.Snaplogic.com/home/qatest/user_manual.pdf",
        "result": "overwritten",
        "original": {
            "content-type" : "application/json"
        }
    }

Snap Settings

Video Demonstration

The following video helps to understand the new AWS canned ACL property.

Watch the following video to understand how to configure the File Writer Snap to write to the S3 Bucket.

Additional Information

Example

• For SFTP files, if you attempt to write a file into a directory where you do not have a write access right, the write operation will fail with "access denied" error. When you get an SFTP account credential, it is also important to know where your home directory is, for example, sftp://ftp.snaplogic.com/home/mrtest for username "mrtest"
• HDFS Example

For HDFS file access, please use a SnapLogic on-premises Groundplex and make sure that its instance is within the Hadoop cluster and SSH authentication has already been established. You can access HDFS files in the same way as other file protocols in File Reader and File Writer Snaps. There is no need to use any account in the Snap.

An example for HDFS is: 

hdfs://<hostname>:<port number>/<path to folder>/<filename>

hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/sample.csv

Troubleshooting

Downloads

Related Links

• Video: File Reader and Writer Demo
• Files
• Binary Snap Pack