In this article

Table of Contents

maxLevel	2
exclude	Older Versions\|Additional Resources\|Related Links\|Related Information

Overview

You can use the File Writer Snap to read a binary data stream from the input view and write it to a specified file destination. Possible file destinations include: SLDB, HTTP, S3, FTP, SFTP, FTPS, or HDFS. If you provide file permissions for the file, the Snap sets those permissions to the file.

Multiexcerpt include macro

name	Migrating from Binary to Amazon S3
templateData	[]
page	File Reader
addpanel	false

Snap Type

File Writer Snap is a Write-type Snap that writes data to the SnapLogic database or external target database.

Prerequisites

Multiexcerpt include macro

name	EC2Prerequisite
page	File Reader

Multiexcerpt include macro

name	FTPS_Prerequisite
templateData	eJyLjgUAARUAuQ==
page	Directory Browser
addpanel	false

Support for Ultra Pipelines

Works in Ultra Pipelines.

Limitations

Files uploaded to SLFS have a 100-MB per file limit. This limit does not apply when writing to external storage.
Do not use sldbSLDB as a file system or storage. File Assets are intended only for specialized files that a pipeline uses to reference certain data, such as accounts, expressions, or JAR files. Use a Cloud storage provider to store production data. File Assets should not be used as a file source or as a destination in production pipelines. When you configure the Writer Snaps, set the file path to a cloud provider or external file system.

Known Issues

This Snap does not create an output file when using the input from SAS Generator Snap configured with only the DELETE SAS permission. This is not the case when the target file exists.
This Snap does not fail and turns green after execution despite providing an expired SAS URI. As a workaround, select Validate after write to fail the Snap in case of invalid credentials.
This Snap Pack does not natively support SHA1-based algorithms to connect to SFTP endpoints. With the August 2023 GA release, you can now leverage the properties specified in the Configuration settings for Snaps to add support for ones that are disabled on your Snaplex.

Snap Views

View Type

View Format

Number of views

Examples of Upstream/Downstream Snaps

Description

Input

Binary

Min: 1

Max: 1

Any Snap with a binary output view can be connected upstream, such as CSV Formatter, JSON Formatter, XML Formatter, or File Reader.

Any binary data stream.

Output

Document

Min: 0

Max: 1

Downstream Snap is optional. Any Snap with a document input view can be connected downstream.

The output view for this Snap is optional. If an output view is open and the file write action is successful, the output view provides a document with information on the filename, result, and original data. An example is:

Code Block
{ "filename": "ftp://ftp.Snaplogic.com/home/qatest/user_manual.pdf", "result": "overwritten", "original": { "content-type" : "application/json" } }

The value of the "result" field can be "overwritten", "created", "ignored", or "appended". The value "ignored" indicates that the Snap did not overwrite the existing file because the value of the File action property is "IGNORE".

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

Stop Pipeline Execution: Stops the current pipeline execution when the Snap encounters an error.
Discard Error Data and Continue: Ignores the error, discards that record, and continues with the rest of the records.
Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

Info

Asterisk ( * ): Indicates a mandatory field.
Suggestion icon ( ): Indicates a list that is dynamically populated based on the configuration.
Expression icon ( ): Indicates whether the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
Add icon ( ): Indicates that you can add fields in the fieldset.
Remove icon (): Indicates that you can remove fields from the fieldset.

Specify the URI of the destination file to which the data (binary input from the upstream input view) is written. It may start with one of the following protocols:

http:

https:

s3:

sftp:

ftp:

ftps:

hdfs:

sldb:

smb:

file: (only for use with a Groundplex)

wasb:

wasbs:

gs:

adl:

This Snap supports S3 Virtual Private Cloud (VPC) endpoints. For example:

s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com

This Snap supports Oracle Object Storage endpoints when used with pre-authenticated requests. For example:

https://objectstorage.us-sanjose-1.oraclecloud.com/p/123AbcdEFG12345_xyz123/n/MyNamespace/b/snaplogic-academy/o/sample.json

To create a pre-authenticated request, refer to the instructions in the following Oracle article:
Using Pre-Authenticated Requests.

Field Field Type Description

Label*

Default Value: File Writer
Example: File Writer

String

Excerpt
Specify a name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

File name*

Default Value: None

Examples:

s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>sftp://ftp.snaplogic.com:22/dir/filename
smb://smb.Snaplogic.com:445/test_files/csv/input.csv

String

Info

icon	false
title	File Patterns

A key-value pair with "filename" key can be defined as a Pipeline parameter: _filename
If the Snap is executed in the Windows Groundplex and needs to access D: drive, the format should be: file:///D:/testFolder/
To write 'sample.csv' file into the 'testDir' folder in the 'Snaplogic' container: wasb:///snaplogic/testDir/sample.csv
To read 'test.csv' file in the 'csv/' folder of the 'mybucket' bucket): gs:///mybucket/csv/test.csv
To read the file from a location of the storage: adl://storename/folder/filename
For region names and their details, see AWS Regions and Endpoints.
Example: s3:///mybucket@s3.eu-west-1.amazonaws.com/test.json
Ensure the file name does not contain '?' character, because it is not supported. The Snap fails with an error if you include the '?' character.

Info

title	Syntax for writing files

For SL DB: <filename>.<file extension>. For example, employee_details.json, contacts.csv
For S3: S3\\<project name>\\folder. For example, s3://snaplogic/test
For Staging: stageName. For example, public.my_s3_stage

Multiexcerpt macro

name	Acceptable File Paths

Acceptable File Paths

Relative paths
- filename.json: Saves the file in the project.
- ../shared/filename.json: Saves the file in the Project Shared Space.
- ../../shared/filename.json: Saves the files in the Org Shared project.
Absolute path
- /<org>/<projectSpace>/<project>/filename.json

Multiexcerpt macro

name	Lint Warning

Lint Warning

The Snap displays a Lint Warning in your Pipeline in the following scenarios:

INCORRECT_FILE_PATH: When you provide an incorrect file path to write a file, such as not following the given relative paths pattern or absolute path patterns.
INCORRECT_ORG_PATH: When you create files or directories in a different org other than the one where the Pipeline is executing.

Therefore, we recommend that you confirm to any of the acceptable relative paths. Otherwise, use an absolute path—that is provide a file path that belongs to the same org where you want to write the file, or use the File Upload icon to specify the File path. Multiexcerpt macro

name	Writing Files in SLDB

falseWriting Files in SLDB

Keep in mind the following for SLDB files:

If you enter a file name, such as file.csv, then it writes the file to: /<org>/projects/<pipeline project>/file1.csv (where <org> is your organization name and <pipeline project> is the project where the Pipeline is stored), if the Pipeline is in a project other than the shared project.
If you enter shared/file1.csv, then it writes the file to: /<org>/shared/file1.csv.

Note: The Snap can write a file to its own project directory or the shared project, and cannot write it to another project directory.

Info

icon	false
title	Writing Files in S3

To write files in S3, your account must have full access.

Note

When using expressions to build a file name, ensure that the resulting filename does not contain characters that are not supported by the target platform.
- The following characters are illegal in a share name: \ / [ ] : | < > + = ; , * ? "
- The following characters are not allowed in SMB share name: " \ / : | < > * ?

Learn more about referencing SMB file names: Microsoft's documentation.

Learn more: Additional Information.

Info

Reading files from Project and Shared Project Spaces

If a Pipeline is created in a project other than the shared project and you want to read the "asset.json" file from the same project, enter "asset.json" or "sldb:///asset.json".

If a Pipeline is created in the shared project and you want to read the "asset.json" file from the shared project, enter "asset.json" or "sldb:///asset.json".

Info

title	Syntax for writing files

For SL DB: <filename>.<file extension>. For example, employee_details.json, contacts.csv
For S3: S3\\<project name>\\folder. For example, s3://snaplogic/test
For Staging: stageName. For example, public.my_s3_stage

Multiexcerpt macro

name	Acceptable File Paths

Acceptable File Paths

Relative paths
- filename.json: Saves the file in the project.
- ../shared/filename.json: Saves the file in the Project Shared Space.
- ../../shared/filename.json: Saves the files in the Org Shared project.
Absolute path
- /<org>/<projectSpace>/<project>/filename.json

Multiexcerpt macro

name	Lint Warning

Lint Warning

The Snap displays a Lint Warning in your Pipeline in the following scenarios:

INCORRECT_FILE_PATH: When you provide an incorrect file path to write a file, such as not following the given relative paths pattern or absolute path patterns.
INCORRECT_ORG_PATH: When you create files or directories in a different org other than the one where the Pipeline is executing.

Therefore, we recommend that you confirm to any of the acceptable relative paths. Otherwise, use an absolute path—that is provide a file path that belongs to the same org where you want to write the file, or use the File Upload icon to specify the File path.

Multiexcerpt macro

name	Writing Files in SLDB

falseWriting Files in SLDB

Keep in mind the following for SLDB files:

If you enter a file name, such as file.csv, then it writes the file to: /<org>/projects/<pipeline project>/file1.csv (where <org> is your organization name and <pipeline project> is the project where the Pipeline is stored), if the Pipeline is in a project other than the shared project.
If you enter shared/file1.csv, then it writes the file to: /<org>/shared/file1.csv.

Note: The Snap can write a file to its own project directory or the shared project, and cannot write it to another project directory.

falseWriting Files in S3

To write files in S3, your account must have full access.

String

Specify the URI of the destination file to which the data (binary input from the upstream input view) is written. It may start with one of the following protocols:

http:
https:
s3:
sftp:
ftp:
ftps:
hdfs:
sldb:
smb:
file: (only for use with a Groundplex)
wasb:
wasbs:
gs:
adl:

This Snap supports S3 Virtual Private Cloud (VPC) endpoints. For example:

s3://my-bucket@bucket.vpce-028b7814794578709-vu0vvauy.s3.us-west-2.vpce.amazonaws.com

This Snap supports Oracle Object Storage endpoints when used with pre-authenticated requests. For example:

https://objectstorage.us-sanjose-1.oraclecloud.com/p/123AbcdEFG12345_xyz123/n/MyNamespace/b/snaplogic-academy/o/sample.json

To create a pre-authenticated request, refer to the instructions in the following Oracle article:
Using Pre-Authenticated Requests.

Info

icon	false
title	File Patterns

A key-value pair with "filename" key can be defined as a Pipeline parameter: _filename
If the Snap is executed in the Windows Groundplex and needs to access D: drive, the format should be: file:///D:/testFolder/
To write 'sample.csv' file into the 'testDir' folder in the 'Snaplogic' container: wasb:///snaplogic/testDir/sample.csv
To read 'test.csv' file in the 'csv/' folder of the 'mybucket' bucket): gs:///mybucket/csv/test.csv
To read the file from a location of the storage: adl://storename/folder/filename
For region names and their details, see AWS Regions and Endpoints.
Example: s3:///mybucket@s3.eu-west-1.amazonaws.com/test.json
Ensure the file name does not contain '?' character, because it is not supported. The Snap fails with an error if you include the '?' character.

Note

When using expressions to build a file name, ensure that the resulting filename does not contain characters that are not supported by the target platform.
- The following characters are illegal in a share name: \ / [ ] : | < > + = ; , * ? "
- The following characters are not allowed in SMB share name: " \ / : | < > * ?

Learn more about referencing SMB file names: Microsoft's documentation.

Learn more: Additional Information.

Info

Reading files from Project and Shared Project Spaces

falseWriting Files in SLDB

Keep in mind the following for SLDB files:

If you enter a file name, such as file.csv, then it writes the file to: /<org>/projects/<pipeline project>/file1.csv (where <org> is your organization name and <pipeline project> is the project where the Pipeline is stored), if the Pipeline is

If a Pipeline is created in a project other than the shared project and you want to read the "asset.json" file from the shared same project, enter "shared/asset.json" or "sldb:///shared/asset.json".
Ensure the file name, folder name, or the file path does not contain '?' character because it is not fully supported and when present, the Snap might fail.

Multiexcerpt macro

name	Writing Files in SLDB

If a Pipeline is created in the shared project and you want to read the "asset.json" file from the shared project, enter "asset.json" or "sldb:///asset.json".
If a Pipeline is created in a project other than the shared project

.

If you enter shared/file1.csv, then it writes the file to: /<org>/shared/file1.csv.

Note: The Snap can write a file to its own project directory or the shared project, and cannot write it to another project directory

and you want to read the "asset.json" file from the shared project, enter "shared/asset.json" or "sldb:///shared/asset.json".
Ensure the file name, folder name, or the file path does not contain '?' character because it is not fully supported and when present, the Snap might fail.

Prevent URL encoding

Checkbox

Select this checkbox to use the file path value as-is without encoding the URL. This prevents the Snap from encoding the file path URL (including the query string if any) automatically. Deselect this checkbox to automatically encode the characters in the File URL.

File action*

Default Value: Overwrite
Example: Overwrite

Dropdown list

Specify the action to perform if the file already exists. The available options are:

Overwrite - The Snap attempts to write the file without checking for the file's existence for a better performance, and the file is overwritten.
Append - Append is supported for file, FTP, FTPS and SFTP protocols only.
Ignore - The Snap does not overwrite the file and will do nothing but write the status and file name to its output view.
Error - The error displays in the Pipeline Run Log. If an error view is defined, the error will be written there as well.

Note
In case you are selecting Append as File action, then ensure that the ‘Number of retries’ is 0 or not greater than 0. For wasb:// and wasbs:// file protocols, only the Overwrite file action is supported.

Info

The Append operation is supported for FILE, SFTP, FTP, FTPS and ADL protocols only. For any other protocols that are not supported by Append, we recommend that you use the File Operations, File Writer, and File Delete Snaps and follow this procedure.

Copy the blob file to your local drive from the endpoint.
Append additional data to the local file.
Delete the original file on the endpoint.
Copy the modified temporary file back from the source to the target.

Note: This approach might involve disk overhead, therefore ensure that you have enough disk space in your system.

Write empty file

Default Value: Not selectedDeselected

Checkbox

Select this checkbox to write an empty file when the incoming binary document has empty data. If there is no incoming document at the input view of the Snap, no file is written regardless of the value of the property.

Write header file

Default Value: Not selected Deselected

Checkbox

Select this checkbox to allow the Snap to write a header file by appending ".header" to the value of the File name property. The same header information is also included in the output view data, as shown in the "Expected output" section above, under the key original.

Note
The binary data stream in the input view may contain header information about the binary data in the form of a document with key-value-pair map data. If the header has no keys other than Content-Type or Content-Encoding, the .header file will not be written.

Validate after write

Default Value: Deselected

Checkbox

Select this checkbox to enable the Snap to check if the file exists after the completion of the file write. This may delay a few more seconds for the validation.

Number of retries

Default Value:0
Example:3

Integer/Expression

Specify the maximum number of retry attempts to make when the Snap fails to write. If the value is larger than 0, the Snap first stores the input data in a temporary local file before writing to the target file.

Info
Ensure that the local drive has sufficient free disk space as large as the expected target file size. Owing to the nature of retry mechanism, the Snap does not support the Append operation if the Number of retries is set to more than zero.

Minimum value: 0

Retry interval (seconds)

Default Value: 1
Example: 3

Integer/Expression

Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

Minimum value: 1

File permissions for various users

Use this field set to provide any combination of permissions to the available users.

Note
Supported for sftp, ftp, ftps, file, and hdfs protocols only. FTP/FTPS servers on Windows machines are not supported.

User type

Default Value: N/A
Example: owner

String/Expression/Suggestion

Choose one of the three standard user types. The available options are:

owner
group
others

Each row can have only one user type and each user type should appear only once. Select one from the suggested list. Specify at most one row per user type.

File permissions

Default Value: N/A
Example: read+write+execute

String/Expression/Suggestion

Specify the privilege to provide to the user on the file. The available permissions are:

read
write
execute
read+write
read+execute
write+execute
read+write+execute

Create directory if not present

Default Value: Deselected

Checkbox

Select this checkbox to enable the Snap to create a new directory if the specified directory path does not exist.
This field supports FTP, SFTP, and SMB protocols and protocols—they allow creation of directories when they are not present. However, this field does not support HTTP, HTTPS, and SLDB file protocols.

Info

file:/// protocol is not supported in Windows Operating System, hence do not select this checkbox, else the Snap displays an error.
For some file protocols such as, file:///(Linux), s3:///, wasb:///, and wasbs:/// if you deselect this checkbox, the Snap automatically creates a new directory as per the corresponding file system when there is no directory present. So, we recommend you not to select this checkbox if you are using the afore mentioned file protocols.

Flush interval (kB)

Default Value:-1
Example:100

Integer

Specify the flush interval in kilobytes during the file upload.

Info

The Snap can flush a given size of data output stream written to the target file server. If the value is zero, the Snap flushes in maximum frequency after each byte block is written. Larger the value is, the less frequent flushes the Snap performs. Leave the property at default -1 for no flush during the upload. This property may help if the file upload experiences an intermittent failure. However, more frequent flushes will result in a slower file upload.

Advanced properties

Use this field set to add advanced properties, such as SAS URI.

Properties

Default Value: SAS URI
Example: SAS URI

String

Multiexcerpt include macro

name	SASURI_Description
page	File Reader

Values

Default Value: N/A
Example:
https://myaccount.blob.core.windows.net/sascontainer/
sasblob.txt?sv=2015-04-05&st=2015-04-29T22%3A18%3A26Z&seD

String/Expression

Specify the value for the SAS URI.

Note
When you specify the SAS URI value in the Snap settings, then the settings provided in the account (if any account is attached) are ignored.

AWS Canned ACL

Default Value: None
Example: PublicRead

Dropdown list

This field appears only when your account type is set to AWS S3.

Select the predefined ACL grant (from AWS) to use when writing a file to S3. Choose a Canned ACL from the available options:

None
Private
PublicRead
PublicReadWrite
AuthenticatedRead
LogDeliveryWrite
BucketOwnerRead
BucketOwnerFullControl
AwsExecRead

Watch the video below for more information about AWS Canned ACL. Learn more: AWS Canned ACLs.

Snap Execution

Default Value: Execute only
Example: Validate & Execute

Dropdown list

Multiexcerpt include macro

name	Execution_Detail_Write
page	SOAP Execute

Video Demonstration

The following video helps to understand the new AWS canned ACL property.

View file

name	Set Canned ACL.mp4
height	250

Watch the following video to understand how to configure the File Writer Snap to write to the S3 Bucket.

View file

name	Configure Snap to Write to S3 Bucket.mp4
height	250

Additional Information

Info

icon	false

You can also access the fields in a binary header when specifying a file name. For example, if you have a File Reader Snap upstream of a File Writer Snap, you can access the "content-location" header field to get the original path of the file. You can then use a new file name based on the old one, for instance, to make a backup file:
$['content-location'].match('/([^/]+)$')[1] + '.backup'

For http: and https: protocols, the Snap uses http PUT method only. This property should have the following syntax:

[protocol]://[host][:port]/[path]

Note

"://" is a separator between the file protocol and the rest of the URL and the host name and the port number should be between "://" and "/". If the port number is left empty, a default port number for the protocol is used. The hostname and port number are left empty in the sldb and S3 protocols. The value in this field should be an absolute path for all protocols except SLDB.
When using the file:/// protocol, the file is accessed based on the permissions of the user associated with the Snaplex (Default: Snapuser).
For HDFS, if you want to be able to suggest information, use the HDFS Writer Snap.

Warning
Use the file system access with caution, and ensure to clean up the file system after use.

Example

For SFTP files, if you attempt to write a file into a directory where you do not have a write access right, the write operation will fail with "access denied" error. When you get an SFTP account credential, it is also important to know where your home directory is, for example, sftp://ftp.snaplogic.com/home/mrtest for username "mrtest"
HDFS Example

For HDFS file access, please use a SnapLogic on-premises Groundplex and make sure that its instance is within the Hadoop cluster and SSH authentication has already been established. You can access HDFS files in the same way as other file protocols in File Reader and File Writer Snaps. There is no need to use any account in the Snap.

Note
HDFS 2.4.0 is supported for the hdfs protocol.

An example for HDFS is:

Code Block
hdfs://<hostname>:<port number>/<path to folder>/<filename>

If Cloudera Hadoop Name node is installed in AWS EC2 and its hostname is "ec2-54-198-212-134.compute-1.amazonaws.com" and its port number is 8020, then you would enter:

Code Block
hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/sample.csv

Expand

title	Example pipeline file for an sldb file write

Example pipeline file for an SLDB file writer is as shown below:

Image Modified

Troubleshooting

Error Reason Resolution

Algorithm negotiation fail: algorithmName="server_host_key" jschProposal="<algorithms>" serverProposal="ssh-rsa"

The library that we use for SFTP connections no longer supports deprecated signature protocols by default. (This changed with the 4.33 GA release.)

Add the algorithm to the serverProposal in the global.properties file.

You can also enable support for RSA-SHA1 authentication in the Node Properties tab on the Update Snaplex dialog in SnapLogic Manager.

In the Node Properties tab of your target Snaplex, add the following key/value pair under Global Properties:
Key: jcc.jvm_options
Value: -Dsftp.server_host_key=ssh-rsa -Dsftp.client_pubkey=ssh-rsa
Click Update, and then restart the Snaplex node.

Learn more: Configuration Options

Could not evaluate expression: filepath

Mismatched input ':' expecting {<EOF>, '||', '&&', '^', '==', '!=', '>', '<', '>=', '<=', '+', '-', '*', '/', '%', '?', '[', PropertyRef}.

The expression toggle (=) is selected on the File name field, so it is trying to evaluate the file path as an expression.

Check the expression syntax.

Disable the expression toggle to remove the field out of expression mode.

Failure:
filename is undefined
filename was not found in the containing object.

The expression toggle (=) is selected on the File name field, so it is trying to evaluate the filename as an expression.

Check expression syntax and data types.

Disable the expression toggle to remove the field value out of expression mode.

Downloads

Multiexcerpt include macro

name	download_instructions
page	OpenAPI

Attachments

upload	false
old	false
patterns	.slp, .zip

Versions Compared

Old Version 133

New Version 134

Key

Overview

Snap Type

Prerequisites

Support for Ultra Pipelines

Limitations

Known Issues

Snap Views

Snap Settings

Video Demonstration

Additional Information

Example

Troubleshooting

Downloads

Related Links

Page Comparison

Versions Compared

Old Version 133

New Version 134

Key

Overview

Snap Type

Prerequisites

Support for Ultra Pipelines

Limitations

Known Issues

Snap Views

Snap Settings

Video Demonstration

Additional Information

Example

Troubleshooting

Downloads

Related Links