File Writer

In this article

Overview

You can use the File Writer Snap to read a binary data stream from the input view and write it to a specified file destination. Possible file destinations include: SLDB, HTTP, S3, FTP, SFTP, FTPS, or HDFS. If you provide file permissions for the file, the Snap sets those permissions to the file.

Important

We plan to introduce additional S3 features exclusively in Amazon S3 Snaps, while Binary Snaps with S3 support will not contain these updates. Therefore, we recommend you to use the Amazon S3 Snap Pack for all your S3 operations within your pipelines. However, Binary Snaps will be retained as is to maintain backward compatibility, but be aware that we will no longer provide S3 support for the Binary Snaps.

Learn more: Migration from Binary Snaps to Amazon S3 Snaps.

Snap Type

File Writer Snap is a Write-type Snap that writes data to the SnapLogic database or external target database.

Prerequisites

IAM Roles for Amazon EC2

The 'IAM_CREDENTIAL_FOR_S3' feature is used to access S3 files from EC2 Groundplex, without Access-key ID and Secret key in the AWS S3 account in the Snap. The IAM credential stored in the EC2 metadata is used to gain access rights to the S3 buckets. To enable this feature, the following line should be added to global.properties and the jcc (node) restarted:
jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE

Please note this feature is supported in the EC2-type Groundplex only.

For more information on IAM Roles, see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html


Connect to FTP server:

To connect to the FTP server that needs to reuse the session for data transfer over TLS protocol, add:

-DFTPS_SSL_TLS_PROTOCOL=TLSV1.2  (or) TLSV1.3property as a JVM option under the Global properties of the Node Properties tab:

Support for Ultra Pipelines

Works in Ultra Pipelines.

Limitations

  • Files uploaded to SLFS have a 100-MB per file limit. This limit does not apply when writing to external storage. 
  • Do not use SLDB as a file system or storage. File Assets are intended only for specialized files that a pipeline uses to reference certain data, such as accounts, expressions, or JAR files. Use a Cloud storage provider to store production data. File Assets should not be used as a file source or as a destination in production pipelines. When you configure the Writer Snaps, set the file path to a cloud provider or external file system.

Known Issues

  • This Snap does not create an output file when using the input from SAS Generator Snap configured with only the DELETE SAS permission. This is not the case when the target file exists.
  • This Snap does not fail and turns green after execution despite providing an expired SAS URI. As a workaround, select Validate after write to fail the Snap in case of invalid credentials.
  • This Snap Pack does not natively support SHA1-based algorithms to connect to SFTP endpoints. With the August 2023 GA release, you can now leverage the properties specified in the Configuration settings for Snaps to add support for ones that are disabled on your Snaplex. 

Snap Views

View TypeView FormatNumber of viewsExamples of Upstream/Downstream SnapsDescription
InputBinary
Min: 1
Max: 1

Any Snap with a binary output view can be connected upstream, such as CSV Formatter, JSON Formatter, XML Formatter, or File Reader.

Any binary data stream.
OutputDocument
Min: 0
Max: 1
Downstream Snap is optional. Any Snap with a document input view can be connected downstream.

The output view for this Snap is optional. If an output view is open and the file write action is successful, the output view provides a document with information on the filename, result, and original data. An example is:

{
        "filename": "ftp://ftp.Snaplogic.com/home/qatest/user_manual.pdf",
        "result": "overwritten",
        "original": {
            "content-type" : "application/json"
        }
    }

The value of the "result" field can be "overwritten", "created", "ignored", or "appended". The value "ignored" indicates that the Snap did not overwrite the existing file because the value of the File action property is "IGNORE".

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution when the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the rest of the records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk ( * ): Indicates a mandatory field.
  • Suggestion icon ( ): Indicates a list that is dynamically populated based on the configuration.
  • Expression icon (  ): Indicates whether the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
  • Add icon (  ): Indicates that you can add fields in the fieldset.
  • Remove icon (): Indicates that you can remove fields from the fieldset.


FieldField TypeDescription

Label*


Default ValueFile Writer
Example
File Writer

String

Specify a name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

File name*


Default ValueNone

Examples

  • s3:///<S3_bucket_name>@s3.<region_name>.amazonaws.com/<path>sftp://ftp.snaplogic.com:22/dir/filename

  • smb://smb.Snaplogic.com:445/test_files/csv/input.csv


Syntax for writing files

  • For SL DB: <filename>.<file extension>. For example, employee_details.json, contacts.csv

  • For S3: S3\\<project name>\\folder. For example, s3://snaplogic/test

  • For Staging: stageName. For example, public.my_s3_stage

Acceptable File Paths

  • Relative paths

    • filename.json: Saves the file in the project.

    • ../shared/filename.json: Saves the file in the Project Shared Space.

    • ../../shared/filename.json: Saves the files in the Org Shared project.

  • Absolute path

    • /<org>/<projectSpace>/<project>/filename.json

Lint Warning

The Snap displays a Lint Warning in your Pipeline in the following scenarios:

  • INCORRECT_FILE_PATH: When you provide an incorrect file path to write a file, such as not following the given relative paths pattern or absolute path patterns.

  • INCORRECT_ORG_PATHWhen you create files or directories in a different org other than the one where the Pipeline is executing.

Therefore, we recommend that you confirm to any of the acceptable relative paths. Otherwise, use an absolute path—that is provide a file path that belongs to the same org where you want to write the file, or use the File Upload icon to specify the File path.

Writing Files in SLDB

Keep in mind the following for SLDB files:

  • If you enter a file name, such as file.csv, then it writes the file to: /<org>/projects/<pipeline project>/file1.csv (where <org> is your organization name and <pipeline project> is the project where the Pipeline is stored), if the Pipeline is in a project other than the shared project.

  • If you enter shared/file1.csv, then it writes the file to: /<org>/shared/file1.csv.

Note: The Snap can write a file to its own project directory or the shared project, and cannot write it to another project directory.

Writing Files in S3

To write files in S3, your account must have full access.