File Writer
In this article
Overview
You can use the File Writer Snap to read a binary data stream from the input view and write it to a specified file destination. Possible file destinations include: SLDB, HTTP, S3, FTP, SFTP, FTPS, or HDFS. If you provide file permissions for the file, the Snap sets those permissions to the file.
Important
We plan to introduce additional S3 features exclusively in Amazon S3 Snaps, while Binary Snaps with S3 support will not contain these updates. Therefore, we recommend you to use the Amazon S3 Snap Pack for all your S3 operations within your pipelines. However, Binary Snaps will be retained as is to maintain backward compatibility, but be aware that we will no longer provide S3 support for the Binary Snaps.
Learn more: Migration from Binary Snaps to Amazon S3 Snaps.
Snap Type
File Writer Snap is a Write-type Snap that writes data to the SnapLogic database or external target database.
Prerequisites
IAM Roles for Amazon EC2
The 'IAM_CREDENTIAL_FOR_S3' feature is used to access S3 files from EC2 Groundplex, without Access-key ID and Secret key in the AWS S3 account in the Snap. The IAM credential stored in the EC2 metadata is used to gain access rights to the S3 buckets. To enable this feature, set the Global properties (Key-Value parameters) and restart the JCC:jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE
This feature is supported in the EC2-type Groundplex only. Learn more.
Connect to FTP server:
To connect to the FTP server that needs to reuse the session for data transfer over TLS protocol, add:
-DFTPS_SSL_TLS_PROTOCOL=TLSV1.2 (or) TLSV1.3
property as a JVM
option under the Global properties of the Node Properties tab:
Support for Ultra Pipelines
Works in Ultra Pipelines.
Limitations
- Files uploaded to SLFS have a 100-MB per file limit. This limit does not apply when writing to external storage.
- Do not use SLDB as a file system or storage. File Assets are intended only for specialized files that a pipeline uses to reference certain data, such as accounts, expressions, or JAR files. Use a Cloud storage provider to store production data. File Assets should not be used as a file source or as a destination in production pipelines. When you configure the Writer Snaps, set the file path to a cloud provider or external file system.
Known Issues
- This Snap does not create an output file when using the input from SAS Generator Snap configured with only the DELETE SAS permission. This is not the case when the target file exists.
- This Snap does not fail and turns green after execution despite providing an expired SAS URI. As a workaround, select Validate after write to fail the Snap in case of invalid credentials.
- This Snap Pack does not natively support SHA1-based algorithms to connect to SFTP endpoints. With the August 2023 GA release, you can now leverage the properties specified in the Configuration settings for Snaps to add support for ones that are disabled on your Snaplex.
Snap Views
View Type | View Format | Number of views | Examples of Upstream/Downstream Snaps | Description |
---|---|---|---|---|
Input | Binary | Min: 1 Max: 1 | Any Snap with a binary output view can be connected upstream, such as CSV Formatter, JSON Formatter, XML Formatter, or File Reader. | Any binary data stream. |
Output | Document | Min: 0 Max: 1 | Downstream Snap is optional. Any Snap with a document input view can be connected downstream. | The output view for this Snap is optional. If an output view is open and the file write action is successful, the output view provides a document with information on the filename, result, and original data. An example is: { "filename": "ftp://ftp.Snaplogic.com/home/qatest/user_manual.pdf", "result": "overwritten", "original": { "content-type" : "application/json" } } The value of the "result" field can be "overwritten", "created", "ignored", or "appended". The value "ignored" indicates that the Snap did not overwrite the existing file because the value of the File action property is "IGNORE". |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Learn more about Error handling in Pipelines. |
Snap Settings
- Asterisk ( * ): Indicates a mandatory field.
- Suggestion icon ( ): Indicates a list that is dynamically populated based on the configuration.
- Expression icon ( ): Indicates whether the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.
- Add icon ( ): Indicates that you can add fields in the fieldset.
- Remove icon (): Indicates that you can remove fields from the fieldset.
Field | Field Type | Description | ||||
---|---|---|---|---|---|---|
Label* Default Value: File Writer | String | Specify a name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline. | ||||
File name* Default Value: None Examples:
Syntax for writing files
Acceptable File Paths
|