In this article
You can use the File Writer Snap to read a binary data stream from the input view and write it to a specified file destination. Possible file destinations include: SLDB, HTTP, S3, FTP, SFTP, FTPS, or HDFS. If you provide file permissions for the file, the Snap sets those permissions to the file.
File Writer Snap is a Write-type Snap that writes data to the SnapLogic database or external target database.
Works in Ultra Pipelines.
View Type | View Format | Number of views | Examples of Upstream/Downstream Snaps | Description | |
---|---|---|---|---|---|
Input | Binary | Min: 1 Max: 1 | Any Snap with a binary output view can be connected upstream, such as CSV Formatter, JSON Formatter, XML Formatter, or File Reader. | Any binary data stream. | |
Output | Document | Min: 0 Max: 1 | Downstream Snap is optional. Any Snap with a document input view can be connected downstream. | The output view for this Snap is optional. If an output view is open and the file write action is successful, the output view provides a document with information on the filename, result, and original data. An example is:
The value of the "result" field can be "overwritten", "created", "ignored", or "appended". The value "ignored" indicates that the Snap did not overwrite the existing file because the value of the File action property is "IGNORE". | |
Error | Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the When errors occur list under the Views tab:
Learn more about Error handling in Pipelines. |
|
Field | Field Type | Description | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Label* Default Value: File Writer | String |
| |||||||||||||||
File name* Default Value: None Examples:
| String | Specify the URI of the destination file to which the data (binary input from the upstream input view) is written. It may start with one of the following protocols:
This Snap supports S3 Virtual Private Cloud (VPC) endpoints. For example:
This Snap supports Oracle Object Storage endpoints when used with pre-authenticated requests. For example:
To create a pre-authenticated request, refer to the instructions in the following Oracle article:
Learn more: Additional Information.
| |||||||||||||||
Prevent URL encoding | Checkbox | Select this checkbox to use the file path value as-is without encoding the URL. This prevents the Snap from encoding the file path URL (including the query string if any) automatically. Deselect this checkbox to automatically encode the characters in the File URL. | |||||||||||||||
File action* Default Value: Overwrite | Dropdown list | Specify the action to perform if the file already exists. The available options are:
| |||||||||||||||
Write empty file Default Value: Deselected | Checkbox | Select this checkbox to write an empty file when the incoming binary document has empty data. If there is no incoming document at the input view of the Snap, no file is written regardless of the value of the property. | |||||||||||||||
Write header file Default Value: Deselected | Checkbox | Select this checkbox to allow the Snap to write a header file by appending ".header" to the value of the File name property. The same header information is also included in the output view data, as shown in the "Expected output" section above, under the key
| |||||||||||||||
Validate after write Default Value: Deselected | Checkbox | Select this checkbox to enable the Snap to check if the file exists after the completion of the file write. This may delay a few more seconds for the validation. | |||||||||||||||
Number of retries Default Value:0 | Integer/Expression | Specify the maximum number of retry attempts to make when the Snap fails to write. If the value is larger than 0, the Snap first stores the input data in a temporary local file before writing to the target file.
Minimum value: 0 | |||||||||||||||
Retry interval (seconds) Default Value: 1 | Integer/Expression | Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure. Minimum value: 1 | |||||||||||||||
File permissions for various users | Use this field set to provide any combination of permissions to the available users.
| ||||||||||||||||
User type Default Value: N/A | String/Expression/Suggestion | Choose one of the three standard user types. The available options are:
Each row can have only one user type and each user type should appear only once. Select one from the suggested list. Specify at most one row per user type. | |||||||||||||||
File permissions Default Value: N/A | String/Expression/Suggestion | Specify the privilege to provide to the user on the file. The available permissions are:
| |||||||||||||||
Create directory if not present Default Value: Deselected | Checkbox | Select this checkbox to enable the Snap to create a new directory if the specified directory path does not exist.
| |||||||||||||||
Flush interval (kB) Default Value:-1 | Integer | Specify the flush interval in kilobytes during the file upload.
| |||||||||||||||
Advanced properties | Use this field set to add advanced properties, such as SAS URI. | ||||||||||||||||
Properties Default Value: SAS URI | String | ||||||||||||||||
Values Default Value: N/A | String/Expression | Specify the value for the SAS URI.
| |||||||||||||||
AWS Canned ACL Default Value: None | Dropdown list | This field appears only when your account type is set to AWS S3. Select the predefined ACL grant (from AWS) to use when writing a file to S3. Choose a Canned ACL from the available options:
Watch the video below for more information about AWS Canned ACL. Learn more: AWS Canned ACLs. | |||||||||||||||
Snap Execution Default Value: Execute only | Dropdown list |
The following video helps to understand the new AWS canned ACL property.
Watch the following video to understand how to configure the File Writer Snap to write to the S3 Bucket.
You can also access the fields in a binary header when specifying a file name. For example, if you have a File Reader Snap upstream of a File Writer Snap, you can access the "content-location" header field to get the original path of the file. You can then use a new file name based on the old one, for instance, to make a backup file: For http: and https: protocols, the Snap uses http PUT method only. This property should have the following syntax:
|
For HDFS file access, please use a SnapLogic on-premises Groundplex and make sure that its instance is within the Hadoop cluster and SSH authentication has already been established. You can access HDFS files in the same way as other file protocols in File Reader and File Writer Snaps. There is no need to use any account in the Snap.
HDFS 2.4.0 is supported for the hdfs protocol. |
An example for HDFS is:
hdfs://<hostname>:<port number>/<path to folder>/<filename> |
hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/sample.csv |
Example pipeline file for an SLDB file writer is as shown below: |
Error | Reason | Resolution |
---|---|---|
| The library that we use for SFTP connections no longer supports deprecated signature protocols by default. (This changed with the 4.33 GA release.) | Add the algorithm to the serverProposal in the global.properties file.
Learn more: Configuration Options |
Could not evaluate expression: filepath Mismatched input ':' expecting {<EOF>, '||', '&&', '^', '==', '!=', '>', '<', '>=', '<=', '+', '-', '*', '/', '%', '?', '[', PropertyRef}. | The expression toggle (=) is selected on the File name field, so it is trying to evaluate the file path as an expression. | Check the expression syntax. Disable the expression toggle to remove the field out of expression mode. |
Failure: | The expression toggle (=) is selected on the File name field, so it is trying to evaluate the filename as an expression. | Check expression syntax and data types. Disable the expression toggle to remove the field value out of expression mode. |