Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

On this Page

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

Snap type:

Write

Description:

This Snap reads a binary data stream from the input view and writes it to a specified file destination. Possible file destinations include: SLDB, HTTP, S3, FTP, SFTP, FTPS, or HDFS. If File permissions for the file are provided, the Snap set those permissions to the file.

Info

This Snap has the ability to use an MD5 checksum that automatically checks for data integrity and corruption while uploading the file.  

Expected upstream Snaps:

In this article

Table of Contents
maxLevel2
excludeOlder Versions|Additional Resources|Related Links|Related Information

Overview

You can use this Snap to read a binary data stream from the input view and write it to a specified file destination. Possible file destinations include: SLDB, HTTP, S3, FTP, SFTP, FTPS, or HDFS. If File permissions for the file are provided, the Snap set those permissions to the file. This is a Write Snap type.

Info

This Snap has the ability to use an MD5 checksum that automatically checks for data integrity and corruption while uploading the file.  

Prerequisites

Multiexcerpt include macro
nameEC2Prerequisite
pageFile Reader

Support for Ultra Pipelines

Works in Ultra Pipelines.

Limitations

Files uploaded to SLFS have a 100-MB per file limit. This limit does not apply when writing to external storage. 

Known Issues

  • This Snap does not create an output file when using the input from SAS Generator Snap configured with only the DELETE SAS permission. This is not the case when the target file exists.
  • This Snap does not fail and turns green after execution despite providing an expired SAS URI. As a workaround, select Validate after write to fail the Snap in case of invalid credentials.

Snap Input and Output

Input/OutputType of viewNumber of viewsExamples of Upstream/Downstream SnapsDescription
InputBinary
Min: 1
Max: 1

Any Snap with a binary output view can be connected upstream, such

as CSV

as CSV Formatter, JSON Formatter, XML Formatter, or File Reader.

Expected downstream Snaps: Downstream

Any binary data stream
OutputDocument
Min: 0
Max: 1
Downstream Snap is optional. Any Snap with a document input view can be connected downstream.
  • Expected input: Any binary data stream
  • Expected output: The

    The output view for this Snap is optional. If an output view is open and the file write action is successful, the output view provides a document with information on the filename, result, and original data. An example is:

    Code Block
    {
            "filename": "ftp://ftp.Snaplogic.com/home/qatest/user_manual.pdf",
            "result": "overwritten",
            "original": {
                "content-type" : "application/json"
            }
        }

    The value of the "result" field can be "overwritten", "created", "ignored", or "appended". The value "ignored" indicates that the Snap did not overwrite the existing file because the value of the File action property is "IGNORE".

    Warning

    Do not use the 423patches7923 build for the File

    Writer Snap, because it causes connectivity issues with the Snaplex nodes.Prerequisites:

    Multiexcerpt include macro
    nameEC2Prerequisite
    pageFile Reader

    Support and limitations:
  • Works in Ultra Pipelines.
  • Files uploaded to SLFS have a 100-MB per file limit. 

    Note
    titleNote

    This limit does not apply when writing to external storage. 

    Known Issues:This Snap does not create an output file when using the input from SAS Generator Snap configured with only the DELETE SAS permission. This is not the case when the target file exists.Account: 

    Writer Snap, because it causes connectivity issues with the Snaplex nodes.


    Account

    This Snap uses account references created on the Accounts page of SnapLogic Manager to handle access to this endpoint. This Snap supports several account types, as listed in the table below, or no account. See Configuring Binary Accounts for information on setting up accounts that work with this Snap.

    Account types supported by each protocol are as follows:

    Protocol
     Account
    Account types
    sldbno account
    s3AWS S3, S3 Dynamic
    ftpBasic Auth
    sftpBasic Auth, SSH Auth 
    ftpsBasic Auth
    hdfsno account
    httpno account
    httpsoptional account
    smbSMB
    fileno account
    wasbAzure Storage
    wasbs

    Azure Storage

    gs

    Google Storage

    adl

    Azure Data lake 


    Note

    The FTPS file protocol works only in explicit mode. The implicit mode is not supported.

    Required settings for account types are as follows:

     
    Account
    Type
    type
     Settings
    Settings
    Basic Auth

    Username, Password

    AWS S3Access-key ID, Secret key, Server-side encryption
    S3 DynamicAccess-key ID, Secret key, Security token, Server-side encryption
    SSH AuthUsername, Private key, Key Passphrase
    SMBDomain, Username, Password
    Azure StorageAccount name, Primary access key
    Google StorageApproval prompt, Application scope, Auto-refresh token
    Azure Data Lake Tenant ID, Access ID, Secret Key
      


    Note

    SnapLogic automatically appends "azuredatalakestore.net" to the store name you specify when using Azure Data Lake; therefore, you do not need to add 'azuredatalakestore.net' to the URI while specifying the directory.

    Views:Input

    This Snap has exactly one binary input view, where it gets the binary data to be written to the file specified in the File Name property.OutputThis Snap has at most one document output view.ErrorThis Snap has at most one document error view and produces zero or more documents in the view.Settings

    Snap Settings

    File permissions for various users

    Set the file permission to assign to the file.
    Note:

    • Supported for sftp, ftp, ftps, file, and hdfs protocols only.
    • FTP/FTPS servers on Windows machines may not be supported.
    FieldField TypeDescription

    Label*


    String


    Excerpt

    Specify a name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your Pipeline.

    Default Value: File Writer
    Example: File Writer

    File name*

    String

    Specify the URI of the destination file to which the the data (binary input from the upstream input view) is written. It may start with one of the following protocols:

    • http:
    • https:
    • s3:
    • sftp:
    • ftp:
    • ftps:
    • hdfs:
    • sldb:
    • smb:
    • file: (only for use with a Groundplex)

    • wasb:

    • wasbs:

    • gs:

    • adl: 


    Info
    titleWriting Files in SLDB

    For SLDB files, if you enter:

    • a file name, such as file.csv, then it writes the file to: /<org>/projects/<pipeline project>/file1.csv (where <org> is your organization name and <pipeline project> is the project where the Pipeline is stored), if the Pipeline is in a project other than the shared project.

    • shared/file1.csv, then it writes the file to: /<org>/shared/file1.csv.

    The Snap can write a file to its own project directory or the shared project, and cannot write it to another project directory.


    Info
    titleWriting Files in S3

    To write files in S3, your account must have full access. 


    Info

    You can also access the fields in a binary header when specifying a file name. For example, if you have a File Reader Snap upstream of a File Writer Snap, you can access the "content-location" header field to get the original path of the file. You can then use a new file name based on the old one, for instance, to make a backup file:
    $['content-location'].match('/([^/]+)$')[1] + '.backup'

    For http: and https: protocols, the Snap uses http PUT method only. This property should have the following syntax:

    [protocol]://[host][:port]/[path]

    Note
    • "://" is a separator between the file protocol and the rest of the URL and the host name and the port number should be between "://" and "/". If the port number is left empty, a default port number for the protocol is used. The hostname and port number are left empty in the sldb and S3 protocols. The value in this field should be an absolute path for all protocols except SLDB.
    • The file:/// protocol is supported only for Groundplex. In Cloudplex configurations, use SLDB or other file protocols. When using the file:/// protocol, the file is accessed based on the permissions of the user associated with the Snaplex is running (Default: Snapuser). 

    • For HDFS, if you want to be able to suggest information, use the HDFS Writer Snap. 


    Warning

    Use the file system access with caution, and ensure to clean up the file system after use.



    Note
    • When using expressions to build a file name, ensure that the resulting filename does not contain characters that are not supported by the target platform. 
      • The following characters are illegal in a share name: \ / [ ] : &#124; < > + = ; , * ? "

      • The following characters are not allowed in SMB share name: " \ / : | < > * ?

    For more information about referencing SMB file names, see Microsoft's documentation

    Default value[None]

    Examples

    File action*


    Dropdown list

    Specify the action to perform if the file already exists. The available options are:

    • Overwrite - If Overwrite is selected, the Snap attempts to write the file without checking for the file's existence for a better performance, and the "fileAction" field will be "overwritten" in the output view data.
    • Append - Append is supported for file, FTP, FTPS and SFTP protocols only.
    • Ignore - If Ignore, it will not overwrite the file and will do nothing but write the status and file name to its output view.
    • Error -  If Error is selected, the error displays in the Pipeline Run Log. If an error view is defined, the error will be written there as well.
    Note
    • In case you are selecting Append as File action, then ensure that the ‘Number of retries’ is 0 or not greater than 0.
    • For wasb:// and wasbs:// file protocols, only the Overwrite file action is supported.

    Default valueOverwrite

    Write empty fileCheckbox

    Select this checkbox to write an empty file when the incoming binary document has empty data. If there is no incoming document at the input view of the Snap, no file is written regardless of the value of the property.

    Default value: Not selected.

    Write header file


    String

    The binary data stream in the input view may contain header information about the binary data in the form of a document with key-value-pair map data. If this property is checked, the Snap writes a header file whose name is generated by appending ".header" to the value of the File name property. The same header information is also included in the output view data, as shown in the "Expected output" section above, under the key "original". Note that if the header has no keys other than Content-Type or Content-Encoding, the .header file will not be written

    Default value: Not selected 

    Validate after write


    Checkbox

    Select this checkbox to enable the Snap to check if the file exists after the completion of the file write. This may delay a few more seconds for the validation.

    Default value: Not selected 

    Number of retriesInteger

    Specify the maximum number of retry attempts when the Snap fails to write. If the value is larger than 0, the Snap first stores the input data in a temporary local file before writing to the target file.

    Info
    • Ensure that the local drive has sufficient free disk space as large as the expected target file size.
    • Owing to the nature of retry mechanism, the Snap does not support the Append operation if the Number of retries is set to more than zero.


    Minimum value
    : 0

    Default valueValue: 0
    Example:
      3

    Multiexcerpt include macro
    nameretries
    pageFile Reader

    Retry interval (seconds)Integer

    Specify the minimum number of seconds for which the Snap must wait before attempting recovery from a network failure.

    Minimum value: 1

    Default value: 1
    Example:  3

    User type

    It should be 'owner' or 'group' or 'others'. Each row can have only one user type and each user type should appear only once. Please select one from the suggested list.

    Example: owner, group, others

    File permissions

    It can be any combination of {read, write, execute} separated by '+' character. Please select one from the suggested list.

    Example: read, write, execute, read+write, read+write+execute
     

    User type

    The user type to assign the permissions. This field is case-insensitive. Suggestible

    before attempting recovery from a network failure.

    Minimum value: 1

    Default Value: 1
    Example: 3

    File permissions for various users

    Use this field set to provide any combination of permissions to the available users. 

    Note
    • Supported for sftp, ftp, ftps, file, and hdfs protocols only.
    • FTP/FTPS servers on Windows machines may not be supported.


    User typeString/Expression

    Choose one of the three standard user types. The available options are:

    • owner
    • group
  • others
  • Note
    • others

    Each row can have only one user type and each user type should appear only once. Select one from the suggested list. Specify at most one row per user type.

    Default Value: N/A
    Example: owner

    File permissionsSpecify any combination of {read, write, execute}, separated by a plus sign (+).  This field is case-insensitive. Suggestible options String/Expression/Suggestion

    Specify the privilege to provide to the user on the file. The available permissions are:

    • read
    • write
    • execute
    • read+write
    • read+execute
    • write+execute
    • read+write+execute

    Default Value: N/A
    Example
    : read+write+execute

    Create directory if not present  


    Checkbox

    Select this checkbox to enable the Snap to create a new directory if the specified directory path does not exist. This field is not supported for HTTP, HTTPS, SLDB and SMB file protocols.

    Default value: Not selected

    Flush interval (kB)Integer

    Flush interval in kilobytes during the file upload. The Snap can flush a given size of data output stream written to the target file server. If the value is zero, the Snap flushes in maximum frequency after each byte block is written. Larger the value is, the less frequent flushes the Snap performs. Leave the property at default -1 for no flush during the upload. This property may help if the file upload experiences an intermittent failure. However, more frequent flushes will result in a slower file upload.

    Default value: -1
    Example: 100

    Multiexcerpt include macro
    nameSASURI_Name
    pageFile Reader

    Multiexcerpt include macro
    nameSASURI_Description
    pageFile Reader
    Multiexcerpt include macroname

    Snap ExecutionpageSOAP Execute


    Dropdown list

    Multiexcerpt include macro
    nameExecution_Detail_Write
    pageSOAP Execute

    Examples

    • For SFTP files, if you attempt to write a file into a directory where you do not have a write access right, the write operation will fail with "access denied" error. When you get an SFTP account credential, it is also important to know where your home directory is, for example, sftp://ftp.snaplogic.com/home/mrtest for username "mrtest"
    • HDFS Example

    For HDFS file access, please use a SnapLogic on-premises Groundplex and make sure that its instance is within the Hadoop cluster and SSH authentication has already been established. You can access HDFS files in the same way as other file protocols in File Reader and File Writer Snaps. There is no need to use any account in the Snap.

    Note

    HDFS 2.4.0 is supported for the hdfs protocol.


    An example for HDFS is: 

    Code Block
    hdfs://<hostname>:<port number>/<path to folder>/<filename>


    If Cloudera Hadoop Namenode is installed in AWS EC2 and its hostname is "ec2-54-198-212-134.compute-1.amazonaws.com" and its port number is 8020, then you would enter:

    Code Block
    hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/sample.csv


    Expand
    titleExample pipeline file for an sldb file write

    Example pipeline file for an sldb file write as shown below: 

    Image Modified

    Troubleshooting


    ErrorReasonResolution

    Could not evaluate expression: filepath

    Mismatched input ':' expecting {<EOF>, '||', '&&', '^', '==', '!=', '>', '<', '>=', '<=', '+', '-', '*', '/', '%', '?', '[', PropertyRef}.


    The expression toggle (=) is selected on the File name field, so it is trying to evaluate the filepath as an expression.


    Check the expression syntax.

    Click on the toggle to take the field out of expression mode.

    Failure:
    filename is undefined

    filename was not found in the containing object.

    The expression toggle (=) is selected on the File name field, so it is trying to evaluate the filename as an expression.


    Check expression syntax and data types.

    Click on the toggle to take the field out of expression mode.

    Downloads

    Attachments
    uploadfalse
    oldfalse
    patterns*.slp, *.zip


    See Also

    Insert excerpt
    Binary Snap Pack
    Binary Snap Pack
    nopaneltrue