{"type":"doc","content":[{"type":"paragraph","content":[{"text":"In this article","type":"text","marks":[{"type":"strong"}]}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{"maxLevel":{"value":"2"},"exclude":{"value":"Older Versions|Additional Resources|Related Links|Related Information"}},"macroMetadata":{"macroId":{"value":"fa1f1ed3-4d15-49af-b1fb-099651fd55e4"},"schemaVersion":{"value":"1"},"title":"org.randombits.confluence.toc.toc.label"}},"localId":"e4e7daa8-b594-4912-bdfb-c19f2b990ebc"}},{"type":"heading","attrs":{"level":1},"content":[{"text":"Overview","type":"text"}]},{"type":"paragraph","content":[{"text":"This Snap reads a binary data stream from its input view and writes a file in HDFS (Hadoop File System). It also helps pick a file by suggesting a list of directories and files. For the HDFS protocol, use a SnapLogic on-premises Groundplex and ensure its instance is within the Hadoop cluster and SSH authentication has already been established. The Snap also supports writing to a Kerberized cluster through the HDFS protocol. This Snap supports HDFS, ABFS (Azure Data Lake Storage Gen 2 ), and WASB (Azure storage) protocols. HDFS 2.4.0 is supported for the HDFS protocol. It also supports reading from ","type":"text"},{"text":"HDFS Encryption","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/50462722"}}]},{"text":".","type":"text"}]},{"type":"mediaSingle","attrs":{"layout":"align-start","width":633,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":633,"alt":"hdfs-writer-overview.png","id":"e5ec26ad-109a-451c-8997-b3152ee24f09","collection":"contentId-1438235","type":"file","height":500}}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Snap Type","type":"text"}]},{"type":"paragraph","content":[{"text":"The HDFS Writer Snap is a Write-type Snap.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Prerequisites","type":"text"}]},{"type":"paragraph","content":[{"text":"None.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Support for Ultra Pipelines","type":"text"}]},{"type":"paragraph","content":[{"text":"Works in ","type":"text"},{"text":"Ultra Pipelines","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/x/A-EV"}}]},{"text":".","type":"text"},{"text":" ","type":"text","marks":[{"type":"strong"},{"type":"em"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Limitations","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Append (File action) is supported for ADL protocol only.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"File names with the following special characters are not supported in the HDFS Writer Snap: '+', '?', '/', ':'.","type":"text"}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Known Issues","type":"text"}]},{"type":"paragraph","content":[{"text":"The upgrade of Azure Storage library from v3.0.0 to v8.3.0 has caused the following issue when using the WASB protocol:","type":"text"},{"type":"hardBreak"},{"text":"When you use invalid credentials for the WASB protocol in Hadoop Snaps (","type":"text"},{"text":"HDFS Reader","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438234"}}]},{"text":", ","type":"text"},{"text":"HDFS Writer","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438235"}}]},{"text":", ","type":"text"},{"text":"ORC Reader","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438232"}}]},{"text":", ","type":"text"},{"text":"Parquet Reader","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438228"}}]},{"text":", ","type":"text"},{"text":"Parquet Writer","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/3021963451"}}]},{"text":"), the pipeline does not fail immediately, instead it takes 13-14 minutes to display the following error:","type":"text"}]},{"type":"paragraph","content":[{"text":"reason=The request failed with error code null and HTTP code 0. , status_code=error","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"SnapLogic® is actively working with Microsoft®Support to resolve the issue.","type":"text"}]},{"type":"paragraph","content":[{"text":"Learn more about ","type":"text"},{"text":"Azure Storage library upgrade","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/3393192072"}}]},{"text":".","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Snap Views","type":"text"}]},{"type":"table","attrs":{"layout":"full-width","width":1800.0,"localId":"ec163d7c-9426-425d-b3ff-99b1652d3b1a"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#f4f5f7","rowspan":1,"colwidth":[137.0]},"content":[{"type":"paragraph","content":[{"text":"Type","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"background":"#f4f5f7","rowspan":1,"colwidth":[115.0]},"content":[{"type":"paragraph","content":[{"text":"Format","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"background":"#f4f5f7","rowspan":1,"colwidth":[181.0]},"content":[{"type":"paragraph","content":[{"text":"Number of Views","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"background":"#f4f5f7","rowspan":1,"colwidth":[255.0]},"content":[{"type":"paragraph","content":[{"text":"Examples of Upstream and Downstream Snaps","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"background":"#f4f5f7","rowspan":1,"colwidth":[565.0]},"content":[{"type":"paragraph","content":[{"text":"Description","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[137.0]},"content":[{"type":"paragraph","content":[{"text":"Input","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[115.0]},"content":[{"type":"paragraph","content":[{"text":"Binary","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[181.0]},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Min: 1","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Max: 1","type":"text"}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[255.0]},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"CSV Formatter","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"JSON Formatter","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"XML Formatter","type":"text"}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[565.0]},"content":[{"type":"paragraph","content":[{"text":"Binary input data.","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[137.0]},"content":[{"type":"paragraph","content":[{"text":"Output","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[115.0]},"content":[{"type":"paragraph","content":[{"text":"Document","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[181.0]},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Min: 0","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Max: 1","type":"text"}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[255.0]},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Maper","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"File Reader","type":"text"}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[565.0]},"content":[{"type":"paragraph","content":[{"text":"The following is an example of the output document map data:","type":"text"}]},{"type":"paragraph","content":[{"text":"{ ","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"\"filename\": \"hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/sample.csv\", 3 \"fileAction\": \"overwritten\" }","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"The value of the \"fileAction\" field can be \"overwritten\" or \"created\" or \"ignored\". The value \"ignored\" indicate that the Snap did not overwrite the existing file because the value of the File action property is \"IGNORE\".","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[137.0]},"content":[{"type":"paragraph","content":[{"text":"Error","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":4,"rowspan":1,"colwidth":[115.0,181.0,255.0,565.0]},"content":[{"type":"paragraph","content":[{"text":"Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter while running the Pipeline by choosing one of the following options from the ","type":"text"},{"text":"When errors occur ","type":"text","marks":[{"type":"strong"}]},{"text":"list under the ","type":"text"},{"text":"Views ","type":"text","marks":[{"type":"strong"}]},{"text":"tab:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Stop Pipeline Execution","type":"text","marks":[{"type":"strong"}]},{"text":": Stops the current pipeline execution if the Snap encounters an error.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Discard Error Data and Continue","type":"text","marks":[{"type":"strong"}]},{"text":": Ignores the error, discards that record, and continues with the remaining records.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Route Error Data to Error View","type":"text","marks":[{"type":"strong"}]},{"text":": Routes the error data to an error view without stopping the Snap execution.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Learn more about ","type":"text"},{"text":"Error handling in Pipelines","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/81526859"}}]},{"text":".","type":"text"}]}]}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Snap Settings","type":"text"}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Asterisk ( ","type":"text"},{"text":"* ","type":"text","marks":[{"type":"strong"}]},{"text":"): Indicates a mandatory field.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Suggestion icon (","type":"text"},{"type":"emoji","attrs":{"id":"2fa452fe-3127-4e25-a103-1daf1d018f2f","text":":Suggest_Icon:","shortName":":Suggest_Icon:"}},{"text":"): Indicates a list that is dynamically populated based on the configuration.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Expression icon (","type":"text"},{"type":"emoji","attrs":{"id":"eafa7218-7a8b-4f28-ac02-063a11f06808","text":":Expression_Enabler:","shortName":":Expression_Enabler:"}},{"text":" ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about ","type":"text"},{"text":"Using Expressions in SnapLogic","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438042/Understanding+Expressions+in+SnapLogic"}}]},{"text":".","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Add icon ( ","type":"text"},{"type":"emoji","attrs":{"id":"246af6d7-b82b-40db-9d59-39d08287d22b","text":":Plus:","shortName":":Plus:"}},{"text":" ): Indicates that you can add fields in the field set.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Remove icon ( ","type":"text"},{"type":"emoji","attrs":{"id":"1f6c9214-8d14-4bf4-ae18-f0d4ed595846","text":":Minus:","shortName":":Minus:"}},{"text":"): Indicates that you can remove fields from the field set.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Upload icon (","type":"text"},{"type":"emoji","attrs":{"id":"4bf1d948-078a-47fa-b108-a7239e972f62","text":":Upload_icon:","shortName":":Upload_icon:"}},{"text":" ): Indicates that you can upload files.","type":"text"}]}]}]}]},{"type":"table","attrs":{"layout":"full-width","width":1800.0,"localId":"b6a6e7c7-2d50-4da2-b5cd-ac953a8c3d79"},"content":[{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#f4f5f7","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"Field","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"background":"#f4f5f7","rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"Field Types","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"background":"#f4f5f7","rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Description","type":"text","marks":[{"type":"strong"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"Label","type":"text","marks":[{"type":"strong"}]},{"text":"*","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"HDFS Writer","type":"text","marks":[{"type":"em"}]},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"HDFS Writer","type":"text","marks":[{"type":"em"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"String","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Specify a unique name for the Snap. ","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"Directory","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"hdfs://:/","type":"text","marks":[{"type":"em"}]},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"hdfs://ec2-54-198-212-134.compute-1.amazonaws.com:8020/user/john/input/","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"wasb:///snaplogic/testDir/","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"wasbs:///snaplogic/testDir/","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"$dirname","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"adl://snapqa/ ","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"abfs(s):///filesystem2/dir1","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"abfs(s)://filesystem2@snaplogicaccount.dfs.core.windows.net/dir1","type":"text","marks":[{"type":"em"}]}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"String/Expression/Suggestion","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Specify the URL for HDFS directory. It should start with hdfs file protocol in the following format:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"hdfs://://","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"wasb://///","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"wasbs://///","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"adl://// ","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"abfs(s):///filesystem//","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"abfs(s)://filesystem@accountname.endpoint/","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"The Directory property is not used in the Pipeline execution or preview and used only in the Suggest operation. When you click the Suggestion icon, the Snap displays a list of subdirectories under the given directory. It generates the list by applying the value of the Filter property.","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"SnapLogic automatically appends ","type":"text"},{"text":"azuredatalakestore.net","type":"text","marks":[{"type":"code"}]},{"text":" to the store name you specify when using Azure Data Lake; therefore, you do not have to add ","type":"text"},{"text":"azuredatalakestore.net","type":"text","marks":[{"type":"code"}]},{"text":" to the URI while specifying the directory.","type":"text"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"File filter","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": *","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"String/Expression","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Specify the Glob filter pattern.","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Use glob patterns to display a list of directories or files when you click the ","type":"text"},{"text":"Suggest","type":"text","marks":[{"type":"strong"}]},{"text":" icon in the Directory or File property. A complete glob pattern is formed by combining the value of the Directory property with the Filter property. If the value of the Directory property does not end with \"/\", the Snap appends one, so that the value of the Filter property is applied to the directory specified by the Directory property.","type":"text"}]}]},{"type":"paragraph"},{"type":"nestedExpand","attrs":{"title":"Glob Pattern Interpretation Rules"},"content":[{"type":"paragraph","content":[{"text":"The following rules are used to interpret glob patterns:","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" The * character matches zero or more characters of a name component without crossing directory boundaries. For example, the *.csv pattern matches a path that represents a file name ending in .csv, and *.* matches all file names that contain a period.","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" The ** characters match zero or more ","type":"text"},{"text":"characters","type":"text","marks":[{"type":"link","attrs":{"href":"http://docs.oracle.com/javase/7/docs/api/java/lang/Character.html"}}]},{"text":" across directories; therefore, it matches all files or directories in the current directory and in its subdirectories. For example, /home/** matches all files and directories in the /home/ directory.","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" The ? character matches exactly one character of a name component. For example, 'foo.?' matches file names that start with 'foo.' and are followed by a single-character extension.","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" The \\ character is used to escape characters that would otherwise be interpreted as special characters. The expression \\\\ matches a single backslash, and \\{ matches a left brace, for example.","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" The ! character is used to exclude matching files from the output. ","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" The [ ] characters form a bracket expression that matches a single character of a name component out of a set of characters. For example, '[abc]' matches 'a', 'b', or 'c'. The hyphen (-) may be used to specify a range, so '[a-z]' specifies a range that matches from 'a' to 'z' (inclusive). These forms can be mixed, so '[abce-g]' matches 'a', 'b', 'c', 'e', 'f' or 'g'. If the character after the [ is a ! then it is used for negation, so '[!a-c]' matches any character except 'a', 'b', or 'c'.","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" Within a bracket expression, the '*', '?', and '\\' characters match themselves. The '-' character matches itself if it is the first character within the brackets, or the first character after the !, if negating.","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" The '{ }' characters are a group of sub-patterns where the group returns a match if any sub-pattern in the group matches the contents of a target directory. The ',' character is used to separate sub-patterns. Groups cannot be nested. For example, the pattern '*.{csv, json}' matches file names ending with '.csv' or '.json'.","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" Leading dot characters in a file name are treated as regular characters in match operations. For example, the '*' glob pattern matches file name \".login\".","type":"text"}]},{"type":"paragraph","content":[{"type":"emoji","attrs":{"id":"2611","text":"☑️","shortName":":ballot_box_with_check:"}},{"text":" All other characters match themselves.","type":"text"}]},{"type":"paragraph","content":[{"text":"Examples:","type":"text"}]},{"type":"paragraph","content":[{"text":"'*.csv' matches all files with a csv extension in the current directory only.","type":"text"}]},{"type":"paragraph","content":[{"text":"'**.csv' matches all files with a csv extension in the current directory and in all its subdirectories.","type":"text"}]},{"type":"paragraph","content":[{"text":"*[!{.pdf,.tmp}] excludes all files with the extension PDF or TMP.","type":"text"}]}]},{"type":"paragraph"}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"File","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph"},{"type":"paragraph","content":[{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": N/A","type":"text"},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"sample.csv","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"tmp/another.csv","type":"text","marks":[{"type":"em"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"$filename","type":"text","marks":[{"type":"em"}]}]}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"String/Expression/Suggestion","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Specify the filename or a relative path to a file under the directory given in the Directory property. It should not start with a URL separator \"/\". The File property can be a JavaScript expression which will be evaluated with values from the input view document. When you click the Suggest icon, it will display a list of regular files under the directory in the Directory property. It generates the list by applying the value of the Filter property.","type":"text"},{"type":"hardBreak"},{"type":"hardBreak"}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"Flush interval (kB)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}},{"type":"strong"}]},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"},{"text":" ","type":"text","marks":[{"type":"strong"}]},{"text":"-1","type":"text","marks":[{"type":"em"}]},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"0","type":"text","marks":[{"type":"em"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"Interval","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Specify the flush interval in kilobytes to flush a specified size of data during the file upload. This Snap can flush the output stream each time a given size of data is written to the target file server.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"If the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]},{"text":"Flush interval","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}},{"type":"strong"}]},{"text":" is 0, the Snap flushes maximum frequency after each byte block is written. The larger the flush interval, the less frequent are the flushes. This field is useful if the file upload experiences an intermittent failure. However, more frequent flushes result in slower file upload. The default value of -1 indicates no flush during the upload.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"Number Of Retries","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}},{"type":"strong"}]},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":":","type":"text"},{"text":" ","type":"text","marks":[{"type":"strong"}]},{"text":"0","type":"text","marks":[{"type":"em"}]},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"1","type":"text","marks":[{"type":"em"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"Integer/Expression","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Specify the maximum number of attempts to be made to receive a response. ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"The request is terminated if the attempts do not result in a response.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Retry operation occurs only when the Snap loses the connection with the server.","type":"text"}]}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"Retry Interval (seconds)","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}},{"type":"strong"}]},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"1","type":"text","marks":[{"type":"em"}]},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"5","type":"text","marks":[{"type":"em"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"Integer/Expression","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Specify the time interval between two successive retry requests. A retry happens only when the previous attempt resulted in an exception.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#000000"}}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"File action*","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Overwrite","type":"text","marks":[{"type":"em"}]},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Append","type":"text","marks":[{"type":"em"}]},{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"type":"hardBreak"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"Dropdown list","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Select an action to perform if the specified file already exists:","type":"text"}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Overwrite","type":"text","marks":[{"type":"em"}]},{"text":" - The Snap attempts to write the file without checking for the file's existence for a better performance, and the \"fileAction\" field will be \"overwritten\" in the output view data.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Append","type":"text","marks":[{"type":"em"}]},{"text":" - The Snap appends records in the incoming documents to the existing file.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Ignore","type":"text","marks":[{"type":"em"}]},{"text":" - If the file already exists, the Snap does not throw an exception and does not overwrite the file, but writes an output document indicating that it has been 'ignored'.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Error","type":"text","marks":[{"type":"em"}]},{"text":" - The error displays in the Pipeline Run Log if the file already exists.","type":"text"}]}]}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"The ","type":"text"},{"text":"Append ","type":"text","marks":[{"type":"em"}]},{"text":"operation is supported for FILE, SFTP, FTP, FTPS and ADL protocols only. For any other protocols that are not supported by ","type":"text"},{"text":"Append","type":"text","marks":[{"type":"em"}]},{"text":", we recommend that you use the ","type":"text"},{"text":"File Operations","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/187072700"}}]},{"text":", ","type":"text"},{"text":"File Writer","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438927/File+Writer?search_id=e990f534-6210-4c62-a3b6-4d86b8631a4c"}}]},{"text":", and ","type":"text"},{"text":"File Delete","type":"text","marks":[{"type":"link","attrs":{"href":"https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438769/File+Delete?search_id=144a1758-0782-497e-936f-d54448d327fb"}}]},{"text":" Snaps and follow this procedure. ","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Copy the blob file to your local drive from the endpoint.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Append additional data to the local file.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Delete the original file on the endpoint.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Copy the modified temporary file back from the source to the target.","type":"text"}]}]}]},{"type":"paragraph","content":[{"text":"Note","type":"text","marks":[{"type":"strong"}]},{"text":": This approach might involve disk overhead, therefore ensure that you have enough disk space in your system.","type":"text"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"File permissions for various users","type":"text","marks":[{"type":"strong"}]}]}]},{"type":"tableCell","attrs":{"colspan":2,"background":"#f4f5f7","rowspan":1,"colwidth":[246.0,738.0]},"content":[{"type":"paragraph","content":[{"text":"Use this field set to select the user and the desired file permissions.","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Limitations with File Permissions for Various Users","type":"text","marks":[{"type":"strong"}]}]},{"type":"bulletList","content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"With \"File permissions for various users\": File names with the following special characters are not supported in the HDFS Writer Snap: '+', '?', '/', ':'.","type":"text"}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Without \"File permissions for various users\": File names with the following special characters are not supported in the HDFS Writer Snap: ':', '/'.","type":"text"}]}]}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"User type","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": N/A","type":"text"},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":": owner, group, others","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"String/Expression/Suggestion","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"It should be 'owner' or 'group' or 'others'. Each row can have only one user type and each user type should appear only once. Please select one from the suggested list.","type":"text"}]},{"type":"paragraph"}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"File permissions","type":"text","marks":[{"type":"strong"}]},{"type":"hardBreak"}]},{"type":"paragraph","content":[{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": N/A","type":"text"},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":": r","type":"text"},{"text":"ead, write, execute, read+write, read+write+execute","type":"text","marks":[{"type":"em"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"String/Expression/Suggestion","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"It can be any combination of {read, write, execute} separated by '+' character. Please select one from the suggested list.","type":"text"}]},{"type":"paragraph"}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"User Impersonation","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Deselected","type":"text","marks":[{"type":"em"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"Checkbox","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Select this check box to enable user impersonation.","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Hadoop allows you to configure proxy users to access HDFS on behalf of other users; this is called impersonation. When user impersonation is enabled on the Hadoop cluster, any jobs submitted using a proxy are executed with the impersonated user's existing privilege levels rather than those of the superuser associated with the cluster. For more information on user impersonation in this Snap, see the section on User Impersonation below.","type":"text"}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"Output for each file written","type":"text","marks":[{"type":"textColor","attrs":{"color":"#172b4d"}},{"type":"strong"}]}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Deselected","type":"text","marks":[{"type":"em"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"Checkbox","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Enables you to produce a different output document for each file that is written. If the Snap receives multiple binary input data and the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#333333"}}]},{"text":"File","type":"text","marks":[{"type":"textColor","attrs":{"color":"#333333"}},{"type":"strong"}]},{"text":" expression property is dynamically evaluated to a filename by using the ","type":"text","marks":[{"type":"textColor","attrs":{"color":"#333333"}}]},{"text":"Content-Location","type":"text","marks":[{"type":"code"}]},{"text":" field from the input metadata, each binary data can be written to a different target file.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#333333"}}]}]},{"type":"paragraph","content":[{"text":"By default, the Snap produces only one output document with a filename that corresponds to the last file that was written.","type":"text","marks":[{"type":"textColor","attrs":{"color":"#333333"}}]}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"Write empty file","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"text":" ","type":"text"}]},{"type":"paragraph","content":[{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Deselected","type":"text","marks":[{"type":"em"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"Checkbox","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"text":"Select this checkbox to write an empty file to all the supported protocols when the binary input document has no data.","type":"text"}]}]}]},{"type":"tableRow","content":[{"type":"tableHeader","attrs":{"colspan":1,"background":"#ffffff","rowspan":1,"colwidth":[269.0]},"content":[{"type":"paragraph","content":[{"text":"Snap Execution","type":"text","marks":[{"type":"strong"}]}]},{"type":"paragraph","content":[{"type":"hardBreak"},{"text":"Default Value","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Validate & Execute","type":"text","marks":[{"type":"em"}]},{"type":"hardBreak"},{"text":"Example","type":"text","marks":[{"type":"strong"}]},{"text":": ","type":"text"},{"text":"Execute only","type":"text","marks":[{"type":"em"}]}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[246.0]},"content":[{"type":"paragraph","content":[{"text":"Dropdown list","type":"text"}]}]},{"type":"tableCell","attrs":{"colspan":1,"rowspan":1,"colwidth":[738.0]},"content":[{"type":"paragraph","content":[{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.macro.core","extensionKey":"multiexcerpt-include-macro","parameters":{"macroParams":{"name":{"value":"Execution_Detail_Write"},"page":{"value":"SOAP Execute"}},"macroMetadata":{"macroId":{"value":"85e2c354-0c2e-46e8-846d-df226d696dbf"},"schemaVersion":{"value":"1"},"placeholder":[{"type":"icon","data":{"url":"https://raw.githubusercontent.com/fuegokit/appfire-design-systems-brand-svg/main/single-color-brand-icons/af-logo-multi-excerpt-24.svg"}}],"title":"MultiExcerpt Include"}},"localId":"d58c1e7a-9467-4ca3-9c65-876df39544db"}}]}]}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Troubleshooting","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.macro.core","extensionKey":"excerpt-include","parameters":{"macroParams":{"":{"value":"Hadoop Directory Browser"},"nopanel":{"value":"true"}},"macroMetadata":{"macroId":{"value":"22214997-33bd-498e-9766-6e1c9a97717c"},"schemaVersion":{"value":"1"},"title":"Insert excerpt"}},"localId":"d2125250-77be-4c72-8133-c2e7cf490b27"}}]},{"type":"heading","attrs":{"level":1},"content":[{"type":"hardBreak"},{"text":"Snap Pack History","type":"text"}]},{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"excerpt-include","parameters":{"macroParams":{"":{"value":"Hadoop Snap Pack"}},"macroMetadata":{"macroId":{"value":"691e1d75-126d-46b7-9913-aedcbdf405a1"},"schemaVersion":{"value":"1"},"title":"Insert excerpt"}},"localId":"4a4bc668-aa14-46cc-b301-9a38d70b544a"}}],"version":1}