Overview
This is a Transform type Snap that is used to detect and convert the data type of the incoming documents. It has the ability to automatically derive the type of each value. You can also specify your preference for handling certain fields in the input document.
Input and Output
Expected input: A document input. Data types in the sub-levels in a nested JSON may not be detected.
Expected output: A document output where the data types have been converted based on the Snap's settings.
Expected upstream Snaps: Any Snap that provides a document stream output. For example, Mapper, or a combination of File Reader and JSON Parser.
Expected downstream Snaps: Any Snap that accepts a document input. For example, Profile, Mapper, or a combination of JSON Formatter and File Writer.
Prerequisites
None
Configuring Accounts
Accounts are not used with this Snap.
Configuring Views
Input | This Snap has exactly one document input view. |
---|---|
Output | This Snap has exactly one document output view. |
Error | This Snap has at most one document error view. |
Troubleshooting
None
Limitations and Known Issues
None
Modes
- Ultra pipelines: Works in Ultra pipelines.
- Spark mode: Does not work in Spark mode.
Snap Settings
Label | Required. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline. |
---|---|
Auto | Required. If selected, the Snap detects the data types automatically. Default value: Selected |
Policy | Specify your preferences about data types of fields if the auto-mode does not work as expected. |
Field | The field that the type has to be converted. This is a suggestible property and suggests all fields in the dataset. Default value: None |
Type | The type of transformation to be performed on the selected column. Four options are available:
Default value: String |
Format | This is only applicable for datetime type. Specify the datetime format. See Simple Date Format for details. Default value: None Example: yyyy-MM-dd'T'HH:mm:ss.SSS'Z' |
Example
Automatic Type Conversion
In this example, the input dataset's data types are automatically converted to their correct data types based on their value.
Download this pipeline.
Downloads