Type Converter

On this Page

Overview

This is a Transform type Snap that is used to detect and convert the data type of the incoming documents. It has the ability to automatically derive the type of each value. You can also specify your preference for handling certain fields in the input document. 

Input and Output

Expected input: A document input. Data types in the sub-levels in a nested JSON may not be detected.

Expected output: A document output where the data types have been converted based on the Snap's settings.

Expected upstream Snaps: Any Snap that provides a document stream output. For example, Mapper, or a combination of File Reader and JSON Parser.

Expected downstream Snaps: Any Snap that accepts a document input. For example, Profile, Mapper, or a combination of JSON Formatter and File Writer.

Prerequisites

None

Configuring Accounts

Accounts are not used with this Snap.

Configuring Views

Input

This Snap has exactly one document input view.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view.

Troubleshooting

None

Limitations and Known Issues

None

Modes


Snap Settings


LabelRequired. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline.
Auto

Required. If selected, the Snap detects the data types automatically. 

Default value: Selected

Policy

Specify your preferences about data types of fields if the auto-mode does not work as expected.

Field

The field that the type has to be converted. This is a suggestible property and suggests all fields in the dataset.

Default value: None

Type

The type of transformation to be performed on the selected column. Four options are available: 

  • Integer
  • Decimal
  • String
  • Date Time

Default value: String

Format

Only applicable for date time type, specify the format. See Simple Date Format for details and Date Functions and Properties for expression examples. 

Default value: None

Example: yyyy-MM-dd'T'HH:mm:ss.SSS'Z'

Snap Execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.


Example


Automatic Type Conversion

In this example, the input dataset's data types are automatically converted to their correct data types based on their value.

Download this pipeline.

 Understanding the pipeline

The input is a CSV document generated by the CSV Generator Snap. This document contains fields of various data types. Below is a preview of this document:

The Copy Snap is used to create two streams of this document. One stream goes into the Type Converter Snap and the other goes into the Type Inspector Snap. The Type Inspector is not necessarily required in this case but it is used to identify the fields in the document that have the wrong data type. Below is a preview of the output from the Type Inspector Snap:

The field $Deadline has the wrong data type. To correct this the Type Converter Snap is to be used.

Generally, the $ID field is supposed to be integer data type, but in this case since it is a categorical field it should be string data type. This is why the Type Converter Snap is configured to let the $ID field remain a string. The data type of the fields not listed in the Policy properties is automatically converted based on its value.

The Type Converter Snap is configured as shown below:

Based on its configuration, the Snap detects and converts data types. Below is a preview of the output from the Type Converter Snap:

A Type Inspector Snap is used downstream to demonstrate that the data type has indeed been converted by the Type Converter Snap. Below is a preview of the output from the Type Inspector Snap:


Download this pipeline.


Downloads

  File Modified

File Type_Converter_Auto.slp

Nov 21, 2018 by Mohammed Iqbal

Snap Pack History

 Click to view/expand

4.27 (427patches13948)

4.27 (main12833)

  • No updates made.

4.26 (main11181)

  • No updates made.

4.25 (425patches10994)

  • Fixed an issue when the Deduplicate Snap where the Snap breaks when running on a locale that does not format decimals with Period (.) character. 

4.25 (main9554)

  • No updates made.

4.24 (main8556)

  • No updates made.

4.23 (main7430)

  • No updates made.

4.22 (main6403)

  • No updates made.

4.21 (snapsmrc542)

  • Introduces the Mask Snap that enables you to hide sensitive information in your dataset before exporting the dataset for analytics or writing the dataset to a target file.
  • Enhances the Match Snap to add a new field, Match all, which matches one record from the first input with multiple records in the second input. Also, enhances the Comparator field in the Snap by adding one more option, Exact, which identifies and classifies a match as either an exact match or not a match at all.
  • Enhances the Deduplicate Snap to add a new field, Group ID, which includes the Group ID for each record in the output. Also, enhances the Comparator field in the Snap by adding one more option, Exact, which identifies and classifies a match as either an exact match or not a match at all.
  • Enhances the Sample Snap by adding a second output view which displays data that is not in the first output. Also, a new algorithm type, Linear Split, which enables you to split the dataset based on the pass-through percentage.

4.20 Patch mldatapreparation8771

  • Removes the unused jcc-optional dependency from the ML Data Preparation Snap Pack.

4.20 (snapsmrc535)

  • No updates made.

4.19 (snapsmrc528)

  • New Snap: Introducing the Deduplicate Snap. Use this Snap to remove duplicate records from input documents. When you use multiple matching criteria to deduplicate your data, it is evaluated using each criterion separately, and then aggregated to give the final result.

4.18 (snapsmrc523)

  • No updates made.

4.17 Patch ALL7402

  • Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17 (snapsmrc515)

  • New Snap: Introducing the Feature Synthesis Snap, which automatically creates features out of multiple datasets that share a one-to-one or one-to-many relationship with each other.
  • New Snap: Introducing the Match Snap, which enables you to automatically identify matched records across datasets that do not have a common key field.
  • Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16 (snapsmrc508)

  • Added a new Snap, Principal Component Analysis, which enables you to perform principal component analysis (PCA) on numeric fields (columns) to reduce dimensions of the dataset.

4.15 (snapsmrc500)

  • New Snap Pack. Perform preparatory operations on datasets such as data type transformation, data cleanup, sampling, shuffling, and scaling. Snaps in this Snap Pack are: 
    • Categorical to Numeric
    • Clean Missing Values
    • Date Time Extractor
    • Numeric to Categorical
    • Sample
    • Scale
    • Shuffle
    • Type Converter