Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Did edits

On this page

Table of Contents
maxLevel2
absoluteUrltrue

Overview

Use this the Mask Snap to hide sensitive information in your dataset before you export the dataset for analytics. With the Mask Snap, you can ensure You can protect sensitive data protection by using masking algorithms that the Snap provides out of the box.

Image Modified

Snap Input and Output

Input/OutputType of ViewNumber of ViewsCompatible Upstream and Downstream SnapsDescription
Input Document
  • Min: 1
  • Max: 1
  • Mapper Snap
  • MySQL - Select
  • REST Get
A dataset where some of the data must be masked.
OutputDocument
  • Min: 1
  • Max: 1
  • MySQL - Insert
  • JSON Formatter
  • CSV Formatter

A dataset where the specified data is masked. 

Snap Settings

Parameter NameData TypeDescriptionDefault ValueExample 
LabelStringRequired. The name for the Snap. Modify this to be more specific, especially if there is more than one of the same Snap in the Pipeline.N/AMask
Mask PolicyN/ARequired. Enables you to specify the policies that you want to use to mask data in the input dataset.N/AN/A

Field

String

The field/parent field in the input dataset that contains the data to be masked.

N/Aname
Search ModeN/A

The mode the Snap must use to search sensitive data.

Select from the following options:

  • Exact Path: The Snap searches for the exact field name.
  • Recursive: The Snap searches through all the levels in the nested structure of the specified Field.

For example, if you want to mask $credit_card field, you must enter $credit_card in Field for Exact Path Search Mode. However, if you are using the Recursive option as the Search Mode, then you can just enter $ in Field.

N/AExact Path
Match FieldString

The type of field in the input data to be matched. Select from the following options:

  • Key: Select if the data to be matched is the field name.
  • Value: Select if the data to be matched is the field value.
Info

The Snap fails to validate if the Match Field is Key and the Mask Field is also Key. In such a case, select Value as the Mask Field.


N/AValue
Match ConditionN/A

The match condition that determines whether the Match Field should be matched. For example, if your input dataset contains $credit_card Field and you enter the Match Field as Value and Match Condition as Number, then the Snap masks all $credit_card fields that contain a number as the value. If the field contains text as the value, then the Snap skips masking that value.

Select from the following options:

  • Regex Match: Matches the Key or Value when the key or value is the expression entered in Match Pattern. If Match Pattern is blank, the Snap matches all values. 
  • Credit Card: Matches the Key or Value when the key or value is the number pattern used in major credit cards. The supported credit card types are: Visa, MasterCard, JCB, American Express, Discover, and Diner's Club.

    The Snap supports both long digits or groups of 4-digits that are separated by a dash or space. For example, 4111 1111 1111 1111, 3589-0731-0185-9601, and 6011000000000004.

  • SSN: Matches the Key or Value if the key or value is in the Social Security Number (SSN) pattern. The Snap supports values entered in the SSN format only, which is, XXX-YY-ZZZZ. You can use blanks or dashes as separators. For example, 123-12-1234 and 123 12 1234.
  • Number: Matches the Value when the value is an integer or decimal.
  • Number (Text): Matches the Key or Value when the key or value is a number in text format. The Snap does not support commas within the number text. 
  • Date: Matches the Value when the value is a date object. 
  • Date (Text): Matches the Key or Value when the key or value is a date in text or a pattern entered in Match Pattern. You can specify the format in Match Pattern. The Snap supports the following formats:
    • "2018-08-12"
    • "2018-08-12T12:34:56"
    • "2018/08/12 12:34:56.780"
    • "2018/08/12 12:34:56.78"
    • "2018-08-12T12:34:56.78"
    • "2018-08-12 12:34:56"
N/ADate
Match PatternStringThe expression of the information pattern to be matched in the input dataset. This is applicable only when the Match Condition is Regex Match or Date (Text).N/AHello
Mask FieldN/A

The field that contains sensitive data and will be masked if the matching conditions are met. Select from the following options:

  • Key: Select if the data to be masked is the key.
  • Value: Select if the data to be masked is the value.
Info

The Snap fails to validate if the Match Field is Key and the Mask Field is also Key. In such a case, select Value as the Mask Field.


N/AValue
Mask MethodN/A

The method to use to mask sensitive information. Select from the following options:

  • Replace: Replaces the matched Value with the value you enter in Mask Value
  • Shuffle: Shuffles the matched data Value randomly.
  • Remove: Removes the matched Key or Value.

    • If the Mask Field is Value, the Snap returns null as the new value. If the Mask Field is Key, the Snap removes the whole Key-Value pair.
    • With arrays and Recursive Search Mode, if the Mask Field is Value, the value is removed from the array. If all values are removed, the Snap returns an empty array. However, if the Mask Field is Key and the Snap removes the array along with the key.
  • Random: Replaces the matched Value with a new random value. The Snap derives the random value based on the data type of the matched value and behaves as described below:
    • Text: Randomly replaces the value with alphanumeric characters having the same length as the original data.
    • Integer: Randomly replaces the value with an integer value in the range between 0 to an integer of the same digit as the actual data. The new value is always different from the original value. For example, if the original data is 120, the possible replacement value is between 0 to 999, except for 120.
    • Decimal: Randomly replaces the value with a value that is based on the precision of the original value from 0 to the precision on the same digit. For example, if the original value is 0.023, the possible replacement value is between 0.000 to 0.099, except for 0.023.
    • Boolean: Randomly replaces the value with either true or false.
  • Start of Month: Replaces the matched Value with the first day of the month. If the value is a text, the Snap converts the text to date using the same way as the Auto mode in the Type Converter Snap. In this mode, the Snap automatically converts text to Date.
  • Start of Year: Replaces the matched Value with the first day of the same year. If the value is a text, the Snap converts the text to date using the same way as the Auto mode in the Type Converter Snap. In this mode, the Snap automatically converts text to Date.
N/ARemove
Mask Value
  • String
  • Numeric
The value that must replace the sensitive information in the input dataset. You can enter either a fixed value or an expression. This is applicable only when the Mask Method is Replace.N/A0
Execute during previewN/A

Specify the execution type from the following options:

  • Validate & Execute: Performs limited execution of the Snap (up to 50 records) during Pipeline validation; performs full execution of the Snap (unlimited records) during Pipeline execution.
  • Execute only: Performs full execution of the Snap during Pipeline execution; does not execute the Snap during Pipeline validation.
  • Disabled: Disables the Snap and, by extension, its the downstream Snaps.

Validate & ExecuteValidate & Execute

Example

This Pipeline demonstrates how the Mask Snap helps you hide sensitive information in a dataset. In this example, the input dataset is a demographic of the Oscar award winners. We want to pass on the dataset to a third party for some to derive the analytics. However, before we pass on the data, we want to hide sensitive information from the dataset. We use the Mask Snap to identify and replace/remove all sensitive information from the Oscar award winners list. While the dataset here is public data, you can apply the masking policies that the Mask Snap provides to hide confidential or sensitive data from your organization.

Download the Pipeline.

The dataset is derived from Kaggle and is a demographic of Oscar award winners in the Best Director category from 1927 through 1976. The input document is picked up using the File Reader Snap and is passed through the CSV Parser Snap. A preview of the dataset in CSV format is as followsper the following:

Image RemovedImage Added

The dataset is passed to the Mask Snap to hide sensitive data. In this example, we want to mask the following fields:

  • date_of_birth: Replace all dates all dates of birth with the first day of the respective year of birth
  • bio_url: Delete all HTTP/HTTPS URLs from the dataset
  • person: Replace the name of the winner with the text "Winner name is masked"

For this purpose, we have configured the Mask Snap is configured as followsper the following:

Image RemovedImage Added

We have added three rows under Mask Policy, and each row enables us to mask one of the fields discussed above. 

...

The second row under Mask Policy enables us to mask the bio_url field. We specify the Field as $ and select the Search Mode as Recursive. Recursive mode will search for all possible fields that could contain a URL that matches the regular expression, https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&\/\/=]*), in the Match Pattern. This regular expression matches all HTTP and HTTPS URLs in the dataset. In this case, the Match Field is Value, the Mask Field is Key, the Match Condition is Regex Match, and the Mask Method is Remove. The Snap completely deletes all keys that contain an HTTP/HTTPS value. Hence, we do not see the bio_url field in the output.

...

After the dataset is passed through the Mask Snap and the different mask conditions are applied to it, the output dataset is as followsper the following screenshot:

Image RemovedImage Added

In the output dataset preview, we can see that the date_of_birth field  field shows the first date of a year instead of the exact birth date, the bio_url field is removed, and the person field shows Winner name is masked instead of the real name. 

The masked dataset output of the Mask Snap is then converted to CSV format using the CSV Formatter Snap and then passed to a File Writer Snap.

...

See Also

...