Cross Validator -- Classification

On this Page

Overview

This is a Transform type Snap that performs K-fold Cross Validation on a classification dataset. Cross validation is a technique for evaluating ML algorithms by splitting the original dataset into K equally-sized chunks. K is the number of folds. In each of the K iterations, K-1 chunks are used to train the model while the last chunk is used as a test set. The average accuracy and other statistics are computed to be used to select the most suitable algorithm for the dataset.

In the settings, you can select the algorithm, specify parameters, and the number of folds. If you want to perform K-fold Cross Validation on regression dataset, use the Cross Validator – Regression Snap instead.

Input and Output

Expected input: The classification dataset.

Expected output: A statistical information about the performance of the selected algorithm on the dataset.

Expected upstream Snap: Any Snap that generates a classification dataset document. For example, CSV Generator, JSON Generator, or a combination of File Reader and JSON Parser.

Expected downstream Snap: CSV/JSON Formatter Snap and File Writer Snap can be used to write the output statistics to file.

Prerequisites

  • The data from upstream Snap must be in tabular format (no nested structure). 
  • This Snap automatically derives the schema (field names and types) from the first document. Therefore, the first document must not have any missing values.

Configuring Accounts

Accounts are not used with this Snap.

Configuring Views

Input

This Snap has exactly one document input view.
OutputThis Snap has exactly one document output view.
ErrorThis Snap has at most one document error view.

Troubleshooting

None.

Limitations and Known Issues

None.

Modes


Snap Settings


LabelRequired. The name for the Snap. Modify this to be more specific, especially if there are more than one of the same Snap in the pipeline.
Label field

Required. The label or output field in the dataset. This must be a categorical type represented as text (string data type). This is the field that the ML model will be trained to predict.

Default value: None

Example: $class

Algorithm

Required. The classification algorithm to be used to build the model. There are eight classification algorithms available currently:

  • Decision Tree
  • K-Nearest Neighbors
  • Logistic Regression
  • Naive Bayes
  • Support Vector Machines
  • Decision Stump
  • Random Forests
  • Multilayer Perceptron.

The implementations are from WEKA, an open source machine learning library in Java.

Default value: Decision Tree

Option

The parameters to be applied on the selected classification algorithm. Each algorithm has a different set of parameters to be configured in this property. If this property is left blank, the default values are applied for all the parameters. If specifying multiple parameters, separate them with a comma ",".

See Options for Algorithms section below for details.

Default value: None

Examples:

  • batch_size = 120
  • batch_size = 120, collapse_tree = true
Fold

Required. The number of folds.

Minimum value: 2

Default value: 10

Use random seed

If selected, Random seed is applied to the randomizer in order to get reproducible results.

Default value: Selected

Random seed

RequiredNumber used as static seed for randomizer.

Default value: 12345

Snap Execution

Select one of the three modes in which the Snap executes. Available options are:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during Pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during Pipeline runtime.
  • Execute only: Performs full execution of the Snap during Pipeline execution without generating preview data.
  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Options for Algorithms


 Options for algorithms

This section lists all applicable parameters for each algorithm along with their default values. 

Decision Tree

Option Name
Option Data Type
Default Value
Description
batch_sizeint100Set the preferred batch size for batch prediction.
binary_splitsbooleanfalseEnable binary splits.
collapse_treebooleantrueSet to allow collapsing the tree.
confidence_factorfloat0.25Set the confidence threshold for pruning.
do_not_check_capabilitiesbooleanfalseEnable capability-check.
do_not_mask_split_point_actual_valuebooleanfalseSet whether the split point actual value is to be masked. 
min_num_objint2Set the minimum number of instances per leaf.
num_decimal_placesint2Set the number of decimal places.
num_foldsint3Set the number of folds for reduced error pruning. One fold is used as pruning set.
reduced_error_pruningbooleanfalseEnable reduced error pruning. This is false if the unpruned parameter is set to true.
save_instance_databooleanfalseSet whether the instance data is to be saved. If set to true, it does not clean up after the tree has been built.
seedint1Set the seed for random data shuffling.
subtree_raisingbooleantrueEnable sub-tree raising.
unprunedbooleanfalseEnable using unpruned tree. This is false if the reduced_error_pruningparameter is set to true.
use_laplacebooleanfalseAllow Laplace smoothing for predicted probabilities.
use_md_correctionbooleantrueAllow MDL correction for info gain on numeric attributes.

K-Nearest Neighbors

Option Name
Option Data Type
Default Value
Description
knnint1Set the number of neighbors the learner will use.
batch_sizeint100Set the preferred batch size for batch prediction.
cross_validatebooleanfalseSet whether the hold-one-out cross-validation will be used to select the best k-value.
distance_weightingenum 
[weight_none, weight_inverse, weight_similarity]
weight_noneSet the distance weighting method used. 
do_not_check_capabilitiesbooleanfalseEnable capability-check.
mean_squaredbooleanfalseSet whether the mean squared error is used rather than mean absolute error when doing cross-validation.
nearest_neighbours_search_algorithmenum 
[LinearNNSearch, BallTree, CoverTree, KDTress, FilteredNeighbourSearch]
LinearNNSearchSet the nearestNeighbourSearch algorithm to be used for finding nearest neighbor(s).
num_decimal_placesint2Set the number of decimal places.
window_sizeint0 (no window size)Set the maximum number of instances allowed in the training pool.

Logistic Regression

Option Name
Option Data Type
Default Value
Description
batch_sizeint100Set the preferred batch size for batch prediction.
do_not_check_capabilitiesbooleanfalseEnable capability-check.
matx_itsint-1Set the maximum number of iterations.
num_decimal_placesint4Set the number of decimal places.
ridgedouble1.00E-08Sets the ridge in the log-likelihood.
use_conjugate_gradient_descentbooleanfalseSets whether conjugate gradient descent is used.

Naive Bayes

Option Name
Data Type
Default Value
Description
batch_sizeint100Set the preferred batch size for batch prediction.
do_not_check_capabilitiesbooleanfalseEnable capability-check.
num_decimal_placesint2Set the number of decimal places.
use_kernel_estimatorbooleanfalseUse kernel density estimator rather than normal distribution for numeric attributes. This is false if use_supervised_discretization is set to true.
use_supervised_discretizationbooleanfalseUse supervised discretization to process numeric attributes. This is false if use_kernel_estimator is set to true.

Support Vector Machines

Option Name
Data Type 
Default Value
Description
batch_sizeint100Set the preferred batch size for batch prediction.
cdouble1Set the complexity constant C.
build_calibration_modelsbooleanfalseSet whether to fit calibration models to SVM outputs.
checks_turned_offbooleanfalseDisable all checks.
do_not_check_capabilitiesbooleanfalseEnable capability-check.
epsilondouble1.00E-12Set the epsilon for round-off error.
filter_typeenum 
[filter_none, filter_normalize, filter_standardize]
filter_normalizeSet how the training data will be transformed.
kernelenum 
[PolyKernel, NormalizedPolyKernel, PrecomputedKernelMatrixKernel, Puk, RBFKernel, StringKernel]
PolyKernelSpecify the Kernel to use.
num_decimal_placesint2Set the number of decimal places.
num_foldsint-1Set the number of folds for the internal cross-validation.
random_seedint1Set the random number seed.
tolerance_parameterdouble1.00E-03Set the tolerance parameter.

Decision Stump

Option Name
Data Type
Default Value
Description
batch_sizeint100Set the preferred batch size for batch prediction.
do_not_check_capabilitiesbooleanfalseEnable capability-check.
num_decimal_placesint2Set the number of decimal places.

Random Forests

Option Name
Data Type
Default Value
Description
bag_size_percentint100Specify the size of each bag, as a percentage of the training set size.
batch_sizeint100Set the preferred batch size for batch prediction.
break_ties_randomlybooleanfalseSet whether to break ties randomly when several attributes look equally good.
calc_out_of_bagbooleanfalseSet whether to calculate the out-of-bag error.
compute_attribute_importancebooleanfalseSet whether compute and output attribute importance (mean impurity decrease method)
do_not_check_capabilitiesbooleanfalseEnable capability-check.
max_depthint0Set the maximum depth of the tree, 0 for unlimited.
num_decimal_placesint2Set the number of decimal places.
num_execution_slotsint1Set the  number of execution slots.
num_featuresint0Set the  number of attributes to randomly investigate.
num_iterationsint100Set the number of iterations.
seedint1Set the seed for random number generator.
store_out_of_bag_predictionsbooleanfalseSet whether to store out-of-bag predictions in internal evaluation object.

Multilayer Perceptron

Option Name
Data Type
Default Value
Description
batch_sizeint100Set the preferred batch size for batch prediction.
do_not_check_capabilitiesbooleanfalseEnable capability-check.
learning_ratedouble0.3Set the learning rate for the backward propagation. The value should be between 0 - 1.
momentumdouble0.2Set the momentum rate for the backward propagation algorithm.The value should be between 0 - 1.
num_decimal_placesint2Set the number of decimal places.
seedint0Set the value used to seed the random number generator.
validation_set_sizeint0
Set the percentage size of validation set to use to terminate training. The value should be between 0 - 1.
validation_thresholdint20
Set the number of consecutive increases of error allowed for validation testing before training terminates.


Examples

Weight Balance Classification – Cross Validation

This pipeline demonstrates a typical cross validation exercise for a dataset before an model is trained for predictions. The dataset is a record of the weight on each side of a weighing scale, the distance of each scale from the ground, and its status of balance. The cross validation is to validate the model's ability to predict this status of balance. 

Download this pipeline.

 Understanding the pipeline

Input

The input document is generated by the CSV Generator Snap and is composed of four fields, one classification field, and three numeric fields:

  • Balance Class: The classification field to denote the status of the weighing scale. B for Balanced, L for Left-inclined, and R for Right-inclined.
  • Left Weight
  • Left Distance
  • Right Weight
  • Right Distance


This dataset has been sourced from UCI Dataset Archive.


Objective

Use Cross Validator (Classification) Snap to evaluate how each ML algorithm performs in this dataset.

Data Preparation

This input document is passed through the Type Converter Snap that is configured to automatically detect and convert the data types. In any ML pipeline, you must first analyze the input document using the Profile Snap and the Type Inspector Snap to ensure that there are no null values or that the data types are accurate. This step is skipped in this example for simplicity's sake.

Cross Validation

After preparing the data, the first thing to do is K-fold Cross Validation. Cross Validator (Classification) Snap takes the full dataset and randomly splits the dataset into training set and test set which are used to evaluate the selected ML algorithm.

Below is the configuration of the Cross Validator (Classification) Snap:

The output from this Snap is as shown below, it can be seen that the selected algorithm performs at 92% accuracy for the provided dataset.

Optionally, you can write the output from the Cross Validator (Classification) Snap into a file using the downstream File Writer Snap.

You can now train the model using this algorithm in the Trainer (Classification) Snap. See Weight Balance Classification – Model Training for details. 

Additional Example

The following use case demonstrates a real-world scenario for using this Snap:

Downloads

  File Modified

File Weight_Balance_Classification_Cross_Validation.slp

Nov 08, 2018 by Vidya Patil


See Also

Snap Pack History

 Click to view/expand

4.27 (main12833)

  • No updates made.

4.26 (main11181)

  • No updates made.

4.25 (main9554)

  • No updates made.

4.24 (main8556)

  • No updates made.

4.23 (main7430)

  • No updates made.

4.22 (main6403)

  • No updates made.

4.21 (snapsmrc542)

  • No updates made.

4.20 Patch mlcore8770

  • Adds the log4j dependency to the ML Core Snaps to resolve the "Could not initialize class org.apache.log4j.LogManager" error. 

4.20 (snapsmrc535)

  • No updates made.

4.19 (snapsmrc528)

  • No updates made.

4.18 (snapsmrc523)

  • No updates made.

4.17 Patch ALL7402

  • Pushed automatic rebuild of the latest version of each Snap Pack to SnapLogic UAT and Elastic servers.

4.17 (snapsmrc515)

  • New Snap: Introducing the Clustering Snap that performs exploratory data analysis to find hidden patterns or groupings in data.
  • Enhanced the AutoML Snap. You can now:
    • Select algorithms to derive the top models.
    • Input the best model generated by another AutoML Snap from a previous execution.
    • View an interactive HTML report that contains statistics of up to 10 models.
  • Added the Snap Execution field to all Standard-mode Snaps. In some Snaps, this field replaces the existing Execute during preview check box.

4.16 (snapsmrc508)

  • New Snap: Introducing the AutoML Snap, which lets you automate the process of selecting machine learning algorithms and tuning hyperparameters. This Snap gives the best predictive model within the specified time limit.

4.15 (snapsmrc500)

  • New Snap Pack. Perform data modeling operations such as model training, cross-validation, and model-based predictions. Additionally, you can also execute Python scripts remotely. Snaps in this Snap Pack are: 
    • Cross Validator -- Classification
    • Cross Validator -- Regression
    • Predictor -- Classification
    • Predictor -- Regression
    • Remote Python Script
    • Trainer -- Classification
    • Trainer -- Regression
  • Releases the Remote Python Executor account and the Remote Python Executor Dynamic account for the Remote Python Script Snap.