On this Page

1 Problem Scenario
2 Dataset
3 Process Summary
4 Key Snaps
- 4.1 Profiling
- 4.2 Cross Validation
- 4.3 Model Building
- 4.4 Model Hosting
- 4.5 API Testing
- 4.6 Visualization API
5 Profit Analysis
- 5.1 Without ML
- 5.2 With ML
- 5.3 Downloads

Problem Scenario

LendingClub provides peer-to-peer lending services. Someone who wants to invest can open an investor account, or someone who requires money can avail a loan. The platform lends money from investors to borrowers. LendingClub has been operating since 2007 and over 1.5M loans have been approved so far.

Approving a loan is challenging. Firstly, a loan can either be approved or rejected. Then, the approved loan can end up being fully paid or charged off. A strict policy would reduce the charged off rate but would also mean a lesser number of loan approvals. This issue is not only for LendingClub but applies to all financial institutes.

In this use case, we want to build a machine learning model that can identify loans that are likely to end up being charged off. If we succeed, we will be able to reduce the loss from charged off loans and use that money to invest in something else.

The live demo is available at our Machine Learning Showcase.
Check the Profit Analysis section to see how much money we can save with machine learning.

Dataset

There are 2 datasets in this model: approved loans and rejected loans. You can download the datasets from here. In this use case, we focus only on the approved loans and only include fully paid and charged off loans. This dataset contains 646902 fully paid loans and 168084 charged off loans since 2007. Since the term of a loan can be either 36 or 60 months, we have used loans approved until 2014 as a training set, and loans approved in 2015 as a test set. Most of the loans approved after 2015 are in progress and we do not know whether they will be fully paid or charged off, and hence, they are excluded.

The following image shows the preview of the test set. The training set is in the same format except that it does not contain $total_pymnt and $profit. To keep things simple, we compute the profit by subtracting $funded_amnt from $total_pymnt. Those two are used in the profit analysis. The training set and test set can be found here.

Process Summary

Pipeline	Description

Pipeline	Description
	Profiling. This pipeline reads the training set and test set from GitHub, performs type conversion, and computes data statistics which is saved to SnapLogic File System (SLFS) in a JSON format.
	Cross Validation. We have 2 pipelines in this step. The first pipeline (child pipeline) performs k-fold cross validation using a specific ML algorithm. The second pipeline (parent pipeline) uses Pipeline Execute Snap to automate the process of performing k-fold cross validation on multiple algorithms. The Pipeline Execute Snap spawns and executes child pipeline multiple times with different algorithms. Instances of child pipeline can be executed sequentially or in parallel to speed up the process by taking advantage of multi-core processor. The Aggregate Snap applies max function to find the algorithm with the best result.

	Model Building. Based on the cross validation result, there is no best algorithm on this dataset. Most of them perform at the same level. Trainer (Classification) Snap trains Random Forests model which is formatted to JSON and compressed. The compressed JSON is written to SLFS.
	Model Hosting. This pipeline is scheduled as an Ultra Task to provide REST API to external applications. The request comes into an open input view. The core Snap in this pipeline is Predictor (Classification) which hosts the ML model from JSON Parser Snap. Filter Snap drops the requests with an invalid token. Extract Params (Mapper) Snap extracts input from the request. See more information here.
	API Testing. JSON Generator Snap contains a sample request including token and text. REST Post Snap sends a request to the Ultra Task (API). Mapper Snap extracts prediction from the response body. See more information here.
	Visualization API. This pipeline is scheduled as an Ultra Task to provide REST API to external applications. Remote Python Script Snap stores the dataset (from the bottom flow) in memory and generates visualization for each incoming request from the top flow. Filter Snap drops the requests with an invalid token. Extract Params (Mapper) Snap extracts input from the request. See more information here.
	Profit Analysis. This pipeline uses the trained model to predict the charged-off rate of loans in the test set. The Filter Snap rejects some of the loans based on the confidence level. The Aggregate Snaps compute statistics (before and after applying the ML model) including the number of approved loans, total fund, total profit, and average profit per loan. Finally, Mapper Snap computes the percentage of improvement.

Key Snaps

Profiling

Type Converter

File Reader (Read Train / Test Set) Snaps read the datasets which are in CSV format. CSV file does not maintain the data type so the CSV Parser Snap will output data as text represented by String data type. This dataset contains 7 numeric fields (with additional two fields in the test set): loan_amnt, funded_amnt, int_rate, installment, annual_inc, fico_score, and dti. Hence, we need to use the Type Converter Snap to convert them. The output contains the dataset which looks similar to the output of the CSV Parser Snap. However, the numeric fields are now represented as BigInteger and BigDecimal data types.

Profile

This Snap computes data statistics of the dataset. You can see the output of this Snap for training set and for the test set .

The output contains data statistics of this dataset. The following images show the profile of $addr_state and $loan_status in the training set. As you can see, California state has the most loans.

There are 185131 (84.17%) fully paid loans and 34824 (15.83%) charged-off loans.

Cross Validation

Cross Validator (Classification)

This Snap performs 10-fold cross validation using the algorithm specified by _algorithm which is a pipeline parameter passed into this pipeline by the Pipeline Execute Snap in the parent pipeline. You can set the default values for pipeline parameters in the pipeline settings.

The output contains statistics of the cross validation result. $correctly_classified_instances_ratio is the accuracy.

This is the result of the pipeline validation based on the first 50 documents in the dataset. The result based on the full dataset is generated when the pipeline is executed.

CSV Generator

This Snap contains a list of algorithms we want to try. In this case, we specify 5 algorithms. You may try Support Vector Machines or Multilayer Perceptron which will take a longer time to complete.

Pipeline Execute

This Snap spawns and executes the child pipeline (Lending Club - 2 - Cross validation) 5 times, each time with a different algorithm as specified by $algorithm from the CSV Generator Snap. You can specify the Pool Size to be greater than 1 to execute multiple child pipeline instances in parallel.

The output contains statistics of the cross validation results from the child pipeline executions. The $original includes the input of the Pipeline Execute Snap. In this case, it contains $algorithm from the CSV Generator Snap.

This is the result of the pipeline validation based on the first 50 documents in the dataset. The result based on the full dataset will be generated when the pipeline is executed.

Mapper

This Snap outputs the algorithm name together with the accuracy which is defined as $correctly_classified_instances_ratio. The rest of the statistics are removed. You can change this to the metric of your choice.

The output contains the algorithm name together with the accuracy. We use CSV Formatter and File Writer Snaps to write the results to the SnapLogic File System (SLFS) in CSV format. As you can see, decision tree, logistic regression, and decision stump perform at about 84% accuracy. We know that this training set contains 84.17% of fully paid loans. So, it could be that the ML algorithms are biased towards "fully paid" and rarely predict "charged off". Accuracy is not a good metric in this case. Random forests algorithm seems to try harder in predicting "charged off" even though it has slightly lower accuracy.

However, our goal is to reduce the loss and maximize the profit, we will compute the profit improvement based on the test set and apply threshold adjustment in Profit Analysis section. According to the profit analysis, logistic regression and random forests perform the best. In the next section, we will build the random forests model and host it as an API.

Aggregate

This Snap applies max function to find the best accuracy.

Model Building

Trainer (Classification)

This Snap trains the ML model using the random forests algorithm. You can see all the available algorithms with their options here.

The output contains $metadata and $model which are serialized. Since we select Readable in the Trainer (Classification) Snap, the output includes the model in a readable format. You can use Mapper, Document to Binary, and File Writer Snaps to extract $readable and write it out as a text file. Compress Snap is optional, it helps reduce the size of the model file.