Illustrating Classification

Note

Today’s example will build on material from the previous lecture (earlier this week).

Predicting Defaults

Today, we will work with UC Irvine’s Credit Default Dataset. I have made a condensed version of it to work with.

library(tidyverse)
library(caret)
Default = read_csv('https://ec242.netlify.app/data/UCI_credit.csv')

About the data

I’ve imported the data from the UCI source. Note that the first column is just a leftover row number from the export – read_csv will call it ...1 – so drop it before you fit anything with default ~ .. Columns PAY_0, PAY_2, PAY_3, PAY_4, PAY_5, and PAY_6 show the last 6 months of payment status (there is no PAY_1 – that’s how UCI shipped it; PAY_0 is the most recent month), where -2 means “no payment required”, -1 and 0 mean paid on time, and 1 and above mean 1 month behind, 2 months behind, etc. Bill and Payment amounts as labeled, with 1 being most recent, and 6 being six months ago. All the dollar figures, LIMIT_BAL included, are New Taiwan dollars – so a LIMIT_BAL of 20000 is not a $20,000 credit line.

Note

In our first breakout:

  1. Check to make sure the default column is a binary indicator for default. Check to make sure there are no surprise NA values (there are!). Then, construct any additional variables you might think useful (hint: the PAY_X variables don’t really have an intuitive numeric interpretation. Should we categorize some of them together?)

  2. Explore the data as we did on Tuesday to get an idea of useful predictors.

  3. Build a logistic model to predict default using any combination of variables and interactions in the data. For now, just use your best judgement for choosing the variables and interactions.

  4. Use a Bayes Classifier cutoff of .50 to generate your classifier output.

Let’s look at how we did. What variables were most useful in explaining default?

Note

In our second breakout, we will create a ROC curve manually. To do this

  1. Take your model from the first breakout, and using a loop (or sapply), step through a large number of possible cutoffs for classification ranging from 0 to 1. Careful at the ends: a cutoff below every fitted probability (or above every one) predicts a single class, so table() gives you a one-row table and confusionMatrix() won’t accept it – the same problem we ran into on Tuesday with the all-“No” classifier. Keep your cutoffs inside the range of your predicted probabilities.

  2. For each cutoff, generate a confusion matrix with accuracy, sensitivity and specificity.

  3. Combine the cutoff with the sensitivity and specificity results and make a ROC plot. Use ggplot for your plot and map the color aesthetic to the cutoff value.

  4. Calculate the AUC (the area under the curve). This is a little tricky but can be done with your data.