Decision Surface And Plotting

Implementation

We will be using Breast Cancer Wisconsin(Diagnostic) dataset for our work.

Importing all the necessary libraries

import pandas as pd import numpy as np pd.set_option("display.max_rows", None, "display.max_columns", None)  from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.decomposition import IncrementalPCA from sklearn.model_selection import GridSearchCV from sklearn.linear_model import LogisticRegression from sklearn import metrics import matplotlib.pyplot as plt import seaborn as sns %matplotlib inline

 

Reading the dataset and displaying first 10 rows.

df = pd.read_csv('data.csv') df.head(10)

 

 

Performing Label Encoding on Target Feature

df['diagnosis'] = df['diagnosis'].map({'B':0, 'M':1})

 

Dropping the Unwanted Columns

df.drop(['id', 'Unnamed: 32'], axis = 1, inplace = True)

 

Differentiating Dependent and Independent Features

x = df.iloc[:, 1:] y = df['diagnosis']

 

x

 

 

Splitting the Input Dataset

x_train, x_test, y_train, y_test = train_test_split(x, y, train_size = 0.6,test_size = 0.4, random_state = 1234)

 

We split the dataset into a 60-40 ratio.

 

Feature Scaling

columns = x_train.columns scalerx = StandardScaler() x_train_scaled = scalerx.fit_transform(x_train) x_train_scaled = pd.DataFrame(x_train_scaled, columns = columns) x_test_scaled = scalerx.transform(x_test) x_test_scaled = pd.DataFrame(x_test_scaled, columns = columns)

 

x_train_scaled

 

 

PCA

We reduce the dimension of the feature to two,as we can plot  decision surface for 2-d grid.

pca = IncrementalPCA(n_components = 2) x_train_pca = pca.fit_transform(x_train_scaled) x_test_pca = pca.transform(x_test_scaled)

Incremental Principal Component Analysis selects two features to explain as much variance as possible. We apply PCA in both testing and training data.

 

Plotting the Scatter-Plot for both Training and Testing dataset

plt.figure(figsize = (20, 6)) plt.subplot(1, 2, 1) plt.scatter(x_train_pca[:,0], x_train_pca[:,1], c = y_train) plt.xlabel('Training 1st Principal Component') plt.ylabel('Training 2nd Principal Component') plt.title('Training Set Scatter Plot with labels indicated by colors, (0)-Violet,(1)-Yellow') plt.subplot(1, 2, 2) plt.scatter(x_test_pca[:,0], x_test_pca[:,1], c = y_test) plt.xlabel('Test 1st Principal Component') plt.ylabel('Test 2nd Principal Component') plt.title('Test Set Scatter Plot with labels indicated by colors, (0)-Violet (1)-Yellow') plt.show()

 

We can see the distinction between the two classes and possibly imagine the decision surface, maybe a correct diagonal between the two.

 

Performing Cross-Validation

params = {'C':[0.01, 0.1, 1, 10, 100]} clf = LogisticRegression() folds = 5 model_cv = GridSearchCV(estimator = clf,                          param_grid = params,                          scoring= 'accuracy',                          cv = folds,                         return_train_score=True,                         verbose = 3) model_cv.fit(x_train_pca, y_train)

We perform a 5-fold grid-search cross-validation on logistic regression classifier on the training set.

 

Best Hyperparameter from Grid-Search CV performed Above

print(model_cv.best_params_)

 

Output

{'C': 10}

 

Re-Training the model with best parameters

model = LogisticRegression(C = 10).fit(x_train_pca, y_train)

 

We re-train our model with c=10 obtained after performing hyperparameter tuning.

 

Predictions

y_train_pred = model.predict(x_train_pca) y_test_pred = model.predict(x_test_pca)

 

Performance Analysis of the model in terms of different performance metrics

print('Training Accuracy of the Model: ', metrics.accuracy_score(y_train, y_train_pred)) print('Test Accuracy of the Model: ', metrics.accuracy_score(y_test, y_test_pred)) print() print('Training Precision of the Model: ', metrics.precision_score(y_train, y_train_pred)) print('Test Precision of the Model: ', metrics.precision_score(y_test, y_test_pred))

 

 

Visualization of Decision Surface

We can create a decision boundary by fitting the model on the training data, then using the same model to make predictions for a grid of values for the input domain.

Once we have the grid of predictions, we can plot the values and their class label.

The best possible approach to visualize decision boundaries is to use a contour plot that can interpolate the colors between the points. We can use the contourf()function for plotting the decision surface.

We have to follow specific steps.

Firstly, we need to define the grid points in the whole feature space.

To do this, first, we find the maximum and minimum values for each feature and increase it by one step beyond that to ensure that the whole feature space is covered.

x_min, x_max = x_train_pca[:, 0].min() - 1, x_train_pca[:, 0].max() + 1 y_min, y_max = x_train_pca[:, 1].min() - 1, x_train_pca[:, 1].max() + 1

 

An arrange() function creates a uniform sample at a particular resolution across each dimension. We will use the meshgrid() function to create a grid of the two input vectors.

xx_train, yy_train = np.meshgrid(np.arange(x_min, x_max, 0.1),                                 np.arange(y_min, y_max, 0.1)) Z_train = model.predict(np.c_[xx_train.ravel(), yy_train.ravel()]) Z_train = Z_train.reshape(xx_train.shape)

 

Similarly for test dataset.

x_min, x_max = x_test_pca[:, 0].min() - 1, x_test_pca[:, 0].max() + 1 y_min, y_max = x_test_pca[:, 1].min() - 1, x_test_pca[:, 1].max() + 1 xx_test, yy_test = np.meshgrid(np.arange(x_min, x_max, 0.1),                               np.arange(y_min, y_max, 0.1)) Z_test = model.predict(np.c_[xx_test.ravel(), yy_test.ravel()]) Z_test = Z_test.reshape(xx_test.shape)

 

Now we have grid values across the feature space.

The contourf() function takes different grids for each of the axes. Then, we plot the decision surface with a two-color colormap.

plt.figure(figsize = (20, 6)) plt.subplot(1, 2, 1) plt.contourf(xx_train, yy_train, Z_train) plt.scatter(x_train_pca[:, 0], x_train_pca[:, 1], c = y_train, s = 30, edgecolor = 'k') plt.xlabel('Training 1st Principal Component') plt.ylabel('Training 2nd Principal Component') plt.title('Scatter Plot with Decision Boundary for the Training Set') plt.subplot(1, 2, 2) plt.contourf(xx_test, yy_test, Z_test) plt.scatter(x_test_pca[:, 0], x_test_pca[:, 1], c = y_test, s = 30, edgecolor = 'k') plt.xlabel('Test 1st Principal Component') plt.ylabel('Test 2nd Principal Component') plt.title('Scatter Plot with Decision Boundary for the Test Set') plt.show()

 

 

So we can see how the contourf() function plotted a beautiful decision boundary. With the help of the above screenshot, we can visualize how the input features are assigned their class labels.

 

Frequently Asked Questions

1. How is the optimal decision boundary determined?
   A classification problem is a rule that partitions the features and assigns features of a partition to the same class. The ‘boundary’ of this partitioning is the decision boundary of the rule. The boundary that this rule produces is the optimal decision boundary.
    
2. How do you determine decision boundaries in logistic regression?
   The logistic regression decision boundary is the set of all points that satisfy the equation P(y=1|x)=P(y=0|x)=½.
    
3. How does decision tree decision boundary differs from that of logistic regression?
   Logistic regression decision boundary divides the feature space into precisely two halves with the help of a single line, whereas decision trees divide the space into smaller and smaller areas.
    
4. What kind of decision boundary is built by a logistic regression classifier?
   In the case of logistic regression, the decision boundary is a straight line,i.e., it comes up with a hyperplane live SVM that divides the feature space into two different classes.

 

Key Takeaways

Let us brief the article,

Firstly, we saw a decision boundary, enhancing our visualization and providing a clear explanation of how data inputs are classified. Lastly, we saw how to implement a decision boundary using logistic regression classifier.

I recommend you all apply the same steps using another classification model to understand it better. Thus we can use decision surface and performance metrics to evaluate the model's performance.

That is the end of the article. Stay tuned for more exciting articles.

Keep Learning Ninjas!