# Support Vector Machine (SVM) Classifier

November 24, 2020 2021-02-13 13:47## Support Vector Machine (SVM) Classifier

So far, we have learned about various techniques that help predict the class based on its features and their relationship. In this lesson, we will be addressing the concept of an algorithm based on Support Vectors that can be used for classifying data. Support-vector machines are supervised learning models with associated learning algorithms that analyze data used for classification and regression analysis.

An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on the side of the gap on which they fall. The objective of the support vector machine algorithm is to find a hyperplane in N-dimensional space that distinctly classifies the data points. A Support Vector Classification model uses the following hyperparameters in its model that determine the performance of the model.

**Hyper Plane**: The separation line between the data classes.**Decision****Boundary line**: The boundary lines are essentially the decision boundaries of the hyperplane. The support vectors can be on the Boundary lines or outside it. The best fit line is determined on the basis of how well the hyperplane separates the maximum number of points outside its boundary line.**Support Vectors**are the data points that are closest to the decision boundary. The distance of the points from the boundary is minimum or least.

**Support Vector Classification in Python**

Now that we know the basic idea of support vectors, we will now discuss a step-wise Python implementation of the algorithm.

**1. Importing necessary libraries**

Before we begin to develop a model and understand it, let us import some essential Python libraries for mathematical calculations, data loading, preprocessing, and model development and prediction.

# Importing the libraries import numpy as np import pandas as pd import matplotlib.pyplot as plt %matplotlib inline # scikit-learn modules from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.svm import SVC from sklearn.metrics import confusion_matrix, accuracy_score, classification_report # For plotting the classification results from mlxtend.plotting import plot_decision_regions

**2. Importing the dataset**

For this problem, we will be loading the Breast Cancer dataset from scikit-learn. The dataset consists of data related to breast cancer patients and their diagnosis (malignant or benign).

# Importing the dataset dataset = load_breast_cancer() # Converting to pandas dataframe df = pd.DataFrame(dataset.data, columns = dataset.feature_names) df['target'] = pd.Series(dataset.target) df.head()

print("Total samples in our dataset is: {}".format(df.shape[0]))

Total samples in our dataset is: 569

dataset.describe()

**3. Separating the features and target variable**

After loading the data set, the independent variable ($x$) and the dependent variable ($y$) need to be separated. Our concern is to find the relationships between the features and the target variable from the above dataset.

For this implementation example, we will only be using the ‘mean perimeter’ and ‘mean texture’ features but you can certainly use all of them.

# Selecting the features features = ['mean perimeter', 'mean texture'] x = df[features] # Target Variable y = df['target']

**4. Splitting the data set into training and test set **

After separating the independent variables ($x$) and dependent variable $(y)$, these values are split into train and test sets to train and evaluate the linear model. We use the train_test_split() module of scikit-learn for splitting the available data into an 80-20 split. We will be using twenty percent of the available data as the test set and the remaining data as the train set.

# Splitting the dataset into the training and test set x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.20, random_state = 25 )

**5. Fitting the model to the training set**

After splitting the data into dependent and independent variables, the Support Vector Classifier model is fitted with the training data using the SVC() class from scikit-learn.

# Fitting SVC to the Training set model = SVC(kernel = 'linear', random_state = 0) model.fit(x_train, y_train)

SVC(kernel='linear', random_state=0)

**6. Predicting the test results**

Finally, the model is tested on the data to get the predictions.

# Predicting the results y_pred = model.predict(x_test)

**7. Evaluating the model**

Let us now evaluate the model using the confusion matrix and calculate its classification accuracy. Confusion matrix determines the performance of the predicted model. Other metrics such as the precision, recall and f1-score are given by the classification report module of scikit-learn.

**Precision** defines the ratio of correctly predicted positive observations of the total predicted positive observations. It defines how accurate the model is. **Recall** defines the ratio of correctly predicted positive observations to all observations in the actual class. **F1 Score** is the weighted average of Precision and Recall and is often used as a metric in place of accuracy for imbalanced datasets.

# Confusion matrix print("Confusion Matrix") matrix = confusion_matrix(y_test, y_pred) print(matrix) # Classification Report print("\nClassification Report") report = classification_report(y_test, y_pred) print(report) # Accuracy of the model accuracy = accuracy_score(y_test, y_pred) print('SVC Accuracy of the model: {:.2f}%'.format(accuracy*100))

Confusion Matrix [[28 11] [ 8 67]] Classification Report precision recall f1-score support 0 0.91 0.77 0.83 39 1 0.89 0.96 0.92 75 accuracy 0.89 114 macro avg 0.90 0.86 0.88 114 weighted avg 0.90 0.89 0.89 114 SVC Accuracy of the model: 89.47%

Hence, the model is working quite well with an accuracy of *89.47%*.

**8. Plotting the decision boundary**

We will now plot the decision boundary of the model on test data.

# Plotting the decision boundary plot_decision_regions(x_test.values, y_test.values, clf = model, legend = 2) plt.title("Decision boundary using SVC (Test)") plt.xlabel("mean_perimeter") plt.ylabel("mean_texture");

Hence, the plot shows the distinction between the two classes as classified by the Support Vector Classification algorithm in Python.

**Putting it all together**

The final code for the implementation of **Support Vector Classification in Python** is as follows.

# Importing the libraries import numpy as np import pandas as pd import matplotlib.pyplot as plt %matplotlib inline # scikit-learn modules from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.svm import SVC from sklearn.metrics import confusion_matrix, accuracy_score, classification_report # Plotting the classification results from mlxtend.plotting import plot_decision_regions # Importing the dataset dataset = load_breast_cancer() # Converting to pandas DataFrame df = pd.DataFrame(dataset.data, columns = dataset.feature_names) df['target'] = pd.Series(dataset.target) print("Total samples in our dataset is: {}".format(df.shape[0])) # Describe the dataset df.describe() # Selecting the features features = ['mean perimeter', 'mean texture'] x = df[features] # Target variable y = df['target'] # Splitting the dataset into the training and test set x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.20, random_state = 25 ) # Fitting SVC to the Training set model = SVC(kernel = 'linear', random_state = 0) model.fit(x_train, y_train) # Predicting the results y_pred = model.predict(x_test) # Confusion matrix print("Confusion Matrix") matrix = confusion_matrix(y_test, y_pred) print(matrix) # Classification Report print("\nClassification Report") report = classification_report(y_test, y_pred) print(report) # Accuracy of the model accuracy = accuracy_score(y_test, y_pred) print('SVC Accuracy of the model: {:.2f}%'.format(accuracy*100)) # Plotting the decision boundary plt.figure(figsize=(10,6)) plot_decision_regions(x_test.values, y_test.values, clf = model, legend = 2) plt.title("Decision boundary using SVC (Test)") plt.xlabel("mean_perimeter") plt.ylabel("mean_texture")

In this lesson, we discussed the concept of Support Vector Classifier along with its implementation in Python. In the next lesson, we will discuss classifiers based on Stochastic Gradient Descent algorithm.