In the previous lesson, we discussed Decision Trees and their implementation for classification in Python. It is known that the downside of using Decision trees is their tendency to overfit and high sensitivity to small changes in data.
In this lesson, we will learn about Random Forests that are essentially a collection of many decision trees. Random forests or random decision forests are an ensemble learning method that uses multiple learning algorithms to obtain better predictive performance. It operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (majority).
What is Random Forest Classifier?
Random Forest Classification works by combining various decision trees to result in a final class prediction. The use of multiple trees gives stability to the algorithm and reduces variance. The random forest algorithm is a commonly used model due to its ability to work well for large and most kinds of data.
The algorithm creates each tree from a different sample of input data. Each subset of data undergoes a decision tree process that gives a class prediction. At each node, a different sample of features is selected for splitting and the trees run in parallel without any interaction. The best result is then chosen from the predictions by majority voting, i.e., the class with the majority in predictions is predicted as the final class prediction of the Random Forest.
Random Forest Classifier in Python
Now that we know the basic idea of Random Forest Classification, we will now discuss a step-wise Python implementation of the algorithm.
1. Importing necessary libraries
Before we begin to build our model, let us import some essential Python libraries for mathematical calculations, data loading, preprocessing, and model development and prediction.
# Importing the libraries import numpy as np import pandas as pd import matplotlib.pyplot as plt %matplotlib inline # scikit-learn modules from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import confusion_matrix, accuracy_score, classification_report # For plotting the classification results from mlxtend.plotting import plot_decision_regions
2. Importing the dataset
For this problem, we will be loading the Breast Cancer dataset from scikit-learn. The dataset consists of data related to breast cancer patients and their diagnosis (malignant or benign).
# Importing the dataset dataset = load_breast_cancer() # Converting to pandas dataframe df = pd.DataFrame(dataset.data, columns = dataset.feature_names) df['target'] = pd.Series(dataset.target) df.head()
print("Total samples in our dataset is: {}".format(df.shape[0]))
Total samples in our dataset is: 569
dataset.describe()
3. Separating the features and target variable
After loading the data set, the independent variable ($x$) and the dependent variable ($y$) need to be separated. Our concern is to find the relationships between the features and the target variable from the above dataset.
For this implementation example, we will only be using the ‘mean perimeter’ and ‘mean texture’ features but you can certainly use all of them.
# Selecting the features features = ['mean perimeter', 'mean texture'] x = df[features] # Target Variable y = df['target']
4. Splitting the dataset into training and test set
After separating the independent variables ($x$) and dependent variable $(y)$, these values are split into train and test sets to train and evaluate the linear model. We use the train_test_split() module of scikit-learn for splitting the available data into an 80-20 split. We will be using twenty percent of the available data as the test set and the remaining data as the train set.
# Splitting the dataset into the training and test set x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.20, random_state = 25 )
5. Fitting the model to the training set
After splitting the data into dependent and independent variables, the Random Forest Classifier model is fitted with the training data using the RandomForestClassifier() class from scikit-learn.
# Fitting Random Forest Classifier to the Training set model = RandomForestClassifier(n_estimators = 10, criterion = 'entropy', random_state = 0) model.fit(x_train, y_train)
RandomForestClassifier(criterion='entropy', n_estimators=10, random_state=0)
6. Predicting the test results
Finally, the model is tested on the data to get the predictions.
# Predicting the results y_pred = model.predict(x_test)
7. Evaluating the model
Let us now evaluate the model using confusion matrix and calculate its classification accuracy. Confusion matrix determines the performance of the predicted model. Other metrics such as the precision, recall and f1-score are given by the classification report module of scikit-learn.
Precision defines the ratio of correctly predicted positive observations of the total predicted positive observations. It defines how accurate the model is. Recall defines the ratio of correctly predicted positive observations to all observations in the actual class. F1 Score is the weighted average of Precision and Recall and is often used as a metric in place of accuracy for imbalanced datasets.
# Confusion matrix
print("Confusion Matrix")
matrix = confusion_matrix(y_test, y_pred)
print(matrix)
# Classification Report
print("\nClassification Report")
report = classification_report(y_test, y_pred)
print(report)
# Accuracy of the model
accuracy = accuracy_score(y_test, y_pred)
print('Random Forest Classification Accuracy of the model: {:.2f}%'.format(accuracy*100))
Confusion Matrix
[[28 11]
[ 8 67]]
Classification Report
precision recall f1-score support
0 0.78 0.72 0.75 39
1 0.86 0.89 0.88 75
accuracy 0.83 114
macro avg 0.82 0.81 0.81 114
weighted avg 0.83 0.83 0.83 114
Random Forest Classification Accuracy of the model: 83.33%
Hence, the model is working quite well with an accuracy of 83.33%.
8. Plotting the decision boundary
We will now plot the decision boundary of the model on test data.
# Plotting the decision boundary
plot_decision_regions(x_test.values, y_test.values, clf = model, legend = 2)
plt.title("Decision boundary using Random Forest Classification (Test)")
plt.xlabel("mean_perimeter")
plt.ylabel("mean_texture")
Hence, the plot shows the distinction between the two classes as classified by the Random Forest Classification algorithm in Python.
Putting it all together
The final code for the implementation of Random Forest Classification in Python is as follows.
# Importing the libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
# scikit-learn modules
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
# Plotting the classification results
from mlxtend.plotting import plot_decision_regions
# Importing the dataset
dataset = load_breast_cancer()
# Converting to pandas DataFrame
df = pd.DataFrame(dataset.data, columns = dataset.feature_names)
df['target'] = pd.Series(dataset.target)
print("Total samples in our dataset is: {}".format(df.shape[0]))
# Describe the dataset
df.describe()
# Selecting the features
features = ['mean perimeter', 'mean texture']
x = df[features]
# Target variable
y = df['target']
# Splitting the dataset into the training and test set
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.20, random_state = 25 )
# Fitting Random Forest Classifier to the Training set
model = RandomForestClassifier(n_estimators = 10, criterion = 'entropy', random_state = 0)
model.fit(x_train, y_train)
# Predicting the results
y_pred = model.predict(x_test)
# Confusion matrix
print("Confusion Matrix")
matrix = confusion_matrix(y_test, y_pred)
print(matrix)
# Classification Report
print("\nClassification Report")
report = classification_report(y_test, y_pred)
print(report)
# Accuracy of the model
accuracy = accuracy_score(y_test, y_pred)
print('Random Forest Classification Accuracy of the model: {:.2f}%'.format(accuracy*100))
# Plotting the decision boundary
plt.figure(figsize=(10,6))
plot_decision_regions(x_test.values, y_test.values, clf = model, legend = 2)
plt.title("Decision boundary using Random Forest Classification (Test)")
plt.xlabel("mean_perimeter")
plt.ylabel("mean_texture")
In this lesson, we discussed the concept of Random Forest Classifier along with its implementation in Python. In the next lesson, we will discuss classifiers based on Support Vector Machines.
