It is standard practice in the field of machine learning to break a dataset into two separate sets. These sets are set for preparation and set for testing. Holding the training and testing data apart is preferable.
If we don’t divide the dataset into training and testing sets, then we end up testing our model on the same data and training it. We tend to get good precision when we test on the same data we trained our model on.
This does not, however, mean that the model will work as well on unseen results. This is referred to in the field of machine learning as overfitting.
Overfitting is the case when the training dataset is depicted a little too accurately by your model. This means that it blends too closely into your model.
Overfitting while training a model is an undesirable phenomenon. So there’s underfitting.
Underfitting is when the model is not even capable of representing in the training dataset the data points.
How to split a dataset
Import the dataset
Let’s start by importing a dataset into our Python notebook.
In this tutorial, we are going to use the titanic dataset as the sample dataset. You can import the titanic dataset from the seaborn library in Python.
import seaborn as sns titanic = sns.load_dataset('titanic') titanic.head()
Input and output vectors
We need to prepare input and output vectors from the dataset before we continue to break the dataset into training and testing sets.
Let’s consider the column ‘Survived’ as an output. This implies that this model will be equipped to predict whether or not a surviving individual will survive.
y = titanic.survived print(y)
We also need to remove ‘survived‘ column from the dataset to get the input vector.
The split ratio is what portion of the knowledge is going to the training set and what portion of it is going to the research set. Almost always the training set is greater than the research set.
70:30 is the most common split ratio used by data scientists.
A 70:30 split ratio means that 70% of the knowledge will go to the training set and 30% of the dataset will go to the testing set.
We will use train test split from the Sklearn library to split the results.
Train-test-split randomly distributes your information according to the ratio given in training and testing.
We will use 70:30 as the split ratio.
We need to import train-test-split from Sklearn first.
from sklearn.model_selection import train_test_split
To perform the split use :
import seaborn as sns from sklearn.model_selection import train_test_split #import dataset titanic = sns.load_dataset('titanic') #output vector y = titanic.survived #input vector x=titanic.drop('survived',axis=1) #split x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3) #verify print("shape of original dataset :", titanic.shape) print("shape of input - training set", x_train.shape) print("shape of output - training set", y_train.shape) print("shape of input - testing set", x_test.shape) print("shape of output - testing set", y_test.shape)
If you liked reading this article and want to read more, continue to follow the site! We have a lot of interesting articles upcoming in the near future. To stay updated on all the articles, don’t forget to join us along on Twitter and sign up for some interesting reads!