Handling Missing Values in Python

Real world data is messy and often contains a lot of missing values. There could be multiple reasons for the missing values but primarily the reason for missingness can be attributed to

Either way we need to address this issue before we proceed with the modeling stuff. It is also important to note that some algorithms like XGBoost and LightGBM can treat missing data without any preprocessing.

Objective¶

The objective of this notebook is to detect missing values and then go over some of the methods used for imputing them.

Data¶

There are two publically available datasets which will be used to explain the concepts:

• 1. Titanic Dataset for Non Time Series problem
• 2. Air Quality Data in India (2015 - 2020) for Time Series problem

Loading necessary libraries and datasets¶

Reading in the dataset¶

• Reading in the Titanic Dataset.

Examining the Target column¶

For each passenger in the test set, we need to create a model that predicts whether or not the passengers survived the sinking of the Titanic. Hence Survived is that target column in the dataset. Let’s examine the Distribution of the target column

Here:

• 0: Did not Survive while
• 1: Survived.

Clearly, less people survived the accident.

Since the focus of the notebook is to detect and handle missing values, we’ll jump directly into it. Let’s now look at a step by step process to manage the missing values in a dataset.

Detecting Missing values¶

Detecting missing values numerically¶

The first step is to detect the count/percentage of missing values in every column of the dataset. This will give an idea about the distribution of missing values.

Both the train and test set have the same proportion of the missing values.

Detecting missing data visually using Missingno library¶

To graphically analyse the missingness of the data, let’s use a library called Missingno It is a package for graphical analysis of missing values. To use this library, we need to import it as follows: import missingno as msno

The bar chart above gives a quick graphical overview of the completeness of the dataset. We can see that Age, Cabin and embarked columns have missing values. Next,it would make sense to find out the locations of the missing data.

Visualizing the locations of the missing data¶

The msno.matrix nullity matrix is a data-dense display which lets you quickly visually pick out patterns in data completion.

• The plot appears blank(white) wherever there are missing values. For instance, in Embarked column there are only two instances of missing data, hence the two white lines.

• The sparkline on the right gives an idea of the general shape of the completeness of the data and points out the row with the minimum nullities and the total number of columns in a given dataset, at the bottom.

It is also possible to sample the dataset to pinpoint the exact location of the missing values. For instance let’s check the first 100 rows.

This shows that the Embarked column has no nullity in the first 100 cases.

Reasons for Missing Values¶

Before we start treating the missing values ,it is important to understand the various reasons for the missingness in data. Broadly speaking, there can be three possible reasons:

**1. Missing Completely at Random (MCAR) **

The missing values on a given variable (Y) are not associated with other variables in a given data set or with the variable (Y) itself. In other words, there is no particular reason for the missing values.

**2. Missing at Random (MAR) **

MAR occurs when the missingness is not random, but where missingness can be fully accounted for by variables where there is complete information.

**3. Missing Not at Random (MNAR) **

Missingness depends on unobserved data or the value of the missing data itself.

Now let us look at nullity matrix again to see if can find what type of missingness is present in the dataset.

Finding reason for missing data using matrix plot¶

• The Embarked Column has very few missing values and donot seem to be correlated with any other column, Hence, the missingness in Embarked column can be attributed as Missing Completely at Random.

• Both the Age and the Cabin columns have a lot of missing values.This could be a case of MAR as we cannot directly observe the reason for missingness of data in these columns.

The missingno package also let’s us sort the graph by a particluar column. Let’s sort the values by Age and Cabin column to see if there is a pattern in the missing values

Hence it is clear that here is no relation between the missingness in Age and Cabin column.To cement this conclusion further we can also draw a Heatmap among the different variables in the dataset.

Finding reason for missing data using a Heatmap¶

The heatmap function shows that there are no strong correlations between missing values of different features. This is good; low correlations further indicate that the data are MAR.

Finding reason for missing data using Dendrogram¶

A dendogram is a tree diagram of missingness. It groups the highly correlated variables together.

Let’s read the above dendrogram from a top-down perspective:

• Cluster leaves which linked together at a distance of zero fully predict one another’s presence—one variable might always be empty when another is filled, or they might always both be filled or both empty, and so on(missingno documentation)

• the missingness of Embarked tends to be more similar to Age than to Cabin and so on.However, in this particluar case, the correlation is high since Embarked column has a very few missing values.

This dataset doesn’t have much missing values but if you use the same methodology on datasets having a lot of missing values, some interesting pattern will definitely emerge.

Treating Missing values¶

After having identified the patterns in missing values, it is important to treat them too. Here is a flowchart which could prove handy:

Deletions¶

Deletion means to delete the missing values from a dataset. This is however not recommended as it might result in loss of information from the dataset. We should only delete the missing values from a dataset if their proportion is very small. Deletions are further of three types:

Pairwise Deletion¶

Parwise Deletion is used when values are missing completely at random i.e MCAR. During Pairwise deletion, only the missing values are deleted. All operations in pandas like mean,sum etc intrinsically skip missing values.

Listwise Deletion/ Dropping rows¶

During Listwise deletion, complete rows(which contain the missing values) are deleted. As a result, it is also called Complete Case deletion. Like Pairwise deletion, listwise deletions are also only used for MCAR values.

The Age column doesn’t have any missing values.A major diadvantage of Listwise deletion is that a major chunk of data and hence a lot of information is lost. Hence, it is advisable to use it only when the number of missing values is very small.

Dropping complete columns¶

If a column contains a lot of missing values, say more than 80%, and the feature is not significant, you might want to delete that feature. However, again, it is not a good methodology to delete data.

Imputations Techniques for non Time Series Problems¶

Imputation refers to replacing missing data with substituted values.There are a lot of ways in which the missing values can be imputed depending upon the nature of the problem and data. Dependng upon the nature of the problem, imputation techniques can be broadly they can be classified as follows:

Basic Imputation Techniques¶

• Imputating with a constant value
• Imputation using the statistics (mean, median or most frequent) of each column in which the missing values are located

For this we shall use the The SimpleImputer class from sklearn.

Imputations Techniques for Time Series Problems¶

Now let’s look at ways to impute data in a typical time series problem. Tackling missing values in time Series problem is a bit different. The fillna() method is used for imputing missing values in such problems.

• Basic Imputation Techniques
  • ‘ffill’ or ‘pad’ - Replace NaN s with last observed value
  • ‘bfill’ or ‘backfill’ - Replace NaN s with next observed value
  • Linear interpolation method

Time Series dataset¶

The dataset is called Air Quality Data in India (2015 - 2020) Tand it contains air quality data and AQI (Air Quality Index) at hourly and daily level of various stations across multiple cities in India.The dataset has a lot of missing values and and is a classic Time series problem.

I won’t go much into explaining the data since I have done a lot of relatedw work in my kernel titled 😷 Breathe India: COVID-19 effect on Pollution.In this notebook, let’s keep our focus on the missing values only. As id evident, city_day dataframe consists of daily pollution level data of some of the prominent cities in India.

There are a lot of missing values and some of the columns like Xylene and PM10 have more than 50% of the values missing. Let’s now see how we can impute these missing values.

Above we see, there are 3 missing values in the Xylene column.

Imputing using ffill¶

We can see that all missing values have been filled with the last observed values.

Imputation using bfill¶

We can see that all missing values have been filled with the next observed values.

Imputation using Linear Interpolation method¶

Time series data has a lot of variations against time. Hence, imputing using backfill and forward fill isn’t the ebst possible solution to address the missing value problem. A more apt alternative would be to use interpolation methods, where the values are filled with incrementing or decrementing values.

Linear interpolation is an imputation technique that assumes a linear relationship between data points and utilises non-missing values from adjacent data points to compute a value for a missing data point.

Refer to the official documentation for a complete list of interpolation strategies here

Advanced Imputation Techniques¶

Advanced imputation techniques uses machine learning algorithms to impute the missing values in a dataset unlike the previous techniques where we used other column values to predict the missing values. We shall look at the following two techniques in this notebook:

• Nearest neighbors imputation
• Multivariate feature imputation

K-Nearest Neighbor Imputation¶

The KNNImputer class provides imputation for filling in missing values using the k-Nearest Neighbors approach.Each missing feature is imputed using values from n_neighbors nearest neighbors that have a value for the feature. The feature of the neighbors are averaged uniformly or weighted by distance to each neighbor.

Multivariate feature imputation - Multivariate imputation by chained equations (MICE)¶

A strategy for imputing missing values by modeling each feature with missing values as a function of other features in a round-robin fashion. It performns multiple regressions over random sample ofthe data, then takes the average ofthe multiple regression values and uses that value to impute the missing value. In sklearn, it is implemented as follows:

Algorithms which handle missing values¶

Some algprithms like XGBoost and LightGBM can handle missing values without any preprocessing, by supplying relevant parameters.

• XGBoost

• LightGBm

Conclusion¶

Well, there is no single best way to handle missing values. One needs to experiment with different methods and then decide which method is best for a particular problem.

References and good resources¶

• Dealing with Missing Data in Python
• How to Handle Missing Data