The method cann’t deal with ‘object’ type data
Explore missing values
missing_val_count_by_column = (data.isnull().sum())
print(missing_val_count_by_column[missing_val_count_by_column > 0])
A simple option: Drop columns with Missing values
## drop data
data_without_missing_values = original_data.dropna(axis=1)
## drop the same columns in both training and test data
cols_with_missing = [col for col in training_data.columns
if training_data[col].isnull().any()]
redued_training_data = trainingdata.drop(cols_with_missing, axis=1)
reduced_test_data = test_data.drop(cols_with_missing, axis=1)
## The loses access to this information when the column is dropped. Also, if test data has missing values in places where training data did not, this will result in an error.
A better option: Imputation
Imputation fills in the missing value with some number.
from sklearn.impute import SimpleImputer
my_imputer = SimpleImputer()
data_with_imputed_values = pd.DataFrame(my_imputer.fit_transform(original_data))
## my_imputer.fit_transform(original_data) will be a ndarray of ndarrays and will not be structured like Pandas DataFrame, so use pd.DataFrame .But this will lose the column titles, since the order of columns does not change, so we use code below to get the title.
data_with_imputed_values.columns = original_data.columns
function
def handle_missing_value(original_data):
my_imputer = SimpleImputer()
new_data = pd.DataFrame(my_imputer.fit_transform(original_data))
new_data.columns = original_data.columns
return new_data
An extension to imputation
## make copy to avoid changing original data(when imputing)
new_data = original_data.copy()
## make new columns indicating what will be imputed. These new columns indicate if the value was originally recorded or added in later(by imputation here)
cols_with_missing = (col for col in new_data.columns
if new_data[col].isnull().any())
for col in cols_with_missing:
new_data[col + '_was_missing'] = new_data[col].isnull()
## imputation
my_imputer = SimpleImputer()
new_data = pd.DataFrame(my_imputer.fit_transform(new_data))
new_data.columns = original_data.columns
## In some cases this approach will meaningfully improve results. In other cases, it doesn't help at all.
function
def handle_missing_value(original_data):
new_data = original_data.copy()
cols_with_missing = (col for col in new_data.columns if new_data[col].isnull().any())
for col in cols_with_missing:
new_data[col + '_was_missing'] = new_data[col].isnull()
new_data_columns = new_data.columns
## imputation
my_imputer = SimpleImputer()
new_data = pd.DataFrame(my_imputer.fit_transform(new_data))
new_data.columns = new_data_columns
return new_data