Project 1: Binary Classification Of Sonar Returns

In this project tutorial you will discover how to effectively use the Keras library in your machine learning project by working through a binary classification project step-by-step. After completing this step-by-step tutorial, you will know:

• How to load training data and make it available to Keras.
• How to design and train a neural network for tabular data.
• How to evaluate the performance of a neural network model in Keras on unseen data.
• How to perform data preparation to improve skill when using neural networks.
• How to tune the topology and configuration of neural networks in Keras.

1.1 Sonar Object Classification Dataset

The dataset we will use in this tutorial is the Sonar dataset. This is a dataset that describes sonar chirp returns bouncing off different surfaces. The 60 input variables are the strength of the returns at different angles. It is a binary classification problem that requires a model to differentiate rocks from metal cylinders.

It is a well understood dataset. All of the variables are continuous and generally in the range of 0 to 1. The output variable is a string M for mine and R for rock, which will need to be converted to integers 1 and 0. The dataset contains 208 observations. The dataset is in the bundle of source code provided with this book. Alternatively, you can download the dataset and place it in your working directory with the filename sonar.csv

  A benefit of using this dataset is that it is a standard benchmark problem. This means that we have some idea of the expected skill of a good model. Using cross validation, a neural network should be able to achieve performance around 84% with an upper bound on accuracy for custom models at around 88%. You can learn more about this dataset on the UCI Machine Learning repository.

1.2 Baseline Neural Network Model Performance

Let's create a baseline model and result for this problem. We will start off by importing all of the classes and functions we will need.

 

# import all of the classes and function we will need
import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

Next we can initialize the random number generator to ensure that we always get the same results when executing this code. This will help if we are debugging.

# Initialize the random number generator
seed = 7
np.random.seed(seed)

 Now we can load the dataset using Pandas and split the columns into 60 input variables (X) and 1 output variable (Y ). We use Pandas to load the data because it easily handles strings (the output variable), whereas attempting to load the data directly using NumPy would be more diffcult

# Load The Dataset And Separate Into Input and Output Variable.

# load dataset
dataframe = pd.read_csv("sonar.csv",header=None)
dataset = dataframe.values
# split into input (X) and output (Y) variables
X = dataset[:,0:60].astype(float)
Y = dataset[:,60]

 The output variable is string values. We must convert them into integer values 0 and 1. We can do this using the LabelEncoder class from scikit-learn. This class will model the encoding required using the entire dataset via the fit() function, then apply the encoding to create a new output variable using the transform() function.

# Label Encode Output Variable

# encode class values as integers
encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)

 

for key,value in enumerate(encoder.classes_):
    print(value,"=",key)

 

We are now ready to create our neural network model using Keras. We are going to use scikit-learn to evaluate the model using stratified k-fold cross validation. This is a resampling technique that will provide an estimate of the performance of the model. To use Keras models with scikit-learn, we must use the KerasClassifier wrapper. This class takes a function that creates and returns our neural network model. It also takes arguments that it will pass along to the call to fit() such as the number of epochs and the batch size. Let’s start off by defining the function that creates our baseline model. Our model will have a single fully connected hidden layer with the same number of neurons as input variables. This is a good default starting point when creating neural networks on a new problem. The weights are initialized using a small Gaussian random number. The Rectifier activation function is used. The output layer contains a single neuron in order to make predictions. It uses the sigmoid activation function in order to produce a probability output in the range of 0 to 1 that can easily and automatically be converted to crisp class values. Finally, we are using the logarithmic loss function (binary crossentropy) during training, the preferred loss function for binary classification problems. The model also uses the efficient Adam optimization algorithm for gradient descent and accuracy metrics will be collected when the model is trained.

# MLP for Pima Indians Dataset with 10-fold cross validation via sklearn
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import cross_val_score
import numpy as np


# Define and Compile Baseline Model

# baseline model
def create_baseline():
    # create model
    model = Sequential()
    model.add(Dense(60, input_dim=60,kernel_initializer='normal',activation='relu'))
    model.add(Dense(1, kernel_initializer='normal',activation='sigmoid'))
    # Compile model
    model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
    return model

 

Now it is time to evaluate this model using stratified cross validation in the scikit-learn framework. We pass the number of training epochs to the KerasClassifier, again using reasonable default values. Verbose output is also turned o↵ given that the model will be created 10 times for the 10-fold cross validation being performed.

# evaluate model with standardized dataset
estimator = KerasClassifier(build_fn=create_baseline, epochs=100, batch_size=5, verbose=0)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
results = cross_val_score(estimator, X, encoded_Y, cv=kfold)
print("Baseline: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))

 The full code listing is provided below for completeness.

# import all of the classes and function we will need
import numpy as np
import pandas as pd
from keras.callbacks import History

from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Initialize the random number generator
seed = 7 
np.random.seed(seed)

# Load dataset
dataframe = pandas.read_csv("sonar.csv",header=None)
dataset = dataframe.values

# split into input(X) and output(Y) variable

X = dataset[:,0:60].astype(float)
Y = dataset[:,60]

# Label Encode Output Variable

# encode class values as integers

from sklearn import preprocessing
encoder = preprocessing.LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)

for key, value in enumerate(encoder.classes_):
    print(value,"=",key)
    

#MLP for Pima Indians Dataset with 10-fold cross validation via sklearn

# Function to create model,required for kerasClassifier
def create_baseline():
    # create model
    model = Sequential()
    model.add(Dense(60, input_dim=60,kernel_initializer='normal',activation='relu'))
    model.add(Dense(1, kernel_initializer='normal',activation='sigmoid'))
    # Compile model
    model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
    return model

#evaluate model with standardized dataset

estimator = KerasClassifier(build_fn=create_baseline,epochs=100,batch_size=5, verbose=0)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
results = cross_val_score(estimator, X, encoded_Y, cv=kfold)
print("Results: %.2f%% (%.2f%%)" %(results.mean()*100,results.std()*100))


# evaluate baseline model with standardized dataset
np.random.seed(seed)
estimators=[]
estimators.append(('standardize',StandardScaler()))
estimators.append(('mlp',KerasClassifier(build_fn=create_baseline,epochs=100,batch_size=5,verbose=0)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
results = cross_val_score(pipeline,X, encoded_Y, cv=kfold)
print("Standardized: %.2f%% (%.2f%%)" % (results.mean()*100,results.std()*100))

 

 

Running this code produces the following output showing the mean and standard deviation of the estimated accuracy of the model on unseen data.

Baseline: 80.261906% (8.22%)

1.3 Improve Performance With Data Preparation

It is a good practice to prepare your data before modeling. Neural network models are especially suitable to having consistent input values, both in scale and distribution. An effective data preparation scheme for tabular data when building neural network models is standardization. This is where the data is rescaled such that the mean value for each attribute is 0 and the standard deviation is 1. This preserves Gaussian and Gaussian-like distributions whilst normalizing the central tendencies for each attribute.

We can use scikit-learn to perform the standardization of our Sonar dataset using the StandardScaler class. Rather than performing the standardization on the entire dataset, it is good practice to train the standardization procedure on the training data within the pass of across validation run and to use the trained standardization instance to prepare the unseen test fold. This makes standardization a step in model preparation in the cross validation process and it prevents the algorithm having knowledge of unseen data during evaluation, knowledge that might be passed from the data preparation scheme like a crisper distribution.

       We can achieve this in scikit-learn using a Pipeline class. The pipeline is a wrapper that executes one or more models within a pass of the cross validation procedure. Here, we can define a pipeline with the StandardScaler followed by our neural network model.

# Binary Classification with Sonar Dataset: Standardized
import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# load dataset
dataframe = pd.read_csv("sonar.csv",header=None)
dataset = dataframe.values

# split into input(X) and output(Y) variables
X = dataset[:,0:60].astype(float)
Y = dataset[:,60]

#encode class values as integers
encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)

# baseline model
def create_baseline():
    #create model
    model = Sequential()
    model.add(Dense(60,input_dim=60, kernel_initializer='normal',activation='relu'))
    model.add(Dense(1, kernel_initializer='normal',activation='sigmoid'))
    # Compile model
    model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
    return model

# evaluate baseline model with standardized dataset

np.random.seed(seed)
estimators = []
estimators.append(('standardize',StandardScaler()))
estimators.append(('mlp',KerasClassifier(build_fn=create_baseline,epochs=100,batch_size=5,verbose=0)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
results = cross_val_score(pipeline, X, encoded_Y, cv=kfold)
print("Standardized: %.2f%% (%.2f%%)" %(results.mean()*100,results.std()*100))

 Running this example provides the results below. We do see a small but very nice lift in the mean accuracy.

Standardized: 83.12% (7.37%)

1.4 Tuning Layers and Neurons in The Model

There are many things to tune on a neural network, such as the weight initialization, activation functions, optimization procedure and so on. One aspect that may have an outsized e↵ect is the structure of the network itself called the network topology. In this section we take a look at two experiments on the structure of the network: making it smaller and making it larger. These are good experiments to perform when tuning a neural network on your problem.

11.4.1 Evaluate a Smaller Network

I suspect that there is a lot of redundancy in the input variables for this problem. The data describes the same signal from di↵erent angles. Perhaps some of those angles are more relevant than others. We can force a type of feature extraction by the network by restricting the representational space in the first hidden layer.

        In this experiment we take our baseline model with 60 neurons in the hidden layer and reduce it by half to 30. This will put pressure on the network during training to pick out the most important structure in the input data to model. We will also standardize the data as in the previous experiment with data preparation and try to take advantage of the small lift in performance.

# Binary Classification with Sonar Dataset: Standardized Smaller
import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# load dataset
dataframe = pd.read_csv("sonar.csv", header=None)
dataset = dataframe.values


# split into input(X) and output (Y) variables

X = dataset[:,0:60].astype(float)
Y = dataset[:, 60]

# encode class values as integers
encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)

# smaller model

def create_smaller():
    #create model
    model = Sequential()
    model.add(Dense(30, input_dim=60, kernel_initializer='normal',activation='relu'))
    model.add(Dense(1, kernel_initializer='normal',activation='sigmoid'))
    # compile model
    model.compile(loss='binary_crossentropy',optimizer='adam', metrics=['accuracy'])
    return model

np.random.seed(seed)
estimators = []
estimators.append(('standardize',StandardScaler()))
estimators.append(('mlp', KerasClassifier(build_fn=create_smaller,epochs=100,batch_size=5,verbose=0)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=10,shuffle=True, random_state=seed)
results = cross_val_score(pipeline,X, encoded_Y, cv=kfold)
print("Smaller: %.2f%% (%.2f%%)" % (results.mean()*100,results.std()*100))

Running this example provides the following result. We can see that we have a very slight boost in the mean estimated accuracy and an important reduction in the standard deviation(average spread) of the accuracy scores for the model. This is a great result because we are doing slightly better with a network half the size. which in turn takes half the time to train.

Smaller: 82.67% (7.32%)

1.4.2 Evaluate a Larger Network

A neural network topology with more layers o↵ers more opportunity for the network to extract key features and recombine them in useful nonlinear ways. We can evaluate whether adding more layers to the network improves the performance easily by making another small tweak to the function used to create our model. Here, we add one new layer (one line) to the network that introduces another hidden layer with 30 neurons after the first hidden layer. Our network now has the topology:

60 inputs -> [60 -> 30] -> 1 output

      The idea here is that the network is given the opportunity to model all input variables before being bottlenecked and forced to halve the representational capacity, much like we did in the experiment above with the smaller network. Instead of squeezing the representation of the inputs themselves, we have an additional hidden layer to aid in the process.

# Binary Classification with Sonar Dataset: Standardized Larger
import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# load dataset
dataframe = pd.read_csv("sonar.csv",header=None)
dataset = dataframe.values

# split into input (X) and output (Y) variables

X = dataset[:,0:60].astype(float)
Y = dataset[:,60]

# encode class values as integers
encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)


# larger model

def create_larger():
    # creat model
    model = Sequential()
    model.add(Dense(60, input_dim=60, kernel_initializer='normal',activation='relu'))
    model.add(Dense(30, kernel_initializer='normal',activation='relu'))
    model.add(Dense(1,kernel_initializer='normal',activation='sigmoid'))
    # Compile model
    model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
    return model


np.random.seed(seed)
estimators = []
estimators.append(('standardize',StandardScaler()))
estimators.append(('mlp',KerasClassifier(build_fn=create_larger,epochs=100,batch_size=5,verbose=0)))
pipeline = Pipeline(estimators)
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
results = cross_val_score(pipeline, X, encoded_Y,cv=kfold)
print("Larger: %.2f%% (%.2f%%)" %(results.mean()*100, results.std()*100))

Running this example produces the results below. We can see that we do get a nice lift in the model performance, achieving near state-of-the-art results with very little e↵ort indeed.

Larger: 83.62% (6.35%)

      With further tuning of aspects like the optimization algorithm and the number of training epochs, it is expected that further improvements are possible. What is the best score that you can achieve on this dataset?

1.5 Summary

In this lesson you discovered how you can work through a binary classification problem step-by-step with Keras,specifically:

• How to load and prepare data for use in Keras.
• How to create a baseline neural network model
• How to evaluate a Keras model using scikit-learn and stratified k-fold cross validation
• How data preparation schemes can lift the performance of your models.
• How experiments adjusting the network topology can lift model performance.

1.5.1 Next

You now know how to develop neural network models in Keras for multiclass and binary classification problems. In the next tutorial you will work through a project to develop neural network models for a regression problem.