多类分类器的混淆矩阵_使用混淆矩阵调试分类器

多类分类器的混淆矩阵

介绍 (Introduction)

A confusion matrix is a visual way to inspect the performance of a classification model. Metrics such as accuracy can be inadequate in cases where there are large class imbalances in the data, a problem common in machine learning applications for fraud detection. A confusion matrix can provide us with a more representative view of our classifier’s performance, including which specific instances it is having trouble classifying.

混淆矩阵是检查分类模型性能的一种直观方法。 在数据中存在大量类别不平衡的情况下，诸如准确性之类的指标可能不足，这是用于欺诈检测的机器学习应用程序中常见的问题。 混淆矩阵可以为我们提供分类器性能的更具有代表性的视图，包括分类器在哪些特定情况下遇到问题。

In this post we are going to illustrate two ways in which Comet’s confusion matrix can help debug classification models.

在本文中，我们将说明Comet的混淆矩阵可以用来帮助调试分类模型的两种方法。

For our first example, we will run an experiment similar to the one illustrated in this post on imbalanced data . We’re going to train a classifier to detect fraudulent transactions in an imbalanced dataset and use Comet’s confusion matrix to evaluate our model’s performance. In our second example we will cover classification on unstructured data with a large number of labels using the CIFAR100 dataset and a simple CNN model.

对于第一个示例，我们将针对不平衡数据运行类似于本文中所示的实验。 我们将训练一个分类器，以检测不平衡数据集中的欺诈交易，并使用Comet的混淆矩阵评估模型的性能。 在我们的第二个示例中，我们将使用CIFAR100数据集和一个简单的CNN模型覆盖具有大量标签的非结构化数据的分类。

数据不平衡的混淆矩阵 (Confusion Matrices with Imbalanced Data)

In this example, we’re going to use the Credit Card Fraud Detection dataset from Kaggle to evaluate our classifier. This dataset is highly imbalanced, with only 492 fraudulent transactions present in a dataset with 284,807 transactions in total. Our model is a single fully connected layer with Dropout enabled. We’re going to train our model using the Adam optimizer for 5 epochs, with a batch size of 64 and use 20% of our dataset for validation.

在此示例中，我们将使用Kaggle的信用卡欺诈检测数据集来评估我们的分类器。 此数据集高度不平衡，数据集中仅存在492个欺诈交易，总共284,807个交易。 我们的模型是启用了Dropout的单个完全连接层。 我们将使用Adam优化器训练5个时期的模型，批处理大小为64，并使用数据集的20％进行验证。

def load_data():
    raw_df = pd.read_csv(
        "https://storage.googleapis.com/download.tensorflow.org/data/creditcard.csv"
    )    return raw_dfdef preprocess(raw_df):
    df = raw_df.copy()    eps = 0.01    df.pop("Time")
    df["Log Amount"] = np.log(df.pop("Amount") + eps)    train_df, val_df = train_test_split(df, test_size=0.2)    train_labels = np.array(train_df.pop("Class"))
    val_labels = np.array(val_df.pop("Class"))    train_features = np.array(train_df)
    val_features = np.array(val_df)    scaler = StandardScaler()
    train_features = scaler.fit_transform(train_features)
    val_features = scaler.transform(val_features)    train_features = np.clip(train_features, -5, 5)
    val_features = np.clip(val_features, -5, 5)    return train_features, val_features, train_labels, val_labels

训练分类器 (Training a classifier)

Our model is a single fully connected layer with Dropout enabled. We’re going to train our model using the Adam optimizer for 5 epochs, with a batch size of 64 and use 20% of our dataset for validation.

我们的模型是启用了Dropout的单个完全连接层。 我们将使用Adam优化器训练5个时期的模型，批处理大小为64，并使用数据集的20％进行验证。

def build_model(input_shape, output_bias=None):
    if output_bias is not None:
        output_bias = tf.keras.initializers.Constant(output_bias)
    model = keras.Sequential(
        [
            keras.layers.Dense(16, activation="relu", input_shape=(input_shape,)),
            keras.layers.Dropout(0.5),
            keras.layers.Dense(1, activation="sigmoid", bias_initializer=output_bias),
        ]
    )    model.compile(
        optimizer=keras.optimizers.Adam(lr=1e-3),
        loss=keras.losses.BinaryCrossentropy(),
        metrics=["accuracy"],
    )    return model

Since we’re using Keras as our modeling framework, Comet will automatically log our hyperparameters, and training metrics (accuracy and loss) to the web UI. At the end of every epoch we’re going to log a confusion matrix of the model’s predictions on our validation dataset using a custom Keras callback and Comet’s `log_confusion_matrix` function.

由于我们使用Keras作为建模框架，因此Comet会自动将我们的超参数和训练指标(准确性和损失)记录到Web UI。 在每个时期结束时，我们将使用自定义Keras回调和Comet的`log_confusion_matrix`函数将模型预测的混淆矩阵记录在验证数据集上。

class ConfusionMatrixCallback(Callback):
    def __init__(self, experiment, inputs, targets, cutoff=0.5):
        s