【飞桨】【李宏毅机器学习训练营】作业二（年收入判断）分析

最新推荐文章于 2022-08-09 18:13:54 发布

小机灵鬼~

最新推荐文章于 2022-08-09 18:13:54 发布

阅读量436

点赞数

文章标签：机器学习

本文链接：https://blog.csdn.net/qq_37279375/article/details/115211837

版权

# 飞桨 # paddlepaddle #李宏毅机器学习特训营

课程链接：https://aistudio.baidu.com/aistudio/course/introduce/1978

作业的题目要求：

项目描述：二元分类是机器学习中最基础的问题之一，在这份教学中，你将学会如何实作一个线性二元分类器，来根据人们的个人资料，判断其年收入是否高于50,000美元。我们将以两种方法：logistic regression与generative model，来达成以上目的，你可以尝试了解、分析两者的设计理念及差别。实现二分类任务：·个人收入是否超过50000元。

数据集介绍：这个数据集是由UCI Machine Learning Repository 的 Census-Income (KDD) Data Set 经过一些处理而得来，为了方便训练，我们移除了一些不必要的资讯，并且稍微平衡了正负两种标记的比例。事实上在训练过程中，只有X_train, Y_train, X_test这三个经过处理的档案会被用到，train.csv和test.csv这个两个原始资料则可以提供你一些额外的资讯。（1. 已去除不必要的属性。2. 已经平衡正标和负标数据之间的比例。）

特征格式：1. train.csv，test_no_label.csv。（·基于文本的原始数据 ·去掉不必要的属性，平衡正负比例）2. X_train, Y_train, X_test(测试) （·train.csv中的离散特征==>在X_train中onehot编码（学历、状态··· ···） ·train.csv中的连续特征==>在X_train中保持不变（年龄、资本损失··· ···） ·X_train, X_test: 每一行包括一个510-dim的特征，代表一个样本。 ·Y_train: label=0 表示"<50k"、label=1 表示">50k"。）

（声明：以下代码均为老师给的示例代码，我加入了一些注释和原理）

数据准备：下载资料，并且对每个属性做正规化，处理过后再将其切分为train_set 和 develop_set。

import numpy as np

np.random.seed(0)
X_train_fpath = 'work/data/X_train'
Y_train_fpath = 'work/data/Y_train'
X_test_fpath = 'work/data/X_test'
output_fpath = 'work/output_{}.csv'

# Parse csv files to numpy array
with open(X_train_fpath) as f:
    next(f)
    X_train = np.array([line.strip('\n').split(',')[1:] for line in f], dtype = float)
with open(Y_train_fpath) as f:
    next(f)
    Y_train = np.array([line.strip('\n').split(',')[1] for line in f], dtype = float)
with open(X_test_fpath) as f:
    next(f)
    X_test = np.array([line.strip('\n').split(',')[1:] for line in f], dtype = float)

def _normalize(X, train = True, specified_column = None, X_mean = None, X_std = None):
    # This function normalizes specific columns of X.
    # The mean and standard variance of training data will be reused when processing testing data.
    #
    # Arguments:
    #     X: data to be processed
    #     train: 'True' when processing training data, 'False' for testing data
    #     specific_column: indexes of the columns that will be normalized. If 'None', all columns
    #         will be normalized.
    #     X_mean: mean value of training data, used when train = 'False'
    #     X_std: standard deviation of training data, used when train = 'False'
    # Outputs:
    #     X: normalized data
    #     X_mean: computed mean value of training data
    #     X_std: computed standard deviation of training data

    if specified_column == None:
        specified_column = np.arange(X.shape[1])
    if train:
        X_mean = np.mean(X[:, specified_column] ,0).reshape(1, -1)
        X_std  = np.std(X[:, specified_column], 0).reshape(1, -1)

    X[:,specified_column] = (X[:, specified_column] - X_mean) / (X_std + 1e-8)
     
    return X, X_mean, X_std

def _train_dev_split(X, Y, dev_ratio = 0.25):
    # This function spilts data into training set and development set.
    train_size = int(len(X) * (1 - dev_ratio))
    return X[:train_size], Y[:train_size], X[train_size:], Y[train_size:]

# Normalize training and testing data
X_train, X_mean, X_std = _normalize(X_train, train = True)
X_test, _, _= _normalize(X_test, train = False, specified_column = None, X_mean = X_mean, X_std = X_std)
    
# Split data into training set and development set
dev_ratio = 0.1
X_train, Y_train, X_dev, Y_dev = _train_dev_split(X_train, Y_train, dev_ratio = dev_ratio)

train_size = X_train.shape[0]
dev_size = X_dev.shape[0]
test_size = X_test.shape[0]
data_dim = X_train.shape[1]
print('Size of training set: {}'.format(train_size))
print('Size of development set: {}'.format(dev_size))
print('Size of testing set: {}'.format(test_size))
print('Dimension of data: {}'.format(data_dim))

$\begin{array}{l} P({c_1}|x) = \frac{{P(x|{c_1})P({c_1})}}{{P(x|{c_1})P({c_1}) + P(x|{c_2})P({c_2})}}\\ = \frac{1}{{1 + \frac{{P(x|{c_2})P({c_2})}}{{P(x|{c_1})P({c_1})}}}} = \frac{1}{{1 + \exp ( - z)}} \end{array}\$

$z = \ln \frac{{P(x|{C_1})}}{{P(x|{C_2})}} + \ln \frac{{P({C_1})}}{{P({C_2})}}$

$\frac{{P({C_1})}}{{P({C_2})}} = \frac{{\frac{{{N_1}}}{{{N_1} + {N_2}}}}}{{\frac{{{N_2}}}{{{N_1} + {N_2}}}}} = \frac{{{N_1}}}{{{N_2}}}$

$f(X) = \frac{1}{{{{(2\pi )}^{d/2}}|\Sigma {|^{1/2}}}}\exp [ - \frac{1}{2}{(X - \mu )^T}{\Sigma ^{ - 1}}(X - \mu )],X = ({x_1},{x_2}, \cdots ,{x_n})$

$P(x|{C_1}) = \frac{1}{{{{(2\pi )}^{D/2}}}}\frac{1}{{|{\Sigma ^1}{|^{1/2}}}}\exp \{ - \frac{1}{2}{(x - {\mu ^1})^T}{({\Sigma ^1})^{ - 1}}(x - {\mu ^1})\}$

$P(x|{C_2}) = \frac{1}{{{{(2\pi )}^{D/2}}}}\frac{1}{{|{\Sigma ^2}{|^{1/2}}}}\exp \{ - \frac{1}{2}{(x - {\mu ^2})^T}{({\Sigma ^2})^{ - 1}}(x - {\mu ^2})\}$

$z = {({\mu ^1} + {\mu ^2})^T}{\Sigma ^{ - 1}}x - \frac{1}{2}{({\mu ^1})^T}{({\Sigma ^1})^{ - 1}}{\mu ^1} + \frac{1}{2}{({\mu ^2})^T}{({\Sigma ^2})^{ - 1}}{\mu ^2} + \ln \frac{{{N_1}}}{{{N_2}}}$

def _shuffle(X, Y):
    # This function shuffles two equal-length list/array, X and Y, together.
    randomize = np.arange(len(X))
    np.random.shuffle(randomize)
    return (X[randomize], Y[randomize])

# 用于将数组中的元素控制在一个给定的范围[1e-8, (1-(1e-8))]
def _sigmoid(z):
    # Sigmoid function can be used to calculate probability.
    # To avoid overflow, minimum/maximum output value is set.
    return np.clip(1 / (1.0 + np.exp(-z)), 1e-8, 1 - (1e-8))

def _f(X, w, b):
    # This is the logistic regression function, parameterized by w and b
    #
    # Arguements:
    #     X: input data, shape = [batch_size, data_dimension]
    #     w: weight vector, shape = [data_dimension, ]
    #     b: bias, scalar
    # Output:
    #     predicted probability of each row of X being positively labeled, shape = [batch_size, ]
    return _sigmoid(np.matmul(X, w) + b)

def _predict(X, w, b):
    # This function returns a truth value prediction for each row of X 
    # by rounding the result of logistic regression function.
    return np.round(_f(X, w, b)).astype(np.int)
    
def _accuracy(Y_pred, Y_label):
    # This function calculates prediction accuracy
    acc = 1 - np.mean(np.abs(Y_pred - Y_label))
    return acc

def _cross_entropy_loss(y_pred, Y_label):
    # This function computes the cross entropy.
    #
    # Arguements:
    #     y_pred: probabilistic predictions, float vector
    #     Y_label: ground truth labels, bool vector
    # Output:
    #     cross entropy, scalar
    cross_entropy = -np.dot(Y_label, np.log(y_pred)) - np.dot((1 - Y_label), np.log(1 - y_pred))
    return cross_entropy

# X.T 
def _gradient(X, Y_label, w, b):
    # This function computes the gradient of cross entropy loss with respect to weight w and bias b.
    y_pred = _f(X, w, b)
    pred_error = Y_label - y_pred
    w_grad = -np.sum(pred_error * X.T, 1)
    b_grad = -np.sum(pred_error)
    return w_grad, b_grad

# Zero initialization for weights ans bias
w = np.zeros((data_dim,)) 
b = np.zeros((1,))

# Some parameters for training    
max_iter = 10
batch_size = 8
learning_rate = 0.2

# Keep the loss and accuracy at every iteration for plotting
train_loss = []
dev_loss = []
train_acc = []
dev_acc = []

# Calcuate the number of parameter updates
step = 1

# Iterative training
for epoch in range(max_iter):
    # Random shuffle at the begging of each epoch
    X_train, Y_train = _shuffle(X_train, Y_train)
        
    # Mini-batch training
    for idx in range(int(np.floor(train_size / batch_size))):
        X = X_train[idx*batch_size:(idx+1)*batch_size]
        Y = Y_train[idx*batch_size:(idx+1)*batch_size]

        # Compute the gradient
        w_grad, b_grad = _gradient(X, Y, w, b)
            
        # gradient descent update
        # learning rate decay with time
        w = w - learning_rate/np.sqrt(step) * w_grad
        b = b - learning_rate/np.sqrt(step) * b_grad

        step = step + 1
            
    # Compute loss and accuracy of training set and development set
    y_train_pred = _f(X_train, w, b)
    Y_train_pred = np.round(y_train_pred)
    train_acc.append(_accuracy(Y_train_pred, Y_train))
    train_loss.append(_cross_entropy_loss(y_train_pred, Y_train) / train_size)

    y_dev_pred = _f(X_dev, w, b)
    Y_dev_pred = np.round(y_dev_pred)
    dev_acc.append(_accuracy(Y_dev_pred, Y_dev))
    dev_loss.append(_cross_entropy_loss(y_dev_pred, Y_dev) / dev_size)

print('Training loss: {}'.format(train_loss[-1]))
print('Development loss: {}'.format(dev_loss[-1]))
print('Training accuracy: {}'.format(train_acc[-1]))
print('Development accuracy: {}'.format(dev_acc[-1]))

import matplotlib.pyplot as plt

# Loss curve
plt.plot(train_loss)
plt.plot(dev_loss)
plt.title('Loss')
plt.legend(['train', 'dev'])
plt.savefig('loss.png')
plt.show()

# Accuracy curve
plt.plot(train_acc)
plt.plot(dev_acc)
plt.title('Accuracy')
plt.legend(['train', 'dev'])
plt.savefig('acc.png')
plt.show()

多变量生成模型

# Parse csv files to numpy array
with open(X_train_fpath) as f:
    next(f)
    X_train = np.array([line.strip('\n').split(',')[1:] for line in f], dtype = float)
with open(Y_train_fpath) as f:
    next(f)
    Y_train = np.array([line.strip('\n').split(',')[1] for line in f], dtype = float)
with open(X_test_fpath) as f:
    next(f)
    X_test = np.array([line.strip('\n').split(',')[1:] for line in f], dtype = float)

# Normalize training and testing data
X_train, X_mean, X_std = _normalize(X_train, train = True)
X_test, _, _= _normalize(X_test, train = False, specified_column = None, X_mean = X_mean, X_std = X_std)

# Compute in-class mean
X_train_0 = np.array([x for x, y in zip(X_train, Y_train) if y == 0])
X_train_1 = np.array([x for x, y in zip(X_train, Y_train) if y == 1])

mean_0 = np.mean(X_train_0, axis = 0)
mean_1 = np.mean(X_train_1, axis = 0)  

# Compute in-class covariance
cov_0 = np.zeros((data_dim, data_dim))
cov_1 = np.zeros((data_dim, data_dim))

for x in X_train_0:
    cov_0 += np.dot(np.transpose([x - mean_0]), [x - mean_0]) / X_train_0.shape[0]
for x in X_train_1:
    cov_1 += np.dot(np.transpose([x - mean_1]), [x - mean_1]) / X_train_1.shape[0]

# Shared covariance is taken as a weighted average of individual in-class covariance.
cov = (cov_0 * X_train_0.shape[0] + cov_1 * X_train_1.shape[0]) / (X_train_0.shape[0] + X_train_1.shape[0])

# Compute inverse of covariance matrix.
# Since covariance matrix may be nearly singular, np.linalg.inv() may give a large numerical error.
# Via SVD decomposition, one can get matrix inverse efficiently and accurately.
u, s, v = np.linalg.svd(cov, full_matrices=False)
inv = np.matmul(v.T * 1 / s, u.T)

# Directly compute weights and bias
w = np.dot(inv, mean_0 - mean_1)
b =  (-0.5) * np.dot(mean_0, np.dot(inv, mean_0)) + 0.5 * np.dot(mean_1, np.dot(inv, mean_1))\
    + np.log(float(X_train_0.shape[0]) / X_train_1.shape[0]) 

# Compute accuracy on training set
Y_train_pred = 1 - _predict(X_train, w, b)
print('Training accuracy: {}'.format(_accuracy(Y_train_pred, Y_train)))

# Predict testing labels
predictions = 1 - _predict(X_test, w, b)
with open(output_fpath.format('generative'), 'w') as f:
    f.write('id,label\n')
    for i, label in  enumerate(predictions):
        f.write('{},{}\n'.format(i, label))

# Print out the most significant weights
ind = np.argsort(np.abs(w))[::-1]
with open(X_test_fpath) as f:
    content = f.readline().strip('\n').split(',')
features = np.array(content)
for i in ind[0:10]:
    print(features[i], w[i])

小机灵鬼~

关注

0
点赞
踩
4

收藏

觉得还不错? 一键收藏
0
评论
【飞桨】【李宏毅机器学习训练营】作业二（年收入判断）分析

# 飞桨 #paddlepaddle #李宏毅机器学习特训营课程链接：https://aistudio.baidu.com/aistudio/course/introduce/1978作业二（年收入判断）分析：包括写代码规范和二分类logistic regression，概率生成模型的原理分析，损失函数等。（小声BB：因为不会，所以一边写一边研究，主要抠细节）作业的题目要求：项目描述：二元分类是机器学习中最基础的问题之一，在这份教学中，你将学会如何实作一个线性...
复制链接

扫一扫