Hung-Yi Lee homework[2]:classification
- 一、作业要求
- 二、开始实验
- 2.1 Logistic regression
- 2.1.1 加载数据:
- [addition]open()和with open() as的区别
- [addition]python3 File next()方法
- 2.1.2 归一化
- [addition]python定义函数 返回值只取其中一个
- [addition]np.arange()
- [addition]np.mean()
- 2.1.3 分割训练集-验证集
- 2.1.4 开始训练
- [addition]np.random.shuffle()
- [addition]np.matmul()
- [addition]np.around()
- 2.1.5 绘制Accuracy和Loss的图像
- [addition]python matplotlib线图(plt.plot())
- 2.1.6 在测试集上进行预测
- [addition]numpy.argsort()
- 2.2 Generative model
一、作业要求
鸣谢李宏毅2020机器学习作业2——Classification
资源 提取码:xm8n
根据收集来的资料,判断每个人其年收入是否高于50000美元,用logistic regression和generative model两种方法来实现。
二、开始实验
2.1 Logistic regression
2.1.1 加载数据:
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(0)
# 添加路径
X_train_fpath = 'X_train'
Y_train_fpath = 'Y_train'
X_test_fpath = 'X_test'
output_fpath = 'output_{}.csv'
# 加载数据
with open(X_train_fpath) as f:
next(f)
X_train = np.array([line.strip('\n').split(',')[1:] for line in f], dtype=float)
with open(Y_train_fpath) as f:
next(f)
Y_train = np.array([line.strip('\n').split(',')[1] for line in f], dtype=float)
with open(X_test_fpath) as f:
next(f)
X_test = np.array([line.strip('\n').split(',')[1:] for line in f], dtype=float)
上述代码意思是:不要第一行表头,不要第一列id,其余数据按numpy中ndarray的形式进行存储。
[addition]open()和with open() as的区别
file = open("test.txt","r")
for line in file.readlines():
print line
file.close()
这样直接打开文件,如果出现异常,如读取过程中文件不存在或异常,则直接出现错误,close方法无法执行,文件无法关闭。
with open("test.txt","r") as file:
for line in file.readlines():
print line
用with语句的好处,就是到达语句末尾时,会自动关闭文件,即便出现异常。
[addition]python3 File next()方法
next()方法语法:
next(iterator[,default])
返回值:返回文件下一行
2.1.2 归一化
# 归一化数据
X_train, X_mean, X_std = _normalize(X_train, train=True)
X_test, _, _ = _normalize(X_test, train=False, specified_column=None, X_mean=X_mean, X_std=X_std)
下面是上面代码中提到的_normalize()函数的具体实现:
# 归一化
def _normalize(X, train=True, specified_column=None, X_mean=None, X_std=None):
if specified_column == None:
# 为每个数据添加索值
specified_column = np.arange(X.shape[1])
if train:
# 求取每个数据的平均值和标准差
X_mean = np.mean(X[:, specified_column], 0).reshape(1, -1)
X_std = np.std(X[:, specified_column], 0).reshape(1, -1)
# 归一化数据
X[:, specified_column] = (X[:, specified_column] - X_mean) / (X_std + 1e-8)
# 返回归一化后的数据,均值,标准差
return X, X_mean, X_std
我见识短浅,看到这个specified_column的操作只能直呼“绝了!"
[addition]python定义函数 返回值只取其中一个
python的函数支持返回多个值。返回多个值时,默认以tuple的方式返回
1 def f():
2 return 1,2
3
4 def f():
5 return (1,2)
上面这两种函数的定义是完全等价的。
如果将函数调用的返回值赋值给对应个数的变量,它会一一对应的赋值,这很容易理解。下面是等价的:
1 a, b = f() # a=1, b=2
2 (a, b) = f()
丢弃返回值
很多时候,多个返回值并非全都是所需的,这时需要丢弃某些返回值,python有几种方式:
-
不进行任何赋值,将丢弃所有返回值
-
因为返回值是元组,所以可以通过索引取得某个或某几个返回值
1 a = f()[0] 2 b = f()[1]
-
使用下划线
_
1 # 丢弃第二个返回值 2 a, _ = f()
-
使用双下划线
__
或更多下划线____
[addition]np.arange()
numpy包中的使用arange函数创建数值范围并返回ndarray对象,函数格式如下:
numpy.arange(start, stop, step, dtype)
[addition]np.mean()
返回数组中元素的算术平均值。如果提供了轴,则沿其计算。
2.1.3 分割训练集-验证集
# 设置训练集-验证集
dev_ratio = 0.1
X_train, Y_train, X_dev, Y_dev = _train_dev_split(X_train, Y_train, dev_ratio = dev_ratio)
train_size = X_train.shape[0]
dev_size = X_dev.shape[0]
test_size = X_test.shape[0]
data_dim = X_train.shape[1]
print('Size of training set: {}'.format(train_size))
print('Size of development set: {}'.format(dev_size))
print('Size of testing set: {}'.format(test_size))
print('Dimension of data: {}'.format(data_dim))
其中用到的_train_dev_split()函数实现如下:
# 分割训练集-验证集
def _train_dev_split(X, Y, dev_ratio = 0.25):
# This function spilts data into training set and development set.
train_size = int(len(X) * (1 - dev_ratio))
return X[:train_size], Y[:train_size], X[train_size:], Y[train_size:]
2.1.4 开始训练
# 将w和b初始化为0
w = np.zeros((data_dim,))
b = np.zeros((1,))
# 设置其他超参数(迭代次数,分批次大小,学习率)
max_iter = 10
batch_size = 8
learning_rate = 0.2
# 创建列表用来保存训练集和验证集的损失值和准确度
train_loss = []
dev_loss = []
train_acc = []
dev_acc = []
# 用来更新学习率
step = 1
# 训练
for epoch in range(max_iter):
# 每个epoch都会重新洗牌
X_train, Y_train = _shuffle(X_train, Y_train)
# 分批次训练
for idx in range(int(np.floor(train_size / batch_size))):
X = X_train[idx * batch_size:(idx + 1) * batch_size]
Y = Y_train[idx * batch_size:(idx + 1) * batch_size]
# 计算梯度值
w_grad, b_grad = _gradient(X, Y, w, b)
# 更新参数w和b
# 学习率随着迭代时间增加而减少
w = w - learning_rate / np.sqrt(step) * w_grad
b = b - learning_rate / np.sqrt(step) * b_grad
step = step + 1
# 参数总共更新了max_iter × (train_size/batch_size)次
# 计算训练集的损失值和准确度
y_train_pred = _f(X_train, w, b)
Y_train_pred = np.round(y_train_pred)
train_acc.append(_accuracy(Y_train_pred, Y_train))
train_loss.append(_cross_entropy_loss(y_train_pred, Y_train) / train_size)
# 计算验证集的损失值和准确度
y_dev_pred = _f(X_dev, w, b)
Y_dev_pred = np.round(y_dev_pred)
dev_acc.append(_accuracy(Y_dev_pred, Y_dev))
dev_loss.append(_cross_entropy_loss(y_dev_pred, Y_dev) / dev_size)
print('Training loss: {}'.format(train_loss[-1]))
print('Development loss: {}'.format(dev_loss[-1]))
print('Training accuracy: {}'.format(train_acc[-1]))
print('Development accuracy: {}'.format(dev_acc[-1]))
# 打乱数据顺序,重新为minibatch分配
def _shuffle(X, Y):
# This function shuffles two equal-length list/array, X and Y, together.
randomize = np.arange(len(X))
np.random.shuffle(randomize)
return X[randomize], Y[randomize]
在每次epoch时,都对X和Y进行重新洗牌,从而使得batch-size内的数据可以实现基本上每次不会出现全部一致的情况。
[addition]np.random.shuffle()
在对一些数据处理过程中需要对数据集中的进行顺序打乱,但每条数据中的内容保持不变。
np.random.shuffle可以达到这种效果。
# 计算梯度值
def _gradient(X, Y_label, w, b):
# This function computes the gradient of cross entropy loss with respect to weight w and bias b.
y_pred = _f(X, w, b) # Y_pred是通过激活函数得到的最终的Y的值
pred_error = Y_label - y_pred # Y_label就是我们给出的Y的值
w_grad = -np.sum(pred_error * X.T, 1)
b_grad = -np.sum(pred_error)
return w_grad, b_grad
# 向前传播然后利用sigmoid激活函数计算激活值
def _f(X, w, b):
# This is the logistic regression function, parameterized by w and b
#
# Arguements:
# X: input data, shape = [batch_size, data_dimension]
# w: weight vector, shape = [data_dimension, ]
# b: bias, scalar
# Output:
# predicted probability of each row of X being positively labeled, shape = [batch_size, ]
return _sigmoid(np.matmul(X, w) + b)
[addition]np.matmul()
- 如果两个参数a,b都是2维的,做普通的矩阵相乘。
-
如果某一个参数是N(N>2)维的,该参数被理解为一些矩阵(参数的最后两个维数为矩阵维数)的stack,而且计算时会相应的广播。
首先,对于a,它会被理解成两个 2 × 4 2\times 4 2×4矩阵的stack
同样,对于b,它会被理解成两个 4 × 2 4\times 2 4×2矩阵的stack
对于c,它会被理解成一个 4 × 2 4\times 2 4×2矩阵的stack
-
那么
np.matmul(a,b)
则会将a的第一个矩阵和b的第一个矩阵相乘,将a的第二个矩阵和b的第二个矩阵相乘,最后得到一个 2 × 2 × 2 2\times 2\times 2 2×2×2的结果 -
那么
np.matmul(a,c)
则会将a的第一和第二个矩阵分别与c的一个矩阵相乘,最后得到一个 2 × 2 × 2 2\times 2\times 2 2×2×2的结果 -
如果第一个参数或者第二个参数是1维的,它会提升该参数为矩阵(根据另一个参数维数,给该参数增加一个为1的维数)。矩阵相乘之后会将为1的维数去掉。
上述两种情况在计算矩阵相乘时,会分别将b提升为 2 × 1 2\times 1 2×1的矩阵和 1 × 2 1\times 2 1×2的矩阵
-
乘一个标量是不被允许的,用*代替
matmul与dot的差异——不允许乘标量;stack的矩阵将矩阵按元素对待被一起广播
# sigmoid函数
def _sigmoid(z):
# Sigmoid function can be used to calculate probability.
# To avoid overflow, minimum/maximum output value is set.
return np.clip(1 / (1.0 + np.exp(-z)), 1e-8, 1 - (1e-8))
[addition]np.around()
np.around返回四舍五入后的值,可指定精度。
around(a,decimals=0,out=None)
a 输入数组
decimals 要舍入的小数位数。 默认值为0。 如果为负,整数将四舍五入到小数点左侧的位置
# 准确度
def _accuracy(Y_pred, Y_label):
# This function calculates prediction accuracy
acc = 1 - np.mean(np.abs(Y_pred - Y_label))
return acc
# 交叉熵损失函数
def _cross_entropy_loss(y_pred, Y_label):
# This function computes the cross entropy.
#
# Arguements:
# y_pred: probabilistic predictions, float vector
# Y_label: ground truth labels, bool vector
# Output:
# cross entropy, scalar
cross_entropy = -np.dot(Y_label, np.log(y_pred)) - np.dot((1 - Y_label), np.log(1 - y_pred))
return cross_entropy
2.1.5 绘制Accuracy和Loss的图像
# Loss curve
plt.plot(train_loss)
plt.plot(dev_loss)
plt.title('Loss')
plt.legend(['train', 'dev'])
plt.savefig('loss.png')
plt.show()
# Accuracy curve
plt.plot(train_acc)
plt.plot(dev_acc)
plt.title('Accuracy')
plt.legend(['train', 'dev'])
plt.savefig('acc.png')
plt.show()
[addition]python matplotlib线图(plt.plot())
Matplotlib 是 Python 的绘图库。
以上实例中,np.arange() 函数创建 x 轴上的值。y 轴上的对应值存储在另一个数组对象 y 中。 这些值使用 matplotlib 软件包的 pyplot 子模块的 plot() 函数绘制。
结果如下:
2.1.6 在测试集上进行预测
# 在测试集上进行预测
predictions = _predict(X_test, w, b)
with open(output_path.format('logistic'), 'w') as f:
f.write('id,label\n')
for i, label in enumerate(predictions):
f.write('{},{}\n'.format(i, label))
# Print out the most significant weights
print("weight")
ind = np.argsort(np.abs(w))[::-1]
with open(X_test_path) as f:
content = f.readline().strip('\n').split(',')
features = np.array(content)
for i in ind[0:10]:
print(features[i], w[i])
[addition]numpy.argsort()
numpy.argsort() 函数返回的是数组值从小到大的索引值。
输出结果为:
2.2 Generative model
import numpy as np
np.random.seed(0)
# 添加文件路径
X_train_path = 'X_train'
Y_train_path = 'Y_train'
X_test_path = 'X_test'
output_path = 'output_{}.csv'
# Parse csv files to numpy array
with open(X_train_path) as f:
next(f)
X_train = np.array([line.strip('\n').split(',')[1:] for line in f], dtype=float)
with open(Y_train_path) as f:
next(f)
Y_train = np.array([line.strip('\n').split(',')[1] for line in f], dtype=float)
with open(X_test_path) as f:
next(f)
X_test = np.array([line.strip('\n').split(',')[1:] for line in f], dtype=float)
# 归一化
def _normalize(X, train=True, specified_column=None, X_mean=None, X_std=None):
if specified_column == None:
# 为每个数据添加索值
specified_column = np.arange(X.shape[1])
if train:
# 求取每个数据的平均值和标准差
X_mean = np.mean(X[:, specified_column], 0).reshape(1, -1)
X_std = np.std(X[:, specified_column], 0).reshape(1, -1)
# 归一化数据
X[:, specified_column] = (X[:, specified_column] - X_mean) / (X_std + 1e-8)
# 返回归一化后的数据,均值,标准差
return X, X_mean, X_std
# Normalize training and testing data
X_train, X_mean, X_std = _normalize(X_train, train=True)
X_test, _, _ = _normalize(X_test, train=False, specified_column=None, X_mean=X_mean, X_std=X_std)
# Compute in-class mean
X_train_0 = np.array([x for x, y in zip(X_train, Y_train) if y == 0])
X_train_1 = np.array([x for x, y in zip(X_train, Y_train) if y == 1])
mean_0 = np.mean(X_train_0, axis=0)
mean_1 = np.mean(X_train_1, axis=0)
data_dim = X_train.shape[1]
# Compute in-class covariance
cov_0 = np.zeros((data_dim, data_dim))
cov_1 = np.zeros((data_dim, data_dim))
for x in X_train_0:
cov_0 += np.dot(np.transpose([x - mean_0]), [x - mean_0]) / X_train_0.shape[0]
for x in X_train_1:
cov_1 += np.dot(np.transpose([x - mean_1]), [x - mean_1]) / X_train_1.shape[0]
# Shared covariance is taken as a weighted average of individual in-class covariance.
cov = (cov_0 * X_train_0.shape[0] + cov_1 * X_train_1.shape[0]) / (X_train_0.shape[0] + X_train_1.shape[0])
# sigmoid函数
def _sigmoid(z):
# Sigmoid function can be used to calculate probability.
# To avoid overflow, minimum/maximum output value is set.
return np.clip(1 / (1.0 + np.exp(-z)), 1e-8, 1 - (1e-8))
# 向前传播然后利用sigmoid激活函数计算激活值
def _f(X, w, b):
return _sigmoid(np.matmul(X, w) + b)
# 准确度
def _accuracy(Y_pred, Y_label):
# This function calculates prediction accuracy
acc = 1 - np.mean(np.abs(Y_pred - Y_label))
return acc
# 预测
def _predict(X, w, b):
return np.round(_f(X, w, b)).astype(np.int)
# Compute inverse of covariance matrix.
# Since covariance matrix may be nearly singular, np.linalg.inv() may give a large numerical error.
# Via SVD decomposition, one can get matrix inverse efficiently and accurately.
u, s, v = np.linalg.svd(cov, full_matrices=False)
inv = np.matmul(v.T * 1 / s, u.T)
# Directly compute weights and bias
w = np.dot(inv, mean_0 - mean_1)
b = (-0.5) * np.dot(mean_0, np.dot(inv, mean_0)) + 0.5 * np.dot(mean_1, np.dot(inv, mean_1)) \
+ np.log(float(X_train_0.shape[0]) / X_train_1.shape[0])
# Compute accuracy on training set
Y_train_pred = 1 - _predict(X_train, w, b)
print('Training accuracy: {}'.format(_accuracy(Y_train_pred, Y_train)))
平均值和方差的计算原理:
[addition]python zip()用法
zip() 函数用于将可迭代的对象作为参数,将对象中对应的元素打包成一个个元组,然后返回由这些元组组成的列表。
[addition]numpy.linalg.svd函数
np.linalg.svd(a,full_matrices=1,compute_uv=1)。
参数:
- a:形如(M,N)矩阵
- full_matrices的取值是为0或者1,默认值为1,这时u的大小为(M,M),v的大小为(N,N) 。否则u的大小为(M,K),v的大小为(K,N) ,K=min(M,N)
- compute_uv的取值是为0或者1,默认值为1,表示计算u,s,v。为0的时候只计算s。
返回值:
- 总共有三个返回值u,s,v
- U的大小为(M,M),s的大小为(M,N),v的大小为(N,N)
- A=usv
- 其中s是对矩阵a的奇异值分解。s除了对角元素不为0,其他元素都为0,并且对角元素从大到小排列。s中有n个奇异值,一般排在后面的比较接近0,所以仅保留比较大的r个奇异值。
计算w,b公式的原理如下: