# 过匹配和规范化

>>> import mnist_loader
>>> training_data, validation_data, test_data = \
>>> import network2
>>> net = network2.Network([784, 30, 10], cost=network2.CrossEntropyCost)
>>> net.large_weight_initializer()
>>> net.SGD(training_data[:1000], 400, 10, 0.5, evaluation_data=test_data,
... monitor_evaluation_accuracy=True, monitor_training_cost=True)

>>> import mnist_loader
>>> training_data, validation_data, test_data = \
... mnist_loader.load_data_wrapper()

## 规范化

$$\partial C_0/\partial w$$ 和 $$\partial C_0/\partial b$$ 可以通过反向传播进行计算，和上一章中的那样。所以我们看到其实计算规范化的代价函数的梯度是很简单的：仅仅需要反向传播，然后加上 $$\frac{\lambda}{n} w$$ 得到所有权重的偏导数。而偏差的偏导数就不要变化，所以梯度下降学习规则不会发生变化：

>>> import mnist_loader
>>> training_data, validation_data, test_data = \
>>> import network2
>>> net = network2.Network([784, 30, 10], cost=network2.CrossEntropyCost)
>>> net.large_weight_initializer()
>>> net.SGD(training_data[:1000], 400, 10, 0.5,
... evaluation_data=test_data, lmbda = 0.1,
... monitor_evaluation_cost=True, monitor_evaluation_accuracy=True,
... monitor_training_cost=True, monitor_training_accuracy=True)

>>> net.large_weight_initializer()
>>> net.SGD(training_data, 30, 10, 0.5,
... evaluation_data=test_data, lmbda = 5.0,
... monitor_evaluation_accuracy=True, monitor_training_accuracy=True)

>>> net = network2.Network([784, 100, 10], cost=network2.CrossEntropyCost)
>>> net.large_weight_initializer()
>>> net.SGD(training_data, 30, 10, 0.5, lmbda=5.0,
... evaluation_data=validation_data,
... monitor_evaluation_accuracy=True)

## 为何规范化可以帮助减轻过匹配

I won't show the coefficients explicitly, although they are easy to find using a routine such as Numpy's polyfit
. You can view the exact form of the polynomial in the source code for the graph if you're curious. It's the function p(x)
defined starting on line 14 of the program which produces the graph.

## 规范化的其他技术

L1 规范化：这个方法其实是在代价函数上加上一个权重绝对值的和：

Dropout ：Dropout 是一种相当激进的技术。和 L1、L2 规范化不同，dropout 并不依赖对代价函数的变更。而是，在 dropout 中，我们改变了网络本身。让我在给出为何工作的原理之前描述一下 dropout 基本的工作机制和所得到的结果。

This and the next two graph are produced with the program more_data.py.

Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis, by Patrice Simard, Dave Steinkraus, and John Platt (2003).

## 练习

• 正如上面讨论的那样，一种扩展 MNIST 训练数据的方式是用一些微小的旋转。如果我们允许过大的旋转，则会出现什么状况呢？

## 问题

• 研究问题：我们的机器学习算法在非常大的数据集上如何进行？对任何给定的算法，其实去定义一个随着训练数据规模变化的渐近的性能是一种很自然的尝试。一种简单粗暴的方法就是简单地进行上面图中的趋势分析，然后将图像推进到无穷大。而对此想法的反驳是曲线本身会给出不同的渐近性能。你能够找到拟合某些特定类别曲线的理论上的验证方法吗？如果可以，比较不同的机器学习算法的渐近性能。

===============================================================================================================================

# 权重初始化

## 练习

• 验证 $$z=\sum_j w_j x_j + b$$ 标准差为 $$\sqrt{3/2}$$。下面两点可能会有帮助：（a） 独立随机变量的和的方差是每个独立随即便方差的和；（b）方差是标准差的平方。

>>> import mnist_loader
>>> training_data, validation_data, test_data = \
>>> import network2
>>> net = network2.Network([784, 30, 10], cost=network2.CrossEntropyCost)
>>> net.large_weight_initializer()
>>> net.SGD(training_data, 30, 10, 0.1, lmbda = 5.0,
... evaluation_data=validation_data,
... monitor_evaluation_accuracy=True)

>>> net = network2.Network([784, 30, 10], cost=network2.CrossEntropyCost)
>>> net.SGD(training_data, 30, 10, 0.1, lmbda = 5.0,
... evaluation_data=validation_data,
... monitor_evaluation_accuracy=True)

$$1/\sqrt{n_{in}}$$ 的权重初始化方法帮助我们提升了神经网络学习的方式。其他的权重初始化技术同样也有，很多都是基于这个基本的思想。我不会在这里给出其他的方法，因为 $$1/\sqrt{n_{in}}$$ 已经可以工作得很好了。如果你对另外的思想感兴趣，我推荐你看看在 $$2012$$ 年的 Yoshua Bengio 的论文的 $$14$$ 和 $$15$$ 页，以及相关的参考文献。

Practical Recommendations for Gradient-Based Training of Deep Architectures, by Yoshua Bengio (2012).

## 问题

• 将规范化和改进的权重初始化方法结合使用 L2 规范化有时候会自动给我们一些类似于新的初始化方法的东西。假设我们使用旧的初始化权重的方法。考虑一个启发式的观点：（1）假设$$\lambda$$ 不太小，训练的第一回合将会几乎被权重下降统治。；（2）如果 $$\eta\lambda \ll n$$，权重会按照因子 $$exp(-\eta\lambda/m)$$ 每回合下降；（3）假设 $$\lambda$$ 不太大，权重下降会在权重降到 $$1/\sqrt{n}$$ 的时候保持住，其中 $$n$$ 是网络中权重的个数。用论述这些条件都已经满足本节给出的例子。

# 再看手写识别问题：代码

network.py 一样，主要部分就是 Network 类了，我们用这个来表示神经网络。使用一个  sizes 的列表来对每个对应层进行初始化，默认使用交叉熵作为代价 cost 参数：

class Network(object):

def __init__(self, sizes, cost=CrossEntropyCost):
self.num_layers = len(sizes)
self.sizes = sizes
self.default_weight_initializer()
self.cost=cost

__init__ 方法的和 network.py 中一样，可以轻易弄懂。但是下面两行是新的，我们需要知道他们到底做了什么。

def default_weight_initializer(self):
self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]]
self.weights = [np.random.randn(y, x)/np.sqrt(x)
for x, y in zip(self.sizes[:-1], self.sizes[1:])]

def large_weight_initializer(self):
self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]]
self.weights = [np.random.randn(y, x)
for x, y in zip(self.sizes[:-1], self.sizes[1:])]

class CrossEntropyCost(object):

@staticmethod
def fn(a, y):
return np.sum(np.nan_to_num(-y*np.log(a)-(1-y)*np.log(1-a)))

@staticmethod
def delta(z, a, y):
return (a-y)

class QuadraticCost(object):

@staticmethod
def fn(a, y):
return 0.5*np.linalg.norm(a-y)**2

@staticmethod
def delta(z, a, y):
return (a-y) * sigmoid_prime(z)

"""network2.py
~~~~~~~~~~~~~~

An improved version of network.py, implementing the stochastic
gradient descent learning algorithm for a feedforward neural network.
Improvements include the addition of the cross-entropy cost function,
regularization, and better initialization of network weights.  Note
that I have focused on making the code simple, easily readable, and
easily modifiable.  It is not optimized, and omits many desirable
features.

"""

#### Libraries
# Standard library
import json
import random
import sys

# Third-party libraries
import numpy as np

#### Define the quadratic and cross-entropy cost functions

@staticmethod
def fn(a, y):
"""Return the cost associated with an output a and desired output
y.

"""
return 0.5*np.linalg.norm(a-y)**2

@staticmethod
def delta(z, a, y):
"""Return the error delta from the output layer."""
return (a-y) * sigmoid_prime(z)

class CrossEntropyCost(object):

@staticmethod
def fn(a, y):
"""Return the cost associated with an output a and desired output
y.  Note that np.nan_to_num is used to ensure numerical
stability.  In particular, if both a and y have a 1.0
in the same slot, then the expression (1-y)*np.log(1-a)
returns nan.  The np.nan_to_num ensures that that is converted
to the correct value (0.0).

"""
return np.sum(np.nan_to_num(-y*np.log(a)-(1-y)*np.log(1-a)))

@staticmethod
def delta(z, a, y):
"""Return the error delta from the output layer.  Note that the
parameter z is not used by the method.  It is included in
the method's parameters in order to make the interface
consistent with the delta method for other cost classes.

"""
return (a-y)

#### Main Network class
class Network(object):

def __init__(self, sizes, cost=CrossEntropyCost):
"""The list sizes contains the number of neurons in the respective
layers of the network.  For example, if the list was [2, 3, 1]
then it would be a three-layer network, with the first layer
containing 2 neurons, the second layer 3 neurons, and the
third layer 1 neuron.  The biases and weights for the network
are initialized randomly, using
self.default_weight_initializer (see docstring for that
method).

"""
self.num_layers = len(sizes)
self.sizes = sizes
self.default_weight_initializer()
self.cost=cost

def default_weight_initializer(self):
"""Initialize each weight using a Gaussian distribution with mean 0
and standard deviation 1 over the square root of the number of
weights connecting to the same neuron.  Initialize the biases
using a Gaussian distribution with mean 0 and standard
deviation 1.

Note that the first layer is assumed to be an input layer, and
by convention we won't set any biases for those neurons, since
biases are only ever used in computing the outputs from later
layers.

"""
self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]]
self.weights = [np.random.randn(y, x)/np.sqrt(x)
for x, y in zip(self.sizes[:-1], self.sizes[1:])]

def large_weight_initializer(self):
"""Initialize the weights using a Gaussian distribution with mean 0
and standard deviation 1.  Initialize the biases using a
Gaussian distribution with mean 0 and standard deviation 1.

Note that the first layer is assumed to be an input layer, and
by convention we won't set any biases for those neurons, since
biases are only ever used in computing the outputs from later
layers.

This weight and bias initializer uses the same approach as in
Chapter 1, and is included for purposes of comparison.  It
will usually be better to use the default weight initializer

"""
self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]]
self.weights = [np.random.randn(y, x)
for x, y in zip(self.sizes[:-1], self.sizes[1:])]

def feedforward(self, a):
"""Return the output of the network if a is input."""
for b, w in zip(self.biases, self.weights):
a = sigmoid(np.dot(w, a)+b)
return a

def SGD(self, training_data, epochs, mini_batch_size, eta,
lmbda = 0.0,
evaluation_data=None,
monitor_evaluation_cost=False,
monitor_evaluation_accuracy=False,
monitor_training_cost=False,
monitor_training_accuracy=False):
"""Train the neural network using mini-batch stochastic gradient
descent.  The training_data is a list of tuples (x, y)
representing the training inputs and the desired outputs.  The
other non-optional parameters are self-explanatory, as is the
regularization parameter lmbda.  The method also accepts
evaluation_data, usually either the validation or test
data.  We can monitor the cost and accuracy on either the
evaluation data or the training data, by setting the
appropriate flags.  The method returns a tuple containing four
lists: the (per-epoch) costs on the evaluation data, the
accuracies on the evaluation data, the costs on the training
data, and the accuracies on the training data.  All values are
evaluated at the end of each training epoch.  So, for example,
if we train for 30 epochs, then the first element of the tuple
will be a 30-element list containing the cost on the
evaluation data at the end of each epoch. Note that the lists
are empty if the corresponding flag is not set.

"""
if evaluation_data: n_data = len(evaluation_data)
n = len(training_data)
evaluation_cost, evaluation_accuracy = [], []
training_cost, training_accuracy = [], []
for j in xrange(epochs):
random.shuffle(training_data)
mini_batches = [
training_data[k:k+mini_batch_size]
for k in xrange(0, n, mini_batch_size)]
for mini_batch in mini_batches:
self.update_mini_batch(
mini_batch, eta, lmbda, len(training_data))
print "Epoch %s training complete" % j
if monitor_training_cost:
cost = self.total_cost(training_data, lmbda)
training_cost.append(cost)
print "Cost on training data: {}".format(cost)
if monitor_training_accuracy:
accuracy = self.accuracy(training_data, convert=True)
training_accuracy.append(accuracy)
print "Accuracy on training data: {} / {}".format(
accuracy, n)
if monitor_evaluation_cost:
cost = self.total_cost(evaluation_data, lmbda, convert=True)
evaluation_cost.append(cost)
print "Cost on evaluation data: {}".format(cost)
if monitor_evaluation_accuracy:
accuracy = self.accuracy(evaluation_data)
evaluation_accuracy.append(accuracy)
print "Accuracy on evaluation data: {} / {}".format(
self.accuracy(evaluation_data), n_data)
print
return evaluation_cost, evaluation_accuracy, \
training_cost, training_accuracy

def update_mini_batch(self, mini_batch, eta, lmbda, n):
"""Update the network's weights and biases by applying gradient
descent using backpropagation to a single mini batch.  The
mini_batch is a list of tuples (x, y), eta is the
learning rate, lmbda is the regularization parameter, and
n is the total size of the training data set.

"""
nabla_b = [np.zeros(b.shape) for b in self.biases]
nabla_w = [np.zeros(w.shape) for w in self.weights]
for x, y in mini_batch:
delta_nabla_b, delta_nabla_w = self.backprop(x, y)
nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
self.weights = [(1-eta*(lmbda/n))*w-(eta/len(mini_batch))*nw
for w, nw in zip(self.weights, nabla_w)]
self.biases = [b-(eta/len(mini_batch))*nb
for b, nb in zip(self.biases, nabla_b)]

def backprop(self, x, y):
"""Return a tuple (nabla_b, nabla_w) representing the
gradient for the cost function C_x.  nabla_b and
nabla_w are layer-by-layer lists of numpy arrays, similar
to self.biases and self.weights."""
nabla_b = [np.zeros(b.shape) for b in self.biases]
nabla_w = [np.zeros(w.shape) for w in self.weights]
# feedforward
activation = x
activations = [x] # list to store all the activations, layer by layer
zs = [] # list to store all the z vectors, layer by layer
for b, w in zip(self.biases, self.weights):
z = np.dot(w, activation)+b
zs.append(z)
activation = sigmoid(z)
activations.append(activation)
# backward pass
delta = (self.cost).delta(zs[-1], activations[-1], y)
nabla_b[-1] = delta
nabla_w[-1] = np.dot(delta, activations[-2].transpose())
# Note that the variable l in the loop below is used a little
# differently to the notation in Chapter 2 of the book.  Here,
# l = 1 means the last layer of neurons, l = 2 is the
# second-last layer, and so on.  It's a renumbering of the
# scheme in the book, used here to take advantage of the fact
# that Python can use negative indices in lists.
for l in xrange(2, self.num_layers):
z = zs[-l]
sp = sigmoid_prime(z)
delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
nabla_b[-l] = delta
nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())
return (nabla_b, nabla_w)

def accuracy(self, data, convert=False):
"""Return the number of inputs in data for which the neural
network outputs the correct result. The neural network's
output is assumed to be the index of whichever neuron in the
final layer has the highest activation.

The flag convert should be set to False if the data set is
validation or test data (the usual case), and to True if the
data set is the training data. The need for this flag arises
due to differences in the way the results y are
represented in the different data sets.  In particular, it
flags whether we need to convert between the different
representations.  It may seem strange to use different
representations for the different data sets.  Why not use the
same representation for all three data sets?  It's done for
efficiency reasons -- the program usually evaluates the cost
on the training data and the accuracy on other data sets.
These are different types of computations, and using different
representations speeds things up.  More details on the
representations can be found in

"""
if convert:
results = [(np.argmax(self.feedforward(x)), np.argmax(y))
for (x, y) in data]
else:
results = [(np.argmax(self.feedforward(x)), y)
for (x, y) in data]
return sum(int(x == y) for (x, y) in results)

def total_cost(self, data, lmbda, convert=False):
"""Return the total cost for the data set data.  The flag
convert should be set to False if the data set is the
training data (the usual case), and to True if the data set is
the validation or test data.  See comments on the similar (but
reversed) convention for the accuracy method, above.
"""
cost = 0.0
for x, y in data:
a = self.feedforward(x)
if convert: y = vectorized_result(y)
cost += self.cost.fn(a, y)/len(data)
cost += 0.5*(lmbda/len(data))*sum(
np.linalg.norm(w)**2 for w in self.weights)
return cost

def save(self, filename):
"""Save the neural network to the file filename."""
data = {"sizes": self.sizes,
"weights": [w.tolist() for w in self.weights],
"biases": [b.tolist() for b in self.biases],
"cost": str(self.cost.__name__)}
f = open(filename, "w")
json.dump(data, f)
f.close()

"""Load a neural network from the file filename.  Returns an
instance of Network.

"""
f = open(filename, "r")
f.close()
cost = getattr(sys.modules[__name__], data["cost"])
net = Network(data["sizes"], cost=cost)
net.weights = [np.array(w) for w in data["weights"]]
net.biases = [np.array(b) for b in data["biases"]]
return net

#### Miscellaneous functions
def vectorized_result(j):
"""Return a 10-dimensional unit vector with a 1.0 in the j'th position
and zeroes elsewhere.  This is used to convert a digit (0...9)
into a corresponding desired output from the neural network.

"""
e = np.zeros((10, 1))
e[j] = 1.0
return e

def sigmoid(z):
"""The sigmoid function."""
return 1.0/(1.0+np.exp(-z))

def sigmoid_prime(z):
"""Derivative of the sigmoid function."""
return sigmoid(z)*(1-sigmoid(z))

>>> evaluation_cost, evaluation_accuracy,
... training_cost, training_accuracy = net.SGD(training_data, 30, 10, 0.5,
... lmbda = 5.0,
... evaluation_data=validation_data,
... monitor_evaluation_accuracy=True,
... monitor_evaluation_cost=True,
... monitor_training_accuracy=True,
... monitor_training_cost=True)

## 问题

• 更改上面的代码来实现 L1 规范化，使用 L1 规范化使用 $$30$$ 个隐藏元的神经网络对 MNIST 数字进行分类。你能够找到一个规范化参数使得比无规范化效果更好么？
• 看看 network.py 中的 Network.cost_derivative 方法。这个方法是为二次代价函数写的。怎样修改可以用于交叉熵代价函数上？你能不能想到可能在交叉熵函数上遇到的问题？在 network2.py 中，我们已经去掉了 Network.cost_derivative 方法，将其集成进了 CrossEntropyCost.delta 方法中。请问，这样是如何解决你已经发现的问题的？

©️2019 CSDN 皮肤主题: 技术黑板 设计师: CSDN官方博客