编程与数学

1. 内积与均值

• $\sum_{i}a_i\cdot b_i$：就是内积
• $\frac1n\sum_{i}$：就是均值

2. position & walker

注意理解这两个英文单词所对应的数学含义：

多元高斯概率密度函数：

$$p(\vec x)\propto\exp(-\frac12(\vec x-\vec \mu)^T\Sigma^{-1}(\vec x-\vec \mu))$$

def lnprob(x, mu, icov):
    diff = x - mu
    return -np.dot(diff, icov.dot(diff))/2

It is important that the first argument of the probability function $x$ is the position of a single walker (a $N$-dimensional numpy array).

3. 取模运算

$$x \% y == 0$$

取模运算表达一种周期或者频率（frequency）的意义，

if iter % freq == 0:
    ...

4. 阈值与双重 if

比如，如果这次的改进较上次优化了 %5 以上，则：

best_valid_loss = np.inf
improvement_thresh = 0.95
patience, patience_inc = 5000, 2
for iter in range(n_iters):
    ... 
    if this_valid_loss < best_valid_loss:
        if this_valid_loss < best_valid_loss * improvement_thresh:
            patience = max(patience, iter*patience_inc)
        best_valid_loss = this_valid_loss

5. $e=\lim\limits_{n\to\infty}(1+\frac1n)^n$的收敛性

>>>for n in (2, 4, 10, 50, 100, 1000, 100000):
>>>     print(n, (1./n)**n)
2 2.25
4 2.44140625
10 2.5937424601000023
50 2.691588029073608
100 2.7048138294215285
1000 2.7169239322355936
100000 2.7182682371922975

6. 随机性的实现——各种分布的抽样

二项分布生成掩码，实现对原始数据随机的选择（屏蔽）。

# denoising autoencoder
def get_corrupted_input(self, input, corruption_level):
    mask = self.theano_rng.binomial(n=1,
            p=1-corruption_level,
            size=input.shape, 
            dtype=theano.config.floatX)
    return mask*input
# dropout
def dropout_layer(layer, p_dropout):
    mask = theano_rng.binomial(n=1,
            p=1-p_dropout,
            size=layer.shape,
            dtype=theano.config.floatX)
    return layer*mask

正太分布或者均匀分布（参数由相关定理保证）实现对权值的初始化：

if not W:
    init_W = numpy.asarray(
            numpy_rng.uniform(
                low= -4*numpy.sqrt(6./(n_in+n_out),
                high=4*numpy.sqrt(6./(n_in+n_out)),
                size=(n_in, n_out))
            ),
            dtype=theano.config.floatX
        )
    W = theano.shared(value=init_W, name='W', borrow=True)