自编码实例6：变分自编码

       自编码可以降维重构样本，而变分自编码学习的不再是样本的个体，而是学习样本的规律。这样训练出来的自编码不单具有重构样本的功能，还具有仿照样本的功能。

       变分自编码，其实就是在编码过程中改变了样本的分布（变分可以理解为改变分布）。假设我们知道样本的分布函数，就可以从这个函数中随便取一个样本，然后进行网络解码层前向传导，这样就可以生成一个新的样本。

       为了得到这个样本的分布函数，模型训练的目的将不再是样本本身，而是通过加一个约束项，将网络生成一个服从于高斯分布的数据集，这样按照高斯分布里的均值和方差规则就可以任意取相关的数据，然后通过解码层还原成样本。

实例：使用变分自编码模拟生成MNIST数据。主要公式为KL离散度的计算。编码器为两个全连接层，第一个全连接层有784个维度的输入变化256个维度的输出；第二个全连接层并列连接了两个输出网络（mean和lg_var），每个网络都输出了两个维度的输出。然后将两个输出通过一个公式的计算，输入到以一个2节点 为开始的解码部分，接着后面为两个全连接层的解码器，第一层由两个维度的输入到256个维度的输出，第二层由256个维度的输入到784个维度的输出。

使用KL离散度公式，来计算它所代表的集合与标准的高斯分布集合之间的距离，将这个距离当成误差，让它最小化从而优化网络参数。

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
from scipy.stats import norm

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/data/")#, one_hot=True)

n_input = 784
n_hidden_1 = 256
n_hidden_2 = 2

x = tf.placeholder(tf.float32, [None, n_input])
zinput = tf.placeholder(tf.float32, [None, n_hidden_2])  # 通过它输入分布数据，用来生成模拟样本数据

weights = {
    'w1': tf.Variable(tf.truncated_normal([n_input, n_hidden_1], stddev=0.001)),
    'b1': tf.Variable(tf.zeros([n_hidden_1])),

    'mean_w1': tf.Variable(tf.truncated_normal([n_hidden_1, n_hidden_2],stddev=0.001)),
    'mean_b1': tf.Variable(tf.zeros([n_hidden_2])),
    'log_sigma_w1': tf.Variable(tf.truncated_normal([n_hidden_1, n_hidden_2],stddev=0.001)), 
    'log_sigma_b1': tf.Variable(tf.zeros([n_hidden_2]))
        
    'w2': tf.Variable(tf.truncated_normal([n_hidden_2, n_hidden_1],stddev=0.001)),
    'b2': tf.Variable(tf.zeros([n_hidden_1])),
    
    'w3': tf.Variable(tf.truncated_normal([n_hidden_1, n_input],stddev=0.001)),
    'b3': tf.Variable(tf.zeros([n_input])),
}

h1=tf.nn.relu(tf.add(tf.matmul(x, weights['w1']), weights['b1']))
z_mean = tf.add(tf.matmul(h1, weights['mean_w1']), weights['mean_b1'])
z_log_sigma_sq = tf.add(tf.matmul(h1, weights['log_sigma_w1']), weights['log_sigma_b1'])

 # sample from gaussian distribution
eps = tf.random_normal(tf.stack([tf.shape(h1)[0], n_hidden_2]), 0, 1, dtype = tf.float32)
z =tf.add(z_mean, tf.multiply(tf.sqrt(tf.exp(z_log_sigma_sq)), eps))
h2=tf.nn.relu( tf.matmul(z, weights['w2'])+ weights['b2'])
reconstruction = tf.matmul(h2, weights['w3'])+ weights['b3']

h2out=tf.nn.relu( tf.matmul(zinput, weights['w2'])+ weights['b2'])
reconstructionout = tf.matmul(h2out, weights['w3'])+ weights['b3']

# cost
reconstr_loss = 0.5 * tf.reduce_sum(tf.pow(tf.subtract(reconstruction, x), 2.0))
latent_loss = -0.5 * tf.reduce_sum(1 + z_log_sigma_sq
                                   - tf.square(z_mean)
                                   - tf.exp(z_log_sigma_sq), 1)
cost = tf.reduce_mean(reconstr_loss + latent_loss)
optimizer = tf.train.AdamOptimizer(learning_rate = 0.001).minimize(cost)

training_epochs = 50
batch_size = 128
display_step = 3
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    
    for epoch in range(training_epochs):
        avg_cost = 0.
        total_batch = int(mnist.train.num_examples/batch_size)

        # 遍历全部数据集
        for i in range(total_batch):
            batch_xs, batch_ys = mnist.train.next_batch(batch_size)#取数据
            
    
            # Fit training using batch data
            _,c = sess.run([optimizer,cost], feed_dict={x: batch_xs})
            #c = autoencoder.partial_fit(batch_xs)
        # 显示训练中的详细信息
        if epoch % display_step == 0:
            print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(c))

    print("完成!")
    
    # 测试
    print ("Result:", cost.eval({x: mnist.test.images}))
    
    # 可视化结果
    show_num = 10
    pred = sess.run(
        reconstruction, feed_dict={x: mnist.test.images[:show_num]})

    f, a = plt.subplots(2, 10, figsize=(10, 2))
    for i in range(show_num):
        a[0][i].imshow(np.reshape(mnist.test.images[i], (28, 28)))
        a[1][i].imshow(np.reshape(pred[i], (28, 28)))
    plt.draw()
    
    pred = sess.run(
        z, feed_dict={x: mnist.test.images})
    #x_test_encoded = encoder.predict(x_test, batch_size=batch_size)
    plt.figure(figsize=(6, 6))
    plt.scatter(pred[:, 0], pred[:, 1], c=mnist.test.labels)
    plt.colorbar()
    plt.show()

    # display a 2D manifold of the digits
    n = 15  # figure with 15x15 digits
    digit_size = 28
    figure = np.zeros((digit_size * n, digit_size * n))
    grid_x = norm.ppf(np.linspace(0.05, 0.95, n))
    grid_y = norm.ppf(np.linspace(0.05, 0.95, n))
    
    for i, yi in enumerate(grid_x):
        for j, xi in enumerate(grid_y):
            z_sample = np.array([[xi, yi]])
            x_decoded = sess.run(reconstructionout,feed_dict={zinput:z_sample})
            
            digit = x_decoded[0].reshape(digit_size, digit_size)
            figure[i * digit_size: (i + 1) * digit_size,
                   j * digit_size: (j + 1) * digit_size] = digit
    
    plt.figure(figsize=(10, 10))
    plt.imshow(figure, cmap='Greys_r')
    plt.show()