CS 20SI: Basic Models in TF

Linear and Logistic Regression in TF

假定我们有线性回归的基础知识.那么考虑这样一个问题:我们经常听到保险公司使用诸如火灾和盗窃等因素来评估一个社区邻里的危险程度. 问题是, 火灾和盗窃之间有依赖关系吗? 换句话说, 找出一个函数可以根据火灾数预测盗窃数.

在这里有一个数据集.数据含义在网站中有说明.

线性回归

我们首先假定这个关系是线性的, 即$y = w\cdot x + b$ . 我们需要找出 $w, b$.

# matplotlib 教程: https://liam0205.me/2014/09/11/matplotlib-tutorial-zh-cn/

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import xlrd

DATA_FILE = 'data/slr05.xls'

# Step1: read data from xls file
book = xlrd.open_workbook(DATA_FILE, encoding_override='utf-8')
sheet = book.sheet_by_index(0)
data = np.asarray([sheet.row_values(i) for i in range(1, sheet.nrows)])
n_samples = sheet.nrows - 1

# Step2: create placeholders for X, Y
X = tf.placeholder(tf.float32, name="X")
Y = tf.placeholder(tf.float32, name="Y")

# Step3: create w and b
w = tf.Variable(0.0, name="w")
b = tf.Variable(0.0, name="b")

# Step4: construct model
Y_predicted = X * w + b

# Step5: loss function
loss = tf.square(Y - Y_predicted, name="loss")

# Step6: use GD to minimize loss function
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss)



with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(1000):
        for x,y in data:
            sess.run(optimizer, feed_dict={X:x,Y:y})

    w_value, b_value = sess.run([w,b])
    print(w_value)
    print(b_value)


# plot data
plot(data[:,0],data[:,1],'.')

# plot linear model
x = range(45)
plot(x,w_value*x + b_value)

show()

1.62028
16.9211

通过上面结果的图, 可以看到线性回归是不容易做精确的.

我们再尝试一个二次曲线吧. $y = w*x^2 + u*x + b$

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import xlrd

DATA_FILE = 'data/slr05.xls'

# Step1: read data from xls file
book = xlrd.open_workbook(DATA_FILE, encoding_override='utf-8')
sheet = book.sheet_by_index(0)
data = np.asarray([sheet.row_values(i) for i in range(1, sheet.nrows)])
n_samples = sheet.nrows - 1

# Step2: create placeholders for X, Y
X = tf.placeholder(tf.float32, name="X")
Y = tf.placeholder(tf.float32, name="Y")

# Step3: create w and b
w = tf.Variable(0.0, name="w")
u = tf.Variable(0.0, name="u")
b = tf.Variable(0.0, name="b")

# Step4: construct model
Y_predicted = X*X * w  + X*u + b

# Step5: loss function
loss = tf.square(Y - Y_predicted, name="loss")

# Step6: use GD to minimize loss function
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(10):
        for x,y in data:
            sess.run(optimizer, feed_dict={X:x,Y:y})

    w_value,u_value,b_value = sess.run([w,u,b])
    print(w_value)
    print(u_value)
    print(b_value)


# plot data
plot(data[:,0],data[:,1],'.')

# plot linear model
x = range(45)
plot(x,w_value*x*x+u_value*x + b_value)

show()

nan
nan
nan

按照 note3 里写的, 算出来是无穷, 不知道怎么回事.

如何知道模型是正确的

Logistics Regression

讲线性回归是无论如何不能没有 Logistics Regression 的.下面来考虑一个问题.

MNIST 手写数字库

mnist 手写数字库是一个非常著名的测试数据集.这里就不对它做任何说明了.下面考虑在 TensorFlow 中使用 Logistics Regression 对这个数据集进行分类.

import time
import numpy as np
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

import time

# Define paramaters for the model
learning_rate = 0.01
batch_size = 128
n_epochs = 60

# Step 1: Read in data
# using TF Learn's built in function to load MNIST data to the folder data/mnist
mnist = input_data.read_data_sets('/data/mnist', one_hot=True)

print("read mnist done.")

# Step 2: create placeholders for features and labels
# each image in the MNIST data is of shape 28*28 = 784
# therefore, each image is represented with a 1x784 tensor
# there are 10 classes for each image, corresponding to digits 0 - 9.
# each lable is one hot vector.
X = tf.placeholder(tf.float32, [batch_size, 784], name='X_placeholder')
Y = tf.placeholder(tf.int32, [batch_size, 10], name='Y_placeholder')

# Step 3: create weights and bias
# w is initialized to random variables with mean of 0, stddev of 0.01
# b is initialized to 0
# shape of w depends on the dimension of X and Y so that Y = tf.matmul(X, w)
# shape of b depends on Y
w = tf.Variable(tf.random_normal(shape=[784, 10], stddev=0.01), name='weights')
b = tf.Variable(tf.zeros([1, 10]), name="bias")

# Step 4: build model
# the model that returns the logits.
# this logits will be later passed through softmax layer
logits = tf.matmul(X, w) + b

# Step 5: define loss function
# use cross entropy of softmax of logits as the loss function
entropy = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=Y, name='loss')
loss = tf.reduce_mean(entropy)  # computes the mean over all the examples in the batch

# Step 6: define training op
# using gradient descent with learning rate of 0.01 to minimize loss
optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss)

with tf.Session() as sess:
    # to visualize using TensorBoard
    writer = tf.summary.FileWriter('./graphs/logistic_reg', sess.graph)

    start_time = time.time()
    sess.run(tf.global_variables_initializer())
    n_batches = int(mnist.train.num_examples / batch_size)
    for i in range(n_epochs):  # train the model n_epochs times
        total_loss = 0

        for _ in range(n_batches):
            X_batch, Y_batch = mnist.train.next_batch(batch_size)
            _, loss_batch = sess.run([optimizer, loss], feed_dict={X: X_batch, Y: Y_batch})
            total_loss += loss_batch
        print('Average loss epoch {0}: {1}'.format(i, total_loss / n_batches))

    print('Total time: {0} seconds'.format(time.time() - start_time))
    print('Optimization Finished!')  # should be around 0.35 after 25 epochs

    # test the model

    preds = tf.nn.softmax(logits)
    correct_preds = tf.equal(tf.argmax(preds, 1), tf.argmax(Y, 1))
    accuracy = tf.reduce_sum(tf.cast(correct_preds, tf.float32))  # need numpy.count_nonzero(boolarr) :(

    n_batches = int(mnist.test.num_examples / batch_size)
    total_correct_preds = 0

    for i in range(n_batches):
        X_batch, Y_batch = mnist.test.next_batch(batch_size)
        # Note: 这里返回类型是 list
        accuracy_batch = sess.run([accuracy], feed_dict={X: X_batch, Y: Y_batch})
        total_correct_preds += accuracy_batch[0]

    print('Accuracy {0}'.format(total_correct_preds / mnist.test.num_examples))

    writer.close()

Extracting /data/mnist\train-images-idx3-ubyte.gz
Extracting /data/mnist\train-labels-idx1-ubyte.gz
Extracting /data/mnist\t10k-images-idx3-ubyte.gz
Extracting /data/mnist\t10k-labels-idx1-ubyte.gz
read mnist done.
Average loss epoch 0: 0.36004507513033773
Average loss epoch 1: 0.31073442212500035
Average loss epoch 2: 0.2984745086618505
Average loss epoch 3: 0.2923075084566931
Average loss epoch 4: 0.2913795799121035
Average loss epoch 5: 0.2891138860428916
Average loss epoch 6: 0.29014808597851965
Average loss epoch 7: 0.28410462371146306
Average loss epoch 8: 0.28576387643241635
Average loss epoch 9: 0.2824974775843418
Average loss epoch 10: 0.2823235393958264
Average loss epoch 11: 0.2828033054735386
Average loss epoch 12: 0.2792769326363301
Average loss epoch 13: 0.28006525218001066
Average loss epoch 14: 0.27658977160475307
Average loss epoch 15: 0.2783060657712647
Average loss epoch 16: 0.2814840956854529
Average loss epoch 17: 0.27844067415758117
Average loss epoch 18: 0.2763183492434247
Average loss epoch 19: 0.27833336361004113
Average loss epoch 20: 0.2779957888836564
Average loss epoch 21: 0.27434623718036205
Average loss epoch 22: 0.2721172578337931
Average loss epoch 23: 0.2732378298349514
Average loss epoch 24: 0.2756466605701637
Average loss epoch 25: 0.27510260722065416
Average loss epoch 26: 0.27023478103660664
Average loss epoch 27: 0.2681298737259141
Average loss epoch 28: 0.27585609026938435
Average loss epoch 29: 0.2754791540389705
Average loss epoch 30: 0.2703300770610675
Average loss epoch 31: 0.27030307393004993
Average loss epoch 32: 0.2740716817761675
Average loss epoch 33: 0.27447062384280807
Average loss epoch 34: 0.27057619799661414
Average loss epoch 35: 0.27220666081112255
Average loss epoch 36: 0.2677810919732376
Average loss epoch 37: 0.2698240237737701
Average loss epoch 38: 0.271830904730124
Average loss epoch 39: 0.2713868394797631
Average loss epoch 40: 0.2682036821479986
Average loss epoch 41: 0.2695189748648923
Average loss epoch 42: 0.2750728232992056
Average loss epoch 43: 0.2677695391758218
Average loss epoch 44: 0.2731832834578643
Average loss epoch 45: 0.26805340835219627
Average loss epoch 46: 0.2707252525881679
Average loss epoch 47: 0.27146211670271453
Average loss epoch 48: 0.26967231906317307
Average loss epoch 49: 0.26819548361237583
Average loss epoch 50: 0.27054036178813684
Average loss epoch 51: 0.26732437343394105
Average loss epoch 52: 0.26716546773719707
Average loss epoch 53: 0.2653121001871001
Average loss epoch 54: 0.267393386795926
Average loss epoch 55: 0.26859270453852047
Average loss epoch 56: 0.2714281693335115
Average loss epoch 57: 0.2667178188508097
Average loss epoch 58: 0.26647953001285735
Average loss epoch 59: 0.2707676462768398
Total time: 146.76355934143066 seconds
Optimization Finished!
Accuracy 0.914