focal loss 多分类问题MNIST手写字学习

focal loss参考
https://blog.csdn.net/u011583927/article/details/90716942

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("/media/liao/My Passport/MNIST/", one_hot=True)
def multi_category_focal_loss1(y_true, y_pred):
    epsilon = 1.e-7
    gamma = 1.0
    #alpha = tf.constant([[2],[1],[1],[1],[1]], dtype=tf.float32)
    alpha = tf.constant([[1],[1],[1],[1],[1],[1],[1],[1],[1],[1]], dtype=tf.float32)

    y_true = tf.cast(y_true, tf.float32)
    y_pred = tf.clip_by_value(y_pred, epsilon, 1. - epsilon)
    y_t = tf.multiply(y_true, y_pred) + tf.multiply(1-y_true, 1-y_pred)
    ce = -tf.log(y_t)
    weight = tf.pow(tf.subtract(1., y_t), gamma)
    fl = tf.matmul(tf.multiply(weight, ce), alpha)
    loss = tf.reduce_mean(fl)
    return loss
        
x = tf.placeholder(tf.float32, [None, 784])
y_ = tf.placeholder(tf.float32, [None, 10])
weight = tf.Variable(tf.zeros([784, 10]))
bias = tf.Variable(tf.zeros([10]))
y = tf.matmul(x, weight) + bias
y_softmax=tf.nn.softmax(y)
#cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=y, labels=tf.argmax(y_, 1))
focal_loss=multi_category_focal_loss1(y_,y_softmax)
cross_entropy=focal_loss
#cross_entropy  = -tf.reduce_mean(tf.multiply(weight_x, tf.log(prob)))
#cross_entropy = focal_loss(logits=y, labels=tf.argmax(y_, 1))
train_op = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

with tf.Session() as sess:
    tf.global_variables_initializer().run()

    validate_feed = {x: mnist.validation.images, y_: mnist.validation.labels}
    test_feed = {x: mnist.test.images, y_: mnist.test.labels}

    for i in range(30000):
        if i % 1000 == 0:
            validate_accuracy = sess.run(accuracy, feed_dict=validate_feed)
            print("After %d trainging step(s) ,validation accuracy"
                  "using average model is %g%%" % (i, validate_accuracy * 100))
	    f_loss=sess.run(focal_loss,feed_dict=validate_feed)
	    print(f_loss,f_loss.shape)
        xs, ys = mnist.train.next_batch(100)
        sess.run(train_op, feed_dict={x: xs, y_: ys})

    test_accuracy = sess.run(accuracy, feed_dict=test_feed)
    print("After 30000 trainging step(s) ,test accuracy using average"
          " model is %g%%" % (test_accuracy * 100))

After 0 trainging step(s) ,validation accuracyusing average model is 9.14%
(2.1671617, ())
After 1000 trainging step(s) ,validation accuracyusing average model is 59.38%
(7.243082, ())
After 2000 trainging step(s) ,validation accuracyusing average model is 57.86%
(7.5029922, ())
After 3000 trainging step(s) ,validation accuracyusing average model is 62.8%
(7.0280437, ())
After 4000 trainging step(s) ,validation accuracyusing average model is 61.74%
(6.5108142, ())
After 5000 trainging step(s) ,validation accuracyusing average model is 58.18%
(7.4595795, ())
After 6000 trainging step(s) ,validation accuracyusing average model is 59.26%
(7.498939, ())
After 7000 trainging step(s) ,validation accuracyusing average model is 59.78%
(7.1583071, ())
After 8000 trainging step(s) ,validation accuracyusing average model is 60.56%
(7.7969065, ())
After 9000 trainging step(s) ,validation accuracyusing average model is 59.78%
(8.9209671, ())
After 10000 trainging step(s) ,validation accuracyusing average model is 59.32%
(7.4151506, ())
After 11000 trainging step(s) ,validation accuracyusing average model is 50.9%
(10.775988, ())
After 12000 trainging step(s) ,validation accuracyusing average model is 62.74%
(7.0292954, ())
After 13000 trainging step(s) ,validation accuracyusing average model is 57.54%
(7.7753515, ())
After 14000 trainging step(s) ,validation accuracyusing average model is 62.86%
(6.4984117, ())
After 15000 trainging step(s) ,validation accuracyusing average model is 56.66%
(9.2939491, ())
After 16000 trainging step(s) ,validation accuracyusing average model is 56.52%
(7.5239344, ())
After 17000 trainging step(s) ,validation accuracyusing average model is 60.24%
(7.5586343, ())
After 18000 trainging step(s) ,validation accuracyusing average model is 61.98%
(6.5576291, ())
After 19000 trainging step(s) ,validation accuracyusing average model is 61.1%
(7.1839633, ())
After 20000 trainging step(s) ,validation accuracyusing average model is 61.98%
(7.9907374, ())
After 21000 trainging step(s) ,validation accuracyusing average model is 62.2%
(6.2956591, ())
After 22000 trainging step(s) ,validation accuracyusing average model is 58.12%
(6.9647093, ())
After 23000 trainging step(s) ,validation accuracyusing average model is 58.6%
(7.16395, ())
After 24000 trainging step(s) ,validation accuracyusing average model is 55.66%
(7.9839039, ())
After 25000 trainging step(s) ,validation accuracyusing average model is 59.2%
(8.3046751, ())
After 26000 trainging step(s) ,validation accuracyusing average model is 61.86%
(6.7330093, ())
After 27000 trainging step(s) ,validation accuracyusing average model is 62.24%
(6.7317719, ())
After 28000 trainging step(s) ,validation accuracyusing average model is 59.66%
(8.54006, ())
After 29000 trainging step(s) ,validation accuracyusing average model is 61.54%
(6.5660267, ())
After 30000 trainging step(s) ,test accuracy using average model is 60.31%

好像不如原本使用softmax交叉商损失函数的情况，如下

After 0 trainging step(s) ,validation accuracyusing average model is 9.14%
After 1000 trainging step(s) ,validation accuracyusing average model is 77.88%
After 2000 trainging step(s) ,validation accuracyusing average model is 81.52%
After 3000 trainging step(s) ,validation accuracyusing average model is 82.76%
After 4000 trainging step(s) ,validation accuracyusing average model is 79.52%
After 5000 trainging step(s) ,validation accuracyusing average model is 73.92%
After 6000 trainging step(s) ,validation accuracyusing average model is 81.64%
After 7000 trainging step(s) ,validation accuracyusing average model is 83.04%
After 8000 trainging step(s) ,validation accuracyusing average model is 76.26%
After 9000 trainging step(s) ,validation accuracyusing average model is 76.14%
After 10000 trainging step(s) ,validation accuracyusing average model is 73.8%
After 11000 trainging step(s) ,validation accuracyusing average model is 84.16%
After 12000 trainging step(s) ,validation accuracyusing average model is 81.18%
After 13000 trainging step(s) ,validation accuracyusing average model is 82.68%
After 14000 trainging step(s) ,validation accuracyusing average model is 78.26%
After 15000 trainging step(s) ,validation accuracyusing average model is 80.88%
After 16000 trainging step(s) ,validation accuracyusing average model is 76.52%
After 17000 trainging step(s) ,validation accuracyusing average model is 84.26%
After 18000 trainging step(s) ,validation accuracyusing average model is 77.52%
After 19000 trainging step(s) ,validation accuracyusing average model is 71.3%
After 20000 trainging step(s) ,validation accuracyusing average model is 77.08%
After 21000 trainging step(s) ,validation accuracyusing average model is 78.76%
After 22000 trainging step(s) ,validation accuracyusing average model is 84.44%
After 23000 trainging step(s) ,validation accuracyusing average model is 82.98%
After 24000 trainging step(s) ,validation accuracyusing average model is 83.28%
After 25000 trainging step(s) ,validation accuracyusing average model is 80.84%
After 26000 trainging step(s) ,validation accuracyusing average model is 81.56%
After 27000 trainging step(s) ,validation accuracyusing average model is 69.84%
After 28000 trainging step(s) ,validation accuracyusing average model is 83.28%
After 29000 trainging step(s) ,validation accuracyusing average model is 77.96%
After 30000 trainging step(s) ,test accuracy using average model is 80.59%

Process finished with exit code 0