神经网络的初始误差估计-采用cross-entropy误差时

[size=large]初始状态下，由于网络的权重时随机初始化的，任意一个样本被分配到任意一类概率都是近似相等的。也即输出为[1/N, 1/N, ... , 1/N], 则对应的cross-entropy误差为

- [log(1/N)^1+ log(1/N)^0+log(1/N)^0+... ]=log(N) （注意这部分不包含regularization的误差）

结果记录如下（程序在最后），备忘：[/size]
[quote]2
[ 0.69314718]
3
[ 1.09861231]
4
[ 1.38629436]
5
[ 1.60943794]
6
[ 1.79175949]
7
[ 1.94591022]
8
[ 2.07944155]
9
[ 2.19722462]
10
[ 2.30258512]
DONE
log(2)=0.693147
log(3)=1.098612
log(4)=1.386294
log(5)=1.609438
log(6)=1.791759
log(7)=1.945910
log(8)=2.079442
log(9)=2.197225
log(10)=2.302585[/quote]

from __future__ import print_function
from __future__ import division


import tensorflow as tf
from tensorflow.python.training import moving_averages
import numpy as np


if __name__ == '__main__':
    for i in range(2,11):
        print(i)
        predict=[1.0/i for _ in range(i)]
        output=tf.Variable([predict],tf.float32);

        target=tf.Variable([0],tf.int64)

        loss=tf.nn.sparse_softmax_cross_entropy_with_logits(output, target)



        sess=tf.Session()
        sess.run(tf.initialize_all_variables())
        lo=loss.eval(session=sess)
        sess.close()

        print(lo)
    print('DONE')

    import math
    for i in range(2,11):
        print('log(%d)=%f'%(i,math.log(i)))

[size=large][color=blue]最优分类误差[/color][/size]
分类器能达到的最优分类误差为：
[quote]正样本比例：logLoss
0.05 : 0.1985
0.10 : 0.3251
0.15 : 0.4227
0.20 : 0.5004
0.25 : 0.5623
0.30 : 0.6109
0.35 : 0.6474
0.40 : 0.6730
0.45 : 0.6881
0.50 : 0.6931
0.55 : 0.6881
0.60 : 0.6730
0.65 : 0.6474
0.70 : 0.6109
0.75 : 0.5623
0.80 : 0.5004
0.85 : 0.4227
0.90 : 0.3251
0.95 : 0.1985[/quote]

代码：

x=[float(a)/100 for a in range(5,99,5)]
import math
def logloss(p,T):
    ll=T*math.log(p)+(1-T)*math.log(1-p)
    return -ll

y=[logloss(a, a) for a in x];#  p==T时可以取到最小值

for xx,yy in zip(x,y):
    print('%.2f : %.4f'%(xx,yy))