机器学习简单实验(LMS算法)

LMS为最小均方算法(least mean square)，目标是使得均方误差(MSE）最小，即样本预测输出值与实际输出值之差平方的期望值最小。
下面实现LMS算法学习或运算:

# -*- coding utf-8 -*-
# LMS.py
import numpy as np
b = 1
a = 0.1
x = np.array([[1,1,1],[1,1,0],[1,0,1],[1,0,0]])
d = np.array([1,1,1,0])
w = np.array([b,0,0])
expect_e = 0.005
maxtrycount = 20
def sgn(v):
    if v>0:
        return 1
    else:
        return 0
def get_v(myw,myx):
    return sgn(np.dot(myw.T,myx))
def neww(oldw,myd,myx,a):
    mye  = get_e(oldw,myx,myd)
    return (oldw + a*mye*myx,mye)
def get_e(myw,myx,myd):
    return myd - get_v(myw,myx)
mycount = 0
while True:
    mye = 0
    i = 0
    for xn in x:
        w,e = neww(w,d[i],xn,a)
        i += 1
        mye += pow(e,2)
    mye /= float(i)
    mycount += 1
    print u"After %d times, w: " %mycount
    print w
    print "error: %f" %mye
    if mye<expect_e or mycount>maxtrycount:break
for xn in x:
    print "%d or %d => %d" %(xn[1],xn[2],get_v(w,xn))

结果:

.............
.............
After 11 times, w: 
[-0.1  0.   0. ]
error: 0.250000
After 12 times, w: 
[-0.1  0.1  0.1]
error: 0.500000
After 13 times, w: 
[-0.1  0.1  0.1]
error: 0.000000
1 or 1 => 1
1 or 0 => 1
0 or 1 => 1
0 or 0 => 0

13次迭代之后，error = 0，分类结果正确。
现在实现一个稍微复杂一点的任务，输入矩阵中，若X向量整除结果为6，则为1类，若整除结果为3，为第-1类，在LMS.py的基础上稍作改动:

# -*- coding utf-8 -*-
# LMS2.py
import numpy as np
b = 1
a = 0.1
x = np.array([[1,1,6],[1,2,12],[1,3,9],[1,8,24]])
d = np.array([1,1,-1,-1])
w = np.array([b,0,0])
expect_e = 0.005
maxtrycount = 20
def sgn(v):
    if v>0:
        return 1
    else:
        return -1
def get_v(myw,myx):
    return sgn(np.dot(myw.T,myx))
def neww(oldw,myd,myx,a):
    mye  = get_e(oldw,myx,myd)
    return (oldw + a*mye*myx,mye)
def get_e(myw,myx,myd):
    return myd - get_v(myw,myx)
mycount = 0
while True:
    mye = 0
    i = 0
    for xn in x:
        w,e = neww(w,d[i],xn,a)
        i += 1
        mye += pow(e,2)
    mye /= float(i)
    mycount += 1
    print u"After %d times, w: " %mycount
    print w
    print "error: %f" %mye
    if mye<expect_e or mycount>maxtrycount:break
for xn in x:
    print "%d    %d => %d" %(xn[1],xn[2],get_v(w,xn))
test = np.array([1,9,27])
print "%d     %d => %d" %(test[1],test[2],get_v(w,xn))
test = np.array([1,11,66])
print "%d     %d => %d" %(test[1],test[2],get_v(w,xn))

运行结果:

............
............
After 7 times, w: 
[ 1.2 -2.8 -1.2]
error: 1.000000
After 8 times, w: 
[ 1.4 -2.8  0.6]
error: 3.000000
After 9 times, w: 
[ 1.4 -2.8  0.6]
error: 0.000000
1    6 => 1
2    12 => 1
3    9 => -1
8    24 => -1
9     27 => -1
11     66 => -1

分类正确.
但是，如果加入几个不规则的和错的数据，分类器还能不能正确分类呢？
加入数据
x = np.array([[1,1,6],[1,3,12],[1,3,9],[1,3,21],[1,2,16],[1,3,15]])
d = np.array([1,1,-1,-1,1,-1])

After 200 times, w: 
[ 18.  -17.4  -0.8]
error: 2.666667
After 201 times, w: 
[ 18.4 -16.8   1.8]
error: 2.666667
1    6 => 1
3    12 => -1
3    9 => -1
3    21 => 1
2    16 => 1
3    15 => -1
9     27 => -1
11     66 => -1

可以看到，现在分类结果出现了错误。是什么原因呢？对于错误的数据，分类器以同样的学习率进行学习，结果当然会出错，所以需要动态改变学习率的大小进行学习，所以需要采用模拟退火算法。