10601 学习笔记

week 2

example [ 1,2, 3,4 ] for 1,2,3,4 each has 2  attribute. Output [a,b,c]

Input space C: number of possible unique inputs----2 ^4

Concept space: the space of all possible concepts 3 ^(2^4)

Hypothesis space H: the space of all semantically distinct conjunctions of the type described above 3^4 +1(empty set)

Training samples D;

Target function C:

Version space: VS(H,D) subset of hypotheses from H consistent with all training examples in D 包含了目标概念里所有的合理变形

VS（H,D): version space with respect to  hypothesis space&training samples

Determine: A hypothesis h in H such that h(x) = c(x) for all x in D.

C>H>V

Find-S algorithm

1.Initialize h to the most specific hypothesis in H

2. For each positive training instance x

for each attribute constraint a(i) in h

If the constraint a(i) in h is satisfied by x 

then do nothing

else replace a(i)  in h by the next more general constraint that is satisfied by x

3.output hypothesis h

week3

How surprised you are? 

Answer is I(E) = log2 (1/P(E))

Definition of Entropy (H): average amount of information in observing the output of the source S

H(X)=−∑x∈Xp(x)logap(x)

Non negative;

if p(x) = 1/X (uniform distribution), then max.

The further p is from uniform, the lower the entrophy.

 

P : probability

T,M not independent :

P(T = t, M= m) ~= P(T= t) * P(M = m)

Joint Entropy:

H(X,Y)=−∑x∈X,y∈Yp(x,y)logp(x,y)

Notice that normally H(X,Y) < H(X) + H(Y)

p(x,y) means x and y happens in the same time.

Condition Entropy:

H(Y|X)=∑x∈X p(x)H(Y|X=x)

=−∑x∈X