Course 4
Average Error on testing data
Two main errors:
Error due to bias
Error due to variance
Estimator
Only god knows about the best function
f
^
\hat f
f^
and we can only get a function from training data called
f
∗
f^*
f∗
we say:
f
∗
i
s
a
n
e
s
t
i
m
a
t
o
r
o
f
f
^
f^*\quad is\quad an\quad estimator \quad of\quad \hat f
f∗isanestimatoroff^
and the difference between f* and fhat comes from bias and variance
Bias and Variance of Estimator
Suppose the mean of a variable is μ, and the variance of x is σ2
S
a
m
p
l
e
N
p
o
i
n
t
s
:
{
x
1
,
x
2
,
.
.
.
,
x
3
}
Sample \quad N \quad points:\{x^1,x^2,...,x^3\}
SampleNpoints:{x1,x2,...,x3}
m = 1 N ∑ n x n ≠ μ s 2 = 1 N ∑ n ( x n − m ) 2 m=\frac{1}{N}\sum_n x^n\neq\mu \quad s^2=\frac 1 N \sum_n (x^n-m)^2 m=N1n∑xn=μs2=N1n∑(xn−m)2
E [ m ] = E [ 1 N ∑ n x n ] = 1 N ∑ n E [ x n ] = μ E [ s 2 ] = N − 1 N σ 2 E[m]=E[\frac{1}{N}\sum_n x^n]=\frac 1 N \sum_n E[x^n]=\mu\quad E[s^2]=\frac{N-1}N \sigma ^2 E[m]=E[N1n∑xn]=N1n∑E[xn]=μE[s2]=NN−1σ2
m is a biased estimator of μ, s2 is a biased estimator of σ2
V
a
r
[
m
]
=
σ
2
N
Var[m]=\frac{\sigma^2}{N}
Var[m]=Nσ2
which shows how much m deviates μ and the variance depends on the amount of sample
and the relationship of these parameters is below:
Simple model with small variance, and complicated model with large variance since simpler model is less likely to be influenced by the sampled data.
Diagnosis
- If your model cannot even fit the training data, then you got a large bias, it is Underfitting
- If you can fit training data but got large error on testing data, then you probably got a large variance, it is Overfitting
For bias, redesign your model:
- Add more features as input
- A more complex model maybe needed
For large variance:
- More data is needed(Very effective but not always practical)
- Regularization
Model Selection
There is usually a trade-off between bias and variance
Select a model that balances two kinds of error to minimize total error
Cross Validation could a possible way to make balance:
And an advanced method called N-fold Cross Validation can be used