Minimum Variance Unbiased Estimation (MVU)

Reference:
Kay S M. Fundamentals of statistical signal processing[M]. Prentice Hall PTR, 1993. (Chapter 2)
Slides of ET4386, TUD

An Example

Consider a process e.g., a constant in noise
$$x[n]=A+w[n],n=0,\dots ,N-1$$
where, we assume

• $A$ is deterministic and unknown,
• $w[n]$ is a zero-mean random process with variance ${\sigma }^2$,
• $x[n]$ is the measured data.

Potential estimators for $A$:

• ${\hat{A}}_1=x[0]$
• ${\hat{A}}_2=\frac{1}{N}{\sum }_{n=0}^{N-1}x[n]$
• ${\hat{A}}_3=\frac{a}{N}{\sum }_{n=0}^{N-1}x[n]$, for some constant $a$
• $\cdots$

Which estimator is good (or optimal) ?

Mean Square Error Criterion

In searching for optimal estimators, we need to adopt some optimality criterion. A natural one is the mean square error (MSE), defined as
$$\mathrm{m}\mathrm{s}\mathrm{e}(\hat{\theta })=E\left[(\hat{\theta }-\theta {)}^2\right]$$
To get more insight, we can rewrite MSE as
$$\begin{array}{rl}\mathrm{m}\mathrm{s}\mathrm{e}(\hat{\theta })& =E\left[(\hat{\theta }-E(\hat{\theta })+E(\hat{\theta })-\theta {)}^2\right]\\ & =E[(\hat{\theta }-E(\hat{\theta }){)}^2]+[E(\hat{\theta })-\theta {]}^2\\ & =\mathrm{var}(\hat{\theta })+b^2(\theta )\end{array}$$
which shows that the MSE is composed of errors due to the variance of the estimator as well as the bias. Unfortunately, adoption of this natural criterion leads to unrealizable estimators, ones that cannot be written solely as a function of the data.

For instance, consider the estimator
$$\check{A}=a\frac{1}{N}\sum _{n=0}^{N-1}x[n]$$
for our example with some constant $a$. We will attempt to find the $a$ which results in the minimum MSE. Since $E(\check{A})=aA$ and $\mathrm{var}(\check{A})=a^2{\sigma }^2/N,$ we have
$$\mathrm{mse}(\check{A})=\frac{a^2{\sigma }^2}{N}+(a-1{)}^2{A}^2$$
Differentiating the MSE with respect to $a$ yields
$$\frac{d\mathrm{mse}(\check{A})}{da}=\frac{2a{\sigma }^2}{N}+2(a-1)A^2$$
which upon setting to zero and solving yields the optimum value
$$a_{\mathrm{o}\mathrm{p}\mathrm{t}}=\frac{A^2}{A^2+{\sigma }^2/N}$$
It is seen that, the optimal value of $a$ depends upon the unknown parameter $A$. The estimator is therefore not realizable.

From a practical viewpoint the minimum MSE estimator needs to be abandoned. An alternative approach is to constrain the bias to be zero and find the estimator which minimizes the variance. Such an estimator is termed the minimum variance unbiased (MVU) estimator.

Minimum Variance Unbiased Estimator

Constrain the bias of the MSE to zero, i.e., consider $E(\hat{\theta })=\theta ,$ then
$$\mathrm{m}\mathrm{s}\mathrm{e}(\hat{\theta })=E\left[(\hat{\theta }-E(\hat{\theta }){)}^2\right]+(E(\hat{\theta })-\theta {)}^2=\mathrm{var}(\hat{\theta })$$
where $\hat{\theta }$ is an unbiased estimator, and let
$$\mathrm{var}(\hat{\theta })\le \mathrm{var}(\tilde{\theta })$$
for any other unbiased estimator $\tilde{\theta },$ then $\hat{\theta }$ is the minimum variance unbiased estimator (MVU) for all $\theta$.

For the example, consider a more general estimator
$$\hat{A}=\sum _{n=0}^{N-1}{a}_{n}x[n]$$
To achieve unbiasedness, we should have
$$\sum _{n=0}^{N-1}{a}_n=1$$
The variance of $\hat{A}$ is
$$\mathrm{var}(\hat{A})=\sum _{n=0}^{N-1}{a}_n^2\mathrm{var}(x[n])={\sigma }^2\sum _{n=0}^{N-1}{a}_n^2$$
Use Lagrangian multipliers with unbiasedness as the constraint equation. Let
$$L(\mathbf{a},\lambda )={\sigma }^2{\mathbf{a}}^T\mathbf{a}-\lambda (1^T\mathbf{a})$$
Differentiate $L$ with respect to $\mathbf{a}$ and set the result to zero:
$$2{\sigma }^2\mathbf{a}-\lambda 1=0$$
Combine it with the constraint ${\sum }_{n=0}^{N-1}{a}_n=1$, we obtain
$$\mathbf{a}=\frac{1}{N}1,$$
i.e.,
$$\hat{A}=\frac{1}{N}\sum _{n=0}^{N-1}x[n]$$

---

Existence of the Minimum Variance Unbiased Estimator

The question arises as to whether a MVU estimator exists, i.e., an unbiased estimator with minimum variance for all $\theta$.

In general, the MVU estimator does not always exist.

Another example: Given a single observation $x[0]$ from the distribution $\mathcal{U}[0,1/\theta ]$, it is desired to estimate $\theta$. It is assumed that $\theta >0$. For an unbiased estimator, we must have
$${\int }_0^{1/\theta }\theta g(u)du=\theta ⟺{\int }_0^{1/\theta }g(u)du=1$$
Assume that we can find a function $g(u)$ such that for all $\theta >0$, the condition above will be satisfied. Then for any ${\theta }_1>{\theta }_2>0$, we have
$${\int }_0^{1/{\theta }_1}g(u)du=1,{\int }_0^{1/{\theta }_2}g(u)du=1⟹{\int }_{1/{\theta }_1}^{1/{\theta }_2}g(u)du=0$$
Clearly, we must have $g(u)=0$ for all $u$, which produces a biased estimator.

---

Finding the Minimum Variance Unbiased Estimator

Even if a MVU estimator exists, we may not be able to find it. In the next few chapters we shall discuss several possible approaches. They are:

1. Determine the Cramer-Rao lower bound (CRLB) and check to see if some estimator satisfies it (Chapters 3 and 4).
2. Apply the Rao-Blackwell-Lehmann-Scheffe (RBLS) theorem (Chapter 5).
3. Further restrict the class of estimators to be not only unbiased but also linear. Then, find the minimum variance estimator within this restricted class (Chapter 6).

Appendix: Some Useful Supplements

An estimator is unbiased does not necessarily mean that it is a good estimator. It only guarantees that on the average it will attain the true value. On the other hand, biased estimators are ones that are characterized by a systematic error, which presumably should not be present. A persistent bias will always result in a poor estimator.

It sometimes occurs that multiple estimates of the same parameter are available, i.e., { θ ^ 1 , θ ^ 2 , ⋯   , θ ^ n } \{\hat{\theta}_1,\hat{\theta}_2,\cdots,\hat{\theta}_n\} {θ^1​,θ^2​,⋯,θ^n​}. A reasonable procedure is to combine these estimates into a better one by averaging them to form
$$\theta =\frac{1}{n}\sum _{i=1}^n{\hat{\theta }}_i$$
Assuming the estimators are unbiased, with the same variance, and uncorrelated with each other,
$$E(\hat{\theta })=\theta ,\mathrm{var}(\hat{\theta })=\frac{1}{n^2}\sum _{i=1}^n\mathrm{var}({\hat{\theta }}_i)=\frac{\mathrm{var}({\hat{\theta }}_1)}{n}$$
so that as more estimates are averaged, the variance will decrease. Ultimately, as $n\to \infty ,\hat{\theta }\to \theta$. However, if the estimators are biased, then no matter how many estimators are averaged, $\hat{\theta }$ will not converge to the true value, as is shown in the figure above.

---

The PDF of $\hat{A}=\frac{1}{N}{\sum }_{n=0}^{N-1}x[n]$ given in the example is $\mathcal{N}(A,{\sigma }^2/N)$:

Note that $w[n]\sim \mathcal{N}(0,{\sigma }^2)$, then $x[n]\sim \mathcal{N}(A,{\sigma }^2)$. Since $x[n]$ is independent to each other, $\hat{A}$ follows Gaussian distribution. It is easy to verify that $E(\hat{A})=A,\mathrm{var}(\hat{A})={\sigma }^2/N$. Thus
$$\hat{A}\sim \mathcal{N}(A,{\sigma }^2/N)$$
The estimator can be proved to be consistent, i.e., as $N\to \infty ,\hat{A}\to A$ by showing that
lim ⁡ N → ∞ Pr ⁡ { ∣ A ^ − A ∣ > ϵ } = 0 \lim_{N\to \infty}\Pr\{|\hat A-A|>\epsilon\}=0 N→∞lim​Pr{∣A^−A∣>ϵ}=0
for any $ϵ>0$:

Since
$$\frac{\hat{A}-A}{\sqrt{{\sigma }^2/N}}\sim \mathcal{N}(0,1)$$

lim ⁡ N → ∞ Pr ⁡ { ∣ A ^ − A ∣ > ϵ } = lim ⁡ N → ∞ Pr ⁡ { ∣ A ^ − A σ 2 / N ∣ > ϵ σ 2 / N } = 0 \lim_{N\to \infty}\Pr\{|\hat A-A|>\epsilon\}=\lim_{N\to \infty}\Pr\left\{\left|\frac{\hat A -A}{\sqrt{\sigma^2/N}} \right|>\frac{\epsilon}{\sqrt{\sigma^2/N}}\right\}=0 N→∞lim​Pr{∣A^−A∣>ϵ}=N→∞lim​Pr{∣∣∣∣∣​σ2/N ​A^−A​∣∣∣∣∣​>σ2/N ​ϵ​}=0

---

A probabilistic perspective of minimum variance:

Two unbiased estimators are proposed whose variances satisfy $\mathrm{var}(\hat{\theta })<\mathrm{var}(\check{\theta })$. If both estimators are Gaussian, prove that
Pr ⁡ { ∣ θ ^ − θ ∣ > ϵ } < Pr ⁡ { ∣ θ ˇ − θ ∣ > ϵ } \Pr \{|\hat \theta -\theta|>\epsilon\}<\Pr \{|\check \theta -\theta|>\epsilon\} Pr{∣θ^−θ∣>ϵ}<Pr{∣θˇ−θ∣>ϵ}
for any $ϵ$. This says that the estimator with less variance is to be preferred since its PDF is more concentrated about the true value.

Since
$$\frac{\hat{\theta }-\theta }{\sqrt{\mathrm{var}(\hat{\theta })}}\sim \mathcal{N}(0,1),\frac{\check{\theta }-\theta }{\sqrt{\mathrm{var}(\check{\theta })}}\sim \mathcal{N}(0,1)$$
Let the cumulative distribution function for $\mathcal{N}(0,1)$
$$\Phi (x)={\int }_{-\infty }^x\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}{t}^2}dt$$
Then
Pr ⁡ { ∣ θ ^ − θ ∣ > ϵ } = Pr ⁡ { ∣ θ ^ − θ var ⁡ ( θ ^ ) ∣ > ϵ var ⁡ ( θ ^ ) } = 2 Φ { − ϵ var ⁡ ( θ ^ ) } \Pr\{|\hat \theta -\theta|>\epsilon\}=\Pr\left\{\left|\frac{\hat \theta -\theta}{\sqrt{\operatorname{var}(\hat \theta)}} \right|>\frac{\epsilon}{\sqrt{\operatorname{var}(\hat \theta)}}\right\}=2\Phi\left\{\frac{-\epsilon}{\sqrt{\operatorname{var}(\hat \theta)}} \right\} Pr{∣θ^−θ∣>ϵ}=Pr⎩⎨⎧​∣∣∣∣∣∣​var(θ^) ​θ^−θ​∣∣∣∣∣∣​>var(θ^) ​ϵ​⎭⎬⎫​=2Φ⎩⎨⎧​var(θ^) ​−ϵ​⎭⎬⎫​
If $\mathrm{var}(\hat{\theta })<\mathrm{var}(\check{\theta })$,
$$\Phi \left\{\frac{-ϵ}{\sqrt{\mathrm{var}(\hat{\theta })}}\right\}<\Phi \left\{\frac{-ϵ}{\sqrt{\mathrm{var}(\check{\theta })}}\right\}$$
or Pr ⁡ { ∣ θ ^ − θ ∣ > ϵ } < Pr ⁡ { ∣ θ ˇ − θ ∣ > ϵ } \Pr \{|\hat \theta -\theta|>\epsilon\}<\Pr \{|\check \theta -\theta|>\epsilon\} Pr{∣θ^−θ∣>ϵ}<Pr{∣θˇ−θ∣>ϵ}.

---

What will happen if an unbiased estimator undergoes a nonlinear transformation? For instance, if we choose to estimate the unknown parameter $\theta =A^2$ by
$$\hat{\theta }={\left(\frac{1}{N}\sum _{n=0}^{N-1}x[n]\right)}^2,$$
can we say that the estimator is unbiased? What happens as $N\to \infty$?

We know that
$$\hat{\theta }={\hat{A}}^2\hat{A}\sim \mathcal{N}(A,{\sigma }^2/N)$$
Therefore,
$$E(\hat{\theta })=E({\hat{A}}^2)=\mathrm{var}(\hat{A})+E^2(\hat{A})={\sigma }^2/N+A^2={\sigma }^2/N+\theta \ne \theta$$
which is biased but asymptotically unbiased.

---

In our example, if the value of ${\sigma }^2$ is also unknown, an unbiased estimator is
$$\hat{\mathbf{\theta }}=\left[\begin{array}{c}\hat{A}\\ {\hat{\sigma }}^2\end{array}\right]=\left[\begin{array}{c}\frac{1}{N}{\sum }_{n=0}^{N-1}x[n]\\ \frac{1}{N-1}{\sum }_{n=0}^{N-1}(x[n]-\hat{A}{)}^2\end{array}\right]$$