Practical Estimators (MLE, BLUE, LS)

Reference:
Kay S M. Fundamentals of statistical signal processing[M]. Prentice Hall PTR, 1993. (Ch. 6.1 - 6.5, Ch. 7.1 - 7.6, Ch. 8.1 - 8.5 and 8.8 - 8.9)
Slides of ET4386, TUD

Motivation:

• Determining the MVU requires to knowledge of the PDF.
• Even when knowing the PDF, finding the MVU is not guaranteed.

Sub-optimal estimators:

• MLE: Maximum Likelihood Estimator (needs PDF)
• BLUE: Best Linear Unbiased Estimator (needs the first two moments but not the entire PDF)
• LS: Least Squares (no need of probabilistic assumptions)

Under certain conditions, these sub-optimal estimators equal the MVU, or their variance converges to the variance of the MVU.

Maximum Likelihood Estimator (MLE)

As the name suggests, the MLE for a scalar parameter is defined to be the value of $\theta$ that maximizes $p(\mathbf{x};\theta )$ for $\mathbf{x}$ fixed, i.e., the value that maximizes the likelihood function.

Asymptotic Properties of the MLE

If the PDF $p(\mathbf{x};\theta )$ of the data $x$ satisfies some “regularity” conditions, then the MLE of the unknown parameter $\theta$ is asymptotically distributed (for large data records) according to
$$\hat{\theta }\stackrel{a}{\sim }\mathcal{N}(\theta ,I^{-1}(\theta ))$$
where $I(\theta )$ is the Fisher information evaluated at the true value of the unknown parameter.

The regularity conditions require the existence of the derivatives of the log-likelihood function, as well as the Fisher information being nonzero, as described more fully in Appendix 7B. An outline of the proof for IID observations is also given there.

---

Example: DC-level in White Gaussian Noise

Consider the observed data set
$$x[n]=A+w[n]n=0,1,\cdots ,N-1\text{ where }w[n]\sim \mathcal{N}(0,A)$$

• PDF:
  $$p(\mathbf{x};A)=\frac{1}{(2\pi A{)}^{N/2}}\exp \left[-\frac{1}{2A}\sum _{n=0}^{N-1}(x[n]-A{)}^2\right]$$

• Score:
  $$\frac{\partial \ln p(\mathbf{x};A)}{\partial A}=-\frac{N}{2A}+\frac{1}{A}\sum _{n=0}^{N-1}(x[n]-A)+\frac{1}{2A^2}\sum _{n=0}^{N-1}(x[n]-A{)}^2$$

• CRLB:
  $$\begin{array}{rl}\frac{\partial {\ln }^{2}p(\mathbf{x};A)}{\partial A^2}& =\frac{N}{2A^2}-\frac{1}{A^2}\sum _{n=0}^{N-1}x[n]-\frac{1}{A^2}\sum _{n=0}^{N-1}(x[n]-A)-\frac{1}{A^3}\sum _{n=0}^{N-1}(x[n]-A{)}^2\\ I(\theta )& =-E(\frac{\partial {\ln }^{2}p(\mathbf{x};A)}{\partial A^2})=-\frac{N}{2A^2}+\frac{NA}{A^2}+0+\frac{NA}{A^3}=\frac{N}{2A^2}+\frac{N}{A}\\ \mathrm{var}(\hat{A})& \ge \frac{1}{I(\theta )}=\frac{A^2}{N(A+1/2)}\end{array}$$

However, it is not obvious whether the score function can be put in the form $\frac{\partial \ln p(\mathbf{x};A)}{\partial A}=I(A)(\hat{A}-A)$. Therefore, we try MLE instead by setting score to zero (maximizing loglikelihood function), i.e.,
$$\frac{\partial \ln p(\mathbf{x};A)}{\partial A}=-\frac{N}{2A}+\frac{1}{A}\sum _{n=0}^{N-1}(x[n]-A)+\frac{1}{2A^2}\sum _{n=0}^{N-1}(x[n]-A{)}^2=0$$
We then obtain
$${\hat{A}}^2+\hat{A}-\frac{1}{N}\sum _{n=0}^{N-1}{x}^2[n]=0$$
Solving the above and choosing the positive $\hat{A}$ :
$$\hat{A}=-\frac{1}{2}+\sqrt{\frac{1}{N}\sum _{n=0}^{N-1}{x}^2[n]+\frac{1}{4}}$$
Note that since
$$\begin{array}{rl}E(\hat{A})& =E(-\frac{1}{2}+\sqrt{\frac{1}{N}\sum _{n=0}^{N-1}{x}^2[n]+\frac{1}{4})}\\ & \ne -\frac{1}{2}+\sqrt{E\left(\frac{1}{N}\sum _{n=0}^{N-1}{x}^2[n]\right)+\frac{1}{4}}\text{ for all }A\\ & =-\frac{1}{2}+\sqrt{A+A^2+\frac{1}{4}}\\ & =A\end{array}$$
the estimator is biased. It is nonetheless a reasonable estimator in that as $N\to \infty$, we have by the law of large numbers
$$\frac{1}{N}\sum _{n=0}^{N-1}{x}^2[n]\to E(x^2[n])=A+A^2$$
and therefore
$$\hat{A}\to A$$
Then we are going to show that $\hat{A}$ is a consistent estimator with
$$E(\hat{A})\stackrel{a}{\to }A,\mathrm{var}(\hat{A})\stackrel{a}{\to }\frac{A^2}{N(A+1/2)}$$
i.e., $\hat{A}$ is asymptotically unbiased and asymptotically achieves the CRLB. Hence, it is asymptotically efficient.

Denote
$$\hat{A}=-\frac{1}{2}+\sqrt{\frac{1}{N}\sum _{n=0}^{N-1}{x}^2[n]+\frac{1}{4}}\triangleq g(u)$$
where $u=\frac{1}{N}{\sum }_{n=0}^{N-1}{x}^2[n]$, and therefore
$$g(u)=-\frac{1}{2}+\sqrt{u+\frac{1}{4}}$$
Linearizing about $u_0=E(u)=A+A^2$, we have
$$g(u)\approx g\left(u_0\right)+{\frac{dg(u)}{du}|}_{u=u_0}\left(u-u_0\right)$$
or
$$\hat{A}\approx A+\frac{\frac{1}{2}}{A+\frac{1}{2}}\left[\frac{1}{N}\sum _{n=0}^{N-1}{x}^2[n]-\left(A+A^2\right)\right]$$
It now follows that the asymptotic mean is
$$E(\hat{A})=A$$
so that $\hat{A}$ is asymptotically unbiased. Additionally, the asymptotic variance becomes
$$\begin{array}{rl}\mathrm{var}(\hat{A})& ={\left(\frac{\frac{1}{2}}{A+\frac{1}{2}}\right)}^2\mathrm{var}\left[\frac{1}{N}\sum _{n=0}^{N-1}{x}^2[n]\right]=\frac{\frac{1}{4}}{N{\left(A+\frac{1}{2}\right)}^2}\mathrm{var}\left(x^2[n]\right)\\ & =\frac{\frac{1}{4}}{N{\left(A+\frac{1}{2}\right)}^2}(4A^3+2A^2)=\frac{A^2}{N(A+\frac{1}{2})}\end{array}$$
which asymptotically achieves the CRLB.

---

Example: Linear Gaussian Model

For linear Gaussian model
$$\mathbf{x}=\mathbf{h}\theta +\mathbf{w},\mathbf{w}\sim \mathcal{N}(0,{\mathbf{C}}_w)$$
The likelihood function is given by
$$p(\mathbf{x};\theta )=\frac{1}{(2\pi {)}^{N/2}\det ({\mathbf{C}}_w{)}^{1/2}}\exp \left[-\frac{1}{2}(\mathbf{x}-\mathbf{h}\theta {)}^T{\mathbf{C}}_w^{-1}(\mathbf{x}-\mathbf{h}\theta )\right]$$
It is clear that this function is maximized by solving
$$\hat{\theta }=\underset{\theta }{\mathrm{argmin}}\left[(\mathbf{x}-\mathbf{h}\theta {)}^T{\mathbf{C}}_w^{-1}(\mathbf{x}-\mathbf{h}\theta )\right]$$
Note that since $\mathbf{x}$ is a stochastic variable that can take many values, so is $\hat{\theta }$.

Denote
$$\begin{array}{rl}J& =(\mathbf{x}-\mathbf{h}\theta {)}^T{\mathbf{C}}_w^{-1}(\mathbf{x}-\mathbf{h}\theta )\\ & ={\mathbf{x}}^T{\mathbf{C}}_w^{-1}\mathbf{x}-2{\mathbf{h}}^T{\mathbf{C}}_w^{-1}\mathbf{x}\theta +{\mathbf{h}}^T{\mathbf{C}}_w^{-1}\mathbf{h}{\theta }^2\end{array}$$
Setting the gradient w.r.t. $\theta$ as zero, we have
$$\begin{array}{rl}\frac{\partial J}{\partial \theta }& =-2{\mathbf{h}}^T{\mathbf{C}}_w^{-1}\mathbf{x}+2{\mathbf{h}}^T{\mathbf{C}}_w^{-1}\mathbf{h}\theta \\ \hat{\theta }& =({\mathbf{h}}^T{\mathbf{C}}_w^{-1}\mathbf{h}{)}^{-1}{\mathbf{h}}^T{\mathbf{C}}_w^{-1}\mathbf{x}\end{array}$$
Therefore, for the linear Gaussian model, the MLE is the MVU estimator.

Invariance Property of the MLE

In many instances we wish to estimate a function of $\theta$, the parameter characterizing the PDF. For example, we may not be interested in the value of a DC level $A$ in WGN but only in the power $A^2$. In this case, the MLE of the transformed parameter is found by substituting the MLE of the original parameter into the transformation. This property of the MLE is termed the invariance property.

The MLE of the parameter $\alpha =g(\theta ),$ where the $\mathrm{PDF}p(\mathbf{x};\theta )$ is parametrized by $\theta ,$ is given by
$$\hat{\alpha }=g(\hat{\theta })$$
where $\hat{\theta }$ is the MLE if $\theta$, which is obtained by maximizing $p(\mathbf{x};\theta )$ over $\theta$.

If $g(\cdot )$ is not a one-to-one function, then $\hat{\alpha }$ maximizes some modified likelihood function ${\bar{p}}_T(\mathbf{x};\alpha ),$ defined as
p ˉ T ( x ; α ) = max ⁡ { θ : α = g ( θ ) } p ( x ; θ ) \bar{p}_{T}(\mathbf{x} ; \alpha)=\max _{\{\theta: \alpha=g(\theta)\}} p(\mathbf{x} ; \theta) pˉ​T​(x;α)={θ:α=g(θ)}max​p(x;θ)

Example: Transformed DC Level in WGN

Consider the data
$$x[n]=A+w[n]n=0,1,\dots ,N-1$$
where $w[n]$ is WGN with variance ${\sigma }^2.$ We wish to find the $\mathrm{M}\mathrm{L}\mathrm{E}$ of $\alpha =\exp (A).$ The PDF is given as
$$p(\mathbf{x};A)=\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{N}{2}}}\exp \left[-\frac{1}{2{\sigma }^2}\sum _{n=0}^{N-1}(x[n]-A{)}^2\right]-\infty <A<\infty$$
and is parameterized by the parameter $\theta =A$. However, since a is a one-to-one transformation of $A$, we can equivalently parameterize the PDF as
$$p_T(\mathbf{x};\alpha )=\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{N}{2}}}\exp \left[-\frac{1}{2{\sigma }^2}\sum _{n=0}^{N-1}(x[n]-\ln \alpha {)}^2\right]\alpha >0$$
where the subscript $T$ indicates that the PDF is parameterized according to the transformed parameter. Clearly, $p_T(\mathbf{x};\alpha )$ is the PDF of the data set
$$x[n]=\ln \alpha +w[n]n=0,1,\dots ,N-1$$
The MLE of $\alpha$ is found by maximizing the equivalent PDF over $\alpha .$ Setting the derivative of $p_T(\mathbf{x};\alpha )$ with respect to $\alpha$ equal to zero yields
$$\sum _{n=0}^{N-1}(x[n]-\ln \hat{\alpha })\frac{1}{\hat{\alpha }}=0$$
or
$$\hat{\alpha }=\exp (\bar{x})$$
But $\bar{x}$ is just the MLE of $A,$ so that
$$\hat{\alpha }=\exp (\hat{A})=\exp (\hat{\theta })$$

---

Example: The SNR of DC Level in WGN

For $N$ IID observation from the PDF $\mathcal{N}(A,{\sigma }^2)$, where $A$ and ${\sigma }^2$ are both unknown, find the MLE of the SNR $\alpha =A^2/{\sigma }^2$.

PDF:
$$p(\mathbf{x};\mathbf{\theta })=\frac{1}{(2\pi {\sigma }^2{)}^{N/2}}\exp \left[-\frac{1}{2{\sigma }^2}\sum _{n=0}^{N-1}(x[n]-A{)}^2\right]$$
where $\mathbf{\theta }=[A{\sigma }^2{]}^T$. Then
$$\begin{array}{rl}\frac{\partial \ln p(\mathbf{x};\mathbf{\theta })}{\partial A}& =\frac{1}{{\sigma }^2}\sum _{n=0}^{N-1}(x[n]-A)\\ \frac{\partial \ln p(\mathbf{x};\mathbf{\theta })}{\partial {\sigma }^2}& =-\frac{N}{2{\sigma }^2}+\frac{1}{2{\sigma }^4}\sum _{n=0}^{N-1}(x[n]-A{)}^2\end{array}$$
Take the derivatives to zero, we have the MLE
$$\hat{\mathbf{\theta }}=\left[\begin{array}{c}\overline{x}\\ \frac{1}{N}{\sum }_{n=0}^{N-1}(x[n]-\overline{x}{)}^2\end{array}\right]$$
According to the invariance property, we have
$$\hat{\alpha }=\frac{{\hat{A}}^2}{{\hat{\sigma }}^2}=\frac{\overline{x}}{\frac{1}{N}{\sum }_{n=0}^{N-1}(x[n]-\overline{x}{)}^2}$$

Best Linear Unbiased Estimator (BLUE)

Sometimes we do not know the PDF of the data, thus the MLE is not available. A common approach is to restrict the estimator to be linear in the data and find the linear estimator that is unbiased and has minimum variance. This estimator, which is termed the best linear unbiased estimator (BLUE), can be determined with knowledge of only the first and second moments of the PDF.

Scalar BLUE

Definition: We observe the data set { x [ 0 ] , x [ 1 ] , ⋯   , x [ N − 1 ] } \{x[0],x[1],\cdots,x[N-1]\} {x[0],x[1],⋯,x[N−1]} whose PDF $p(\mathbf{x};\theta )$ depends on an unknown parameter $\theta$. The BLUE restricts the estimator to be linear in the data or
$$\hat{\theta }=\sum _{n=0}^{N-1}{a}_{n}x[n]={\mathbf{a}}^T\mathbf{x}$$
where the $a_n$'s are constants yet to be determined.

Since we are restricting the class of estimators to be linear, the BLUE will be optimal (that is to say, the MVU estimator) only when the MVU estimator turns out to be linear.

Finding the BLUE:

• Unbiased
  $$E(\hat{\theta })=\sum _{n=0}^{N-1}{a}_{n}E(x[n])={\mathbf{a}}^{T}E(\mathbf{x})=\theta$$
  In order to satisfy the unbiased constraint, $E(x[n])$ must be linear in $\theta$
  $$E(x[n])=h[n]\theta$$
  where the $h[n]$'s are known. Or
  $$E(\mathbf{x})=\mathbf{h}\theta ,{\mathbf{a}}^T\mathbf{h}=1.$$
  Note that if we write $x[n]$ as
  $$x[n]=E(x[n])+[x[n]-E(x[n])]$$
  then by viewing $x[n]-E(x[n])$ as noise or $w[n]$ we have
  $$x[n]=\theta h[n]+w[n]$$
  which means that the BLUE is applicable to amplitude estimation of known signals in noise.

• Minimum variance
  var ⁡ ( θ ^ ) = E [ ( θ ^ − E ( θ ^ ) ) 2 ] = E { [ a T ( x − E ( x ) ) ] 2 } = a T E [ ( x − E ( x ) ) ( x − E ( x ) ) T ] a = a T C a \begin{aligned} \operatorname{var}(\hat \theta)=& E[(\hat \theta-E(\hat \theta))^2]=E\{[\mathbf a^T(\mathbf x-E(\mathbf x))]^2\}\\ =& \mathbf a^TE[(\mathbf x-E(\mathbf x))(\mathbf x-E(\mathbf x))^T]\mathbf a\\ =& \mathbf a^T \mathbf C \mathbf a \end{aligned} var(θ^)===​E[(θ^−E(θ^))2]=E{[aT(x−E(x))]2}aTE[(x−E(x))(x−E(x))T]aaTCa​

Therefore, we can formulate the optimization problem
$$\begin{array}{rlrl}& \underset{\mathbf{a}}{\mathrm{minimize}}& & {\mathbf{a}}^T\mathbf{C}\mathbf{a}\\ & \text{subject to}& & {\mathbf{a}}^T\mathbf{h}=1\end{array}$$
which can be solved using the method of Lagrange multipliers.
$$J={\mathbf{a}}^T\mathbf{C}\mathbf{a}+\lambda ({\mathbf{a}}^T\mathbf{h}-1)$$
Setting the gradient w.r.t. $\mathbf{a}$ to zero,
$$\frac{\partial J}{\partial \mathbf{a}}=2\mathbf{C}\mathbf{a}+\lambda \mathbf{h}=0⟹\mathbf{a}=-\frac{\lambda }{2}{\mathbf{C}}^{-1}\mathbf{h}$$
The Lagrange multiplier $\lambda$ is found using the constraint
$${\mathbf{a}}^T\mathbf{h}=-\frac{\lambda }{2}{\mathbf{h}}^T{\mathbf{C}}^{-1}\mathbf{h}=1⟹-\frac{\lambda }{2}=\frac{1}{{\mathbf{h}}^T{\mathbf{C}}^{-1}\mathbf{h}}$$
Thus the optimal $\mathbf{a}$ is given by
$${\mathbf{a}}_{opt}=\frac{{\mathbf{C}}^{-1}\mathbf{h}}{{\mathbf{h}}^T{\mathbf{C}}^{-1}\mathbf{h}}$$
and the variance for this value of $\mathbf{a}$
$$\mathrm{var}(\hat{\theta })={\mathbf{a}}_{opt}^T\mathbf{C}{\mathbf{a}}_{opt}=\frac{1}{{\mathbf{h}}^T{\mathbf{C}}^{-1}\mathbf{h}}$$
We can prove that the ${\mathbf{a}}_{opt}$ is indeed the global minimum and not just a local minimum by verifying
$${\mathbf{a}}^T\mathbf{C}\mathbf{a}=(\mathbf{a}-{\mathbf{a}}_{opt}{)}^T\mathbf{C}(\mathbf{a}-{\mathbf{a}}_{opt})+\frac{1}{{\mathbf{h}}^T{\mathbf{C}}^{-1}\mathbf{h}}$$
Therefore,
$$\hat{\theta }={\mathbf{a}}_{opt}^T\mathbf{x}=\frac{{\mathbf{h}}^T{\mathbf{C}}^{-1}\mathbf{x}}{{\mathbf{h}}^T{\mathbf{C}}^{-1}\mathbf{h}}$$
Since $E(\mathbf{x})=\mathbf{h}\theta$, the BLUE is unbiased.

Also, as we asserted earlier, to determine the BLUE we only require knowledge of

• $E(\mathbf{x})$, the mean or $\mathbf{h}$
• $\mathbf{C}$, the covariance

Example: Scalar BLUE for Linear Model

For linear model
$$\mathbf{x}=\mathbf{h}\theta +\mathbf{w},\text{ with }E(\mathbf{w})=0\text{ and }\mathrm{cov}(\mathbf{w})={\mathbf{C}}_x$$
Since $E(\mathbf{x})=\mathbf{h}\theta$, $\mathbf{C}={\mathbf{C}}_x$, we can directly obtain that
$$\hat{\theta }=({\mathbf{h}}^T{\mathbf{C}}_x^{-1}\mathbf{h}{)}^{-1}{\mathbf{h}}^T{\mathbf{C}}_x^{-1}\mathbf{x}$$
which is exactly the form of MVU if the noise is Gaussian.

---

Vector BLUE

Definition: If the parameter to be estimated is a $p\times 1$ vector parameter, then for the estimator to be linear in the data we require
$${\hat{\theta }}_i=\sum _{n=0}^{N-1}{a}_{in}x[n]i=1,2,\cdots ,p$$
where the $a_{in}$'s are weighting coefficients. In matrix form this is
$$\hat{\mathbf{\theta }}=\mathbf{A}\mathbf{x}$$
where $\mathbf{A}$ is a $p\times N$ matrix.

Finding BLUE:

• Unbiased
  $$E(\hat{\mathbf{\theta }})=\mathbf{A}E(\mathbf{x})=\mathbf{\theta }$$
  To satisfy this condition, we must have
  $$E(\mathbf{x})=\mathbf{H}\mathbf{\theta },\mathbf{A}\mathbf{H}=\mathbf{I}$$
  where
  $$\mathbf{H}=[{\mathbf{h}}_1{\mathbf{h}}_2\cdots {\mathbf{h}}_p],\mathbf{A}=\left[\begin{array}{c}{\mathbf{a}}_1^T\\ {\mathbf{a}}_2^T\\ ⋮\\ {\mathbf{a}}_p^T\end{array}\right]$$
  With these definitions the unbiased constraint reduces to
  $${\mathbf{a}}_i^T{\mathbf{h}}_j={\delta }_{ij},i=1,2,\cdots ,p;j=1,2,\cdots ,p$$

• Minimum variance
  $$\mathrm{var}({\hat{\theta }}_i)={\mathbf{a}}_i^T\mathbf{C}{\mathbf{a}}_i$$

Therefore, we can formulate the optimization problem
minimize ⁡ { a i } i = 1 p a i T C a i subject to a i T h j = δ i j \begin{aligned} &\underset{\{\mathbf a_i\}_{i=1}^p} {\operatorname{minimize}} &&\mathbf a_i^T\mathbf C \mathbf a_i\\ &\text{subject to} && \mathbf a_i^T\mathbf h_j=\delta_{ij} \end{aligned} ​{ai​}i=1p​minimize​subject to​​aiT​Cai​aiT​hj​=δij​​
Unlike the scalar BLUE, now we have $p$ constraints on ${\mathbf{a}}_i$ instead of just one. Since each ${\mathbf{a}}_i$ is free to assume any value, independently of the others, we actually have $p$ separate minimization problems linked only by the constraints. Consider the Lagrangian function for ${\mathbf{a}}_i$,
$$J_i={\mathbf{a}}_i^T\mathbf{C}{\mathbf{a}}_i+\sum _{j=1}^p{\lambda }_j^{(i)}({\mathbf{a}}_i^T{\mathbf{h}}_j-{\delta }_{ij})$$
Setting the gradient of the Lagrangian to zero
$$\frac{\partial J_i}{\partial {\mathbf{a}}_i}=2\mathbf{C}{\mathbf{a}}_i+\sum _{j=1}^p{\lambda }_j^{(i)}{\mathbf{h}}_j=2\mathbf{C}{\mathbf{a}}_i+\mathbf{H}{\mathbf{\lambda }}_i=0⟹{\mathbf{a}}_i=-\frac{1}{2}{\mathbf{C}}^{-1}\mathbf{H}{\mathbf{\lambda }}_i$$
where ${\mathbf{\lambda }}_i=[{\lambda }_1^{(i)}{\lambda }_2^{(i)}\cdots {\lambda }_p^{(i)}{]}^T$.

To find the vector of Lagrangian multipliers we use the constraint equations
$${\mathbf{a}}_i^T{\mathbf{h}}_j={\delta }_{ij}j=1,2,\dots ,p$$
or in combined form
$$\left[\begin{array}{c}{\mathbf{h}}_1^T\\ ⋮\\ {\mathbf{h}}_{i-1}^T\\ {\mathbf{h}}_i^T\\ {\mathbf{h}}_{i+1}^T\\ ⋮\\ {\mathbf{h}}_p^T\end{array}\right]{\mathbf{a}}_i=\left[\begin{array}{c}0\\ ⋮\\ 0\\ 1\\ 0\\ ⋮\\ 0\end{array}\right]$$
Letting ${\mathbf{e}}_i$ denote the vector of all zeros except in the $i$ th place, we have the constraint equation
$${\mathbf{H}}^T{\mathbf{a}}_i=-\frac{1}{2}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}{\mathbf{\lambda }}_i={\mathbf{e}}_i⟹-\frac{1}{2}{\mathbf{\lambda }}_i={\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{e}}_i$$
(assuming invertibility of ${\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}$).

Therefore,
$${\mathbf{a}}_{i_{opt}}={\mathbf{C}}^{-1}\mathbf{H}{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{e}}_i$$
and a corresponding variance of
$$\begin{array}{rl}\mathrm{var}\left({\hat{\theta }}_i\right)& ={\mathbf{a}}_{i_{\text{opt }}}^T{\mathbf{C}\mathbf{a}}_{i_{\text{opt }}}\\ & ={\mathbf{e}}_i^T{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}{\mathbf{C}\mathbf{C}}^{-1}\mathbf{H}{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{e}}_i\\ & ={\mathbf{e}}_i^T{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{e}}_i\\ & =[{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{]}_{ii}\end{array}$$
We can express the vector BLUE in a more compact form by letting
$$\begin{array}{rl}\hat{\mathbf{\theta }}& =\left[\begin{array}{c}{\mathbf{a}}_{1_{\text{opt }}}^T\mathbf{x}\\ {\mathbf{a}}_{2\text{ opt }}^T\mathbf{x}\\ ⋮\\ {\mathbf{a}}_{p_{\text{opt }}}^T\mathbf{x}\end{array}\right]=\left[\begin{array}{c}{\mathbf{e}}_1^T{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{x}\\ {\mathbf{e}}_2^T{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{x}\\ ⋮\\ {\mathbf{e}}_p^T{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{x}\end{array}\right]\\ & =\left[\begin{array}{c}{\mathbf{e}}_1^T\\ {\mathbf{e}}_2^T\\ ⋮\\ {\mathbf{e}}_p^T\end{array}\right]{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{x}\\ & ={\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{x}\end{array}$$

Example: Vector BLUE for General Linear Model (Gauss-Markov Theorem)

If the data are of the general linear model form
$$\mathbf{x}=\mathbf{H}\mathbf{\theta }+\mathbf{w}$$
where $\mathbf{H}$ is a known $N\times p$ matrix, $\mathbf{\theta }$ is a $p\times 1$ vector of parameters to be estimated, and $\mathbf{w}$ is a $N\times 1$ noise vector with zero mean and covariance $\mathrm{C}$ (the PDF of $\mathbf{w}$ is otherwise arbitrary), then the BLUE of $\theta$ is
$$\hat{\mathbf{\theta }}={\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{x}$$
and the minimum variance of ${\hat{\theta }}_i$ is
$$\mathrm{var}\left({\hat{\theta }}_i\right)={\left[{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}\right]}_{ii}$$
In addition, the covariance matrix of $\hat{\mathbf{\theta }}$ is
$${\mathbf{C}}_{\hat{\theta }}=E\left[(\hat{\mathbf{\theta }}-E(\hat{\mathbf{\theta }}))(\hat{\mathbf{\theta }}-E(\hat{\mathbf{\theta }}){)}^T\right]$$
where
$$\begin{array}{rl}\hat{\mathbf{\theta }}-E(\hat{\mathbf{\theta }})& ={\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}(\mathbf{H}\mathbf{\theta }+\mathbf{w})-E(\hat{\mathbf{\theta }})\\ & ={\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{w}\end{array}$$
Thus,
$${\mathbf{C}}_{\tilde{\theta }}=E\left[{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{w}{\mathbf{w}}^T{\mathbf{C}}^{-1}\mathbf{H}{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}\right]$$
since the covariance matrix of $\mathbf{w}$ is $\mathbf{C},$ we have
$$\begin{array}{rl}{\mathbf{C}}_{\dot{\theta }}& ={\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}{\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{C}{\mathbf{C}}^{-1}\mathbf{H}{\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}\\ & ={\left({\mathbf{H}}^T{\mathbf{C}}^{-1}\mathbf{H}\right)}^{-1}\end{array}$$

Least Squares (LS)

Different from MVU, MLE and BLUE, the method of least squares (LS) needs no probabilistic assumptions about the data. However, estimator may not be statistically efficient.

$$\begin{array}{rl}\text{ Measurement vector }& :\mathrm{x}=x[0],x[1],\dots ,x[N-1]\\ \text{ Signal vector }& :\mathrm{s}=s[0],s[1],\dots ,s[N-1]\\ \text{ Unknown parameter }& :\theta \end{array}$$
The least squares estimator (LSE) of $\theta$ chooses the value that makes $s[n]$ closest to the observed data $x[n]$. Closeness is measured by the LS error criterion
$$J(\theta )=\sum _{n=0}^{N-1}(x[n]-s[n]{)}^2$$
The value of $\theta$ that minimizes $J(\theta )$ is the LSE. For a vector parameter $\mathbf{\theta }$ of dimension $p\times 1$, the LS error criterion is almost the same
$$J(\mathbf{\theta })=\sum _{n=0}^{N-1}(x[n]-s[n]{)}^2$$
Example: DC level signal

Assume that the signal model in Figure 8.1 is $s[n]=A$ and we observe $x[n]$ for $n=$ $0,1,\dots ,N-1.$ Then, according to the LS approach, we can estimate $A$ by minimizing
$$J(A)=\sum _{n=0}^{N-1}(x[n]-A{)}^2$$
Differentiating with respect to $A$ and setting the result equal to zero produces
$$\hat{A}=\frac{1}{N}\sum _{n=0}^{N-1}x[n]=\bar{x}$$
or the sample mean estimator. Our familiar estimator, however, cannot be claimed to be optimal in the MVU sense but only in that it minimizes the LS error. We know, however, from our previous discussions that if $x[n]=A+w[n],$ where $w[n]$ is zero mean then the LSE will also be the MVU estimator, but otherwise not.

Linear Least Squares

In applying the linear LS approach for a scalar parameter we must assume that
$$s[n]=\theta h[n]$$
where $h[n]$ is a known sequence. (Compare it with [BLUE](#Scalar BLUE))

Or generalize it with vector parameter $\mathbf{\theta }$
$$\mathbf{s}=\mathbf{H}\mathbf{\theta }$$
where the measurement vector $\mathbf{s}=[s[0],s[1],\cdots ,s[N-1]{]}^T$, and $\mathbf{H}$ is a known $N\times p$ matrix $(N>p)$ of full rank $p$. The matrix $\mathbf{H}$ is referred to as the observation matrix.

The LSE is found by minimizing
$$J(\mathbf{\theta })=\sum _{n=0}^{N-1}(x[n]-s[n]{)}^2=(\mathbf{x}-\mathbf{H}\mathbf{\theta }{)}^T(\mathbf{x}-\mathbf{H}\mathbf{\theta })$$
Setting the derivative of $J(\mathbf{\theta })$ to zero
$$\frac{\partial J(\mathbf{\theta })}{\partial \mathbf{\theta }}=-2{\mathbf{H}}^T\mathbf{x}+2{\mathbf{H}}^T\mathbf{H}\mathbf{\theta }=0⟹\hat{\mathbf{\theta }}=({\mathbf{H}}^T\mathbf{H}{)}^{-1}{\mathbf{H}}^T\mathbf{x}$$
The equations ${\mathbf{H}}^T\mathbf{H}\mathbf{\theta }={\mathbf{H}}^T\mathbf{x}$ to be solved for $\hat{\mathbf{\theta }}$ are termed as the normal equations.

The minimum LS error is
$$\begin{array}{rl}{J}_{min}& =J(\hat{\mathbf{\theta }})=(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }}{)}^T(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }})\\ & ={\mathbf{x}}^T(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }})-{\hat{\mathbf{\theta }}}^T{\mathbf{H}}^T(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }})\\ & \stackrel{a}{=}{\mathbf{x}}^T(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }})\\ & ={\mathbf{x}}^T\mathbf{x}-{\mathbf{x}}^T\mathbf{H}({\mathbf{H}}^T\mathbf{H}{)}^{-1}{\mathbf{H}}^T\mathbf{x}\end{array}$$
where $\stackrel{a}{=}$ follows from the normal equations.

An extension of the linear LS problem is to weighted LS. We include an $N\times N$ positive definite symmetric weighting matrix $\mathbf{W}$, so that
$$J(\mathbf{\theta })=(\mathbf{x}-\mathbf{H}\mathbf{\theta }{)}^T\mathbf{W}(\mathbf{x}-\mathbf{H}\mathbf{\theta })$$
The rationale for introducing weighting factors into the error criterion is to emphasize the contributions of those data samples that are deemed to be more reliable.

The general form of the weighted LSE can be similarly derived by
$$\frac{\partial J(\mathbf{\theta })}{\partial \mathbf{\theta }}=-2{\mathbf{H}}^T\mathbf{W}\mathbf{x}+2{\mathbf{H}}^T\mathbf{W}\mathbf{H}\mathbf{\theta }=0⟹\hat{\mathbf{\theta }}=({\mathbf{H}}^T\mathbf{W}\mathbf{H}{)}^{-1}{\mathbf{H}}^T\mathbf{W}\mathbf{x}$$
and its minimum LS error is
$$\begin{array}{rl}{J}_{min}& =J(\hat{\mathbf{\theta }})=(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }}{)}^T\mathbf{W}(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }})\\ & ={\mathbf{x}}^T\mathbf{W}(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }})-{\hat{\mathbf{\theta }}}^T{\mathbf{H}}^T\mathbf{W}(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }})\\ & ={\mathbf{x}}^T\mathbf{W}(\mathbf{x}-\mathbf{H}\hat{\mathbf{\theta }})\\ & ={\mathbf{x}}^T\mathbf{W}\mathbf{x}-{\mathbf{x}}^T\mathbf{W}\mathbf{H}({\mathbf{H}}^T\mathbf{W}\mathbf{H}{)}^{-1}{\mathbf{H}}^T\mathbf{W}\mathbf{x}\end{array}$$

Geometrical Interpretations

The LS error can be also written as
$$J(\mathbf{\theta })=\Vert \mathbf{x}-\mathbf{H}\mathbf{\theta }{\Vert }^2=\Vert \mathbf{x}-\sum _{i=1}^p{\mathbf{\theta }}_i{\mathbf{h}}_i{\Vert }^2$$
which can be interpreted as the problem of fitting or approximating a data vector $\mathbf{x}$ in $R^N$ by another vector $\hat{\mathbf{s}}$, which is a linear combination of vectors { h 1 , h 2 , . . . , h p } \{\mathbf h_1 , \mathbf h_2 , ... , \mathbf h_p\} {h1​,h2​,...,hp​} that lie in a $p$-dimensional subspace of $R^N$.

Define the error vector as $\mathbf{ϵ}=\mathbf{x}-\mathbf{H}\mathbf{\theta }$. Then the normal equations becomes
$${\mathbf{H}}^T\mathbf{ϵ}=0$$
i.e., the error vector must be orthogonal to the columns of $\mathbf{H}$. This is the well-known orthogonality principle. Therefore, the problem is solved by choosing $\hat{\mathbf{s}}$ in the subspace to be the orthogonal projection of $\mathbf{x}$:
$$\hat{\mathbf{s}}=\mathbf{P}\mathbf{x}=\mathbf{H}({\mathbf{H}}^T\mathbf{H}{)}^{-1}{\mathbf{H}}^T\mathbf{x}$$
where the $N\times N$ matrix $\mathbf{P}=\mathbf{H}({\mathbf{H}}^T\mathbf{H}{)}^{-1}{\mathbf{H}}^T$ is known as the projection matrix. It has the properties

• ${\mathbf{P}}^T=\mathbf{P}$, symmetric
• ${\mathbf{P}}^2=\mathbf{P}$, idempotent

It can be easily verified from the properties given above that ${\mathbf{P}}^{\perp }=\mathbf{I}-\mathbf{P}$ is also a projection matrix. As a result, the error vector $\mathbf{ϵ}=\mathbf{x}-\hat{\mathbf{s}}=(\mathbf{I}-\mathbf{P})\mathbf{x}$ is the projection of $\mathbf{x}$ onto the complement subspace or the subspace orthogonal to the signal subspace. The minimum LS error can then be presented as
$$\begin{array}{rl}{J}_{min}& ={\mathbf{x}}^T\mathbf{x}-{\mathbf{x}}^T\mathbf{H}({\mathbf{H}}^T\mathbf{H}{)}^{-1}{\mathbf{H}}^T\mathbf{x}\\ & ={\mathbf{x}}^T(\mathbf{I}-\mathbf{P})\mathbf{x}\\ & ={\mathbf{x}}^T{\mathbf{P}}^{\perp }\mathbf{x}\\ & ={\mathbf{x}}^T({\mathbf{P}}^{\perp }{)}^T{\mathbf{P}}^{\perp }\mathbf{x}\\ & =\Vert {\mathbf{P}}^{\perp }\mathbf{x}{\Vert }^2\end{array}$$
which is just $\Vert \mathbf{ϵ}{\Vert }^2$.

Constraint Least Squares

At times we are confronted with LS problems whose unknown parameters must be constrained. Such would be the case if we wished to estimate the amplitudes of several signals but knew a priori that some of the amplitudes were equal. Then, the total number of parameters to be estimated should be reduced to take advantage of the a priori knowledge. For this example the parameters are linearly related, leading to a linear least squares problem with linear constraints. We summarize the constraints as
$$\mathbf{A}\mathbf{\theta }=\mathbf{b}$$
where $\mathbf{A}$ is a known $r\times p$ matrix and $\mathbf{b}$ is a known $r\times 1$ vector.

To find the LSE subject to the linear constraints we use the technique of Lagrangian multipliers. We determine ${\hat{\mathbf{\theta }}}_c$ ($c$ denotes the constrained LSE) by minimizing the Lagrangian
$$J_c=(\mathbf{x}-\mathbf{H}\mathbf{\theta }{)}^T(\mathbf{x}-\mathbf{H}\mathbf{\theta })+{\mathbf{\lambda }}^T(\mathbf{A}\mathbf{\theta }-\mathbf{b})$$
where $\mathbf{\lambda }$ is a $r\times 1$ vector of Lagrangian multipliers. Expanding this expression, we have
$$J_c={\mathbf{x}}^T\mathbf{x}-2{\mathbf{\theta }}^T{\mathbf{H}}^T\mathbf{x}+{\mathbf{\theta }}^T{\mathbf{H}}^T\mathbf{H}\mathbf{\theta }+{\mathbf{\lambda }}^T\mathbf{A}\mathbf{\theta }-{\mathbf{\lambda }}^T\mathbf{b}$$
Taking the gradient with respect to $\mathbf{\theta }$ and yields
$$\frac{\partial J_c}{\partial \mathbf{\theta }}=-2{\mathbf{H}}^T\mathbf{x}+2{\mathbf{H}}^T\mathbf{H}\mathbf{\theta }+{\mathbf{A}}^T\mathbf{\lambda }$$
and setting it equal to zero produces
$$\begin{array}{rl}{\hat{\mathbf{\theta }}}_c& ={\left({\mathbf{H}}^T\mathbf{H}\right)}^{-1}{\mathbf{H}}^T\mathbf{x}-\frac{1}{2}{\left({\mathbf{H}}^T\mathbf{H}\right)}^{-1}{\mathbf{A}}^T\mathbf{\lambda }\\ & =\hat{\mathbf{\theta }}-{\left({\mathbf{H}}^T\mathbf{H}\right)}^{-1}{\mathbf{A}}^T\frac{\mathbf{\lambda }}{2}\end{array}$$
where $\hat{\mathbf{\theta }}$ is the unconstrained LSE and $\mathbf{\lambda }$ is yet to be determined. To find $\mathbf{\lambda }$ we impose the constraint so that
$$\mathbf{A}{\mathbf{\theta }}_c=\mathbf{A}\hat{\mathbf{\theta }}-\mathbf{A}{\left({\mathbf{H}}^T\mathbf{H}\right)}^{-1}{\mathbf{A}}^T\frac{\mathbf{\lambda }}{2}=\mathbf{b}$$
and hence
$$\frac{\mathbf{\lambda }}{2}={\left[\mathbf{A}{\left({\mathbf{H}}^T\mathbf{H}\right)}^{-1}{\mathbf{A}}^T\right]}^{-1}(\mathbf{A}\hat{\mathbf{\theta }}-\mathbf{b})$$
Substituting into (8.51) produces the solution
$${\hat{\mathbf{\theta }}}_c=\hat{\mathbf{\theta }}-{\left({\mathbf{H}}^T\mathbf{H}\right)}^{-1}{\mathbf{A}}^T{\left[\mathbf{A}{\left({\mathbf{H}}^T\mathbf{H}\right)}^{-1}{\mathbf{A}}^T\right]}^{-1}(\mathbf{A}\hat{\mathbf{\theta }}-\mathbf{b})$$
where $\hat{\mathbf{\theta }}={\left({\mathbf{H}}^T\mathbf{H}\right)}^{-1}{\mathbf{H}}^T\mathbf{x}.$ The constrained LSE is a corrected version of the unconstrained LSE.

Nonlinear Least Squares

Recall that the LS procedure estimates model parameters $\mathbf{\theta }$ by minimizing the LS error criterion
$$J=(\mathbf{x}-\mathbf{s}(\mathbf{\theta }){)}^T(\mathbf{x}-\mathbf{s}(\mathbf{\theta }))$$
where $\mathbf{s}(\mathbf{\theta })$ is the signal model for $\mathbf{x}$, with its dependence on $\mathbf{\theta }$ explicitly shown. (Note that if $\mathbf{x}-\mathbf{s}(\mathbf{\theta })\sim \mathcal{N}(0,{\sigma }^2\mathbf{I})$, the LSE is also the MLE.) In the linear LS problem the signal takes on the special form $\mathbf{s}(\mathbf{\theta })=\mathbf{H}\mathbf{\theta }$, which leads to the simple linear LSE. In general, $\mathbf{s}(\mathbf{\theta })$ cannot be expressed in this manner but is an $N$-dimensional nonlinear function of $\mathbf{\theta }$.

Before discussing general methods for determining nonlinear LSEs we first describe two methods that can reduce the complexity of the problem. They are

• transformation of parameters

  In this case we seek a one-to-one transformation of $\mathbf{\theta }$ that produces a linear signal model in the new space. To do so we let
  $$\mathbf{\alpha }=\mathbf{g}(\mathbf{\theta })$$
  where $\mathbf{g}$ is a $p$-dimensional function of $\mathbf{\theta }$ whose inverse exists. If a $\mathbf{g}$ can be found so that
  $$\mathbf{s}(\mathbf{\theta }(\mathbf{\alpha }))=\mathbf{s}({\mathbf{g}}^{-1}(\mathbf{\alpha }))=\mathbf{H}\mathbf{\alpha }$$
  then the signal model will be linear in $\mathbf{\alpha }$. We can then easily find the linear LSE of $\mathbf{\alpha }$ and thus the nonlinear LSE of $\mathbf{\theta }$ by
  $$\hat{\mathbf{\theta }}={\mathbf{g}}^{-1}(\hat{\mathbf{\alpha }})$$
  where
  $$\hat{\mathbf{\alpha }}=({\mathbf{H}}^T\mathbf{H}{)}^{-1}{\mathbf{H}}^T\mathbf{x}$$
  This approach relies on the property that the minimization can be carried out in any transformed space that is obtained by a one-to-one mapping and then converted back to the original space.

  Example:

  For a sinusoidal signal model
  $$s[n]=A\cos \left(2\pi f_{0}n+\varphi \right)n=0,1,\dots ,N-1$$
  it is desired to estimate the amplitude $A,$ where $A>0,$ and phase $\varphi .$ The frequency $f_0$ is assumed known. The LSE is obtained by minimizing
  $$J=\sum _{n=0}^{N-1}{\left(x[n]-A\cos \left(2\pi f_{0}n+\varphi \right)\right)}^2$$
  over $A$ and $\varphi$, a nonlinear LS problem. However, because
  $$A\cos \left(2\pi f_{0}n+\varphi \right)=A\cos \varphi \cos 2\pi f_{0}n-A\sin \varphi \sin 2\pi f_{0}n$$
  if we let
  $$\begin{array}{l}{\alpha }_1=A\cos \varphi \\ {\alpha }_2=-A\sin \varphi \end{array}$$
  then the signal model becomes
  $$s[n]={\alpha }_1\cos 2\pi f_{0}n+{\alpha }_2\sin 2\pi f_{0}n$$
  In matrix form this is
  $$\mathbf{s}=\mathbf{H}\mathbf{\alpha }$$
  where
  $$\mathbf{H}=\left[\begin{array}{cc}1& 0\\ \cos 2\pi f_0& \sin 2\pi f_0\\ ⋮& ⋮\\ \cos 2\pi f_0(N-1)& \sin 2\pi f_0(N-1)\end{array}\right]$$
  which is now linear in the new parameters. The LSE of $\mathbf{\alpha }$ is
  $$\hat{\mathbf{\alpha }}={\left({\mathbf{H}}^T\mathbf{H}\right)}^{-1}{\mathbf{H}}^T\mathbf{x}$$
  and to find $\hat{\mathbf{\theta }}$ we must find the inverse transformation ${\mathbf{g}}^{-1}(\mathbf{\alpha }).$ This is
  $$\begin{array}{rl}A& =\sqrt{{\alpha }_1^2+{\alpha }_2^2}\\ \varphi & =\arctan \left(\frac{-{\alpha }_2}{{\alpha }_1}\right)\end{array}$$
  so that the nonlinear LSE for this problem is given by
  $$\hat{\mathbf{\theta }}=\left[\begin{array}{l}\hat{A}\\ \hat{\varphi }\end{array}\right]=\left[\begin{array}{c}\sqrt{{\hat{\alpha }}_1^2+{\hat{\alpha }}_2^2}\\ \arctan \left(\frac{-{\hat{\alpha }}_2}{{\hat{\alpha }}_1}\right)\end{array}\right]$$

• separability of parameters

  Although the signal model is nonlinear, it may be linear in some of the parameters:
  $$\mathbf{s}=\mathbf{H}(\mathbf{\alpha })\mathbf{\beta }$$
  where
  $$\mathbf{\theta }=\left[\begin{array}{c}\mathbf{\alpha }\\ \mathbf{\beta }\end{array}\right]=\left[\begin{array}{c}(p-q)\times 1\\ q\times 1\end{array}\right]$$
  and $\mathbf{H}(\mathbf{\alpha })$ is an $N\times q$ matrix dependent on $\mathbf{\alpha }$. This model is linear in $\mathbf{\beta }$ but nonlinear in $\mathbf{\alpha }$. As a result, the LS error may be minimized with respect to $\mathbf{\beta }$ and thus reduced to a function of $\mathbf{\alpha }$ only. since
  $$J(\mathbf{\alpha },\mathbf{\beta })=(\mathbf{x}-\mathbf{H}(\mathbf{\alpha })\mathbf{\beta }{)}^T(\mathbf{x}-\mathbf{H}(\mathbf{\alpha })\mathbf{\beta })$$
  the $\mathbf{\beta }$ that minimizes $J$ for a given $\mathbf{\alpha }$ is
  $$\hat{\mathbf{\beta }}={\left({\mathbf{H}}^T(\mathbf{\alpha })\mathbf{H}(\mathbf{\alpha })\right)}^{-1}{\mathbf{H}}^T(\mathbf{\alpha })\mathbf{x}$$
  and the resulting LS error is,
  $$J(\mathbf{\alpha },\hat{\mathbf{\beta }})={\mathbf{x}}^T\left[\mathbf{I}-\mathbf{H}(\mathbf{\alpha }){\left({\mathbf{H}}^T(\mathbf{\alpha })\mathbf{H}(\mathbf{\alpha })\right)}^{-1}{\mathbf{H}}^T(\mathbf{\alpha })\right]\mathbf{x}$$
  The problem now reduces to a maximization of
  $${\mathbf{x}}^T\mathbf{H}(\mathbf{\alpha }){\left({\mathbf{H}}^T(\mathbf{\alpha })\mathbf{H}(\mathbf{\alpha })\right)}^{-1}{\mathbf{H}}^T(\mathbf{\alpha })\mathbf{x}$$
  over $\mathbf{\alpha }$.

When these approaches fail, we are forced to tackle the original nonlinear LS problem. It can be done by linearizing the signal model about some nominal $\mathbf{\theta }$ and then apply the linear LS procedure.
$$s[n;\theta ]\approx s[n;{\theta }_0]+{\frac{\partial s[n;\theta ]}{\partial \theta }|}_{\theta ={\theta }_0}(\theta -{\theta }_0)$$
where the dependence of $s[n]$ on $\theta$ is now shown. The LS error becomes
$$\begin{array}{rl}J& =\sum _{n=0}^{N-1}(x[n]-s[n;\theta ]{)}^2\\ & \approx \sum _{n=0}^{N-1}{\left(x[n]-s\left[n;{\theta }_0\right]+{\frac{\partial s[n;\theta ]}{\partial \theta }|}_{\theta ={\theta }_0}{\theta }_0-{\frac{\partial s[n;\theta ]}{\partial \theta }|}_{\theta ={\theta }_0}\theta \right)}^2\\ & ={\left(\mathbf{x}-\mathbf{s}\left({\theta }_0\right)+\mathbf{H}\left({\theta }_0\right){\theta }_0-\mathbf{H}\left({\theta }_0\right)\theta \right)}^T\left(\mathbf{x}-\mathbf{s}\left({\theta }_0\right)+\mathbf{H}\left({\theta }_0\right){\theta }_0-\mathbf{H}\left({\theta }_0\right)\theta \right)\end{array}$$
since $\mathbf{x}-\mathbf{s}\left({\theta }_0\right)+\mathbf{H}\left({\theta }_0\right){\theta }_0$ is known, we have as the $\mathrm{L}\mathrm{S}\mathrm{E}$
$$\begin{array}{rl}\hat{\theta }& ={\left({\mathbf{H}}^T\left({\theta }_0\right)\mathbf{H}\left({\theta }_0\right)\right)}^{-1}{\mathbf{H}}^T\left({\theta }_0\right)\left(\mathbf{x}-\mathbf{s}\left({\theta }_0\right)+\mathbf{H}\left({\theta }_0\right){\theta }_0\right)\\ & ={\theta }_0+{\left({\mathbf{H}}^T\left({\theta }_0\right)\mathbf{H}\left({\theta }_0\right)\right)}^{-1}{\mathbf{H}}^T\left({\theta }_0\right)\left(\mathbf{x}-\mathbf{s}\left({\theta }_0\right)\right)\end{array}$$
If we now iterate the solution, it becomes
$${\theta }_{k+1}={\theta }_k+{\left({\mathbf{H}}^T\left({\theta }_k\right)\mathbf{H}\left({\theta }_k\right)\right)}^{-1}{\mathbf{H}}^T\left({\theta }_k\right)\left(\mathbf{x}-\mathbf{s}\left({\theta }_k\right)\right)$$
The linearization method is termed the Gauss-Newton method and can easily be generalized to the vector parameter case as
$${\mathbf{\theta }}_{k+1}={\mathbf{\theta }}_k+{\left({\mathbf{H}}^T\left({\mathbf{\theta }}_k\right)\mathbf{H}\left({\mathbf{\theta }}_k\right)\right)}^{-1}{\mathbf{H}}^T\left({\mathbf{\theta }}_k\right)\left(\mathbf{x}-\mathbf{s}\left({\mathbf{\theta }}_k\right)\right)$$
where
$$[\mathbf{H}(\mathbf{\theta }){]}_{ij}=\frac{\partial s[i]}{\partial {\theta }_j}$$