Convex Optimization 读书笔记 (6)

Chapter7: Statistical estimation

7.1 Parametric distribution estimation

7.1.1 Maximum likelihood estimation

Define log-likelihood function, and denoted l:
$$l(x)=\log p_x(y)$$A widely used method, called maximum likelihood (ML) estimation, is to estimate $x$ as
$${\hat{x}}_{\mathrm{m}\mathrm{l}}=\arg \underset{x}{\max }{p}_x(y)=\arg \underset{x}{\max }l(x)$$

7.1.2 Maximum a posteriori probability estimation

Maximum a posteriori probability (MAP) estimation can be considered a Bayesian version of maximum likelihood estimation, with a prior probability density on the underlying parameter $x$. We assume that $x$ (the vector to be estimated) and $y$ (the observation) are random variables with a joint probability density $p(x,y)$.
The prior density of $x$ is given by
$$p_x(x)=\int p(x,y)dy$$Similarly,
$$p_y(y)=\int p(x,y)dx$$The conditional density of $y$, given $x$, is given by
$$p_{y\mid x}(x,y)=\frac{p(x,y)}{p_x(x)}$$In the MAP estimation method, our estimate of $x$, given the observation $y$, is given by
$$\begin{array}{rl}{\hat{x}}_{\mathrm{m}\mathrm{a}\mathrm{p}}& =\arg \underset{x}{\max }{p}_{x\mid y}(x,y)\\ & =\arg \underset{x}{\max }{p}_{y\mid x}(x,y)p_x(x)\\ & =\arg \underset{x}{\max }p(x,y)\end{array}$$

7.2 Nonparametric distribution estimation

7.3 Optimal detector design and hypothesis testing

Suppose $X$ is a random variable with values in { 1 , . . . , n } \{1, . . . , n\} {1,...,n}, with a distribution that depends on a parameter θ ∈ { 1 , . . . , m } θ ∈ \{1, . . . , m\} θ∈{1,...,m}. The distributions of $X$, for the $m$ possible values of $\theta$, can be represented by a matrix $P\in {\mathbf{R}}^{n\times m}$, with elements
$$p_{kj}=\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{b}(X=k\mid \theta =j)$$The $j$th column of $P$ gives the probability distribution associated with the parameter value $\theta =j$. The $m$ values of $\theta$ are called hypotheses, and guessing which hypothesis is correct is called hypothesis testing.

7.3.1 Deterministic and randomized detectors

A (deterministic) estimator or detector is a function $\psi$ from { 1 , . . . , n } \{1, . . . , n\} {1,...,n} (the set of possible observed values) into { 1 , . . . , m } \{1, . . . , m\} {1,...,m} (the set of hypotheses).
A randomized detector of $\theta$ is a random variable θ ^ ∈ { 1 , . . . , m } \hat{θ} ∈ \{1, . . . , m\} θ^∈{1,...,m}. A randomized detector can be defined in terms of a matrix $T\in {\mathbf{R}}^{m\times n}$ with
$$t_{ik}=\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{b}(\hat{\theta }=i\mid X=k)$$

7.3.2 Detection probability matrix

For the randomized detector defined by the matrix $T$, we define the detection probability matrix as $D=TP$ . We have
$$D_{ij}=(TP{)}_{ij}=\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{b}(\hat{\theta }=i\mid \theta =j)$$so $D_{ij}$ is the probability of guessing $\hat{\theta }=i$, when in fact $\theta =j$.

7.3.3 Optimal detector design

7.3.4 Multicriterion formulation and scalarization

The optimal detector design problem can be considered a multicriterion problem, and the $m(m-1)$ objectives given by the off-diagonal entries of $D$, which are the probabilities of the different types of detection error:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}(\mathrm{w}.\mathrm{r}.\mathrm{t}.{\mathbf{R}}_{+}^{m(m-1)})& D_{ij},i,j=1,...,m,i\ne j\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& t_k⪰0,1^T{t}_k=1,k=1,...,n\end{array}$$

7.3.5 Binary hypothesis testing

As an illustration, we consider the special case $m=2$, which is called binary hypothesis testing.

7.3.6 Robust detectors

We define the worst-case detection probability matrix $D^{\mathrm{w}\mathrm{c}}$ as
$$D_{ij}^{\mathrm{w}\mathrm{c}}=\underset{P\in \mathcal{P}}{\sup }{D}_{ij},i,j=1,...,m,i\ne j$$and
$$D_{ii}^{\mathrm{w}\mathrm{c}}=\underset{P\in \mathcal{P}}{\inf }{D}_{ii},i=1,...,m$$

7.4 Chebyshev and Chernoff bounds

7.4.1 Chebyshev bounds

Chebyshev bounds give an upper bound on the probability of a set based on known expected values of certain functions. If $X$ is a random variable on $\mathbf{R}$ with $\mathbb{E}X=\mu$ and $\mathbb{E}(X-\mu {)}^2={\sigma }^2$, then we have $\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{b}(|X-\mu |\ge 1)\le {\sigma }^2$, again no matter what the distribution of $X$ is.

7.4.2 Chernoff bounds

Let $X$ be a random variable on $\mathbf{R}$. The Chernoff bound states that
$$\mathbf{p}\mathbf{r}\mathbf{o}\mathbf{b}(X\ge u)\le \underset{\lambda \ge 0}{\inf }\mathbb{E}{e}^{\lambda (X-u)}$$which can be expressed as
log ⁡ p r o b ( X ≥ u ) ≤ inf ⁡ λ ≥ 0 { − λ u + log ⁡ E e λ X } \log\mathbf{prob}(X\geq u)\leq \inf_{\lambda\geq 0} \{-\lambda u +\log \mathbb{E} e^{\lambda X} \} logprob(X≥u)≤λ≥0inf​{−λu+logEeλX}

7.4.3 Example

7.5 Experiment design

We consider the problem of estimating a vector $x\in {\mathbf{R}}^n$ from measurements or experiments
$$y_i=a_{i}x+w_i,i=1,...,m$$The associated estimation error $e=\hat{x}-x$ has zero mean and covariance matrix
$$E=\mathbb{E}e{e}^T=(\sum _{i=1}^m{a}_i{a}_i^T{)}^{-1}$$We suppose that the vectors $a_1,...,a_m$, which characterize the measurements, can be chosen among $p$ possible test vectors $v_1,...,v_p\in {\mathbf{R}}^n$. The goal of experiment design is to choose the vectors $a_i$, from among the possible choices, so that the error covariance $E$ is small (in some sense).

7.5.1 The relaxed experiment design problem

In the case when $m$ is large compared to $n$, however, a good approximate solution can be found by ignoring, or relaxing, the constraint that the $m_i$ are integers. The relaxed experiment design problem is
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}(\mathrm{w}.\mathrm{r}.\mathrm{t}.{\mathbf{S}}_{+}^n)& E=\frac{1}{m}(\sum _{i=1}^p{\lambda }_i{v}_i{v}_i^T{)}^{-1}\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& \lambda ⪰0,1^T\lambda =1\end{array}$$

7.5.2 Scalarizations

$D$-optimal design

The most widely used scalarization is called $D$-optimal design, in which we minimize the determinant of the error covariance matrix $E$.
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& \log \det (\sum _{i=1}^p{\lambda }_i{v}_i{v}_i^T{)}^{-1}\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& \lambda ⪰0,1^T\lambda =1\end{array}$$

$E$-optimal design

In $E$-optimal design, we minimize the norm of the error covariance matrix, the maximum eigenvalue of $E$. The $E$-optimal experiment design problem can be cast as an SDP
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& t\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& \sum _{i=1}^p{\lambda }_i{v}_i{v}_i^T⪰tI\\ & \lambda ⪰0,1^T\lambda =1\end{array}$$

$A$-optimal design

In A-optimal experiment design, we minimize $\mathbf{t}\mathbf{r}E$, the trace of the covariance matrix
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& \mathbf{t}\mathbf{r}(\sum _{i=1}^p{\lambda }_i{v}_i{v}_i^T{)}^{-1}\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& \lambda ⪰0,1^T\lambda =1\end{array}$$