Machine Learning Theory - L1 Concentration Inequalities

Machine Learning Theory - L1 Concentration Inequalities

This is the first lecture of course Machine Learning Theory (AI603). This lecture introduced some basic but useful concentration inequalities.

What are concentration inequalities?

Concentration inequalities furnish us bounds on how random variables deviate from a value (typically, expected value) or help us to understand how well they are concentrated.

—Introduction to Concentration Inequalities

Why we need?

A classification algorithm has an error probability ϵ, i.e., the output of the algorithm misclassifies a randomly sampled data with probability ϵ. What is the probability of the event that the algorithm misclassifies more than 200ϵ data points among 100 data points.

1 Markov Inequality

---

Theorem: If $X$ is a non-negative random variable,
$$\mathbb{P}(X\ge a)\le \frac{\mathbb{E}[X]}{a},\forall a>0$$

---

Proof:

Let $f$ be the probability density function (PDF) of $X$.
$$\begin{array}{rl}\mathbb{E}[X]& ={\int }_0^{\infty }xf(x)dx\\ & \ge {\int }_a^{\infty }xf(x)dx\\ & \ge a{\int }_a^{\infty }f(x)dx\\ & =a\mathbb{P}(X\ge a)\end{array}$$
By rearranging the terms,
$$\mathbb{P}(X\ge a)\le \frac{\mathbb{E}[X]}{a}$$

A simple trick to make positive random variables:

A non-negative strictly increasing function $\varphi (x)=e^{\theta x}$

Then, with Markov Inequality we have Chernoff bound:
$$\begin{gathered} \mathbb{P}(X\ge a)=\mathbb{P}(e^X\ge e^a)=\mathbb{P}(e^{\theta X}\ge e^{\theta a}) \\ \le \frac{\mathbb{E}[e^{\theta X}]}{e^{\theta a}}\forall a,\theta >0 \end{gathered}$$
When $X$ is the sum of independent random variables $X_1,\dots ,X_n$,
$$\mathbb{P}(X\ge a)\le \underset{\theta >0}{\inf }\frac{\mathbb{E}[e^{\theta X}]}{e^{\theta a}}=\underset{\theta >0}{\inf }\frac{{\prod }_{i=1}^n\mathbb{E}[e^{\theta X_i}]}{e^{\theta a}}\forall a>0$$

2 Chernoff-Hoeffding

---

Theorem: Let $X_1,\dots ,X_n$ be i.i.d. Bernoulli random variables $f_X(x)=p^x(1-p{)}^{1-x}=\{\begin{array}{r}[l]p\text{if }x=1,\\ q\text{if }x=0.\end{array}$ with probability $p$. When $0<p\le q$,
$$\mathbb{P}(\sum _{i=1}^n{X}_i\ge nq)\le e^{-nD(q||p)},$$
where $D(q||p)=q\log \frac{p}{q}+(1-q)\log \frac{1-q}{1-p}$ is the KL-divergence. When $0<q\le p$,
$$\mathbb{P}(\sum _{i=1}^n{X}_i\le nq)\le e^{-nD(q||p)}.$$

---

Proof:

From the Markov inequality, for all $\theta >0$
$$\begin{array}{rl}\mathbb{P}(\sum _{i=1}^n{X}_i\ge nq)& \le \frac{\mathbb{E}[e^{\theta {\sum }_{i=1}^n{X}_i}]}{e^{\theta nq}}\\ & =\frac{{\prod }_{i=1}^n\mathbb{E}[e^{\theta X_i}]}{e^{\theta nq}}\\ & =\frac{{\prod }_{i=1}^n(p{e}^{\theta }+1-p)}{e^{\theta nq}}.\\ & =\exp (-\theta nq)(p{e}^{\theta }+1-p{)}^n\end{array}$$
To find the minimization of the above equality, let $\varphi (\theta )=\ln (\exp (-\theta nq)(p{e}^{\theta }+1-p{)}^n)$.
$$\begin{array}{r}\varphi (\theta )=-\theta nq+n\ln (p{e}^{\theta }+1-p)\\ {\varphi }^{\prime }(\theta )=-nq+\frac{np{e}^{\theta }}{p{e}^{\theta }+1-p}=0\\ e^{\theta }=\frac{q^2}{p^2}=\frac{q(1-p)}{p(1-q)}\end{array}$$
Thus, with $\theta =\log (\frac{q(1-p)}{p(1-q)})$,
$$\mathbb{P}(\sum _{i=1}^n{X}_i\ge nq)\le e^{-nD(q||p)}.$$

3 Hoeffding’s inequality

---

Lemma: Let $X\in [a,b]$ be a random variable. Then, for all $\theta >0$,
$$\mathbb{E}[e^{\theta (X-\mathbb{E}[X])}]\le \exp (\frac{{\theta }^2(b-a{)}^2}{8}).$$
Theorem: Let $X_1,\dots ,X_n$ be independent random variables. When $X_i$ is bounded by $[a_i,b_i]$ for all $i$,
$$\mathbb{P}(\frac{1}{n}\sum _{t=1}^n(X_i-\mathbb{E}[X_i])\ge t)\le \exp (-\frac{2n^2{t}^2}{{\sum }_{i=1}^n(b_i-a_i{)}^2}).$$

---

Proof:

Let $Y=X-$