Lecture4-2Probability to rescure

Inferring something unknown

Difficult to infer unknown target $f$ outside $\mathcal{D}$, can we infer something unknown in other scenarios?

How to infer the orange probability?

Sampling

• Bin: ASSUME orange prob $= \mu$, green prob $= 1- \mu$ with $\mu$ unknown
• Sample: $N$ marbles sampled independently, with orange fraction $= \nu$, green fraction $= 1- \nu$, with $\nu$ known.

Possible vs. Probable

• possibly not: sample can be an extreme case, so we cannot possibly learn $f$
• probably yes: in-sample $\nu$ likely close to unknown $\mu$

Hoeffding’s Inequality

With a sample size of $N$, $\nu$ is probably close to $\mu$(within $\epsilon$):

$$\begin{align} \mathbb{P}[|\nu - \mu| > \epsilon] \leq 2\exp(-2\epsilon^2N) \end{align}$$
The probability of big deviation( $\epsilon$) is small.

The statement $\nu=\mu$ is probably approximate correct(PAC).

• valid for all $N$ and $\epsilon$
• doesn’t depend on $\mu$, so we do not need to know $\mu$
• larger size of $N$ or loose gap $\epsilon \Rightarrow$ higher probability for $\mu \approx \nu$

If larger $N$, can probably infer unknown $\mu$ from $\nu$