预测分析笔记1&2

课程笔记：预测分析 2021Spring

参考教材：Murphy, K. P. (2021). Probabilistic Machine Learning: An Introduction. MIT press.

In this class,we’ll cover topics in machine learning from a probabilistic view.

We will also introduce some topics in statistical computing,such as EM,MCMC,varaitional inference,some optimization algorithm.

Chapter 1 Introduction介绍

• Machine Learning:
  • supervised learning（监督学习）
  • unsupervised learning（无监督学习）
  • reinforcement learning（强化学习）

Supervised Learning监督学习

• The most common form

• Data: $(X,y)$

  • $X$: feature（机器学习中）,covariate,predictor（统计中）

  • $y$: response,label

  • Task : learn a mapping $f$ from input $x\in X$ to out put $y\in Y$, i.e. $y=f(X)$.

    the mapping $f$ is unknown,we want to learn from data.

• Supervised Learning

  (depends on type of response $y$.)

  • classification: discrete,unordered and mutually exclusive labels.
  • regression: real valued quantity $y\in R$ instead of a class label.

  Those idea is similar ,except loss function.

In classification分类问题

The goal is to automatically come up with classification model $\hat{f}$:decision tree,neural nets,logistic regressions to reliably predict the labels $y=\hat{f}(X)$.

• How to measure the performance of $\hat{f}$ on this data?

  we look at the misclassification rate:

  L ( θ ) = 1 N ∑ n = 1 N I { y n ≠ f ( x n , θ ) } L(\theta)=\frac{1}{N} \sum_{n=1}^N \mathbb{I}_{\{y_n\neq f(x_n,\theta)\}} L(θ)=N1​n=1∑N​I{yn​​=f(xn​,θ)}​,where $\mathbb{I}(e)$ is the binary indicator function.

  • we might also assume errors are unequal. So we give a more generous definition.

  • Define empirical risk to be the average loss of predictor:

    L ( θ ) = 1 N ∑ n = 1 N l { y n ≠ f ( x n , θ ) } L(\theta)=\frac{1}{N} \sum_{n=1}^N l_{\{y_n\neq f(x_n,\theta)\}} L(θ)=N1​n=1∑N​l{yn​​=f(xn​,θ)}​

  • The zero-one loss is a special case. i.e. $l(y,\hat{y})=\mathbb{I}(y,\hat{y})$

  Our goal is to minimize the expected loss on future data that have not yet seen.

  With a suitably flexible model,we can derive the training loss to zero by simply memorizing the correct out put for each input. But it does not do well in test dataset.

• How can we pick a model of the right complexity?

  If we use training set to evaluate different models,we’ll always pick the most complex model , since we will have the most degree of freedom,and hence will have minimum loss,so insted we should pick the model with minimum test error.

In regression回归问题

we want to predict a real-valued quantity $y\in R$ instead of a class label y ∈ { 1 , 2 , . . . , c } y\in\{1,2,...,c\} y∈{1,2,...,c}.We have different loss function:$$l(y,\hat{y})=(y-\hat{y}{)}^2$$

• $MSE(\theta )=\frac{1}{N}{\sum }_{n=1}^N(y_n-f(x_n,\theta ){)}^2$

Unsupervised Learning无监督学习

• In supervised learning,we assume that each input example $X$ in the training set has an associated set of output targets $y$ , and our goal is to learn the input-output mapping $f$.

• In unsupervised learning,we just get observed “inputs” D = { x n ∣ n = 1 : n } D=\{x_n| n=1:n\} D={xn​∣n=1:n} without any corresponding “outputs”

• From a probabilistic perspective , we can view the task of unsupervised learning as fitting an unconditional model of the form $p(X)$,which can generate new data $X$, whereas supervised learning fitting $p(y|X)$ , which specifies outputs given inputs.

• One simple example of unsupervised learning is clustering. The goal is to partition the input into regions that contain “similar” points.

• How to evaluate unsupervised learning?

  • It is hard to evaluate the quality of the output of an unsupervised learning method , because there is no ground truth to compute to.

  • A common method for evaluating unsupervised models is to measure the probability assigned by the model to unseen test examples.

    $$L(\theta ;D)=-\frac{1}{|D|}\sum _{x\in D}logp(X|\theta )$$

    e.g. mixture models.

Chapter 2 Probabilistic inference 概率推断

Bayesian VS Frequentist贝叶斯学派vs频率学派

introduction and compare介绍与对比

• Frequentist :$\theta$ is deterministic.(为一定值)

  eg. MLE ,MOM

• Bayesian: $\theta orH$ is unknown quantity of interest,is a random variable.（为一随机变量）

• Data:$y$

  Given model ,$p(y|\theta )$

Frequentist频率学派

• Frequentist:

  $\theta =\underset{\theta }{\mathrm{argmax}}P(y|\theta )$ (MLE解)

Bayes贝叶斯学派

• Bayesian: $p(\theta |y)=\frac{p(y|\theta )p(\theta )}{p(y)}$ (product rule of probability)

  • Basis of Bayesian inference

    • Bayes rule: computing the probability distribution over possible values of an unknown quantity $H$ given some observed data $Y=y$.

      $$P(H=h|Y=y)=\frac{P(H=h)P(Y=y|H=h)}{P(Y=y)}$$, we can find that $posterior\propto prior * likelihood $

      $$P(h|y)P(y)=P(h)P(y|h)=P(h,y)$$

      • $P(H)$ : prior distribution,what we know about $H$ before we see any data.
      • $P(Y|H=h)$ : the distribution over possible outcomes $Y$ when we expect to see if $H=h$.
      • $P(H,y)$ : unnormalized posterior distribution.
      • $P(Y=y)$ : marginal likelihood function.
        week 1

    • Posterior predictive：
      $$P(\theta |y)\propto P(y|\theta )\ast P(\theta )$$,i.e. we ignore normalizing constant $p(y)$
      $P(y_{new}|y)=\int p(y_{new}|\theta )\ast p(\theta |y)d\theta$.Bayes model average,we will introduce a bit more later.(在Chapter 4 中讲解)

compare对比（MAP与MLE）

• MAP、MLE and plugin approximation

  As the amount of data increases,the posterior will become concentrated around a single point,namely the posterior mode(后验众数).

  (In MLE,we provide a point estimate. But in many cases,we want to quantify the uncertainty,so we derive the standard deviation, and confidence interval.)

  The posterior mode is defined as the hypothesis with maximum posterior probability:

  ​ $$h_{map}=\underset{h}{\mathrm{argmax}}P(h|D)=\underset{h}{\mathrm{argmax}}[\log P(D|h)+\log P(h)]$$

  h:参数，D:数据集

  This is called the maximum a posterior or MAP estimate.

  • As the data set increases in size, the log likelihood grows in magnitude, but the prior term remains constant. We thus say that the likelihood overwhelms the prior.

    当数据集中数据量在增大时，log似然的绝对值在增大，但是先验是不变的（与数据量无关）。

  • A reasonable approximation to the MAP estimate is to ignore the prior term.(MLE is a reasonable approximate of MAP if the size of data is large,plus the prior distribution is weak.)

    所以在数据量很大时，log似然会起到主导的作用，先验的作用会很小。此时MAP和MLE近似是相等的。

  • Note that MAP is a way to represent posterior distribution,as a point estimate.

    MAP是一个点估计，我们也可以根据方差和置信度给出置信域。

Is Bayes relevant in the “big data era”?贝叶斯学派方法是否能在大数据时代很好的应用？

• As the amount of data increases, the posterior $p(\theta |D)$ often shrinks to a point, since the log likelihood term, $\log p(D|\theta )$, grows with N, whereas the log prior,$logp(\theta )$, is independent of N.

• Consequently, the posterior approaches a delta function at the MLE,$p(\theta |D)\to \delta (\theta -\hat{\theta })$,where $\delta (x)$ is delta function.

  在大数据下，Bayes与频率学派得出的结果区别不是很大（MAP与MLE），但Bayes方法较复杂、计算成本较高（更新后验分布/似然函数不易计算）。故在大数据下，“不需”使用Bayes方法。

• Thus one way think that Bayes is irrelevant in the era of “big data”.