Basic Concepts for Learning from Data

When we talk about learning from data, we talk about models, algorithms, learning and inference procedures. Here I present my understanding about this term 'Learning From Data' in the context of machine learning which can also be seen as part of my preparation for MLPR examination. The content is greatly aided by Amos Storkey.

1.1 Basic Concepts

The first question is: what is the relation between models and algorithms in the context of machine learning?


1) Generally, model encodes our understanding about the data. It is some kind of process of learning from data.
2) Algorithms in the context of machine learning, come from the model(with a bit of maths describing the process), to give the approximation of the real world. Different algorithms outputs different approximations.


Machine learning does not give us something for nothing. Once we apply machine learning approaches to some datasets, we need to have assumptions about the distribution of data and the model. We can simplify this procedure as using out prior beliefs and our model, plus the data to have some posterior beliefs.

Can we do this without a prior model? In fact we can not do this, because without a prior model, we have no connection between data and problems. A prior model is sometimes called the inductive bias.So we can say the term 'model free' is not acceptable. There are also words saying that we might not need prior assumptions or having bias free model. However, from this perspective, all the prior can be treated as some kind of bias.


The procedure of modeling is to construct the general structure of a set of phenomenons, find out the general and common aspects and describe it with some language as well as leaving some level of flexibility to fit different scenarios. Once we are facing some specific instance, we can fit the data within this general framework, and narrow down the space or reduce the uncertainty about this particular problem. Usually, a good model will have some level of this universality.

How can we measure the utility of a model? Which one is a good model and which is a bad one? One simple approach is to use the measure of likelihood.

Suppose we have a series of data, represented asX, here we assume all data are independent identically distributed (i.i.d). Now we have a modelM, we define the likelihood as:

The higher the likelihood, the better the model is, and this approach is called Maximum Likelihood. Note that here we also make an assumption that we know all the parameters of a model. And of course, this measurement contains a underlying assumption that we treat each model as equal.

1.2 Learning with Bayesian Methods
Is the approach of ML enough for us? Any problems about this approach?

Simple answer, Maximum likelihood fails to capture prior or uncertainty. To say it explicitly, ML approach prefers those model with more parameters, because usually more parameters leads to more flexibility of the model, which results a better fitting of the data. In other words, more parameters usually leads to a higher likelihood. However, this might lead to overfitting and model with this will lose its generality.

Do we have other choices? Yes, Bayesian Methods may help. Here I will simply talk about Bayesian Methods used in probabilistic machine learning.

First question, what is a Bayesian Method? Is the approach using Bayes Rule a Bayesian Method? The answer is no. A Bayesian Method:

1 Has a model over some parameters p(X|theta)
2 Introduces the uncertainty over parameters p(theta)
3 Learns the posterior of the parameters from data using Bayesian Rule 

How do we infer with this posterior distribution? We take every possibility into consideration and produce the weighted average of the answer.

How do we compare models with this approach? We now compare the posterior distribution of models

 

Here I just present the basic idea of these complex concepts, in the real world of machine learning, they can be changed into various forms.