8:32 2014-09-29 Monday
start review CalTech machine learning,
video 9, Neural Network
8:32 2014-09-29
gradient descent => stochastic gradient descent
8:53 2014-09-29
SGD == Stochastic Gradient Descent
8:54 2014-09-29
SVM == Support Vector Machine
8:54 2014-09-29
outline:
* Stochastic gradient descent
* Neural network model
* Backpropagation algorithm
8:55 2014-09-29
gradient descent // batch gradient descent
9:00 2014-09-29
SGD // Stochastic Gradient Descent
9:00 2014-09-29
one example per time
9:03 2014-09-29
average direction
9:03 2014-09-29
randomized GD // SGD == Stochastic Gradient Descent
9:04 2014-09-29
Benefits of SGD:
* cheaper computation
* randomization
* simple
9:10 2014-09-29
we're going for cheap computation, and we're
going for this one for free(randomization to
escape local minimum)
9:10 2014-09-29
randomization helps
9:14 2014-09-29
learning rate:
tells us how far we go.
9:14 2014-09-29
SGD in action
9:15 2014-09-29
SGD to solve "movie rating"
9:17 2014-09-29
we're going to match the taste of the user
to the content of the movie
9:18 2014-09-29
batch gradient descent => stochastic gradient descent
9:21 2014-09-29
biological inspiration of neural networks
9:22 2014-09-29
biological function => biological structure
9:24 2014-09-29
maybe if we put a bunch of perceptrons together
in a network, we maybe able to achieve the intelligence
in a learning that a biological system does.
9:26 2014-09-29
combination of perceptrons rather than a single one.
9:31 2014-09-29
combining this very simple unit does achieve something
9:32 2014-09-29
the famous problem where perceptrons failed
9:32 2014-09-29
can we do this with more than one perceptrons
combined in a right way?
9:33 2014-09-29
that's create the full multilayer perceptron
9:38 2014-09-29
this is the original input space
10:12 2014-09-29
so this multilayer perceptron implements the function
that single perceptron failed.
10:14 2014-09-29
"feedforward"
10:15 2014-09-29
I can get a very sophisticated surface under
the constaints of this hierarchical thing.
10:15 2014-09-29
powerful model
10:16 2014-09-29
for us, being powerful is good, but with
2 red flags:
* generalization
* optimization
10:17 2014-09-29
what is the combination of weights that
matches a function.
10:23 2014-09-29
so let's look at the neural networks
10:25 2014-09-29
the neural network looks like this.
10:25 2014-09-29
each layer has a nonlinearity
10:29 2014-09-29
θis used in logistic regression as
a logistic function
10:30 2014-09-29
I'm here used here genetically for any
nonlinearity you want.
10:30 2014-09-29
I could have a label for this depend on
where this happens.
10:31 2014-09-29
follow the rules of derivation from one layer to another
10:34 2014-09-29
the intermediate layers, we're going to call them
"hidden layers", because the user didn't see them.
10:35 2014-09-29
it's a soft threshold, I'm going to use the tanh
(hyperbolic tangent),
10:38 2014-09-29
it is the combination of hard threshold & linearity
10:39 2014-09-29
if your signal is very small, it's as if you're linear
if your signal is extremely large, it's as if you're hard threshold
10:39 2014-09-29
and you get the benefit of one function that is analytic
& very well behaved for good optimization.
10:41 2014-09-29
the notation will be more elaborate than the perceptron.
10:41 2014-09-29
although it's only a notaional view graph, it's an
important view graph to follow, because if you decide to implement
a neural network, you just print this view graph, and
call it. and you have your neural network.
10:43 2014-09-29
the parameters of neural network are called w,
the weights happen to belong to any layer to any
neuron.
10:44 2014-09-29
you keep repeating until you get the final
11:03 2014-09-29
you apply x to the input terminal
11:03 2014-09-29
d is the dimensionality of your network
11:03 2014-09-29
backpropagation algorithm
11:08 2014-09-29
Applying SGD:
all the weights determines h // hypothesis
11:11 2014-09-29
target label: yn
11:11 2014-09-29
get error on example (xn, yn)
e(h(xn), yn) = e(w)
11:12 2014-09-29
to implement SGD, we need the gradient
▽e(w)
11:13 2014-09-29
the idea here is just doing it efficiently
11:14 2014-09-29
backpropagation algorithm:
take one example at a time, apply it to the network,
then adjust all the weights of the network in the
direction of the negative gradient according to that
single example, that is what makes it stochastic.
11:17 2014-09-29
the parameters are all the weights.
11:17 2014-09-29
you have different neurons in different layers,
so this is just a funny array.
11:17 2014-09-29
by definition, I have some error measure,
e(h(xn), yn), and this happens to be the function
of weights of the network.
h is determined by w, because this is active quantity
when we're learning
11:18 2014-09-29
to implement SGD, all you need to implement is
the gradient of this quantity:
▽e(w) // take lots of partial derivative i, j, l
11:21 2014-09-29
so all you need to do is compute this partial
derivative for every i, j, l
11:22 2014-09-29
the gradient vector is a huge vector,
each partial derivative is a component.
11:23 2014-09-29
then you take this entire vector of stuff,
then you move in the space along the negative
of that gradient. // gradient descent
11:23 2014-09-29
there is a bigger difference when you find
an efficient algorithm to do something.
11:24 2014-09-29
let take part of the network,
11:26 2014-09-29
feeding through some weight into this guy.
11:26 2014-09-29
that error will change if you change w,
this what partial e partial w tells us. // partial derivate
// rate of change
11:28 2014-09-29
output is a function of the previous layer
of the previous layer of the previous layer...
until I arrive here
11:30 2014-09-29
I have the network that has tons of weights.
11:30 2014-09-29
a trick for efficient computation
11:35 2014-09-29
but this one is as good as the original one
11:35 2014-09-29
this quantity can be computed recursively.
11:36 2014-09-29
the more trouble one, I'm going to call it δ
11:37 2014-09-29
the interesting thing is that the derivative
of this error respect to weight
11:38 2014-09-29
so let's get δ for the final layer
11:38 2014-09-29
the change in the w will be proportion to these 2 guys:
one of them is x here, and one of them is δ here
11:40 2014-09-29
so we will change weight according to two quanities,
that the weight is sanwiched between, that is a pretty
attractive one.
11:44 2014-09-29
let's get δ for the final layer
11:44 2014-09-29
when we compute the thing, we got x for the 1st layer,
put the input, then we propagate forward, got the output.
11:44 2014-09-29
if you get δ later, you'll be get δ earlier.
11:45 2014-09-29
so this will be propagate backward, hence the name
backpropagation
11:46 2014-09-29
so we're going to start with the δ at the output.
11:46 2014-09-29
so when you look at the output layer,
11:47 2014-09-29
what is e(w)?
e(w) is the error measure between the value of
your hypothesis,that is the value of the neuron network
in its current state with the weights frozen, you apply xn,
you go forward, until you get the output, which is h(xn),
you compare that to the target output, which is the label
of the example yn
11:56 2014-09-29
so we have δ for the final layer
11:59 2014-09-29
the next is to back propagate down to other δs
so this is the essence of the algorithm
12:00 2014-09-29
I'm learning backward.
12:02 2014-09-29
so now I need to take into consideration all
the ways that affect the output,I'm drawing all
relevant of the network.
12:04 2014-09-29
so I'm going to apply the chain rule again.
12:04 2014-09-29
if affect all of these guys
12:05 2014-09-29
we're done, we just have to keep doing this,
and we get all the δs.
12:10 2014-09-29
the form for the δ is interesting.
12:11 2014-09-29
this look exactly like the forward path.
12:12 2014-09-29
you see the reverse, now we're going down.
12:13 2014-09-29
that's what we do in the backward propagation.
12:14 2014-09-29
backpropagation algorithm:
you pick the input, you compute x as forward,
you get the error, you compute the δs backward,
the δ & the x determine the weight in between.
12:16 2014-09-29
update the weights
12:16 2014-09-29
you're perfectly at the top of the hill
12:18 2014-09-29
but as long as you're there, you're not moving
12:19 2014-09-29
we just want to break the symmetry,
we introduce randomness, we shake the food a little bit,
12:22 2014-09-29
one final remark: the hidden layer
12:23 2014-09-29
the hidden layers are just a means for us
to get more sophisticated dependency.
12:23 2014-09-29
if you think what the hidden layers do,
they just do a nonlinear transform.
12:24 2014-09-29
I can look these guys, and consider them
features, these one will be features of features...
12:25 2014-09-29
these are learned features
12:25 2014-09-29
don't look at the data before you choose the transform.
12:26 2014-09-29
the network is looking at the data all it wants.
it's actually adjusting the weights to get the proper
transform the fit the data.
12:27 2014-09-29
this is not bothering me, because I'm already charge
the network for the proper VC
12:28 2014-09-29
the weights here that constitute that guy contribute
to the VC dimension
12:28 2014-09-29
VC dimension is more or less a number of weights, that's
the rule of thumb here.
12:29 2014-09-29
looking at the data without accounting for it is bad
12:29 2014-09-29
it's a nonlinear transformation with a view that
matches very specific dependence that I'm after.
that's the source of efficiency there.
12:30 2014-09-29
can I interpret what the hidden layers are doing?
12:31 2014-09-29
can you please tell what the hidden layers are doing?
12:31 2014-09-29
factor number 7 is important in your case.
12:33 2014-09-29
very common in machine learning, when the learning
algorithm tries to learn, it tries to use the right
hypothesis, it didn't tries to explain to you what the
right hypothesis is. that was the goal.
----------------------------------------------------------
12:37 2014-09-29
batch gradient descent => stochastic gradient descent(SGD)
12:38 2014-09-29
conjugate gradient
12:38 2014-09-29
convex optimization
12:38 2014-09-29
initially the neural networks are going to solve
the problems of the universe.
12:39 2014-09-29
because the simplicity of the network & the simplicity
of the algorithm, people use them in many applications.
it became the standard tool.
12:41 2014-09-29
they have some serious competitors, for example, SVM(Support Vector
Machine), lots of other models, but they still in use.
12:42 2014-09-29
not the top choice nowadays, but every now and then, someone
will publish something and he will use the neural network and
get good results.
12:44 2014-09-29
how to choose the number of layers?
OK, this is "model selection", the neural network
is really a class of hypothese set.
how many layers, how many units per layer.
12:47 2014-09-29
that's not an approximation question, we're
talking about learning question.
12:48 2014-09-29
the real questions is: how many weights can I afford?
because it's reflect directly at the VC dimension &
the number of examples we need.
12:48 2014-09-29
the question of organizing is less severe.
12:50 2014-09-29
given a particular architecture, it tries to kill
some weights in order to reduce the number of parameters
as a method for regularization.
12:51 2014-09-29
basically this is a model selection questions:
validation, regularization
12:52 2014-09-29
I want the analytically property of differentiation.
12:53 2014-09-29
but the whole idea is that you're going to arrive
at a minimum.
12:54 2014-09-29
you just start from different starting point.
different randomization for the presentation.
12:55 2014-09-29
we're dealing with sophisticated model:
12:57 2014-09-29
2 red flags: generalization & optimization
12:58 2014-09-29
when you have a powerful model, the question of generalization comes
in, means you can express a lot of things,
12:59 2014-09-29
the VC dimension summarizes all generalization considerations
13:00 2014-09-29
but at least we're under control, because we
have the numbers that describe it.
13:02 2014-09-29
I'm given the data set: (input, output)
and I have multilayer perceptron, each layer which
is computing a perceptron function of perceptron function of...
13:03 2014-09-29
the VC dimension is roughly the number of parameters,
that has stood the time of practice.
13:05 2014-09-29
the minimum of the error function will happens
at the particular combination of the weights.
13:14 2014-09-29
neural network is a model, genetic algorithm is
an optimization technique, there is no relationship
between them.
13:17 2014-09-29
the learning algorithm looks at the data,
but we already choose the hypothese set.
13:19 2014-09-29
I recommend to use a package for neural networks
13:20 2014-09-29
neural networks have been studies for a great
level of detail
13:21 2014-09-29
if you're very keen on the interpretation aspect.
13:21 2014-09-29
when you're combining perceptrons, you're going
to implement more interesting functions.
13:22 2014-09-29
the structure of multilayer is an interesting model
to study, from then on, it becomes a learning question
13:24 2014-09-29
we have a neural network, we no longer to look at the
target function, try to choose the neuron, we try to
put it as a model, we let the learning algorithm to choose
the weight.that is the backpropagation in this case.
13:25 2014-09-29
could you please describe early stopping?
it's basically a way to prevent overfitting,
// validation, regularization (model selection)
13:27 2014-09-29
what are the tools to deal with overfitting?
validation & regularization
then early stopping will be very easily explained.
13:28 2014-09-29
instead of choose points at random, you choose
random permutation, from 1 to n, and then go through
that in order, for the next epoch, you do another
permutation,...
13:31 2014-09-29
If you do this way, eventually every example will
contribute the same, but an epoch will be difficult
to define,
13:32 2014-09-29
Does have layers no loops limit the power of the
neural network?
13:34 2014-09-29
VC dimension roughly depends on the number of parameters(weights)
start review CalTech machine learning,
video 9, Neural Network
8:32 2014-09-29
gradient descent => stochastic gradient descent
8:53 2014-09-29
SGD == Stochastic Gradient Descent
8:54 2014-09-29
SVM == Support Vector Machine
8:54 2014-09-29
outline:
* Stochastic gradient descent
* Neural network model
* Backpropagation algorithm
8:55 2014-09-29
gradient descent // batch gradient descent
9:00 2014-09-29
SGD // Stochastic Gradient Descent
9:00 2014-09-29
one example per time
9:03 2014-09-29
average direction
9:03 2014-09-29
randomized GD // SGD == Stochastic Gradient Descent
9:04 2014-09-29
Benefits of SGD:
* cheaper computation
* randomization
* simple
9:10 2014-09-29
we're going for cheap computation, and we're
going for this one for free(randomization to
escape local minimum)
9:10 2014-09-29
randomization helps
9:14 2014-09-29
learning rate:
tells us how far we go.
9:14 2014-09-29
SGD in action
9:15 2014-09-29
SGD to solve "movie rating"
9:17 2014-09-29
we're going to match the taste of the user
to the content of the movie
9:18 2014-09-29
batch gradient descent => stochastic gradient descent
9:21 2014-09-29
biological inspiration of neural networks
9:22 2014-09-29
biological function => biological structure
9:24 2014-09-29
maybe if we put a bunch of perceptrons together
in a network, we maybe able to achieve the intelligence
in a learning that a biological system does.
9:26 2014-09-29
combination of perceptrons rather than a single one.
9:31 2014-09-29
combining this very simple unit does achieve something
9:32 2014-09-29
the famous problem where perceptrons failed
9:32 2014-09-29
can we do this with more than one perceptrons
combined in a right way?
9:33 2014-09-29
that's create the full multilayer perceptron
9:38 2014-09-29
this is the original input space
10:12 2014-09-29
so this multilayer perceptron implements the function
that single perceptron failed.
10:14 2014-09-29
"feedforward"
10:15 2014-09-29
I can get a very sophisticated surface under
the constaints of this hierarchical thing.
10:15 2014-09-29
powerful model
10:16 2014-09-29
for us, being powerful is good, but with
2 red flags:
* generalization
* optimization
10:17 2014-09-29
what is the combination of weights that
matches a function.
10:23 2014-09-29
so let's look at the neural networks
10:25 2014-09-29
the neural network looks like this.
10:25 2014-09-29
each layer has a nonlinearity
10:29 2014-09-29
θis used in logistic regression as
a logistic function
10:30 2014-09-29
I'm here used here genetically for any
nonlinearity you want.
10:30 2014-09-29
I could have a label for this depend on
where this happens.
10:31 2014-09-29
follow the rules of derivation from one layer to another
10:34 2014-09-29
the intermediate layers, we're going to call them
"hidden layers", because the user didn't see them.
10:35 2014-09-29
it's a soft threshold, I'm going to use the tanh
(hyperbolic tangent),
10:38 2014-09-29
it is the combination of hard threshold & linearity
10:39 2014-09-29
if your signal is very small, it's as if you're linear
if your signal is extremely large, it's as if you're hard threshold
10:39 2014-09-29
and you get the benefit of one function that is analytic
& very well behaved for good optimization.
10:41 2014-09-29
the notation will be more elaborate than the perceptron.
10:41 2014-09-29
although it's only a notaional view graph, it's an
important view graph to follow, because if you decide to implement
a neural network, you just print this view graph, and
call it. and you have your neural network.
10:43 2014-09-29
the parameters of neural network are called w,
the weights happen to belong to any layer to any
neuron.
10:44 2014-09-29
you keep repeating until you get the final
11:03 2014-09-29
you apply x to the input terminal
11:03 2014-09-29
d is the dimensionality of your network
11:03 2014-09-29
backpropagation algorithm
11:08 2014-09-29
Applying SGD:
all the weights determines h // hypothesis
11:11 2014-09-29
target label: yn
11:11 2014-09-29
get error on example (xn, yn)
e(h(xn), yn) = e(w)
11:12 2014-09-29
to implement SGD, we need the gradient
▽e(w)
11:13 2014-09-29
the idea here is just doing it efficiently
11:14 2014-09-29
backpropagation algorithm:
take one example at a time, apply it to the network,
then adjust all the weights of the network in the
direction of the negative gradient according to that
single example, that is what makes it stochastic.
11:17 2014-09-29
the parameters are all the weights.
11:17 2014-09-29
you have different neurons in different layers,
so this is just a funny array.
11:17 2014-09-29
by definition, I have some error measure,
e(h(xn), yn), and this happens to be the function
of weights of the network.
h is determined by w, because this is active quantity
when we're learning
11:18 2014-09-29
to implement SGD, all you need to implement is
the gradient of this quantity:
▽e(w) // take lots of partial derivative i, j, l
11:21 2014-09-29
so all you need to do is compute this partial
derivative for every i, j, l
11:22 2014-09-29
the gradient vector is a huge vector,
each partial derivative is a component.
11:23 2014-09-29
then you take this entire vector of stuff,
then you move in the space along the negative
of that gradient. // gradient descent
11:23 2014-09-29
there is a bigger difference when you find
an efficient algorithm to do something.
11:24 2014-09-29
let take part of the network,
11:26 2014-09-29
feeding through some weight into this guy.
11:26 2014-09-29
that error will change if you change w,
this what partial e partial w tells us. // partial derivate
// rate of change
11:28 2014-09-29
output is a function of the previous layer
of the previous layer of the previous layer...
until I arrive here
11:30 2014-09-29
I have the network that has tons of weights.
11:30 2014-09-29
a trick for efficient computation
11:35 2014-09-29
but this one is as good as the original one
11:35 2014-09-29
this quantity can be computed recursively.
11:36 2014-09-29
the more trouble one, I'm going to call it δ
11:37 2014-09-29
the interesting thing is that the derivative
of this error respect to weight
11:38 2014-09-29
so let's get δ for the final layer
11:38 2014-09-29
the change in the w will be proportion to these 2 guys:
one of them is x here, and one of them is δ here
11:40 2014-09-29
so we will change weight according to two quanities,
that the weight is sanwiched between, that is a pretty
attractive one.
11:44 2014-09-29
let's get δ for the final layer
11:44 2014-09-29
when we compute the thing, we got x for the 1st layer,
put the input, then we propagate forward, got the output.
11:44 2014-09-29
if you get δ later, you'll be get δ earlier.
11:45 2014-09-29
so this will be propagate backward, hence the name
backpropagation
11:46 2014-09-29
so we're going to start with the δ at the output.
11:46 2014-09-29
so when you look at the output layer,
11:47 2014-09-29
what is e(w)?
e(w) is the error measure between the value of
your hypothesis,that is the value of the neuron network
in its current state with the weights frozen, you apply xn,
you go forward, until you get the output, which is h(xn),
you compare that to the target output, which is the label
of the example yn
11:56 2014-09-29
so we have δ for the final layer
11:59 2014-09-29
the next is to back propagate down to other δs
so this is the essence of the algorithm
12:00 2014-09-29
I'm learning backward.
12:02 2014-09-29
so now I need to take into consideration all
the ways that affect the output,I'm drawing all
relevant of the network.
12:04 2014-09-29
so I'm going to apply the chain rule again.
12:04 2014-09-29
if affect all of these guys
12:05 2014-09-29
we're done, we just have to keep doing this,
and we get all the δs.
12:10 2014-09-29
the form for the δ is interesting.
12:11 2014-09-29
this look exactly like the forward path.
12:12 2014-09-29
you see the reverse, now we're going down.
12:13 2014-09-29
that's what we do in the backward propagation.
12:14 2014-09-29
backpropagation algorithm:
you pick the input, you compute x as forward,
you get the error, you compute the δs backward,
the δ & the x determine the weight in between.
12:16 2014-09-29
update the weights
12:16 2014-09-29
you're perfectly at the top of the hill
12:18 2014-09-29
but as long as you're there, you're not moving
12:19 2014-09-29
we just want to break the symmetry,
we introduce randomness, we shake the food a little bit,
12:22 2014-09-29
one final remark: the hidden layer
12:23 2014-09-29
the hidden layers are just a means for us
to get more sophisticated dependency.
12:23 2014-09-29
if you think what the hidden layers do,
they just do a nonlinear transform.
12:24 2014-09-29
I can look these guys, and consider them
features, these one will be features of features...
12:25 2014-09-29
these are learned features
12:25 2014-09-29
don't look at the data before you choose the transform.
12:26 2014-09-29
the network is looking at the data all it wants.
it's actually adjusting the weights to get the proper
transform the fit the data.
12:27 2014-09-29
this is not bothering me, because I'm already charge
the network for the proper VC
12:28 2014-09-29
the weights here that constitute that guy contribute
to the VC dimension
12:28 2014-09-29
VC dimension is more or less a number of weights, that's
the rule of thumb here.
12:29 2014-09-29
looking at the data without accounting for it is bad
12:29 2014-09-29
it's a nonlinear transformation with a view that
matches very specific dependence that I'm after.
that's the source of efficiency there.
12:30 2014-09-29
can I interpret what the hidden layers are doing?
12:31 2014-09-29
can you please tell what the hidden layers are doing?
12:31 2014-09-29
factor number 7 is important in your case.
12:33 2014-09-29
very common in machine learning, when the learning
algorithm tries to learn, it tries to use the right
hypothesis, it didn't tries to explain to you what the
right hypothesis is. that was the goal.
----------------------------------------------------------
12:37 2014-09-29
batch gradient descent => stochastic gradient descent(SGD)
12:38 2014-09-29
conjugate gradient
12:38 2014-09-29
convex optimization
12:38 2014-09-29
initially the neural networks are going to solve
the problems of the universe.
12:39 2014-09-29
because the simplicity of the network & the simplicity
of the algorithm, people use them in many applications.
it became the standard tool.
12:41 2014-09-29
they have some serious competitors, for example, SVM(Support Vector
Machine), lots of other models, but they still in use.
12:42 2014-09-29
not the top choice nowadays, but every now and then, someone
will publish something and he will use the neural network and
get good results.
12:44 2014-09-29
how to choose the number of layers?
OK, this is "model selection", the neural network
is really a class of hypothese set.
how many layers, how many units per layer.
12:47 2014-09-29
that's not an approximation question, we're
talking about learning question.
12:48 2014-09-29
the real questions is: how many weights can I afford?
because it's reflect directly at the VC dimension &
the number of examples we need.
12:48 2014-09-29
the question of organizing is less severe.
12:50 2014-09-29
given a particular architecture, it tries to kill
some weights in order to reduce the number of parameters
as a method for regularization.
12:51 2014-09-29
basically this is a model selection questions:
validation, regularization
12:52 2014-09-29
I want the analytically property of differentiation.
12:53 2014-09-29
but the whole idea is that you're going to arrive
at a minimum.
12:54 2014-09-29
you just start from different starting point.
different randomization for the presentation.
12:55 2014-09-29
we're dealing with sophisticated model:
12:57 2014-09-29
2 red flags: generalization & optimization
12:58 2014-09-29
when you have a powerful model, the question of generalization comes
in, means you can express a lot of things,
12:59 2014-09-29
the VC dimension summarizes all generalization considerations
13:00 2014-09-29
but at least we're under control, because we
have the numbers that describe it.
13:02 2014-09-29
I'm given the data set: (input, output)
and I have multilayer perceptron, each layer which
is computing a perceptron function of perceptron function of...
13:03 2014-09-29
the VC dimension is roughly the number of parameters,
that has stood the time of practice.
13:05 2014-09-29
the minimum of the error function will happens
at the particular combination of the weights.
13:14 2014-09-29
neural network is a model, genetic algorithm is
an optimization technique, there is no relationship
between them.
13:17 2014-09-29
the learning algorithm looks at the data,
but we already choose the hypothese set.
13:19 2014-09-29
I recommend to use a package for neural networks
13:20 2014-09-29
neural networks have been studies for a great
level of detail
13:21 2014-09-29
if you're very keen on the interpretation aspect.
13:21 2014-09-29
when you're combining perceptrons, you're going
to implement more interesting functions.
13:22 2014-09-29
the structure of multilayer is an interesting model
to study, from then on, it becomes a learning question
13:24 2014-09-29
we have a neural network, we no longer to look at the
target function, try to choose the neuron, we try to
put it as a model, we let the learning algorithm to choose
the weight.that is the backpropagation in this case.
13:25 2014-09-29
could you please describe early stopping?
it's basically a way to prevent overfitting,
// validation, regularization (model selection)
13:27 2014-09-29
what are the tools to deal with overfitting?
validation & regularization
then early stopping will be very easily explained.
13:28 2014-09-29
instead of choose points at random, you choose
random permutation, from 1 to n, and then go through
that in order, for the next epoch, you do another
permutation,...
13:31 2014-09-29
If you do this way, eventually every example will
contribute the same, but an epoch will be difficult
to define,
13:32 2014-09-29
Does have layers no loops limit the power of the
neural network?
13:34 2014-09-29
VC dimension roughly depends on the number of parameters(weights)