Mini-Batch Gradient Descent
1. What is Mini-Batch Gradient Descent?
Mini-Batch Gradient Descent is an algorithm between the Batch Gradient Descent and Stochastic Gradient Descent. Concretly, this use some(not one or all) examples(M) for each iteration.
2. Compute Effort
The compute time of this algorithm depends on the examples. It not stable, but the worst case is like Batch Gradient Descent: O( N2 )
The table below shows the different among these there Gradient Descent
Batch Gradient Descent | Mini-Batch Gradient Descent | Stochastic Gradient Descent |
---|---|---|
use 1 example in each iteration | use some examples | use all example in each iteration |
relative compute loose | somewhat in between | relative compute intensive |
3. Gradient Descent Formula
For all
θi
∂Jθ∂θi=1m∑i=1M[hθ(xi)−yi]⋅(xi)
E.g.,
two parameters θ0,θ1 –> hθ(x)=θ0+θ1x1
For i = 0 :
∂Jθ∂θ0=1m∑i=1M[hθ(xi)−yi]⋅(x0)
For i = 1:
∂Jθ∂θ1=1m∑i=1M[hθ(xi)−yi]⋅(x1)
Note that the datasets need to be shuffled before iteration.