Basics of Neural Network Programming - Derivatives with a Computation Graph

In last class, we worked through an example of using computation graph to compute the function J. In this class, let's show how you can use it to figure out derivative calculation for that function J.

Following is the computation graph:

Figure-1

• We have:

If nudged to 11.001 from 11, then increased to 33.003 from 33.

• Then, what is ?

Suppose , then , then

So, the change of variable propagate to the right of the computation graph. The increase of J is 3x of increase of a. So we have:

Actually, in calculus we have following chain rule:

So, this little illustration shows how by having computed , it can then help you to compute . This is another step of backward propagation.

• Then what is ?

If bumps up from 6 to 6.001, then v goes up to 11.001 from 11. So goes to 33.003 from 33.

• What is ?

If b bumps up from 3 to 3.001, then goes to 6.002 and goes to 11.002. And J goes to 33.006.

• Last, what is ?

•  One notational convention when writing codes to implement backward propagation: we'll use to represent the derivative of your final output (such as J in this case) respect to the various intermediate quantities (a,b,c,u,v etc.).

There usually be some final output variable that we really care about or want to optimize. A lot of computations will be trying to compute the derivatives of that final output variable (J in this case) with various intermediate variables such as a,b,c,u,v, noted as .  Then, what do you call this complex variable name in the codes? Because we always take derivatives respect to this final output variable, we're going to just use the  in order to represent that quantity.

The take away:

The most efficient way to compute derivatives is through a right-to-left backward propagation following the direction of the red arrows.

<end>