深度学习Course2第二周Hyperparameter tuning, Batch Normalization, Programming Frameworks习题整理

Hyperparameter tuning, Batch Normalization, Programming Frameworks

1. Which of the following are true about hyperparameter search?

• Choosing values in a grid for the hyperparameters is better when the number of hyperparameters to tune is high since it provides a more ordered way to search.
• When sampling from a grid, the number of values for each hyperparameter is larger than when using random values.
• When using random values for the hyperparameters they must be always uniformly distributed.
• Choosing random values for the hyperparameters is convenient since we might not know in advance which hyperparameters are more important for the problem at hand.

2. If it is only possible to tune two parameters from the following due to limited computational resources. Which two would you choose?

• β1 , β2 in Adam.
• α
• ϵ in Adam.
• The β parameter of the momentum in gradient descent.

3. Using the “Panda” strategy, it is possible to create several models. True/False?

• False
• True

4. If you think $\beta$ (hyperparameter for momentum) is between 0.9 and 0.99, which of the following is the recommended way to sample a value for beta?

• r = np.random.rand()
  beta = r*0.9 + 0.09
• r = np.random.rand()
  beta = 1-10**(- r - 1)
• r = np.random.rand()
  beta = 1-10**(- r + 1)
• r = np.random.rand()
  beta = r*0.09 + 0.9

5. Finding good hyperparameter values is very time-consuming. So typically you should do it once at the start of the project, and try to find very good hyperparameters so that you don’t ever have to tune them again. True or false?

• False
• True

6. When using batch normalization it is OK to drop the parameter $W^{[l]}$ from the forward propagation since it will be subtracted out when we compute ${\tilde{z}}_{\text{normalize}}^{[l]}$=${\beta }^{[l]}{\hat{z}}^{[l]}$+ ${\gamma }^{[l]}$. True/False?

• True
• False

7. In the normalization formula $z_{norm}^{(i)}=\frac{z^{(i)}-\mu }{\sqrt{{\sigma }^2+\epsilon }}$ , why do we use epsilon?

• To speed up convergence
• To have a more accurate normalization
• In case $\mu$ is too small
• To avoid division by zero

8. **Which of the following statements about $\gamma$ and $\beta$ in Batch Norm are true? **

• They set the mean and variance of the linear variable ${\hat{z}}^{[l]}$ of a given layer.
• There is one global value of $\gamma \in R$ and one global value of $\beta \in R$ for each layer, and these apply to all the hidden units in that layer.
• The optimal values are $\gamma =\sqrt{\sigma 2+\epsilon }$, and $\beta =\mu$.
• They can be learned using Adam, Gradient descent with momentum, or RMSprop, not just with gradient descent.
• $\beta$ and $\gamma$ are hyperparameters of the algorithm, which we tune via random sampling.

9. A neural network is trained with Batch Norm. At test time, to evaluate the neural network on a new example you should perform the normalization using $\mu$ and ${\sigma }^2$ estimated using an exponentially weighted average across mini-batches seen during training. True/false?

• True
• False

10. Which of the following are some recommended criteria to choose a deep learning framework?

• It must be implemented in C to be faster.
• It must run exclusively on cloud services, to ensure its robustness.
• It must use Python as the primary language.
• Running speed.