https://discuss.pytorch.org/t/model-zero-grad-or-optimizer-zero-grad/28426
model.zero_grad() and optimizer.zero_grad() are the same IF all your model parameters are in that optimizer. I found it is safer to call model.zero_grad() to make sure all grads are zero, e.g. if you have two or more optimizers for one model.