batch_size笔记

Batch size is a term used in machine learning and refers to the number of training examples utilised in one iteration. The batch size can be one of three options:

1. batch mode: where the batch size is equal to the total dataset thus making the iteration and epoch values equivalent
2. mini-batch mode: where the batch size is greater than one but less than the total dataset size. Usually, a number that can be divided into the total dataset size.
3. stochastic mode: where the batch size is equal to one. Therefore the gradient and the neural network parameters are updated after each sample.

Terminology explained:

• one epoch = one forward pass and one backward pass of all the training examples
• batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you'll need.
• number of iterations = number of passes, each pass using [batch size] number of examples. To be clear, one pass = one forward pass + one backward pass (we do not count the forward pass and backward pass as two different passes).

Example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.

Advantages of using a batch size < number of all samples:

• It requires less memory. Since you train the network using fewer samples, the overall training procedure requires less memory. That's especially important if you are not able to fit the whole dataset in your machine's memory.

• Typically networks train faster with mini-batches. That's because we update the weights after each propagation. In our example we've propagated 11 batches (10 of them had 100 samples and 1 had 50 samples) and after each of them we've updated our network's parameters. If we used all samples during propagation we would make only 1 update for the network's parameter.

Disadvantages of using a batch size < number of all samples:

• The smaller the batch the less accurate the estimate of the gradient will be. In the figure below, you can see that the direction of the mini-batch gradient (green color) fluctuates much more in comparison to the direction of the full batch gradient (blue color).

 Reference:

1. Batch size (machine learning) | Radiology Reference Article | Radiopaedia.orgBatch size is a term used in machine learning and refers to the number of training examples utilized in one iteration. The batch size can be one of three options:batch mode: where the batch size is equal to the total dataset thus making the ite...https://radiopaedia.org/articles/batch-size-machine-learning#:~:text=Batch%20size%20is%20a%20term,iteration%20and%20epoch%20values%20equivalent
2. python - What is batch size in neural network? - Cross Validatedhttps://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network