[RL 11] Asynchronous Methods for Deep Reinforcement Learning (A3C) (ICML, 2016)

Asynchronous Methods for Deep Reinforcement Learning (A3C) (ICML, 2016)

1. Introduction

1. On-line DRL
   1. problems
      1. related samples lead to unstable
   2. solution
      1. ER
         1. only for off-policy alg
         2. resource cost
      2. Asynchronous Methods

4. Asynchronous RL Framework

1. Design goal
   • train deep neural network policies reliably and without large resource requirements.
2. Main ideas
   1. asynchronous actor-learners
   2. parallel running(to explorate different parts of envs)
3. Techs
   1. accumulate gradients
   2. n-step bootstrap
   3. parameters sharing
   4. exploration
      1. different $ϵ$
      2. entropy for A3C

5. Experiments

1. Atari
   1. n-step is faster
   2. A3C outperformance others
2. Scalability (可拓展性)
   • improvement over actors
   • n-step gets more speed up ratio
     • perhaps due to the reduction of bias
3. Robustness
   • there is usually a range of good learning rate