Adversarial inverse reinforcement learning(AIRL)

Overview

For imitation learning, it is easy for us to have a fit on the expert behavior, However when we encounter a new situation, we can’t solve the problem. So we imagine that there is an expert strategy in the real world, and this expert strategy will give a reward function to guide our agents to learn. AIRL is based on the past GAIL. In GAIL, we directly use GAN to fit the behavior of the agent to the expert behavior, and do not directly give the reward function. This behavior will lead to our learning is not clear and not good. Migration, so we will record the function of the discriminator in GAN as D, and use D to represent our reward function. Below I will write in detail, how this is derived.

the goal of the IRL

Firstly, we define a distribution $p_{\theta }(\tau )$. in another word, it is the expectation for the accumulated over the expert trajectory. so it can be wrote in this form. And D is the demonstrations of the expert.

$p_{\theta }(\tau )\propto p(s_0){\prod }_{t=0}^{T-1}p(s_{t+1}|s_t,a_t)e^{r_{\theta }(s_t,a_t)}$

the goal of IRL can been seen as training a generative model over trajectories as:

max ⁡ θ { J ( θ ) } = max ⁡ θ E τ ∼ D [ log ⁡ p θ ( τ ) ] \max \limits_{\theta}\{J(\theta)\}=\max \limits_{\theta}E_{\tau\sim D}[\log p_{\theta}(\tau)] θmax​{ J(θ)}=θmax​Eτ∼D​[logpθ​(τ)]

Then we calculate the gradient with respect to $\theta$(where $p_{\theta ,t}={\int }_{s_{t^{\prime }}\ne t,a_{t^{\prime }}\ne t}{p}_{\theta (\tau )}$ denote the state-action marginal at time t):
$\frac{\partial }{\partial \theta }J(\theta )={\sum }_{t=0}^T{E}_{\mathcal{D}}\left[\frac{\partial }{\partial \theta }{r}_{\theta }\left(s_t,a_t\right)\right]-E_{p_{\theta ,t}}[\partial \theta$