Transformer in Offline：Waypoint Transformer: Reinforcement Learning via Supervised Learning with Int

GCRL Workshop/ NIPS 2023
paper

Intro

Goal-conditioned RL 结合 Transformer 应用在Offline RL的设定下。

Method

tranformer网络参数通过最小化MSE损失函数优化:
$$\arg \underset{\varphi }{\min }\sum _{\tau \in \mathcal{D}}{L}_{\varphi }(W_{\varphi }(s_t,\omega ),s_{t+K})$$
而对于$W_{\varphi }$通过最小化与平均累计reward的MSE优化：
$$\arg \underset{\varphi }{\min }\sum _{\tau \in \mathcal{D}}({\left[\frac{1}{T-t}\sum _{t^{\prime }=t}^T{\gamma }^t{r}_t\sum _{t^{\prime }=t}^T{\gamma }^t{r}_t\right]}^{\top }-W_{\varphi }(s_t,\omega ){)}^2.$$

Results