Typical Policy Representation in Policy Search Methods

Thanks Jan Peters et al for their great work of A Survey on Policy Search for Robotics.

Policy representation may be categorized into time-independent representation $\pi(x)$ and time-dependent representation $\pi(x,t)$. Since time-dependent representations can use different policies for different time steps, they allow for a simpler structure of the individual policies can be used.

In the following content, we will describe all these representations in their deterministic formulation $\pi_{\theta}(x,t)$. In stochastic formulations, typically a zero-mean Gaussian noise vector $\epsilon_{t}$ is added to $\pi_{\theta}(x,t)$. In this case, the parameter vector $\theta$ typically includes the covariance matrix used for generating the noise $\epsilon_{t}$.

Linear Polices

Linear policy $\pi$:

$$\pi_{\theta}(x)=\theta^{T}\phi(x)$$ where $\phi$ is a basis function vector. Linear polices are always limited to problems where appropriate basis functions are known.

Radial Basis Functions Networks

An RBF policy $\pi_{\theta}$ is given as

$$\pi_{\theta}(x)=w^{T}\phi(x), \quad \phi_{i}(x)=\exp\left(-\frac{1}{2}(x-\mu_{i})^{T}D_{i}(x-\mu_{i})\right)$$ where $D_{i}=\mathrm{diag}(d_{i})$ . The parameters $\beta=\{\mu_{i},d_{i}\}_{i=1,\dots,n}$ of the basis functions are now free parameters to be learned. Hence $\theta=\{w,\beta\}$ .

Dynamic Movement Primitives

DMPs are most widely used time-dependent policy representation in robotics. The key principle is to use a linear spring-damper system which is modulated by a nonlinear forcing function :

$$\ddot{y}_{t}=\tau^{2}\alpha_{y}(\beta_{y}(g-y_{t})-\dot{y}_{t})+\tau^{2}f_{t}$$ where the variable $y_{t}$ specifies the desired joint position. $\tau$ is the time-scaling coefficient, the coefficients $\alpha_{y}$ and $\beta_{y}$ define the spring and damping constants and the goal parameter $g$ is the unique point attractor of the system. The forcing function $f_{t}$ changes the goal attractor $g$ .

One key innovation of the DMP approach is the use of a phase variable $z_{t}$ to scale the execution speed of the movement.

$$\dot{z}=-\tau\alpha_{z}z,\quad z(0)=1$$
For each degree of freedom, an individual spring-damper system and forcing function is used.
$$f(z)=\frac{\sum_{i=1}^{K}\phi_{i}(z)w_{i}}{\sum_{i=1}^{K}\phi_{i}(z)}z,\quad \phi_{i}(z)=\exp\left(-\frac{1}{2\sigma^{2}}(z-c_{i})^{2}\right)$$ The parameters $w_{i}$ are denoted as shape-parameters of the DMP as they modulate the acceleration profile and, hence, indirectly specify the shape of the movement. The nonlinear dynamic system is globally stable. We can think of the fact that the goal paramter $g$ specifies the final position while the shape parameters $w_{i}$ specify how to reach the final position.

A policy $\pi_{\theta}(x_{t},t)$ that is specified by a DMP, directly controls the acceleration of the joint, is given by:

$$\pi_{\theta}(x_{t},t)=\tau^{2}\alpha_{y}(\beta_{y}(g-y_{t})-\dot{y})+\tau^{2}f(z_{t})$$ Note that the DMP policy is linear in the shape parameters $w$ and the goal attractor $g$, but nonlinear in the time-scaling constant $\tau$. Then $\theta=\{w,g,\tau\}$.

Miscellaneous Representations

There exist other representations such as central pattern generators for robot walking and feed-forward neural networks used in simulation.