Stochastic Optimal Control (SOC)

Definition

The SOC problem involves how one computes an optimal policy for a system that is driven by uncertain disturbances, to maximize a certain performance index.

Note

In a continuous-time/continuous-state problem, a nonlinear partial differential equation called the Hamilton-Jacobi-Bellman (HJB) equation needs to be solved, which is computationally intractable in most robotic control applications.

If the continuous SOC problem is discretized, it can be represented as a Markov Decision Process (MDP) and solved using well-developed iterative methods, such as policy iteration and value iteration.

linearly-solvable optimal control (LSOC)

In LSOC, the HJB function can be naturally linearized, and efficient solution methods exist.

In the LSOC, the exponentiated value function, termed the disirability function, is obtained as the principal eigenfunction of the linearized differential operator.

There is a discrete-time equivalent of the LSOC, called the linearly-solvable MDP (LMDP), but the solution methodologies for this class of problem still exhibit lmitations when handling high-dimensional and long-horizon problems.

Solution

multiscale or hierarchical

• [2017 ICRA] Multiscale Abstraction, Planning and Control using Diffusion Wavelets
  for Stochastic Optimal Control Problems