Key Concepts in RL

Key Concepts in RL

for review.
不定期更新。

In a nutshell, RL is the study of agents and how they learn by trial and error. It formalizes the idea that rewarding or punishing an agent for its behavior makes it more likely to repeat or forego that behavior in the future.

​ Agent-environment interaction loop

The environment is the world that the agent lives in and interacts with. At every step of interaction, the agent sees a (possibly partial) observation of the state of the world, and then decides on an action to take. The environment changes when the agent acts on it, but may also change on its own.

The agent also perceives a reward signal from the environment, a number that tells it how good or bad the current world state is. The goal of the agent is to maximize its cumulative reward, called return. Reinforcement learning methods are ways that the agent can learn behaviors to achieve its goal.

More Terminology

DQN (Deep Q Network) 使用神经网络生产Q值

马尔可夫决策过程 MDP

在执行动作前作出的决策为规划 (planning)

但是在强化学习中，agent却不是那么容易知晓MDP中所有的元素的，比如，agent也许不会知道环境将会如何改变当它执行了一个动作后（状态转移概率函数$T$），也不会知道它执行这个动作获得即时的奖励将会是多少（奖励函数$R$），agent能做的就是：根据自己已有的策略$\pi$选择关于当前状态$s$下自己认为好的动作$a$，执行此动作给环境，观察环境给出的反馈$r$和下一个状态$s\prime$，并根据这个反馈$r$调整更新自己的策略$\pi$，这样反复迭代，直到找到一种 最优的策略${\pi }^{\prime }$能够最大限度获得正反馈

那么，当agent不知道转移概率函数$T$和奖励函数$R$，它是如何找到一个好的策略的呢，当然会有很多方法：

Model-based RL

一种方法就是Model-based方法，让agent学习一种模型，这种模型能够从它的观察角度描述环境是如何工作的，然后利用这个模型做出动作规划，具体来说，当agent处于$s_1$状态，执行了$a_1$动作，然后观察到了环境从$s_1$转化到了$s_2$以及收到的奖励$r$, 那么这些信息能够用来提高它对$T(s_2|s_1,a_1)$和$R(s_1,a_1)$的估计的准确性，当agent学习的模型能够非常贴近于环境时，它就可以直接通过一些规划算法来找到最优策略，具体来说：当agent已知任何状态下执行任何动作获得的回报，即$R(s_t,a_t)$已知，而且下一个状态也能通过$T(s_{t+1}|s_t,a_t)$被计算，那么这个问题很容易就通过动态规划算法求解，尤其是当$＝T(s_{t+1}|s_t,a_t)＝1$时，直接利用贪心算法，每次执行只需选择当前状态$s_t$下回报函数取最大值的动作$(ma{x}_{a}R(s,a|s=s_t))$即可，这种采取对环境进行建模的强化学习方法就是Model-based方法。

Model-free RL
但是，事实证明，我们有时候并不需要对环境进行建模也能找到最优的策略，一种经典的例子就是Q-learning，Q-learning直接对未来的回报$Q(s,a)$进行估计，$Q(s_k,a_k)$表示对$s_k$状态下执行动作$a_t$后获得的未来收益总和$E({\sum }_{t=k}^n{\gamma }^k{R}_k)$的估计，若对这个Q值估计的越准确，那么我们就越能确定如何选择当前$s_t$状态下的动作：选择让$Q(s_t,a_t)$最大的$a_t$即可，而Q值的更新目标由Bellman方程定义，更新的方式可以有TD（Temporal Difference）等，这种是基于值迭代的方法，类似的还有基于策略迭代的方法以及结合值迭代和策略迭代的actor-critic方法，基础的策略迭代方法一般回合制更新（Monte Carlo Update），这些方法由于没有去对环境进行建模，因此他们都是Model-free的方法。

所以，如果你想查看这个强化学习算法是model-based还是model-free的，你就问你自己这个问题：在agent执行它的动作之前，它是否能对下一步的状态和回报做出预测，如果可以，那么就是model-based方法，如果不能，即为model-free方法。

https://www.quora.com/What-is-the-difference-between-model-based-and-model-free-reinforcement-learning

States and Observation

A state $s$ is a complete description of the state of the world. There is no information about the world which is hidden from the state. An observation $o$ is a partial description of a state, which may omit information.

When the agent is able to observe the complete state of the environment, we say that the environment is fully observed. When the agent can only see a partial observation, we say that the environment is partially observed.

Action Spaces

action space: The set of all valid actions in a given environment.

discrete action space: only a finite number of moves are available to the agent.

continuous action space: like the agent controls a robot in a physical world, actions are real-valued vectors.

Policies

A policy is a rule by an agent to decide what actions to take. The policy is trying to maximize reward.

Deterministic Policies: $a_t=\mu (s_t)$

Stochastic Policies: $a_t\pi (\cdot |s_t)$

two kinds of stochastic policies:

1. categorical policies: used in discrete action spaces.
2. diagonal Gaussian policies: used in continuous action spaces.

two key computaions:

1. sampling actions from the policy
2. computing log likelihoods of particular actions, $log{\pi }_{\theta }(a|s)$