《Grokking Deep Reinforcement Learning》笔记（Chapter 1-2）

Chapter 1

基本概念：

DRL中的智能体只是做出决策的部分，其他部分都归属于环境environment.

1. learn from sequential feedback会导致temporal credit assignment问题，智能体如何权衡Immediate and long-term goals（chapter 3 will cover this problem，解决办法是动态规划类算法）

2. learn from evaluative feedback会导致“exploration v.s. exploitation trade-off”(chapter 4): 奖赏函数不一定传达某个要实现任务的本质，correctness.

3. learn from sampled feedback会导致智能体不能泛化，因为状态空间和动作空间基本上无限大(chapter 4):

而DRL算法所包含的智能体希望解决上述三种情况：learn from feedback that is simultaneously one-shot (as opposed to sequential), evaluative, and exhaustive (as opposed to sampled).

DRL的优势和劣势（相比DL）：

DRL擅长concrete, well-specified tasks单一任务, 但不擅长泛化到不同的任务中，深度学习也可以激励DRL的发展。

DRL存在sample efficiency的问题，第二个问题是与奖赏函数的设计相关（包括奖赏函数是否能准确反映出任务，稀疏奖赏值问题）。与奖赏函数对应的一个研究方向是intrinsic motivation.

研究决策的一些学科：operation research, psychology, philosophy, neuroscience, control theory, economics.

Chapter 2

RL的核心三个问题：complex sequential decision-making under uncertainty，其中complex说明智囊团的状态和动作空间可能是巨大的；sequential表示智能体得到的奖赏函数值，即做出某个动作的结果通常情况下是延迟奖励的（complex means agents’ state and action spaces are huge; sequential means there are delayed consequences）。uncertainty指的是我们不知道环境会如何根据我们做的决策进行变化。 这三个词分别对应sampled, sequential, evaluative feedback.

RL agent内部的三个核心步骤：如下图，RL的核心三个步骤为：interaction, evaluation of behaviors, improvement of behavior.

环境的表示：一般用MDP进行建模。环境用一些变量来表示，这些变量的集合就构成了状态空间（环境本身的状态空间state space）。智能体能够观测到的信息组成了观测空间observation space. 智能体能够从动作空间中选择动作The set of all actions in all states is referred to as the action space. 环境还有transition function，接受到智能体的动作之后转移到下一个状态，并给出奖赏值作为该动作的响应，即reward function。Environment model包括The set of transition and reward function。

RL环境举例：

bandit walk environment:

• grid-world environment. The bandit walk (BW) is a walk with three states, but only one non-terminal state (environments that have a single non-terminal state are called ‘bandit’ environments. )
• Agent starts in the middle of the walk.
• A “walk” (or ‘corridor’) is a special case of grid-world environments with a single row.
• Action space: left (action 0), right (action 1)
• Transition: left action takes agent to the left, right action moves agent to the right.
• reward: +1 when landing on the rightmost cell

状态转移graph:

The bandit slippery walk:

区别：0.8的概率会按动作移动，0.2的概率将移动到反方向。因此是stochastic environment

环境内部的逻辑：

The set of the observation (or state), the action, the reward, and the new observation (or new state) is called an experience tuple.

Episodic tasks：tasks that have a natural ending, such as a game. (opposed to continuing tasks)

在RL中，智能体假设环境的底层模型是MDP.

如何在程序中实现MDP? 书中的实现方式是采用dictionaries in Python.

MDP的一些特点：

• states是fully observable的，即能够获取internal state of the environment at each time step。

• Markov Property: The probability of the next state, given the current state and action, is independent of the history of interactions （memoryless property）.

问题：To make the MDP completely Markovian, how deep do you have to go？不同的问题所需要的状态空间所包含的variables也不同，有的问题需要的状态变量较少，有的则需要很多。例如frozen lake中只需要每个智能体所在的Location id.

RL问题中状态的种类：The set of all states in the MDP is denoted $S^{+}$（所有环境的状态）, 其中的一部分子集是初始状态starting or initial states, denoted $S^i$。另一种状态叫做absorbing or terminal state. 大部分情况下是多个terminal states. 智能体从Terminal states出发，任何transition只能回到terminal states. 因此要对terminal state做一些设计： 习惯上让终止状态对应的动作能够返回到其自身状态上以便于算法收敛到同一个解（it’s a compatibility convention that allows for all algorithms to converge to the same solution to make all actions available in a terminal state transition from that terminal state to itself with probability 1 and reward 0.）

例如在frozen lake中，

Actions, 可以是离散的动作也可以是连续的（对应的是控制问题）

Transition function:

Reward function: 两种定义方式，一种是基于s,a，另一种是更加完整的版本，complete tuple (s,a,s’)

Horizon相关定义

time step指的是：epoch, cycle, iteration or interaction

episodic task & continuing tasks. 以上两种任务也可以从智能体的角度进行定义，被称为planning horizon.

a finite horizon is a planning horizon in which the agent knows the task will terminate in a finite number of time steps. 一种特殊情况是planning horizon = 1，该情况被称为greedy horizons. 所有的bandit problem都是greedy horizons.

infinite horizon指的是智能体没有预先定义好的time step limit. 但从任务的角度来说，任务可以是episodic的也可以是continuing的。一种实际做法是使用hard time step limit using the transition function. 在indefinite planning horizon中，一个eiposde是一个collection containing all interactions between an initial and a terminal state.

Discount factor的两个作用：第一个，可以用于调整智能体在一个episode内获取的return, 第二，减少不同episode之间的方差。

MDPs的一些扩展形式：

MDP v.s. POMDP