Multi-armed Bandits

The most important feather distinguishing reinforcement learning from other types of learning is that it uses training information that evaluates the actions taken rather than instructs by giving correct actions.

Evaluative Feedback and Instructive Feedback

In there pure forms, these two kinds of feedback are quite distinct: evaluative feedback depends entirely on the action taken, whereas instructive feedback is independent of the action taken.
Studying this case enables us to see most clearly how evaluative feedback differs from, and yet can be combined with, instructive feedback.

1. A k-armed Bandit Problem

Consider the following learning problem. You are faced repeatedly with a choice among $k$ different options, or actions. After each choice you receive a numerical reward chosen from a stationary probability distribution that depends on the action you selected. Your objective is to maximize the expected total reward over some time period, for example, over 1000 action selections, or time steps.

In our $k$-armed bandit problem, each of the $k$ actions has an expected or mean reward given that action is selected; let us call this the value of that action. We denote the action selected on time step $t$ as $A_{t}$, and the corresponding reward as $R_{t}$. The value then of an arbitrary action $a$, denoted $q_{*}(a)$. is the expected reward given that $a$ is selected:

$$q_{*}(a)\doteq \mathbb{E}[R_{t}\mid A_{t}=a]$$
If you knew the value of each action, then it would be trivial to solve the $k$-armed bandit problem: you would always select the action with highest value. We assume that you do not know the action values with certainty, although you may have estimates. We denote the estimated value of action $a$ at time step $t$ as $Q_{t}(a)$ to be close to $q_{*}(a)$.