Paper Reading:
On the Gittins Index for Multiarmed bandits
– By Richard Weber University of Cambridge
本文是 Gittins index 最简单也最 elegant 的proof, 这篇博客我们来看看Weber是怎么proof Gittins index的.
Multiarmed Bandit (MAB) Problem
MAB 问题我们就不在赘述了, 实际上就是一个 n n n dimentional MDP, 即总共有 n n n 个 arms/bandits, 每次只能拉一个. 被拉的arm的states按照一个已知的Markov Chain (MC) 来演化, 没被拉的arms全都freeze. 每次拉都会得到一个immediate reward.
States of the bandits: x = { x 1 ( t ) , x 2 ( t ) , . . . , x n ( t ) } x=\left\{x_1(t), x_2(t), ..., x_n(t)\right\} x={
x1(t),x2(t),...,xn(t)}
Actions: a 1 a_1 a1, a 2 a_2 a2, …, a n a_n an ( a i a_i ai 对应"拉第 i i i个bandit")
目标是找到最优policy π \pi π 最大化 expected total-discounted reward:
V π ( x ) = E π [ ∑ t = 0 ∞ β t R j t ( x j t ( t ) ) ∣ x ( 0 ) = x ] V_\pi(x)=E_\pi\left[ \sum_{t=0}^{\infty} \beta^t R_{j_t}\left( x_{j_t} (t) \right) \left| \right.x(0)=x \right] Vπ(x)=Eπ[t=0∑∞βtRjt(xjt(t))∣x(0)=x]
π ⋆ = arg max π V π ( x ) \pi^\star=\argmax_{\pi}V_\pi(x) π⋆=πargmaxVπ(x)
简言之, V π ( x ) V_\pi(x) Vπ(x)是从任一状态 x x x 开始, 遵循策略 π \pi π走出的所有轨迹的平均return (这里定义为accumulated discounted future rewards), 最优policy π ⋆ \pi^\star π⋆是所有policy中最大化平均return 的那个.
Gittins Index
上述MAB可以用传统 discounted-cost MDP theory来解 (e.g., value iteration), 但是算法复杂度随着M的增加而指数增加, 也就是 curse of dimensionality. 另一方面, Gittins index是一种线性复杂度的解法: 每次选择bandits时, 可以对每个bandits计算一个Gittins index, 然后选index最大的那个user即可. More formally, 在 t t t 时刻选取bandit j j j 是最优的 iff
G j ( x j ( t ) ) = max 1 ≤ i ≤ n G i ( x i ( t ) ) G_j\left(x_j(t) \right)=\max_{1\leq i\leq n}G_i\left(x_i(t) \right) Gj(xj(t))=1≤i≤nmaxGi(xi(t))
G i G_i Gi