The complexity of solving MAB (multi-armed bandit) using Markov decision theory increases exponentially with the number of bandit processes.
Instead of solving the n-dimensional MDP with the state-space ∏ni=1χi , the optimal solution(Gittins Index) is obtained by solving n 1-dimensional optimization problems.
The index is given as,
νi(xi)=supτ>0E[∑τt=0βtri(Xit)|Xi0=xi]E[∑τt=0βt|Xi0=xi]
Off-Line Algorithm for computing Gittins Index
1. Largest-Remaining-Index Algorithm
Initialization: identify the state α1 with the highest Gittins index.
S(α1)=χ , ν(α1)=r(α1)=rα1
choose: α1=argmaxα∈χrα
corresponding Gittins index is: ν(α1)=rα1
Recursion step:
Define the m×m matrix by ∀a,b∈χ
Q(k)a,b={
Pa,b0if b∈C(αk)otherwise
and define the m×1 vectors:
d