n-armed bandit _ ucb1 algorithm

前言：家里发生了一些事情，所以又耽搁了一段时间，这周交的report都有点潦草，好在ucb1本身就不是一个很复杂的算法。
参考文献：《Bandit Algorithms for Website Optimization》
This week, I have studied one of the algorithms in the UCB falmily, which is called the UCB1 algorithm.

The UCB1 algorithm pays attention to not only what it knows, but also how much it knows.

The epsilon-Greedy algorithm I have studied two months ago has the following broad properties.

• Its default choice is to select the arm that currently has the highest estimated value.

• It sometimes decides to explore and chooses an option that isn’t the one that currently seems best. Besides, it explores by selecting from all of the arms completely at random. It make one of these random exploratory decisions with probabilty epsilon.

The UCB algorithm is special for two reasons:

• UCB doesn’t use randomness at all.

• UCB doesn’t have any free parameters.

UCB algorithms take the form of picking the arm which maximizes a surrogate function.

They pick:

$$i=argmax_i \mu_i+P_i$$

where

$\mu_i$: the average function which estimates the mean payoff the arm $i$