TIEMPO

Dynamic programming was developed by Bellman (1957) as an optimisation method for sequential decision problems and which has very general use for optimisation in problems beyond BSDPs and bandits. The term “programming” as used here refers to planning rather than computer programming. It has been used extensively for deterministic problems but will be presented here in its stochastic form. It works by breaking the problem down into a series of smaller problems (sub-problems). Solving these gives the solution to the original problem. This approach reduces the required computation in two ways:

1. The solutions to smaller sub-problems can be re-used in solving larger sub- problems which reduces repeated calculation of shared sections of paths. 2. A policy can be found to be suboptimal and so eliminated without evaluating

CHAPTER 2. BAYES SEQUENTIAL DECISION PROBLEMS 22

the whole length of any paths that result from that policy (see the Principle of Optimality below).

Dynamic programming is based on the calculation of the value V(s, t) of any state

s in the state space S at time t. Here V(s, t) will be taken to be the maximum total expected reward that can be gained when starting from state s at time t (i.e. when an optimal policy is followed). The value is an important concept because of the Principle of Optimality (which holds for Markov decision problems), as stated in Bellman (1957):

“An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.”

So to find an optimal policy from the starting state, only policies that are optimal from each subsequent state need be considered. These are exactly the policies that are needed to calculate the state values. Therefore, to find an optimal policy it is sufficient to calculate the values of all states at all times.

Let r(s, a, t) be a reward function and π(s0 | s, a) be the transition probability from state s to any state s0 after taking actiona, then the value of a state is given by the recursive value function (also known as the Bellman equation),

V(s, t) = max a∈A " r(s, a, t) +γX s0_∈S π(s0 |s, a)V(s0, t+ 1) #

where γ is a constant discount factor an A is the available action space. For our purposes here it will be assumed that the process has time T at which the process terminates. The value of states at this time can be found directly,

V(s, T) = max

a∈A[r(s, a, T)].

Dynamic programming works backwards sequentially from these end states to give the values of all states. This is calledbackward induction. The valueV(s0,0) of the initial

CHAPTER 2. BAYES SEQUENTIAL DECISION PROBLEMS 23

state s0 at time 0 is the optimal reward and the recursion sequence that calculates this value gives the optimal policy. A infinite horizon process may not terminate but, for discounted rewards, a finite horizon approximation can be used which tends to the solution as the horizon tends to infinity since received rewards become small as time increases (Blackwell 1962). Regret-based problems use an infinite horizon with undiscounted rewards so dynamic programming is inappropriate in those cases. Dynamic programming as a solution method for BDSPs has been studied extensively and theoretical results have shown that this method can be used to find the optimal solution (e.g. Rieder 1975). Dynamic programming is a much more efficient method than complete enumeration of paths and has been applied extensively but it still quickly runs into computational limits as the number of calculations required grow exponentially with the number of possible actions and outcomes. A wide range of methods of computation besides backward induction have been developed to calculate the value function of states and the term dynamic programming is often used to refer to this collection of algorithms rather than a single method. For a full overview of DP techniques see Bertsekas (2012).

An even more general family of algorithmic techniques for estimating value functions comes under the term approximate dynamic programming. This is itself an example of a BSDP where paths are explored with resources being progressively concentrated on paths more likely to be part of an optimal policy. The problem has been studied in different fields and the main texts on the subject are Powell (2007), Sutton and Barto (1998) and Bertsekas and Tsitsiklis (1996).

Dynamic programming can be applied to the Bernoulli MAB as given in Section 1.3. For each action there are two possible outcomes where Pr(yt= 1 |at=a, αa,t, βa,t) =

µa,t. The value function for state s= (α1,t, β1,t, . . . , αk,t, βk,t) at time t is,

V(s, t) = max a∈A n µa,t h 1 +γVt+1(s+, t) i + (1−µa,t)γVt+1(s−, t) o .

CHAPTER 2. BAYES SEQUENTIAL DECISION PROBLEMS 24

wheres+ = (αt,1, βt,1, . . . , αt,a+1, βt,a, . . . , αt,k, βt,k) ands−= (αt,1, βt,1, . . . , αt,a, βt,a+

1, . . . , αt,k, βt,k) are the states reached after a success and failure respectively. Using

the finite horizon approximation described earlier we terminate the process at some time T where states have value V(s, T) = maxa∈Aµa,T.

Although there are only two outcomes at each stage the computation required still increases exponentially with the number of available arms and evaluating the value function is impractical for as few as four arms (Gittins et al. 2011). The next section will describe an alternative approach, based on decomposing the dynamic program- ming problem into smaller subproblems, which gives an optimal policy for the MAB.