Transition
Elements of the state transition ˜xt+1(xt) are dependent on each other when the ith element of the next period state ˜xt+1,i(xt) depends on the ith element of the current state and also on the other elements of the current state. An interesting analysis can be applied to the case where the state transition ˜xt+1(xt) is stochastically increasing in xt. Note that the statext has multiple dimensions in this case. Hence, we need a more general definition of the stochastically increasing property.
Definition 2.3. A set of random vectors {x˜(θ), θ ∈Rd} of dimension m is stochas-
tically increasing in θ ∈ Rd if E[u(˜x(θ))] is increasing for all increasing functions
u:Rm →
R.
Given this definition, we have the following result.
Proposition 2.5. WhenMt(xt)is increasing[resp., decreasing] in eachxt,i andx˜t+1(xt)
is stochastically increasing in xt ∈ X for every t, then the following statements are
1. Bt(xt) is increasing [resp., decreasing] in each xt,i.
2. A state-dependent threshold policy that stops the process when xt,i ≤ xt,i(xt,−i)
[resp., xt,i ≥xt,i(xt,−i)] is optimal for each i.
At first sight, the conditions that the one-step benefit function is increasing in xt,i for all elements i and that the state transition is stochastic increasing appear to be somewhat restrictive. Consider, for example, a two-dimensional state space in which a higher value of xt,2 leads to a lower value of xt,1 in the next period. In this case,
one can redefine the state variable as yt,2 = −xt,1 and yt,1 = xt,1, which makes the
state transition ˜yt,1(yt) stochastically increasing in both yt,1 and yt,2. Similarly, if
an initial formulation of Mt(xt) is increasing in xt,1 and decreasing in xt,2, a similar
transformation makes the one-step benefit function Mt(yt) an increasing function. Therefore, the increasing benefit-function condition is not too restrictive. In general, one element of the current state may impact some elements of the next period state but not all of them. Suppose ˜xt+1,i(xt) is independent ofxt,j for somej 6=i. Then, by definition ˜xt+1,i(xt) is stochastically increasing in xt,j. Therefore, the stochastically increasing property of multi-dimensional state transition can also be satisfied easily.
Cases in which the state transition depends only on parts of the state space can be analyzed following the analyses given in this and the previous subsections. For example, consider the case in which the state transition of a three-dimensional state space ˜xt+1(xt) can be separated into ˜xt,(1,2)(xt,(1,2)) and ˜xt,3(xt,3), where the random
transition ˜xt,(1,2)(xt,(1,2)) is stochastically increasing in xt,(1,2). If the one-step benefit
functionMt(xt) is increasing in both xt,1 and xt,2, we can apply a slight modification
of Proposition 2.5 to xt,(1,2) for each fixed xt,3. We omit the proposition and the
analysis to avoid repetition.
2.4.4
Monotonicity Results and Bounds for Optimal Thresh-
olds
We discuss two types of monotonicity results for the optimal thresholds. The first one is the parametric monotonicity of the state-dependent optimal thresholds. The
second one is the time-monotonicity of optimal thresholds. We also provide bounds for the optimal thresholds. The result of this subsection is useful for developing efficient numerical algorithms. They also help characterize how policy parameters respond to the changes in the environment.
Proposition 2.6. The following statements are true for every t:
1. If Bt(xt) is increasing [resp., decreasing] in both xt,i and xt,j for i 6= j, then
xt,i(xt,−i)[resp., xt,i(xt,−i)] is decreasing in xt,j and xt,j(xt,−j)[resp., xt,j(xt,−j)]
is also decreasing in xt,i.
2. If Bt(xt) is increasing in xt,i and decreasing in xt,j for i 6=j, then xt,i(xt,−i) is
increasing in xt,j and xt,j(xt,−j) is also increasing in xt,i.
3. If Bt(xt) is increasing [resp., decreasing] in xt,i and convex in xt,j for i 6= j,
thenxt,j(xt,−j)is increasing [resp., decreasing] inxt,i andxt,j(xt,−j)is decreasing
[resp., increasing] in xt,i.
Next we consider time-monotonicity of optimal thresholds in stationary optimal stopping problems. An optimal stopping problem is stationary if the Markov process
xtis time-homogeneous and the reward functionsC(xt) and S(xt) are time-invariant. It is a well-known result that the value function Vt(x) is decreasing in t for every
x in such problems. See, for example, §4.4 of Bertsekas (2005). As t increases, the decision maker has less opportunity to delay the stopping decision, hence the value function Vt(x) decreases in t. This property directly implies that Bt(x) is decreasing int, which in turn implies the following proposition.
Proposition 2.7. For stationary optimal stopping problems, the following statements are true:
1. xt is increasing in t and xt is decreasing in t.
2. xt,i(xt,−i) is increasing in t and xt,i(xt,−i) is decreasing in t for every xt,−i. Finally, we provide bounds for the optimal thresholds. By definition, Bt(xt) ≥
period t with xt, i.e., Mt(xt) >0, then the optimal policy also continues the process at period t, i.e, Bt(xt)>0. This property implies the following proposition.
Proposition 2.8. The optimal thresholds have the following bounds: 1. xt≤sup{x∈X :Mt(x)≤0} and xt≥inf{x∈X :Mt(x)≤0}.
2. xt,i(xt,−i) ≤ sup{xt,i : Mt(xt,i, xt,−i) ≤ 0, xt ∈ X} and xt,i(xt,−i) ≥ inf{xt,i :
Mt(xt,i, xt,−i)≤0, xt ∈X}.
Note that determining the x that satisfies sup{x ∈ X : Mt(x) ≤ 0} is simple and does not involve recursive computation. Often it can be derived in a closed form. Hence, these bounds together with the monotonicity results considerably help reduce the computational time required to determine the optimal thresholds and resulting expected profit by reducing the search region. They also provide qualitative understanding of a decision process modeled as an optimal stopping problem.