The term Monte Carlo Tree Search [36, 42, 82] describes a group of tree search algorithms
first implemented in 2006. The basic principles involve building a game tree step-by-step,
gradually adding tree nodes, and running playouts from leaf states at each iteration to de-
termine the direction of further growth. (see figure 2.3) From these Playouts (also called
simulations), a reward signal is received from the terminal game state, and the information
is propagated upwards back through each parent node, modifying that parent node’s value
as it does so. The iterative growth of the tree is non-symmetrical, and controlled by a tree
policy (a bandit algorithm) which attempts to balance exploitation against exploration by
selecting potentially high reward nodes.
An extensive survey of Monte Carlo Tree Search and associated information was per-
formed in 2012 by Browne et al. [29].
The basic MCTS algorithm is made up of 4 steps, and works as follows (see figure 2.3):
• Selection: The algorithm moves down through the tree until it reaches a node which
has unexpanded children or a terminal node.
• Expansion: If the selected node has unexpanded child nodes, then one (or more) of
Figure 2.3: Overview of the typical MCTS process [29]
• Simulation: A simulation is run from each of the new child nodes, normally to a
terminal state.
• Back-propagation: The simulation result is backed up through the parent nodes of the
selected node, updating statistics until it reaches the root node.
During this process, the algorithm makes use of two policies; a tree policy and a default
policy. These steps are shown in algorithm 1, with s0representing the initial search state, v0
the root node, ti the current iteration, tmax the iteration budget, v1 the selected child, r1the
rewards from the simulation conducted by the default policy, v1.s the state that is represented
by the node v1, and a is an action that leads to a specific node. The tree policy handles the
selection of nodes during the selection of the algorithm, and also expansion if not every child
of the selected node is being expanded. The default policy controls the algorithm’s action
selection during simulation. MCTS can be customised by specifying different algorithms
for the tree policy and default policy. In most vanilla implementations of MCTS, the tree
policy used is UCB1 (see subsection 2.4.3), and the default policy used is random node
selection. This configuration of MCTS is most often called UCT.
Once the budget for tree simulation has been fulfilled, the best child node of the root is
child with the most visits, but other options exist [29], and more are detailed in our work
here (see chapter 5.)
2.4.2.1 MCTS Strengths
There are three main strengths of MCTS:
• Aheuristic: MCTS techniques do not require domain knowledge in order to function.
• Any time: All values are always up to date following every iteration of the algorithm,
meaning that at any time in the process, the best move determined so far can be ob-
tained.
• Asymmetric: Builds an asymmetric tree, so avoids building areas of the tree that are
unpromising.
Aheuristic As vanilla MCTS uses no domain knowledge, it can in theory be introduced
in an unmodified state to a new game and make somewhat sensible moves. It is important
to note however that while heuristic knowledge is not required, it can greatly improve the
strength of a MCTS player. (see subsection 2.4.2.2). A UCT player is a very strong opponent
without heuristic knowledge in simple deterministic games such as Connect 4, Tic Tac Toe
(Noughts and Crosses), and Checkers (Draughts).
Any-Time MCTS can produce a decision at any time during its operation, and the deci-
sion will be representatively strong of the budget consumed by that time. Each MCTS run
produces a new “best” action, the MCTS process can be stopped at any point and the opti-
mal decision (for that budget) can be retrieved. There is also an associated advantage that all
the MCTS process and also access to data on areas of the decision tree that are not selected
for moving forward.
Asymmetric Tree growth in MCTS is asymmetric, as nodes are selected based on how
promising they appear based on collected statistics from the MCTS process. This results in
a smaller amount of budget being spent on areas of the decision space that are unpromising.
2.4.2.2 MCTS Weaknesses
There are four main weaknesses of MCTS:
• Agent Confusion: MCTS agents can become confused in complex “trap” states.
• Heuristic Agent Strength: MCTS may not behave as well as a heuristic agent specifi-
cally designed for the game in question.
• Lazy Play: MCTS can make poor moves if it determines that the current move is not
“required” to be strong.
• Playout Bias: Bias in simulation from the forward model can greatly reduce MCTS
performance.
Agent Confusion Browne et al. [28] demonstrated that MCTS agents can become con-
fused and lead into trap states in certain situations. This is largely due to the simulations
during the MCTS rollout being performed randomly, as it means that no consideration is
given to the opponent’s ability to move play away from certain states, and can direct tree
growth towards lines of play that would never occur against an actual opponent. This weak-
ditional processing must occur during rollout (although heuristic knowledge can also reduce
the number of rollouts required for a good solution.)
A Trap State is a game state where most moves will lead onwards towards a win, however
opponent skill will drive the gameplay away from those wins and towards a nearly certain
loss. As such, the state may appear very favourable to initial MCTS analysis, and might
be chosen in error. Ramanujan et al. [108] suggest that MCTS/UCT is weaker than other
techniques for such games (particularly Chess).
Heuristic Agent Strength If a decision space has a very strong heuristic agent, it is likely
to be a stronger player than aheuristic MCTS [29]. This is due to the fact that heuristic
agents can usually quickly prune a decision tree without performing any type of search.
However heuristic agents are also very difficult to build for most complex games, and there
are many domains in which aheuristic MCTS outperforms even heuristic agents designed
specifically for those domains.
Lazy Play It has been shown that MCTS can be “lazy” in non-tight situations [7], which
is to say that they can make suboptimal choices when they are not immediately required
to make an optimal choice to receive a reward. This is a natural consequence of all search
algorithms, not just MCTS, which are not incentivised to reach a quick decision, as there
will be no consequence of including additional cyclic moves before the final desired state is
reached. As such, an agent may decide to embark upon a pointless cycle of moves which
have no lasting result on the game state, simply because it assigns no negative value to such
a pointless cycle.
Playout Bias As MCTS relies on a relatively weak reward signal generated by simula-
tion on the forward model, any bias in that forward simulation can greatly reduce agent