DEFINICIÓN DE LAS CATEGORÍAS

The first known combination of Reinforcement Learning and neural networks was made by Anderson in 1986. Anderson’s system used MLPs with Reinforcement Learning algorithm to learn search heuristics. The system was successfully applied to the pole-balancing task and to the Tower of Hanoi puzzle (Anderson 1986). In this section we concentrate on the application of Temporal Difference (TD) learning to neural networks.

4.9.1

TD with Multilayer Perceptrons

There have been many success stories in applying TD with MLPs. Perhaps the most well-known one is that of a Backgammon-playing program called TD-Gammon (Tesauro 1992, 1994, 1995). Initially TD-Gammon had no knowledge of Backgammon strategies, as it was told only the rules of the game and the final outcome – win or a loss. The network received reward of 1 when it won, -1 when it lost and 0 for all other situations. The network used was a standard MLP with one hidden layer and an output indicating the likelihood of the player winning from the current board position.

The network was trained with TD(λ) by playing 1,500,000 games against itself.

During the first few thousand games, the network learned a number of elementary strategies. More sophisticated concepts emerged later. In exhibition matches, the latest version of TD-Gammon was able to equal the top human players, losing just 0.02 points per game. Furthermore, TD-Gammon was able to come up with genuinely novel strategies, which have been adopted by top human players.

Crites and Barto applied RL to elevator dispatching. The system examined is a simulated 10-story building with 4 elevator cars. The size of the state space is large (~1022) with 218 combinations of call buttons, 240 combinations of car buttons, and 184 combinations of positions and directions of the cars. When compared to 8 commercial heuristic systems, the RL system achieved an average improvement of 5% to 15% in system time (Crites & Barto 1996). RL has also been applied in airline revenue management (Gosavi et al. 2002) and marketing (Pednault et al. 2002).

Randlov and Alstrom apply Sarsa(λ) to solve a real-world problem of learning to drive a bicycle (Randlov & Alstrom). Balancing the bicycle is a difficult task to learn, as the agent must take into account the angle and angular velocity of the handle bars, as well as the direction, tilt and velocity of the bicycle. The agent has a choice of just two actions: it can either apply torque to the handle bars or shift its centre of mass. The authors have applied the idea of shaping (borrowed from psychology) to train the agent in a series of increasingly-difficult tasks. The authors note that training through self-play is a form of shaping, because at first the agent plays against a nearly random opponent, thereby solving an easy task. The complexity of the task grows as the agent’s playing ability improves. As the result, the agent first learns to balance the bicycle and then later manages to ride it towards a target location (Randlov & Alstrom).

4.9.2

TD with cascade networks

Surprisingly there has been relatively little application of cascade networks to TD learning. The first such application was made by Rivest and Precup for Tic-Tac-Toe (Rivest & Precup 2003) and then later for Car-rental and Backgammon tasks (Bellemare et al. 2004).

The algorithm uses a look-up table (cache) to store the states that have already been found. Before calculating the estimated value V(s) for state s, we first check whether s is in cache. If it is then its cached value is returned. Otherwise, the network is evaluated for s and the value is saved in cache before being returned. When a new target value is computed for state s, it overwrites the current estimate V(s) in the cache. If a state is revisited then its cached value will be updated multiple times according to the TD update rule. After a certain period, the system is consolidated by training the network on the cache data.

The cascade network used was a variation of Cascor. It uses a sibling-descendant pool of candidates, which allows it to add hidden nodes to the same layer, as well as by adding more layers. The results of the Tic-Tac-Toe project showed that Cascor plays equally well or outperforms MLPs against all three built-in opponents:

random, basic and minimax. Interestingly, the structure of the Cascor networks does not become more complex as the strength of the player they are trained against is increased. This suggests that the same network structure is able to learn simple, as well as complex patterns (Rivest & Precup 2003).

A more detailed study of applying Cascor to TD learning was performed by Vamplew and Ollington in 2005. The proposed algorithm (Cascade-Sarsa) differs from Rivest and Precup in three aspects. The algorithm is online, meaning that the network is trained after each interaction with the environment. Secondly, Cascade-Sarsa uses eligibility traces. Finally, it is based on Cascade2, developed by Fahlman as reported in (Prechelt 1997). Although Cascor is effective for classification tasks, it experiences difficulties with regression tasks. This occurs because the correlation term forces hidden unit activations to extreme values, which prevents the network from producing smoothly varying output. Cascade2 differs from Cascor by training candidate units using the residual error as a target output (Vamplew & Ollington 2005a).

Cascade-Sarsa was compared against three other networks, including MLP with a single hidden layer of neurons using asymmetric sigmoid activation functions. The networks were tested on three benchmark RL problems – Acrobot, Mountain-Car and Puddleworld (Vamplew & Ollington 2005b). These problems have been shown to be difficult to learn by MLPs (Boyan & Moore 1995). On Acrobot and Mountain-Car Cascade-Sarsa was significantly more successful than MLP on both on-line and off- line tests. However, Cascade-Sarsa failed on the Puddleworld problem, not even able to match the relatively poor on-line performance of MLP (Vamplew & Ollington 2005b).

To explain this difference in performance we take a closer look at the problems being learnt. In both Acrobot and Mountain-Car tasks the decision boundaries are linear, and hence each boundary can be learnt by a single sigmoidal hidden unit. Naturally Cascade-Sarsa performs well on these tasks, because each candidate node can specialize on a single decision boundary. In contrast the Puddleworld task requires networks to learn localised regions within the input space, which cannot be achieved by a single candidate node used by Cascade-Sarsa (Vamplew & Ollington 2005b).