Problemas con el APE 70

Single-step look-ahead methods can be seen as greedy approaches to the problem, as an operator which seems inferior at the moment may lead to greater gains later on. Looking two steps ahead (i.e. considering the reward of operator x if it is applied after another operatory), Cowlinget al.(2001) introduced a choice function that selects the next operator based on a weighted average of past performance, time elapsed since last application, and the pervious operator used. The first and third measures favor exploitation, while the second favors exploration. This differs from the Compass algorithm in that it considers the sequence in which operators were applied. The choice-function-based method performed significantly better than random and greedy selection on the sales summit scheduling problem.

Generalizing the choice function idea, Burke et al. (2007) used a TS algorithm to find the best sequence of operators constructing a solution for a timetabling problem. Their algorithm starts with a set of operators, applied in sequence to construct a complete solution. The set is then perturbed, checked against the tabu list, and evaluated based on solution quality. The process continues for a pre-determined number of iterations. Compared to nine other specialized algorithms, the authors showed that their algorithm never outperformed the best, but its results were within range of the others. This work was extended in Qu and Burke (2008) to include other algorithms at the upper level, namely: Iterated Local Search (ILS), Variable Neighborhood Search (VNS), and Steepest Descent (SD). ILS and VNS showed to be superior to the other methods when tested on the same data set.

Another multi-step look-ahead approach is Dynamic Programming (DP), in which a non-myopic control policy can be learnt to account for the future effects of selecting an operator. While this seems appealing, there are three difficulties: first, fully describing the state space is not trivial, especially if it is continuous. Second, the action-reward relationship maybe complex and could require approximation with a learning function. Third, the policy

one and can restrict its applicability to similar test instances. Such difficulties have been addressed by a few researchers, but generally speaking a DP-based approach has not yet been fully developed, and the experimental studies conducted thus far indicate that learning an optimal, or near optimal, policy is a hard task.

Nareyek (2004) proposed ten reinforcement learning strategies to update the weights of a set of heuristics, from which selection is made probabilistically. Five were used in case of solution quality improvement, and the remaining in case of a decline. Different combinations of improving and non-improving strategies were tested on the Orc Quest problem and a modified Logistic Domain problem1. The authors concluded that, generally, a low update rate of weights is better in case of an improvement, whereas a high rate is better in case of a decline.

In Battiti and Campigotto (2008), the authors modeled the algorithm configuration problem as a Markov Decision Process, and used the Least Squares Policy Iteration (LSPI) algorithm, adopted from Reinforcement Learning (RL), to learn a near optimal control policy through a training set. The policy was used to learn how to change the prohibition length parameter for the Reactive Tabu Search (RTS) algorithm of Battiti and Protasi (1997). Comparing RTS with LSPI-RTS revealed that LSPI-RTS could not outperform its non- adaptive counterpart; however, it was not significantly worse. Comparison to three other SAT solvers showed superior performance of LSPI-RTS.

Extending the above work in Battiti and Campigotto (2011), the authors compared three LSPI-based algorithms to their original versions using default, and tuned, parameter settings. Tuning was carried out with ParamILS (Section 2.4.1.2 later) of Hutter et al.

(2009a). The algorithms were: Hamming Reactive Tabu Search (HRTS), Adaptive WalkSAT, and Scaling and Probabilistic Smoothing (SAPS) together with its Reactive version (R-SAPS). When testing against default parameter settings, the LSPI-based HRTS,

1_{Detailed description of both problems can be found in Nareyek, A., 2001.Constraint-based agents: an architecture for}

SAPS, and R-SAPS showed superior performance on a set of benchmark random MAX-3- SAT problems; however, the LSPI-based Adaptive WalkSAT performed worse. Repeating the same comparison on other structured MAX-SAT industrial instances, the control policy degraded the performance of SAPS and R-SAPS, while the other two algorithms performed just as well or slightly better. When testing against tuned parameter settings, LSPI-based SAPS was not significantly worse than SAPSoffline, and the LSPI-based HRTS was either significantly worse than HRTSoffline or just as good. The authors contribute the inferior performance of the LSPI-based methods to its inability to converge to a single policy during training.

Mcclymont and Keedwell (2011) attempted to learn the best sequence of applying a number of operators. Their method, Markov Chain Hyper-heuristic (MCHH), uses Markov chains to model the transition probabilities between operators, and RL to update the transition probabilities online based on the operators’ performance. MCHH was applied within a (1+1)-Evolutionary Strategy (ES) to determine which variation operator to select next. The authors compared a (1+1)-ES using MCHH to random selection and a prohibition- based algorithm proposed by Burkeet al.(2005). They found their algorithm to be superior to random selection, and it performed just as well as the prohibition-based one. However, the authors noted that while their method converges at a reasonable rate, higher population diversity is still needed.

Gaspero and Urli (2012) created three RL-based heuristic selection policies: tabular RL, where the possible actions and rewards are relatively few and can be read from a table; Multi-layer Perceptron RL, where the actions and rewards are related through a function that is approximated with Artificial Neural Networks; and Eligibility Traces RL, where each action on the trajectory to a reward is assigned a share of that reward, not just based on the last action. Each of the policies introduced its own hyper-parameter which the authors set using the offline tunerF-Race (Section2.4.1.2later). However, all three policies performed

rather poorly when tested on the Cross-domain Heuristic Search Challenge2 (CHeSC) test- bed, see Burke et al. (2011), due to the limited number of training instances available to learn the policies.

A slightly different approach to determine which operator(s) to apply for the nextn- steps is landscape analysis, which seeks to characterize the topology surrounding the current solution. Proper characterization of the landscape can offer guidance to whether the algorithm should diversify or intensify its search, and select its operators accordingly. Introductions to the topic can be found in Merz and Freisleben (2000), Kallel et al.(2001), and Battitiet al.(2009).

Ideally, the algorithm should use problem independent measures to characterize the landscape. One such measure is ruggedness, first introduced by Weinberger (1990), which calculates the autocorrelation of a time series of fitness values generated by a random walk algorithm. A related measure is the correlation length, which estimates the largest distance, or time lag, between two points at which the value of one point can still provide information about the expected value of the other one (Hordijk, 1996). Put differently, it estimates the largest time lagtfor which one can still expect some correlation between two points,tsteps apart.

Another common measure is the fitness distance correlation, first introduced by Jones (1995), which measures the extent to which solution quality values are correlated with the distance to the global optimum. Additional measures include: the number of global and local optima, the distribution (uniform or random) of the global and local optima, and the structure of the basins of attraction (Weinberger, 1991). These measures have also been extended to multi-objective optimization applications, see Deb (1999), and Knowles and Corne (2002).

Problem specific measures have also been created. Smith-Miles and Lopes (2011) review such measures for assignment, routing, knap-sack, bin packing, graph coloring, and timetabling problems. Unfortunately, real optimization problems are highly dimensional and are very complex to analyze or measure. This limits landscape analysis to simple and artificial landscapes such as theN-klandscape (Kauffman and Levin, 1987). Early attempts to measure the relationship between solutions within an N-k landscape (using the auto- correlation function and the auto-correlation length) can be found in Weinberger (1990), and Hordijk (1996). Recently, ideas based on complex network analysis, which not only relates solutions within the same landscape, but also relates different landscapes to each other, can be found in Ochoaet al.(2008) Tomassiniet al.(2008), and Ochoaet al.(2011).