Aspectos negativos-
GRUPO PARTICIPANTES
Section 8.3 discussed the results with the underlying assumption that one gener- ation was equivalent to the next. For canonical genetic programming this is true. For incremental evolution there is good reason why it is not true. This section analyses the results given the assumption that the computational requirements for a generation are proportional to the number of fitness cases evaluated in that generation.
8.5.1
Addition of ADFs
Under adjusted generations, the addition of ADFs can again be seen to represent a benefit to incremental evolution.
As was done in section 8.3.1 the efficiency ratio of two efficiency ratios was studied and very similar results were obtained. Especially for even-5, incremen- tal evolution benefited from the addition of ADFs above what benefits ADFs normally bring. As we discussed, this was most likely caused by the code en- capsulation offered by the automatically defined functions and just as was done previously, the following analysis will focus on the results with ADFs.
8.5.2
Allocation of Generations
The objective of this section is to discover the optimal number of generations to allocate to stage one, given this new “adjusted generations” light. In comparison to the equivalent discussion using unit-cost generations (see section 8.3) the re- sults are notably different. The most important difference is that this technique can sometimes be seen to be beneficial.
When the success effort measure was considered, performance tended to im- prove with the use of this technique for: even-5 with 16 fitness cases in the first
10 20 30 40 50 650 600 550 500 450 400
Even−5−parity with ADFs with 16 fitness cases in the first stage
using adjusted generations
Generations allocated to first stage
Min. comp. effort (x1000)
10 20 30 40 50
160
140
120
100
Even−5−parity with ADFs with 16 fitness cases in the first stage
using adjusted generations
Generations allocated to first stage
Success effort
Figure 8.8: Performance as measured by minimum computational effort and suc- cess effort given adjusted generations. Note the differing trends.
stage, even-6 with 32 fitness cases, and for all three experimental configurations involving even-7. The graphs that demonstrate these trends can be found in the electronic appendix (see appendix D).
Unfortunately the same cannot be said about the results when viewed with the minimum computational effort measure. At best, one might claim that there was a slight benefit for even-5 with 8 fitness cases and even-6 with 32, but in general the results showed either a downward trend or a flat trend (which indicated incremental evolution did not have any notable impact on performance.)
This difference in opinion between success effort and minimum computational effort shows up in the average correlation coefficient between the two measures: 0.51—still a strong association, but considerably lower than section 8.3’s corre- lation of 0.91. In fact, in one configuration a negative correlation (-0.13) was observed (see figure 8.8).
If we analyse these adjusted-generations results solely on success effort then, in terms of the optimal number of generations to allocate to stage one, we see mixed results across the different configurations. However, “mixed results” should be
Cost of failure Cost of success
0 10 20 30 40 50
Even−5−parity with ADFs with 16 fitness cases in the first stage
using adjusted generations
Adjusted generations
Figure 8.9: The reduced cost of failure using adjusted generations.
considered an improvement—they suggest the optimal number of generations is not clear. That result is an improvement given the obvious deterioration that pre- viously occurred when the number of generations allocated to stage one increased (see section 8.3.2).
How is it that success effort performance can be seen to improve even as the generations allocated to stage one increase to 50 (the total number of generations allocate for a run) while at the same time minimum computational effort can show the opposite trend? This can be explained by considering the cost of failure.
If 50 generations (the maximum for a run) are allocated to stage one and a run fails to find a solution to the first stage, then it will never spend any time in stage two. If stage one considers half of the fitness cases in stage two (as is the case in figure 8.8) then, using adjusted generations, the cost of a generation in stage one is only half that of a generation in stage two. A run that fails to find a solution to stage one will therefore cost only 50×0.5 = 25 adjusted-generations to execute to termination. In contrast consider a run that finds, within say 10 generations, a solution to all of the fitness cases in stage one, and then spends 30 generations in stage two before finding a solution there. It will cost 10×0.5 + 30 = 35 adjusted-generations to find that solution. In other words it is possible that the cost of failure could be less than the cost of success. Figure 8.9 plots this effect for even-5 with 16 fitness cases in the first stage.
How is it that this effect can impact the success effort and minimum compu- tational effort measures? Success effort directly includes the cost of failure in the numerator of its formula (see equation 4.1). As a result success effort demon- strates the performance advantage offered by adjusted generations. Minimum computational effort on the other hand does not incorporate this benefit.
Koza’s measure finds the minimum adjusted-generation based on the implicit assumption that the runs would continue were they not stopped at the optimal minimum generation. Given that the use of adjusted generations upsets this assumption, what impact does this have? If the minimum cost of failure (in adjusted-generations) is above the minimum generation (in adjusted-generations) then there will be no impact on this measure. As an increase occurs in the number of failures that use fewer adjusted-generations than the minimum generation, so the observed minimum computational effort will be larger than the true value. In these situations the observed minimum computational effort should be considered as an upper bound for the true minimum computational effort. However, as we will see next, even without this modification, Koza’s measure is already an upper bound.
Without modification, Koza’s minimum computational effort is not capable of indicating a reduction in the effort required when the minimum cost-of-failure drops below the minimum generation. Success effort on the other hand is able to give this indication.
Koza’s Measure is an Upper Bound
Minimum computational effort, according to Koza’s first book on genetic pro- gramming, gives the number of individuals that must be processed in order to yield a solution with 99% probability by generation j [71, chapter 8], where j is what we term the minimum generation. Koza’s method actually finds the upper bound to this number.
The key assumption in Koza’s method is that, for a probability of successP(i), one must execute i generations. This is not true. Instead, one must execute at most that many generations. On average, to obtain a probability of success of
P(i), one must execute xgenerations where
x=
Ri
k=0(Y(k)·k)·dk
Ri
k=0Y(k)·dk
To illustrate this visually, imagine a graph that plots any cumulative probabil- ity of success curve. Mark the probability of success at the minimum generation with an ‘X’. Koza’s measure will not be able to distinguish between this curve and any other that both (a) passes through ‘X’ and (b) is below the original curve. Yet for the original curve, there is a greater chance of finding a solution within fewer generations, and so it should naturally have a lower computational effort.
When is this issue of concern? As the required number of runs R(P(j)) increases, Koza’s measure becomes closer and closer to the real minimum com- putational effort because the cost of success becomes a less and less important component of the total minimum computational effort while the cost of failure— controlled by the specification of minimum generation—becomes the more im- portant component. However, for runs where R(P(j)) = 1 Koza’s upper bound is furthest from the real value.
And how much of an impact does this actually have? The difference between the “real” value and the value obtained when Koza’s method is followed is the difference between the mean and the maximum of the generations-to-success that are lower than the minimum generation.
Measuring what Koza Intended to Measure
It is not difficult to re-define Koza’s measure so that it gives the real value, rather than an upper bound, for the expected number of individuals to be processed. The definition need only become
I0(i) = (X(i) + 1)·M·R(P(i))
whereX(i) is the mean of the generations-to-termination—with each generations- to-termination observation truncated such that the number of generations in the observation is at most i.
Making this modification would also solve the issue associated with a variable cost-of-failure given that X(i) accounts for this.
Unfortunately, the impact of this modification on the coverage rates of the methods to produced confidence intervals is unknown. Further, it is inadvisable to compare minimum computational efforts produced using different methods. These two issues are sufficient disadvantage that we would hesitate to recommend the use of this modification.
Nonetheless users of Koza’s measure should be aware that, on two counts, it is only an upper bound: (i) whenR(P(j)) is low and (ii) when the cost-of-failure is below the minimum generation.
8.5.3
Fitness Cases in Stage One
The section looks at the how the number of fitness cases in the first stage im- pacted performance. Again, just like the unit-cost analysis of section 8.3.3, be-
cause there was no obvious manner with which to choose the “best” number of generations to allocate to stage one, we considered the average across each of the ten experiments executed for each configuration. But again because of the difference in configuration between the two experiments with even-4, we limited the comparison to even-5, -6, and -7.
The ordering is similar to that seen in section 8.3.3. It is better to use more fitness cases in the first stage: for even-5, 16 fitness cases in the first stage outperforms 8; for even-6, 32 fitness cases outperforms 16, which does better than 8; and for even-7, the best choice was 64, followed by 32, and then 16. This trend is still fairly clear with an average confidence that a difference truly exists of some 66% (averaged over each of the seven comparisons)
What is not clear is why such a trend exists. Given that there is an advantage to be had in terms of cost-of-failure, one might assume that this trend would peak at some maximum proportion of fitness cases (although the proportion would most likely be problem-dependent). Further experiments (beyond the maximum proportion of 0.5 used in this work) would be required to assess if this hypothesis is true.
On the other hand, it is possible that this trend occurred as an indication that, despite the adjusted-generations light, performance is still generally inferior to direct evolution. A comparison of performance can be found in section 8.5.5.
8.5.4
Problem Difficulty
We now attempt to discover how this technique, under adjusted generations, performed as the problems became more difficult. Analysis similar to that of section 8.3.4 was performed, but this time the results are encouraging.
The most interesting result can be seen in figure 8.10—incremental evolution was shown to offer increased benefits as the problem became more difficult. When considered by the number of allocated generations, 25 of the 30 graphs showed an increase or flat trend as the difficulty increased. These are very interesting results; they indicate that, under the adjusted generations light, incremental evolution may become more beneficial as the problem domains become harder.
This result is only true if success effort was considered. We have seen that final success proportion decreases with the use of this technique and that neither success proportion nor minimum computational effort appreciate the reduced cost of runs that terminate before the minimum generation, so limiting the analysis to success effort is quite reasonable.
0.6 0.8 1.0 1.2 1.4 1.6 One−half Even−n
Success effort ratio
4 5 6 7 0.6 0.8 1.0 1.2 1.4 1.6 One−quarter Even−n
Success effort ratio
5 6 7 0.4 0.6 0.8 1.0 1.2 1.4 One−eighth Even−n
Success effort ratio
6 7
Efficiency ratios of even−n−parity with ADFs against direct evolution by problem difficulty given the proportion of fitness cases in the first stage
using adjusted generations
In line with the analysis in section 8.3.4, if the number of fitness cases in the first stage is kept constant then incremental evolution decreases in performance as the problem increases in difficulty.
8.5.5
Comparison to Direct Evolution
When a comparison is made to direct evolution two things become clear. The first is that we are reminded that the probability of finding a solution consistently decreased as the number of generations allocated to stage one increased (remem- ber that adjusted generations have no impact on final success proportion). But despite this raw reduction in performance, in five of the ten configurations with ADFs, there were occurrences where the required success effort was reduced. So incremental evolution with adjusted generations can offer a benefit. This is an interesting result.
How does this happen? The reason is that both the cost-of-failure and the cost-of-success decrease with this method. The decreases are sufficient to over- come the reduced success proportion.
However, it is not true that this technique is generally beneficial. None of the ten configurations had (over the ten settings for allocated generations) an average success effort ratio that was above one. Given that there was no obvious way to select the optimal (or even near-optimal) number of generations to allocate to stage one, we thus have no rule-of-thumb to predict whether use of this technique would be beneficial.
8.5.6
Summary
What can we take from the result under adjusted generations? The most interest- ing result was that fitness-based incremental evolution was sometimes beneficial. The other points were:
• The benefits of adding ADFs were again beyond what is normally seen.
• If the technique was beneficial then it was best to allocate as many genera- tions as possible to the first stage. This was shown to decrease the cost of failure, which we then showed that minimum computational effort may not be able to measure. Koza’s statistic was also demonstrated to be an upper bound.
• It was shown that performance improved as the number of fitness cases increased. This occurred despite the benefits of reduced cost-of-failure as- sociated with fewer fitness cases in the first stage.
• Incremental evolution was shown to improve in performance as the problem domain became more difficult.