2.5. LA INTRAHISTORIA DE LA RED DE EVALUACIÓN FORMATIVA
2.5.1. RECONSTRUCCIÓN LONGITUDINAL DE LA INTRAHISTORIA DE LA RED
2.5.1.2. Trabajo desarrollado en la Red: proyectos, congresos, líneas de investigación y utilización de
This section answers the question “what is the coverage like when the true success rate approaches 100%?”. We will show that the method’s performance is accept- able when compared with the performance of other confidence interval generating methods.
Method
For this question it would be preferable to use data from real genetic programming runs. Obtaining such data is, unfortunately, currently too demanding computa-
8The variablez is introduced in section 2.3.1 and, as is common, is set to 0.99 throughout
tionally. Instead we have chosen to simulate GP data using the assumption that the success proportion curves are either normally or log-normally distributed— the two most accurate approximations that we found to real GP data (see sec- tion 3.2.3).
Normal For the normally-distributed experiments, we simulated the success probability curves using three different means (25, 100, 500 generations) and three different standard deviations of 1
16, 1 8, and
1
4 times the mean. Each of these
curves were multiplicatively scaled by a “success factor”—making success factor equivalent to the true cumulative success probability after an infinite number of generations. Nine success factors were used (90%, 95%, 96%, 97%, 98%, 99%, 99.5%, 99.9%, and 100%). Each of these 81 configurations were repeated for six different run sizes (25, 50, 75, 100, 200, and 500 runs), giving a total of 486 configurations.
For each configuration of mean, standard deviation, success factor, and run size, one million simulated runs were generated using a normal distribution. For success factors of 95%, 99%, 99.5%, 99.9%, and 100%, from these simulated runs, 10,000 samples of the specified number of runs were randomly selected (with replacement). For the other success factors 1,000 samples were obtained. For each sample a 95% confidence interval for the sample’s computational effort was calculated using the Wilson-Dependent method. Whether the confidence interval included the true minimum computational effort (the coverage) was recorded.
The true computational effort was obtained by calculating
min
i (i+ 1)·R(P(i)·p)·M
where: R is the function for calculating the number of runs required (equa- tion 2.7); P(i) is the cumulative proportion given by the normal distribution function; p is the specified success factor; i is the generation which ranged from 0 to 1000; and M is the population size (effectively just a scaling factor; 500 was used).
Log-Normal For the log-normally distributed experiments, a very similar pro- cedure was followed. The only differences were that:
• The distribution was log-normal.
• Ten success factors were considered (90%, 95%, 96%, 97%, 98%, 99% 99.25%, 99.5%, 99.75% and 100%).
0.80 0.85 0.90 0.95 1.00 0.80 0.85 0.90 0.95 1.00 Normal Success factor
True success proportion at true minimum generation
0.80 0.85 0.90 0.95 1.00 0.80 0.85 0.90 0.95 1.00 Log−normal Success factor
True success proportion at true minimum generation
Figure 3.6: Success factor versus true success proportion at the true minimum generation for the Normal and Log-normal experiments.
• 1,000 samples were taken for each of the configurations.
• Only one standard deviation scaler was used (12). Results and Discussion
It is of use to note the relationship between success factor and the true cumu- lative success at the true minimum generation. For the normally distributed experiments, the specified success factor is an acceptable indicator but for the log-normal experiments this approximation is less accurate. Figure 3.6 graphs these relationships. It should be noted that the true success proportion at the true minimum generation cannot be greater than the value for success factor.
Figure 3.7 plots the observed coverage rates for the different success factors for the normal and log-normal experiments. Both curves show good coverage rates near the desired 95% level up to a success effort of about 99% at which point they both drop markedly. However, when they drop, although they fall quickly, at worst the drop was to 78% when 95% was specified. Although this is far from ideal, this range of coverage is not unlike that observed by Newcombe [91]
0.90 0.92 0.94 0.96 0.98 1.00 75 80 85 90 95 100 Success factor Coverage Normal Log−normal
Figure 3.7: Observed coverage against success factor for the Normal and Log- normal experiments. Confidence intervals are less than ±1.5 percentage points for Normal and ±2.0 percentage points for Log-normal.
regarding Wilson’s method (where the minimum observed coverage was 83%) and the “Likelihood based” method (where the minimum was 80%), and it is far superior to the frequently used normal-approximation method with a minimum observed coverage of 0.02%!
Further, it is not uncommon for confidence interval measures to be unreliable at extreme values. A commonly known example would be the rule-of-thumb for normal-approximation confidence intervals for proportions: one should satisfy the equations np >5 andn(1−p)>5 before the intervals are valid [24]. But for extreme values of p (close to either zero or one) the products can be very small, thus reducing the quality of the underlying assumptions which has a direct result on the quality of the confidence intervals.
An analysis by both run size and success factor shows that the drop at high success factors is greater for higher run sizes, thus for those with a high success factor, increasing the run size decreases the coverage rates. Figure 3.8 graphs this effect.
Runs 100 200 300 400 500 Success Factor 0.90 0.92 0.94 0.96 0.98 1.00 Coverage (%) 70 75 80 85 90 95 100 Normal Runs 100 200 300 400 500 Success Factor 0.90 0.92 0.94 0.96 0.98 1.00 Coverage (%) 70 75 80 85 90 95 100 Log−normal
Figure 3.8: Observed coverage and run size against success factor for the Normal and Log-normal experiments.
The minimum generation ranged from 29 to 848 generations but because of the bias towards very high success proportions, the mean minimum generation was just 285.
An interesting effect occurs when the observed probability of success is greater than z and either a very large number of runs was executed or the confidence (1−α) level is reduced. In some cases it is possible that the lower limit of the confidence interval for cumulative probability of success does not drop below z. When this occurs the function R(p, z) is clamped to one for both the upper and lower limits, thus producing a zero width confidence interval for the minimum computational effort. Zero width intervals are an obvious indication of failure in any method to generate confidence intervals. Fortunately, this is an unlikely scenario. Of the experiments, only those at 500 runs demonstrated this issue, and even then only at very low rates: from 0.2% at a success factor of 99.5% up to 2% when the success factor was 100%.