Computing only the average objective values of the metaheuristics over multiple data does not provide a full comparison between them. Averages are susceptible to outliers: they can allow excellent performance on some datasets to compensate for an overall bad performance. There may be situations in which such behaviour is desired. However, in general we prefer algorithms that behave well on as many problems as possible.
We have carried out tests to determine the statistical significance of differences between the performances of the metaheuristics (Hollander and Wolfe, 1999). The issue of statistical tests for comparison of algorithms on multiple datasets was theoretically and empirically reviewed by Demšar (2006).
The null-hypothesis being tested is that the metaheuristics have equal mean performance and the observed differences are merely random. The alternative hypothesis is that the algorithms have different mean performances of statistical significance.
The most common statistical method for testing differences between more than two algorithms is Analysis of Variance (ANOVA) (see (Hollander and Wolfe, 1999) and (Demšar, 2006) for more details).
Since ANOVA is based on assumptions that are violated in this context, we make use of the Friedman Test, the non-parametric equivalent of ANOVA (Friedman, 1937, 1940), and its corresponding Nemenyi Post-hoc Test (Nemenyi, 1963).
According to the Friedman test, the statistical significance of differences between the metaheuristics is examined by testing whether the measured average ranks are significantly different from the overall mean rank. In particular, we use the version of the Friedman test developed by Iman and Davenport (1980), which considers a powerful test statistic FF (Appendix A). If the equivalence of the algorithms is rejected, the Nemenyi post-hoc test is applied in order to perform pairwise comparisons. For more details, see Appendix A.
To perform the Friedman and Nemenyi tests, the ranks of the algorithms for each dataset are evaluated, with a rank of 1 assigned to the best performing algorithm, rank 2 to the second best one, and so on. The
average ranks for each metaheuristic among the 48 datasets are then computed. The metaheuristics are ordered with respect to the average ranks from the best one to the worst one, as follows:
B-VNS GS-VNS GRASP PILOT MGA
1.9 2 2.26 4.4 4.45
According to the ranking, B-VNS is the best performing algorithm, immediately followed by GS-VNS, and GRASP, with PILOT and MGA achieving the worst results.
Now, we analyse the statistical significance of differences between these ranks. Consider the Iman and Davenport (1980)’s version of the Friedman test for k=5 algorithms and N=48 datasets. The value of the FF
test statistic, which is distributed according to the F-distribution with (k−1, (k−1)(N−1)) = (4, 188) degrees of freedom, is computed (Appendix A). This value is 100.4, which is greater than the critical value (3.42 for α=1%, where α is the significance level of the test expressed as percentage). Thus, a significant difference between the performance of the metaheuristics exists, according to the Friedman test.
As the equivalence of the algorithms is rejected, we proceed with the Nemenyi post-hoc test (Appendix A). Considering a significance level α=1%, the critical value is q0.01≅3.26. The critical difference (CD) for the Nemenyi test is
. 05 . 48 1 6
6 26 5
.
3 ≅
⋅
⋅ ⋅
= CD
The differences between the average ranks of the metaheuristics are reported in Table 5.
- INSERT TABLE 5 -
From this table, we can identify two groups of metaheuristics. The first group includes B-VNS, GS-VNS, and GRASP, while the second group includes PILOT and MGA. Considering a significance level α=1%, the algorithms within each group have comparable performance according to the Nemenyi test since, in each case, the value of the test statistic is less than the critical difference. Conversely, two algorithms belonging to different groups have significantly different performance according to the Nemenyi test.
Summarizing, from the Friedman and Nemenyi statistical tests, B-VNS, GS-VNS, and GRASP have comparable performance, and they are the best performing algorithms. On the other hand, PILOT and MGA have comparable performance, but worse than B-VNS, GS-VNS, and GRASP.
Another way to compare the performance of the algorithms is to count the number of times they generate the optimal solution. In particular, counting the overall number of exact solutions obtained is a good approach to estimate the diversification capability of each metaheuristic. The Exact Method obtains the exact solution for all problem instances of 32 datasets, among the overall 48 datasets; for the remaining sets NF is reported. Therefore, the total number of instances having the exact solution is: 32×10=320.
The percentages of the number of optimal solutions obtained by the metaheuristics among the 320 instances are the following (ranking from the best to the worst algorithm):
B-VNS GS-VNS GRASP MGA PILOT 100 100 99.7 99.7 96.7
B-VNS and GS-VNS obtain all the optimal solutions, underlying a high exploration capability even for complex instances. In the same way, GRASP and MGA offer very good results, missing only 1 solution out of 320, although MGA is extremely time consuming. With 11 cases (out of 320), PILOT fails to find the global optimum and becomes trapped at a local optimum.
Furthermore, some optima reached by the metaheuristics require a greater computational time than required by the Exact Method, thus nullifying the purpose of the metaheuristics. In this sense the best performances are obtained by B-VNS, GS-VNS, and GRASP, all of which require less computational time than the Exact Method among the 32 datasets. In contrast, PILOT and MGA obtain the optimal solution but in a time that exceeds that of the Exact Method in 9 and 18 datasets, respectively. Although MGA reaches more exact solutions than PILOT, it is computationally more burdensome.
From this further analysis, the results reinforce the conclusion that B-VNS, GS-VNS, and GRASP are effective metaheuristics for the MLST problem. Furthermore, the algorithm which appears to be the most suitable for the proposed problem is B-VNS, thanks to the following features: ease of implementation, user-friendly code, high quality of the solutions, and shorter computational running times.
5. Conclusions
In this paper, we have studied metaheuristics for the minimum labelling spanning tree (MLST) problem.
In particular, we have examined and implemented the metaheuristics recommended in the literature: the Modified Genetic Algorithm (MGA) of Xiong et al. (2006), and the Pilot Method (PILOT) of Cerulli et al.
(2005). Furthermore, some new implementations for the MLST problem have been proposed: a Greedy Randomized Adaptive Search Procedure (GRASP), a Basic Variable Neighbourhood Search (B-VNS), and a VNS-based version that we have called Group-Swap Variable Neighbourhood Search (GS-VNS).
Computational experiments were performed using different instances of the MLST problem to evaluate how the algorithms are influenced by the parameters, the structure of the network and the distribution of the labels on the edges. Applying the nonparametric statistical tests of Friedman (1937, 1940), and Nemenyi (1963), we concluded that B-VNS, GS-VNS, and GRASP have significantly better performance than the other methods recommended in the literature with respect to solution quality and running time.
Furthermore, this result has been reinforced by comparing the metaheuristics with an exact approach. B-VNS, GS-VNS and GRASP obtain a large number of optimal or near-optimal solutions, showing an
enhanced diversification capability.
All the results allow us to state that B-VNS, GS-VNS, and GRASP are fast and extremely effective metaheuristics for the MLST problem. In addition, B-VNS is particularly recommended for the proposed problem because its simplicity and its ability to obtain high-quality solutions in short computational running times.
Appendix A. Statistical tests
Friedman Test (Friedman, 1937, 1940): The Friedman test is a non-parametric statistical test that examines the existence of significant differences between the performances of multiple algorithms over different datasets. Given k algorithms and N datasets, it ranks the algorithms for each dataset separately, and tests whether the measured average ranks are significantly different from the mean rank. The statistic used by Friedman (1937, 1940) is
( ) ( )
which follows a Chi-Square distribution with (k-1) degrees of freedom.
Iman and Davenport (1980) developed a more powerful version of the Friedman test by considering the following statistic:
which is distributed according to the F-distribution with (k−1) and (k−1)(N−1) degrees of freedom. For more details, see Demšar (2006).
Nemenyi Test (Nemenyi, 1963): The Nemenyi test is used to perform pairwise comparisons of multiple algorithms over different datasets (Nemenyi, 1963). The performance of two algorithms is considered significantly different if the corresponding average ranks differ by at least the critical difference (CD)
( )
where k is the number of the metaheuristics, N the number of datasets, qα the critical value, and α the significance level of the statistical test. For more details, see Demšar (2006).
Acknowledges
Sergio Consoli was supported by an E.U. Marie Curie Fellowship for Early Stage Researcher Training (EST-FP6) under grant number MEST-CT-2004- 006724 at Brunel University (project NET-ACE).
José A. Moreno-Pérez was supported by the projects TIN2005-08404-C04-03 of the Spanish Government (with financial support from the European Union under the FEDER project) and PI042005/044 of the Canary Government.
We gratefully acknowledge this support.
References
[1] D. Avis, A. Hertz and O. Marcotte, Graph Theory and Combinatorial Optimization, New York:
Springer-Verlag, 2005.
[2] T. Brüggemann, J. Monnot and G. J. Woeginger, “Local search for the minimum label spanning tree problem with bounded colour classes,” Operations Research Letters, vol. 31, pp. 195–201, 2003.
[3] C. Blum and A. Roli, “Metaheuristics in combinatorial optimization: Overview and conceptual comparison,” ACM Computing Surveys, vol. 35(3), pp. 268–308, 2003.
[4] R. Cerulli, A. Fink, M. Gentili and S. Voss, “Metaheuristics comparison for the minimum labelling spanning tree problem,” in The Next Wave on Computing, Optimization, and Decision Technologies, B.
L. Golden, S. Raghavan and E.A. Wasil (Eds), New York: Springer-Verlag, 2005, pp. 93–106.
[5] R. S. Chang and S. J. Leu, “The minimum labelling spanning trees,” Information Processing Letters, vol. 63, no. 5, pp. 277–282, 1997.
[6] S. Consoli and K. Darby-Dowman, “Combinatorial optimization and metaheuristics,” Brunel University, West London, UK, Tech. Rep. TR/01/06, Nov. 2006. Available:
http://hdl.handle.net/2438/503.
[7] S. Consoli, “Test datasets for the Minimum Labelling Spanning Tree problem [online]”. Available at http://people.brunel.ac.uk/~mapgssc/MLSTP.htm [accessed 12 March 2007].
[8] J. Demšar, “Statistical Comparison of Classifiers over multiple Data Sets,” Journal of Machine Learning Research, vol. 7, pp. 1–30, 2006.
[9] C. Duin and S. Voss, “The Pilot Method: A strategy for heuristic repetition with applications to the Steiner problem in graphs,” Wiley InterScience, vol. 34(3), pp. 181–191, 1999.
[10] M. Friedman, “A comparison of alternative tests of significance for the problem of m rankings,”
Annals of Mathematical Statistics, vol. 11, pp. 86–92, 1940.
[11] M. Friedman, “The use of ranks to avoid the assumption of normality implicit in the analysis of variance,” Journal of the American Statistical Association, vol. 32, pp. 675–701, 1937.
[12] F. Glover and G. A. Kochenberger, Handbook of metaheuristics, Norwell, MA: Kluwer Academic Publishers, 2003.
[13] D. E. Goldberg, K. Deb, and B. Korb, “Don’t worry, be messy,” in Proc. 4th Intern. Conf. on Genetic
Algorithms, Morgan-Kaufmann, La Jolla, CA, 1991.
[14] P. Hansen and N. Mladenović, “Variable neighbourhood search,” in Handbook of metaheuristics, F.
Glover and G. A. Kochenberger (Eds), Norwell, MA: Kluwer Academic Publishers, 2003, ch. 6, pp.
145–184.
[15] P. Hansen and N. Mladenović, “‘Variable neighbourhood search: Principles and applications,”
European Journal of Operational Research, vol. 130, pp. 449–467, 2001.
[16] P. Hansen and N. Mladenović, “Variable neighbourhood search”, Computers and Operations Research, vol. 24, pp. 1097–1100, 1997.
[17] M. Hollander and D. A. Wolfe, Nonparametric statistical methods, 2nd ed., New York: John Wiley &
Sons, 1999.
[18] R. L. Iman and J. M. Davenport, “Approximations of the critical region of the Friedman statistic,”
Communications in Statistics, vol. 9, pp. 571–595, 1980.
[19] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by simulated annealing,” Science, vol.
220, no. 4598, pp. 671–680, 1983.
[20] S. O. Krumke and H. C. Wirth, “On the minimum label spanning tree problem”, Information Processing Letters, vol. 66, no. 2, pp. 81–85, 1998.
[21] P. B. Nemenyi, “Distribution-free multiple comparisons,” Ph.D. thesis, Princeton Univ., New Jersey, 1963.
[22] L. S. Pitsoulis and M. G. C. Resende, “Greedy randomized adaptive search procedure,” in Handbook of Applied Optimization, P. Pardalos and M. G. C. Resende (Eds), Oxford University Press, 2002, pp.
168–183.
[23] M. G. C. Resende and C. C. Ribeiro, “Greedy randomized adaptive search procedures”, in Handbook in Metaheuristics, F. Glover and G. Kochenberger (Eds), Kluwer Academic Publishers, 2002, pp. 219–
249.
[24] R. Sedgewick, Algorithms in C++: Fundamentals, Data Structures, Sorting, Searching, and Graph Algorithms, Boston: Addison-Wesley, 2001.
[25] A. S. Tanenbaum, Computer Networks, Englewood Cliffs, New Jersey: Prentice-Hall, 1989.
[26] R. Van Nes, Design of multimodal transport networks: A hierarchical approach, Delft, The Netherlands: Delft University Press, 2002.
[27] S. Voss, A. Fink, and C. Duin, “Looking ahead with the Pilot Method,” Annals of Operations Research, vol. 136, pp. 285–302, 2004.
[28] S. Voss, S. Martello, I. H. Osman, and C. Roucairol, Meta-Heuristics. Advanced and Trends Local Search Paradigms for Optimization, Norwell, MA: Kluwer Academic Publishers, 1999.
[29] Y. Wan, G. Chen and Y. Xu, “A note on the minimum label spanning tree,” Information Processing Letters, vol. 84, pp. 99–101, 2002.
[30] Y. Xiong, B. Golden, and E. Wasil, “Improved Heuristics for the Minimum Labelling Spanning Tree Problem,” IEEE Transactions on Evolutionary Computation, vol. 10, no. 6, pp. 700–703, Dec. 2006.
[31] Y. Xiong, B. Golden, and E. Wasil, “A One-Parameter Genetic Algorithm for the Minimum Labelling Spanning Tree Problem,” IEEE Transactions on Evolutionary Computation, vol. 9, no. 1, pp. 55–60, Feb. 2005.
[32] Y. Xiong, B. Golden, and E. Wasil, “Worst case behavior of the MVCA heuristic for the minimum labelling spanning tree problem,” Operations Research Letters, vol. 33, no. 1, pp. 77–80, 2005.
[33] Y. Xiong, “The Minimum Labelling Spanning Tree Problem and some variants,” Ph.D. thesis, Grad.
School Maryland Univ., USA, 2005.
Table 1
Computational results for Group 1 (max-CPU-time for heuristics = 1000 ms) Parameters Average objective function values
n ℓ d EXACT PILOT MGA GRASP B-VNS GS-VNS
0.8 2.4 2.4 2.4 2.4 2.4 2.4
0.5 3.1 3.2 3.1 3.1 3.1 3.1
20 20
0.2 6.7 6.7 6.7 6.7 6.7 6.7
0.8 2.8 2.8 2.8 2.8 2.8 2.8
0.5 3.7 3.7 3.7 3.7 3.7 3.7
30 30
0.2 7.4 7.5 7.4 7.4 7.4 7.4
0.8 2.9 2.9 2.9 2.9 2.9 2.9
0.5 3.7 3.7 3.7 3.7 3.7 3.7
40 40
0.2 7.4 7.7 7.4 7.4 7.4 7.4
0.8 3 3 3 3 3 3
0.5 4 4.1 4.1 4 4 4
50 50
0.2 8.6 8.6 8.6 8.6 8.6 8.6
TOTAL: 55.7 56.3 55.8 55.7 55.7 55.7
Parameters Computational times (milliseconds)
n ℓ d EXACT PILOT MGA GRASP B-VNS GS-VNS
0.8 0 3.2 15.6 1.6 0 0
0.5 0 3 22 0 1.6 0
20 20
0.2 11 3.2 23.4 0 1.6 0
0.8 0 6.3 9.4 1.6 0 1.5
0.5 0 7.8 26.5 0 0 0
30 30
0.2 138 9.3 45.4 1.5 1.4 3.1
0.8 2 17.2 12.5 1.5 0 1.5
0.5 3.2 18.9 28.2 1.5 1.6 3.1
40 40
0.2 100.2ּ103 21.8 120.3 15.6 6.2 6.2
0.8 3.1 26.5 21.8 3 1.4 3.1
0.5 21.9 34.3 531.3 9.4 3.2 6.2
50 50
0.2 66.3ּ103 34.4 93.6 3.2 7.7 8
TOTAL: 166.7ּ103 185.9 950 38.9 24.7 32.7
Table 2
Computational results for Group 2 with n = 100 (max-CPU-time for heuristics = 20 ּ103 ms) Parameters Average objective function values
n ℓ d EXACT PILOT MGA GRASP B-VNS GS-VNS
0.8 1.8 1.8 1.8 1.8 1.8 1.8
0.5 2 2 2 2 2 2
25
0.2 4.5 4.5 4.5 4.5 4.5 4.5
0.8 2 2 2 2 2 2
0.5 3 3.1 3 3 3 3
50
0.2 6.7 6.9 6.7 6.7 6.7 6.7
0.8 3 3 3 3 3 3
0.5 4.7 4.7 4.7 4.7 4.7 4.7
100
0.2 NF 10.2 9.9 9.8 9.7 9.7
0.8 4 4 4 4 4 4
0.5 5.2 5.4 5.2 5.2 5.2 5.2
100
125
0.2 NF 11.3 11.1 11 11 11
TOTAL: - 58.9 57.9 57.7 57.6 57.6
Parameters Computational times (milliseconds)
n ℓ d EXACT PILOT MGA GRASP B-VNS GS-VNS
0.8 9.4 9.4 26.5 0 0 0
0.5 14 14.1 29.7 4.6 1.6 4.5
25
0.2 34.3 21.9 45.3 9.3 1.5 4.8
0.8 17.8 48.3 23.5 6.4 3.2 12.6
0.5 23.5 70.3 106.2 51.6 20.3 21.7
50
0.2 10.2ּ103 67.2 148.3 57.8 22 26.5
0.8 142.8 229.6 254.7 61 132.9 146.9
0.5 2.4ּ103 242.2 300 28.2 59.1 75.9
100
0.2 NF 242.1 9.4ּ103 1.2ּ103 460.8 514
0.8 496.9 340.7 68.7 9.4 9.3 20.2
0.5 179.6ּ103 359.3 759.4 595.4 490.7 345.4 100
125
0.2 NF 353.1 2ּ103 562.9 448.2 1.2ּ103 TOTAL: - 2ּ103 13.2ּ103 2.6ּ103 1.7ּ103 2.4ּ103
Table 3
Computational results for Group 2 with n = 200 (max-CPU-time for heuristics = 60 ּ103 ms) Parameters Average objective function values
n ℓ d EXACT PILOT MGA GRASP B-VNS GS-VNS
0.8 2 2 2 2 2 2
0.5 2.2 2.2 2.2 2.2 2.2 2.2
50
0.2 5.2 5.2 5.2 5.2 5.2 5.2
0.8 2.6 2.6 2.6 2.6 2.6 2.6
0.5 3.4 3.4 3.4 3.4 3.4 3.4
100
0.2 NF 8.3 8.3 8.1 7.9 7.9
0.8 4 4 4 4 4 4
0.5 NF 5.5 5.4 5.4 5.4 5.4
200
0.2 NF 12.4 12.4 12.2 12 12
0.8 4 4.1 4 4.1 4 4
0.5 NF 6.3 6.3 6.3 6.3 6.3
200
250
0.2 NF 14 14 13.9 13.9 13.9
TOTAL: - 70 69.8 69.4 68.9 68.9
Parameters Computational times (milliseconds)
n ℓ d EXACT PILOT MGA GRASP B-VNS GS-VNS
0.8 29.7 42.2 26.5 20.5 0 0
0.5 32.7 92.1 68.8 14.2 10.9 34.4
50
0.2 5.4ּ103 207.9 326.6 37.5 170.4 232.8
0.8 138.6 492.0 139.3 45.3 106.2 140.8
0.5 807.8 746.9 1.6ּ103 176.6 140.5 159.4
100
0.2 NF 851.6 2.2ּ103 667.2 2.5ּ103 2.9ּ103
0.8 22.5ּ103 3.2ּ103 204.6 43.6 29.4 79.7
0.5 NF 3.3ּ103 16.1ּ103 885.6 2.2ּ103 876.1 200
0.2 NF 3.2ּ103 12.7ּ103 9.4ּ103 29.8ּ103 33.7ּ103 0.8 20.6ּ103 5ּ103 2.2ּ103 4.9ּ103 1.4ּ103 1.5ּ103 0.5 NF 5.2ּ103 17.6ּ103 506.0 2.6ּ103 2.3ּ103 200
250
0.2 NF 4.6ּ103 26.4ּ103 1.4ּ103 1.6ּ103 1.5ּ103 TOTAL: - 26.9ּ103 79.6ּ103 18.1ּ103 40.6ּ103 43.4ּ103
Table 4
Computational results for Group 2 with n = 500 (max-CPU-time for heuristics = 300 ּ103 ms) Parameters Average objective function values
n ℓ d EXACT PILOT MGA GRASP B-VNS GS-VNS
0.8 2 2 2 2 2 2
0.5 2.6 2.6 2.6 2.6 2.6 2.6
125
0.2 NF 6.3 6.2 6.2 6.2 6.2
0.8 3 3 3 3 3 3
0.5 NF 4.2 4.3 4.2 4.1 4.1
250
0.2 NF 10 10.1 9.9 9.9 9.9
0.8 NF 4.7 4.7 4.7 4.7 4.7
0.5 NF 6.7 7.1 6.5 6.5 6.5
500
0.2 NF 15.9 16.6 15.9 15.8 15.8
0.8 NF 5.1 5.4 5.1 5.1 5.1
0.5 NF 7.9 8.3 7.9 7.9 7.9
500
625
0.2 NF 18.5 19.1 18.4 18.3 18.3
TOTAL: - 86.9 89.4 86.4 86.1 86.1
Parameters Computational times (milliseconds)
n ℓ d EXACT PILOT MGA GRASP B-VNS GS-VNS
0.8 370 1.2ּ103 18 152 21.9 45
0.5 597 2.9ּ103 2.6ּ103 455 860.8 560
125
0.2 NF 7.3ּ103 57.1ּ103 4ּ103 3.9ּ103 3.7ּ103
0.8 5.3ּ103 20.9ּ103 516 248 67.1 490
0.5 NF 29.6ּ103 28ּ103 583 96.2ּ103 26.9ּ103 250
0.2 NF 30.4ּ103 181.2ּ103 3.3ּ103 16.1ּ103 10.2ּ103 0.8 NF 128.3ּ103 117.5ּ103 28.1ּ103 15.1ּ103 8.6ּ103 0.5 NF 131.9ּ103 170.9ּ103 90.9ּ103 26ּ103 110.2ּ103 500
0.2 NF 115.9ּ103 241.8ּ103 20.2ּ103 235.7ּ103 50.3ּ103 0.8 NF 204ּ103 51.9ּ103 4.9ּ103 67.9ּ103 970 0.5 NF 200.9ּ103 222.2ּ103 35.7ּ103 132.7ּ103 33.9ּ103 500
625
0.2 NF 181.7ּ103 297.8ּ103 53.1ּ103 175.9ּ103 60ּ103 TOTAL: - 1055ּ103 1371.5ּ103 213.8ּ103 770.4ּ103 395.9ּ103
Table 5
Pairwise differences of the average ranks of the algorithms Algorithm
(rank) B-VNS
(1.9) GS-VNS
(2) GRASP
(2.26) PILOT
(4.4) MGA (4.45)
B-VNS (1.9) - 0.1 0.36 2.5 2.55
GS-VNS (2) - - 0.26 2.4 2.45
GRASP (2.26) - - - 2.14 2.19
PILOT (4.4) - - - - 0.05
MGA (4.45) - - -
Figure 1: The top two graphs show a sample graph and its optimal solution. The bottom three graphs show some feasible solutions.
1st ITERATION:
LABEL 1 2 3 4
NUMBER OF COMPONENTS 3 4 4 5
2st ITERATION:
LABEL 1 2 3 4
NUMBER OF COMPONENTS - 2 2 2 3st ITERATION:
LABEL 1 2 3 4
NUMBER OF COMPONENTS - 1 - 1
Figure 2: Example illustrating the steps of the revised MVCA.
Figure 3: Example illustrating the steps of Group-Swap VNS.