Conversión al sistema decimal errónea

In this work, we presented techniques for enabling testing of SPLs based on FMs using SB approaches combined with constraint solvers.

In the mono-objective configuration generation part, we have introduced a scalable technique based on a similarity heuristic for generating configurations with respect to combinatorial testing. By providing a partial but scalable t-wise approach, the proposed approach outperforms the state of the art tools and allow to scale to large SPLs encompassing more than 6,000 features and with high interaction strengths (t ≥ 6). We then introduced the first mutation-based approach for generating configurations.

In the third part, we have extended the configuration approaches to support multiple testing objectives. The first chapter of this part introduced a significant improvement over the state of the art to support multiple objectives targeting a single configuration. The following chapter has introduced an improvement to the multi-objective generation technique by supporting objectives target a whole configuration suite. Both these techniques have been evaluated on a large set of SPLs and proven to be effective.

The fourth part was focusing and the evaluation of a given configuration suite. We have first introduced a mutation based approach and linked the mutation criterion to the similarity heuristic introduced in Chapter4. The following chapter has evaluated the correlation of the mutation criterion with fault detection, demonstrating its ability to be an alternative to CIT.

In the fifth part, the first chapter has introduced an approach for reverse-engineering a SPL and its FM from the source code of SPVs. It is the first fully automated and language-independent technique. Chapter 11has introduced a search-based technique for automatically evaluating whether reverse- engineered FMs were reflecting the system they model, and when it is not the case, automatically fix them. The approach has been evaluated on Linux and proved its ability to fix actual errors in the constraints of the FM.

Finally, the previous chapter has introduced an industrial application of the proposed configuration and evaluation techniques to the CETREL company. By modeling credit card authorizations as a SPL FM, we enabled the use of the introduced methods, thus reducing the testing effort as long as increasing the level of confidence in the test test suites used by the experts.

13.2

Future work and open research questions

This section describes potential future research directions.

13.2.1

Search-based techniques applied to software product lines

Improving the search process. Future work will investigate alternative ways to improve the search process. Specifically, practices like parameter tuning [AF13], supervised search [JCHP13] or hybrid

13.2. Future work and open research questions

approaches involving both constraint-driven and genetic search will be considered. The objectives are to (1) investigate the effects of the parameters on the effectiveness and efficiency of the proposed approach, (2) Find criteria for deciding when to stop the evolution process.

Search-based approaches for SPLs as a service. In this dissertation, we proposed several approaches based on search-based techniques. Although we made the implementations publicly available in most of the cases, it is not that easy for the community or industrial to reuse them. In this context, we plan to include them in framework or application programming interface where the user could set his own fitness functions and testing objectives and select different techniques to apply. This framework will also allow to compare different search techniques for a specific problem.

Beyond Boolean FMs. One limitation of the work presented in this dissertation is that it relies of FMs. These models are Boolean and thus cannot encompass complex and non-Boolean constraints. In future work, we plan to work with SMT solvers in order to handle other type of constraints. In particular, we will analyze whether flattening non-Boolean models to Boolean ones as an impact of the solving process.

13.2.2

Combinatorial interaction testing in practice

CIT in the cloud. Recently, the zcov platform has been released, proposing combinatorial testing for SPLs as a service using the cloud of Amazon and parallel computing. The first performances results are promising, managing to generate the 3-wise interactions for Linux in less than 8 hours, according to the last reporti. It could be interesting to design or adapt our search-based techniques for these architecture and compare them with CIT techniques.

Interaction faults in practice. There are too few approaches which evaluate the interaction faults in practice. One limitation of the proposed approaches is that they do not evaluate whether they can in practice find interaction faults. Thus, there is no need to evaluate the proposed approaches in practice, and also to compare with other techniques.

A repository with SPL including interaction faults To support the previous future work, we aim at providing a SPL repository containing code of SPLs with actual interaction faults. It requires the manual of interactions faults or the collect of existing ones. Such a repository will be useful to the community for applying their approaches or comparing them with other techniques.

i_{http://zcov.net/2015/01/15/beta-3-wise-performance/}

List of papers, tools & services

Papers included in the dissertation: • 2015

– Christopher Henard, Mike Papadakis, Mark Harman, and Yves Le Traon. Combining multi-

objective search and constraint solving for configuring large software product lines. In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE 2015), 2015

• 2014

– C Henard, M Papadakis, G Perrouin, J Klein, P Heymans, and Y Le Traon. Bypassing the

combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans. Software Eng., 40(7):650–670, 2014

– Christopher Henard, Mike Papadakis, and Yves Le Traon. Mutation-based generation of software

product line test configurations. In Claire Le Goues and Shin Yoo, editors, Search-Based Soft- ware Engineering, volume 8636 of Lecture Notes in Computer Science, pages 92–106. Springer International Publishing, 2014

– Tewfik Ziadi, Christopher Henard, Mike Papadakis, Mikal Ziane, and Yves Le Traon. Towards a

language-independent approach for reverse-engineering of software product lines. In Proceedings of the 29th Annual ACM Symposium on Applied Computing, SAC ’14, pages 1064–1071, New York, NY, USA, 2014. ACM

– Mike Papadakis, Christopher Henard, and Yves Le Traon. Sampling program inputs with mutation analysis: Going beyond combinatorial interaction testing. In Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation, ICST ’14, pages 1–10, Washington, DC, USA, 2014. IEEE Computer Society

– Christopher Henard, Mike Papadakis, and Yves Le Traon. Mutalog: A tool for mutating logic

formulas. In Proceedings of the 2014 IEEE International Conference on Software Testing, Veri- fication, and Validation Workshops, ICSTW ’14, pages 399–404, Washington, DC, USA, 2014. IEEE Computer Society

• 2013

– Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. Multi-

objective test generation for software product lines. In Proceedings of the 17th International Software Product Line Conference, pages 62–71. ACM, 2013

– Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. Towards

automated testing and fixing of re-engineered feature models. In Software Engineering (ICSE), 2013 35th International Conference on, pages 1245–1248. IEEE, 2013

– Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. Assessing

software product line testing via model-based mutation: An application to similarity testing. In Software Testing, Verification and Validation Workshops (ICSTW), 2013 IEEE Sixth International Conference on, pages 188–197. IEEE, 2013

– Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. Pledge:

A product line editor and test generation tool. In Proceedings of the 17th International Software Product Line Conference Co-located Workshops, SPLC ’13 Workshops, pages 126–129, New York, NY, USA, 2013. ACM

List of papers, tools & services

Papers not included in the dissertation:

• Antonia Bertolino, Said Daoudagh, Donia El Kateb, Christopher Henard, Yves Le Traon, Francesca Lonetti, Eda Marchetti, Tejeddine Mouelhi, and Mike Papadakis. Similarity testing for access control. Information and Software Technology, 58:355–372, 2015

• Christopher Henard, Mike Papadakis, and Yves Le Traon. Flattening or not of the combinatorial interaction testing models? In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on, pages 1–4. IEEE, 2015

Papers published prior PhD:

• Thibault Cholez, Christopher Henard, Isabelle Chrisment, Olivier Festor, Guillaume Doyen, and Rida Khatoun. A first approach to detect suspicious peers in the kad p2p network. In Network and Information Systems Security (SAR-SSI), 2011 Conference on, pages 1–8. IEEE, 2011

Software developed during PhD:

• PLEDGE: A product line editor and configuration generation tool (http://research.henard.net/ SPL/PLEDGE/)

• MutaLog: A tool for mutating logic formulas (http://research.henard.net/SPL/MutaLog/)

• ExtractorPL: A tool for reverse-engineering a SPL from software product variants (http:// pagesperso-systeme.lip6.fr/Tewfik.Ziadi/sac14/)

Services:

• Journal reviewer: STVR (in 2014), IST (in 2015) • Journal co-reviewer: SQJ (in 2014)

• Conference co-reviewer: ISARCS 2012, ICST 2012 & 2013, SAC SVT 2013, ISSRE 2013, ICSE SEIP 2013, MODELS 2013 & 2014, SPLC 2014, ICSE 2014, ICSE NIER 2014.

• Workshop reviewer: JLDP 2012, Mutation 2014 • Webmaster of the ICST 2013 conference website

Bibliography

[AB11] Andrea Arcuri and Lionel Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pages 1–10, New York, NY, USA, 2011. ACM.

[AB12] Andrea Arcuri and Lionel Briand. Formal analysis of the probability of interaction fault detection using random testing. IEEE Trans. Softw. Eng., 38(5):1088–1099, September 2012.

[ABBJ14] Mathieu Acher, Benoit Baudry, Olivier Barais, and Jean-Marc Jézéquel. Customization and 3d printing: A challenging playground for software product lines. In Proceedings of the 18th International Software Product Line Conference - Volume 1, SPLC ’14, pages 142–146, New York, NY, USA, 2014. ACM.

[ABHPW10] Shaukat Ali, Lionel C. Briand, Hadi Hemmati, and Rajwinder Kaur Panesar-Walawege. A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng., 36(6):742–762, November 2010.

[ABL05] J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In Proceedings of the 27th International Conference on Software Engineering, ICSE ’05, pages 402–411, New York, NY, USA, 2005. ACM.

[ABLN06] J.H. Andrews, L.C. Briand, Y. Labiche, and A.S. Namin. Using mutation analysis for assessing and comparing testing coverage criteria. Software Engineering, IEEE Transactions on, 32(8):608–624, Aug 2006.

[ACC+_{11] Mathieu Acher, Anthony Cleve, Philippe Collet, Philippe Merle, Laurence Duchien, and}

Philippe Lahire. Reverse engineering architectural feature models. In Proceedings of the 5th European Conference on Software Architecture, ECSA’11, pages 220–235, Berlin, Heidelberg, 2011. Springer-Verlag.

[ACC+14] Mathieu Acher, Anthony Cleve, Philippe Collet, Philippe Merle, Laurence Duchien, and Philippe Lahire. Extraction and evolution of architectural variability models in plugin-based systems. volume 13, pages 1367–1394. Springer Berlin Heidelberg, 2014.

[ACP+12] Mathieu Acher, Anthony Cleve, Gilles Perrouin, Patrick Heymans, Charles Vanbeneden, Philippe Collet, and Philippe Lahire. On extracting feature models from product de- scriptions. In Proceedings of the Sixth International Workshop on Variability Modeling of Software-Intensive Systems, VaMoS ’12, pages 45–54, New York, NY, USA, 2012. ACM.

[AF13] Andrea Arcuri and Gordon Fraser. Parameter tuning or default values? an empiri- cal investigation in search-based software engineering. Empirical Software Engineering, 18(3):594–623, 2013.

[AHH11] Ebrahim Khalil Abbasi, Arnaud Hubaux, and Patrick Heymans. A toolset for feature-based configuration workflows. In Proceedings of the 2011 15th International Software Product Line Conference, SPLC ’11, pages 65–69, Washington, DC, USA, 2011. IEEE Computer Society.

[AKL09] Sven Apel, Christian Kastner, and Christian Lengauer. Featurehouse: Language- independent, automated software composition. In Proceedings of the 31st International Conference on Software Engineering, ICSE ’09, pages 221–231, Washington, DC, USA, 2009. IEEE Computer Society.

[ASB+08] Vander Alves, Christa Schwanninger, Luciano Barbosa, Awais Rashid, Peter Sawyer, Paul Rayson, Christoph Pohl, and Andreas Rummler. An exploratory study of information retrieval techniques in domain analysis. In Proceedings of the 2008 12th International Software Product Line Conference, SPLC ’08, pages 67–76, Washington, DC, USA, 2008. IEEE Computer Society.

Bibliography

[Bat05] Don Batory. Feature models, grammars, and propositional formulas. In Proceedings of the 9th International Conference on Software Product Lines, SPLC’05, pages 7–20, Berlin, Heidelberg, 2005. Springer-Verlag.

[BC07] Renée C. Bryce and Charles J. Colbourn. One-test-at-a-time heuristic search for interaction test suites. In Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, GECCO ’07, pages 1082–1089, New York, NY, USA, 2007. ACM.

[BDEK+15] Antonia Bertolino, Said Daoudagh, Donia El Kateb, Christopher Henard, Yves Le Traon, Francesca Lonetti, Eda Marchetti, Tejeddine Mouelhi, and Mike Papadakis. Similarity testing for access control. Information and Software Technology, 58:355–372, 2015. [BFN08] Dimo Brockhoff, Tobias Friedrich, and Frank Neumann. Analyzing hypervolume indicator

based algorithms. In Proceedings of the 10th International Conference on Parallel Problem Solving from Nature: PPSN X, pages 651–660, Berlin, Heidelberg, 2008. Springer-Verlag.

[BM07] Renée C. Bryce and Atif M. Memon. Test suite prioritization by interaction coverage. In Workshop on Domain Specific Approaches to Software Test Automation: In Conjunction with the 6th ESEC/FSE Joint Meeting, DOSTA ’07, pages 1–7, New York, NY, USA, 2007. ACM.

[BP10] Daniel Le Berre and Anne Parrain. The sat4j library, release 2.2. JSAT, 7(2-3):59–6, 2010. [BRN+_{13] Thorsten Berger, Ralf Rublack, Divya Nair, Joanne M. Atlee, Martin Becker, Krzysztof}

Czarnecki, and Andrzej Wąsowski. A survey of variability modeling in industrial practice. In Proceedings of the Seventh International Workshop on Variability Modelling of Software- intensive Systems, VaMoS ’13, pages 7:1–7:8, New York, NY, USA, 2013. ACM.

[BRR10] Christian Berger, Holger Rendel, and Bernhard Rumpe. Measuring the ability to form a product line from existing products. In Fourth International Workshop on Variability Modelling of Software-Intensive Systems, Linz, Austria, January 27-29, 2010. Proceedings, pages 151–154, 2010.

[BSL+10] Thorsten Berger, Steven She, Rafael Lotufo, Andrzej Wąsowski, and Krzysztof Czarnecki. Variability modeling in the real: A perspective from the operating systems domain. In Pro- ceedings of the IEEE/ACM International Conference on Automated Software Engineering, ASE ’10, pages 73–82, New York, NY, USA, 2010. ACM.

[BSL+_{12] Thorsten Berger, Steven She, Rafael Lotufo, Andrezj Wasowski, and Krzysztof Czarnecki.}

Variability modeling in the systems software domain. Technical report, Generative Software Development Laboratory, University of Waterloo, 2012.

[BSPM11] RenéeC. Bryce, Sreedevi Sampath, JanB. Pedersen, and Schuyler Manchester. Test suite prioritization by cost-based combinatorial interaction coverage. International Journal of System Assurance Engineering and Management, 2(2):126–134, 2011.

[BSRC10] David Benavides, Sergio Segura, and Antonio Ruiz-Cortés. Automated analysis of feature models 20 years later: A literature review. Inf. Syst., 35(6):615–636, September 2010. [BTRC05] David Benavides, Pablo Trinidad, and Antonio Ruiz-Cortés. Automated reasoning on fea-

ture models. In Proceedings of the 17th International Conference on Advanced Information Systems Engineering, CAiSE’05, pages 491–503, Berlin, Heidelberg, 2005. Springer-Verlag.

[BYL+12] Mehra N. Borazjany, Linbin Yu, Yu Lei, Raghu Kacker, and Rick Kuhn. Combinatorial testing of acts: A case study. In Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, ICST ’12, pages 591–600, Washington, DC, USA, 2012. IEEE Computer Society.

[CCL03] Myra B. Cohen, Charles J. Colbourn, and Alan C. H. Ling. Augmenting simulated annealing to build interaction test suites. In Proceedings of the 14th International Symposium on Software Reliability Engineering, ISSRE ’03, pages 394–, Washington, DC, USA, 2003. IEEE Computer Society.

Bibliography

[CCMJ12] Stephen Creff, Joel Champeau, Arnaud Monegier, and Jean-Marc Jézéquel. Relationships formalization for model-based product lines. In Proceedings of the 2012 19th Asia-Pacific Software Engineering Conference - Volume 01, APSEC ’12, pages 158–163, Washington, DC, USA, 2012. IEEE Computer Society.

[CDFP97] David M. Cohen, Siddhartha R. Dalal, Michael L. Fredman, and Gardner C. Patton. The aetg system: An approach to testing based on combinatorial design. IEEE Trans. Softw. Eng., 23(7):437–444, July 1997.

[CDH13] John A. Clark, Haitao Dan, and Robert M. Hierons. Semantic mutation testing. Sci. Comput. Program., 78(4):345–363, April 2013.

[CDS06] Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. Coverage and adequacy in software product line testing. In Proceedings of the ISSTA 2006 Workshop on Role of Software Architecture for Testing and Analysis, ROSATEA ’06, pages 53–63, New York, NY, USA, 2006. ACM.

[CDS07] Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. Interaction testing of highly- configurable systems in the presence of constraints. In Proceedings of the 2007 International Symposium on Software Testing and Analysis, ISSTA ’07, pages 129–139, New York, NY, USA, 2007. ACM.

[CDS08] Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Trans. Softw. Eng., 34(5):633–650, September 2008.

[CG08] Andrea Calvagna and Angelo Gargantini. A logic-based approach to combinatorial testing with constraints. In Proceedings of the 2Nd International Conference on Tests and Proofs, TAP’08, pages 66–83, Berlin, Heidelberg, 2008. Springer-Verlag.

[CH06] K. Czarnecki and S. Helsen. Feature-based survey of model transformation approaches. IBM Syst. J., 45(3):621–645, July 2006.

[CHC+_{11] Thibault Cholez, Christopher Henard, Isabelle Chrisment, Olivier Festor, Guillaume Doyen,}

and Rida Khatoun. A first approach to detect suspicious peers in the kad p2p network. In Network and Information Systems Security (SAR-SSI), 2011 Conference on, pages 1–8. IEEE, 2011.

[CHS08] Andreas Classen, Patrick Heymans, and Pierre-Yves Schobbens. What’s in a feature: A requirements engineering perspective. In Proceedings of the Theory and Practice of Soft- ware, 11th International Conference on Fundamental Approaches to Software Engineering, FASE’08/ETAPS’08, pages 16–30, Berlin, Heidelberg, 2008. Springer-Verlag.

[CHW98] J. Coplien, D. Hoffman, and D. Weiss. Commonality and variability in software engineering. Software, IEEE, 15(6):37–45, Nov 1998.

[Cli96] Norman Cliff. Ordinal Methods for Behavioral Data Analysis. Psychology Press, New-York, USA, September 1996.

[CN01] Paul C. Clements and Linda Northrop. Software Product Lines: Practices and Patterns. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2001.

[DAVV98] Gary B. Lamont David A. Van Veldhuizen. Multiobjective evolutionary algorithm research: A history and analysis. Technical report, tr-98-03, Department of Electrical and Computer Engineering, Air Force Institute of Technology, 1998.

[DDH+13] Jean-Marc Davril, Edouard Delfosse, Negar Hariri, Mathieu Acher, Jane Cleland-Huang, and Patrick Heymans. Feature model extraction from large collections of informal product descriptions. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, pages 290–300, New York, NY, USA, 2013. ACM.

[DER05] Hyunsook Do, Sebastian Elbaum, and Gregg Rothermel. Supporting controlled experi- mentation with testing techniques: An infrastructure and its potential impact. Empirical Softw. Engg., 10(4):405–435, October 2005.

Bibliography

[DJW02] Stefan Droste, Thomas Jansen, and Ingo Wegener. On the analysis of the (1+ 1) evolu- tionary algorithm. Theor. Comput. Sci., 276(1-2):51–81, April 2002.

[DMB11] Leonardo De Moura and Nikolaj Bjørner. Satisfiability modulo theories: Introduction and applications. Commun. ACM, 54(9):69–77, September 2011.

[DMSNCMM+_{11] Paulo Anselmo Da Mota Silveira Neto, Ivan Do Carmo Machado, John D. Mcgregor,}

Eduardo Santana De Almeida, and Silvio Romero De Lemos Meira. A systematic mapping study of software product lines testing. Inf. Softw. Technol., 53(5):407–423, May 2011. [DN11] Juan J. Durillo and Antonio J. Nebro. jmetal: A java framework for multi-objective

optimization. Adv. Eng. Softw., 42(10):760–771, October 2011.

[DO91] Richard A. DeMillo and A. Jefferson Offutt. Constraint-based automatic test data generation. IEEE Trans. Softw. Eng., 17(9):900–910, September 1991.

[DPAM02] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: Nsga-ii. Trans. Evol. Comp, 6(2):182–197, April 2002.

[DR06] Hyunsook Do and Gregg Rothermel. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans. Softw. Eng., 32(9):733–752, September 2006.

[EBG12] Faezeh Ensan, Ebrahim Bagheri, and Dragan Gašević. Evolutionary search-based test generation for software product line feature models. In Proceedings of the 24th International Conference on Advanced Information Systems Engineering, CAiSE’12, pages 613–628, Berlin, Heidelberg, 2012. Springer-Verlag.

[ER11] Emelie Engström and Per Runeson. Software product line testing - a systematic mapping study. Inf. Softw. Technol., 53(1):2–13, January 2011.

[FHLS98] P.G. Frankl, R.G. Hamlet, Bev Littlewood, and L. Strigini. Evaluating testing methods by delivered reliability [software]. Software Engineering, IEEE Transactions on, 24(8):586–601, Aug 1998.

[FKBA07] P. Frenzel, R. Koschke, A.P.J. Breu, and K. Angstmann. Extending the reflexion method