One of the main issues of mutation analysis is its computational cost [JH11]. This is due to the need to introduce and execute a vast number of mutants. However, in our context the computational cost of mutation analysis is not very important due to the following three reasons. First, we only check whether a mutant violates the Boolean constraints of the mutated models. This is a simple SAT verification process which is actually quite fast. Second, the number of mutants is small. Actually much smaller than the number of interactions, see Table9.1. For 2-wise, the number of mutants is 5 times lower than the number of interactions. As a result, mutation requires less operations than CIT. Third, we do not execute the system. Test selection and evaluation based on the input model is much faster than the actual program execution. Furthermore, in practice, testers will have to verify the program behavior, i.e., resolving the oracle problem, which is a typical manual activity. Hence, human time dominates the computational expenses of the approach.
Finally, it should be mentioned that equivalent mutants, i.e., mutants that can not be killed by any test case do not introduce a big overhead. Actually, the number of such mutants is much less than
Chapter 9. Mutation analysis as a potential alternative to combinatorial interaction testing
the number of invalid pairs (Table9.1). Therefore, the computational cost of mutation is lower than the cost of CIT.
9.6.3
Threats to validity
There are several influencing factors that can threaten the validity of the conducted experiment. Regarding the generalization of the findings, i.e., external validity, it is possible that the selected programs are not representative. This may also be the case for the utilized test suites and the faulty versions. Thus, on larger or other types of programs, mutants and input interactions might not be the most appropriate choice. Similarly, the examined approaches might not being effective in revealing other faults. However, the chosen subjects are real world programs widely used in the literature e.g. [PYCH13, PLT13, YHC13]. Additionally, both the test suites and faults were developed by researchers independently of the present study. The problem we are facing is the absence of programs with high quality test cases, well defined input models or specifications, and faulty versions. Clearly, more studies are in need to answer this concern with confidence.
With respect to the confidence on the reported results, i.e., internal validity, issues on the utilized input models, the employed test sets and the correctness of the used tools can be identified. It is possible that errors on the input models and the used tools may have influenced the reported results. To reduce this threat we performed several manual checks on both the implementation and the employed input models. Here, it must be noted that the input models were independently developed by other researchers [PYCH13]. They were also checked by us in order to give confidence about their correctness. Additionally, as already mentioned, the employed suites are widely used in software engineering experiments e.g. [PYCH13, PLT13,YHC13].
Finally, some threats regarding the evaluation metrics used, i.e., construct validity, can be identified. It is likely that the number of faults found by the approaches do not express the real fault detection ability of the test suites. Additionally, it is possible that the faults number and difficulty can influence the significance of the performed statistical analysis. To reduce this threat, we employed the Kendall τ coefficient which is a non-parametric hypothesis test, i.e,. it does not require a very big sample size, and it measures the similarity of the data order when ranked by the studied effectiveness measures, i.e., the mutants found or the interactions covered. We also measured the correlations after executing every test case of various test suite sizes to eliminate the effects of the test suite size.
9.7
Conclusions
In this chapter, we proposed a mutation analysis approach as an alternative technique to CIT. We conducted a correlation analysis between a) the CIT and b) the mutation approach with their actual fault detection. Our results suggest that our mutants have a stronger correlation with code-level faults than the input interactions of the CIT approach. Therefore, mutation forms a valid measure of the test suites quality.
Part
V
Reverse-engineering and
re-engineering
10
From software product variants
to a software product line: a
preliminary approach
In the previous parts, we presented methods for generating and evaluating configurations. However, a software product line is not always available. The goal of this chapter is to present a preliminary but fully automated approach for reverse-engineering software product lines and their feature model from the source code of software product variants.
This chapter is based on the work that has been published in the following paper:
• Tewfik Ziadi, Christopher Henard, Mike Papadakis, Mikal Ziane, and Yves Le Traon. Towards a language-independent approach for reverse-engineering of software product lines. In Proceedings of the 29th Annual ACM Symposium on Applied Computing, SAC ’14, pages 1064–1071, New York, NY, USA, 2014. ACM
Contents
10.1 Introduction . . . . 128 10.2 The ExtractorPL approach. . . . 128
10.2.1 The banking system example . . . 129 10.2.2 Abstraction of the software product variants . . . 129 10.2.3 Automatic identification of the software product line features . . . 131 10.2.4 Feature code generation . . . 133
10.3 Case study . . . . 133
10.3.1 Accuracy of the extracted software product line (research question 1) . . . 134 10.3.2 Quality of the extracted software product line (research question 2) . . . 135 10.3.3 Threats to validity . . . 137
10.4 Discussion . . . . 137
10.4.1 Benefits . . . 137 10.4.2 Limitations . . . 138
Chapter 10. From software product variants to a software product line: a preliminary approach
10.1
Introduction
SPLE can be implemented as a top-down approach. In that case, features and variability are first specified at the design stage and then software products are built. The top-down process is useful when SPLE is adopted from the beginning. However, current practices are different. Indeed, as reported by Berger et al. [BRN+13], most of the industrial practitioners first implement several software products and then try to manage them. These SPVs are created using ad-hoc techniques, e.g., copy-paste-modify. This is a bad practice leading to a complex management and a low SP quality. Thus, migrating such SPVs to a SPL is the challenge faced by extractive approaches in SPLE [Kru02].
The need for reverse-engineering the full implementation of a SPL. Over the past decade, several studies have investigated such approaches. Some of them deal only with the extraction of features from textual requirements [ASB+08], architectural artifacts [ACC+14, KK12, RPK10], or SPV descriptions [ACP+12]. We advocate that these techniques can go beyond the feature identification step. Indeed, they can refactor and migrate SPVs into a SPL. In other words, a full implementation of a SPL can be reverse-engineered. Therefore, not only features but also their associated assets, e.g, code units, should be identified and extracted.
Contributions of this chapter. This chapter introduces an automated technique, called ExtractorPL, capable of performing a reverse-engineering of a SPL. ExtractorPL infers a full implementation of a SPL given the source code of SPVs. The main challenge of this task is to analyze the source code of the SPVs in order to:
1. Identify the variability among the SPVs, 2. Associate them with features,
3. Regroup the features into a variability model, 4. Map the code units to each feature.
The proposed approach is language-independent and only uses as input the source code of the SPVs. In addition, ExtractorPL is implemented as a publicly available prototype tool and a case study performed on existing SPLs assesses the feasibility and the practicality of the introduced technique. The remainder of this chapter is organized as follows. Section 10.2 and10.3 respectively present the ExtractorPL approach and a case study to evaluate it. Section 10.4discusses the benefits and the limitations of the approach. Finally, Section 10.5concludes the chapter and present our future work.