Protección de datos

study was conducted as part of the course Model-driven Software De- velopment within the context of the M.Sc. study path held in winter term 2013/14 at Philipps-University Marburg, Germany.

Preliminary Studies In a 45 minutes theoretical lecture we intro-

duced the participants to the topic of software model quality and model quality assurance. In particular, we introduced the quality as- surance techniques model metrics, model smells, and model refactor- ing. In a further 45 minutes lecture we demonstrated the application of existing techniques as well as the specification of new techniques using the functionality of EMF Refactor. Finally, in a 15 minutes in- troduction immediately ahead of the experiment we presented the example language SWM.

Study design We asked participants to specify new model quality

assurance techniques for textual SWM models using EMF Refactor. The grammar of SSM can be found in Listing 6.1 on page 68. The corresponding meta model is shown in Figure13.1on page 138. The

experiment consisted of three main tasks. The time slots for process- ing the tasks were set to 40 minutes each. In task Ex_Spec_M, the participants were asked to specify 5 new SWM metrics of increasing complexity (sub tasksEx_Spec_M_1toEx_Spec_M_5) using either Java or OCL as concrete specification language. In taskEx_Spec_S, the par- ticipants were asked to specify 4 new SWM model smells of increas- ing complexity (sub tasksEx_Spec_S_1toEx_Spec_S_4) using Java as concrete specification language. In task Ex_Spec_R, the participants were asked to specify 3 new SWM model refactorings of increasing complexity (sub tasksEx_Spec_R_1toEx_Spec_R_3) using Java as con- crete specification language. The participants were provided with an Eclipse IDE and a customized version of EMF Refactor including the SWM language. Finally, we provided a brief summary how to use the code generation functionality of EMF Refactor, three predefined plu- gin projects to be used as targets for the appropriate generated code, and an example SWM that can be used for testing purposes.

Data collected We collected several measurements during the ex-

periment. First, in a pre-study questionnaire we asked the participants about their personal skills, more precisely about their proficiency with Java and OCL as well as their experience with EMF. Here, we used a ten-point Likert scale ranging from values 1 (beginner) to 10 (expert). Second, the participants were asked to send the pro- vided Eclipse projects containing the specified techniques after fin- ishing the tasks Ex_Spec_M, Ex_Spec_S, and Ex_Spec_R to the exper- imenter. Third, in an inter-study questionnaire the participants were asked to assess the difficulty of each sub task. Here, we used a ten- point Likert scale ranging from values1(very simple) to10(very dif- ficult). Fifth, in a post-study questionnaire the participants were asked how much they appreciated (1) the possibility to use either Java or OCL for specifying new SWM metrics and (2) the code generation functionality of EMF Refactor. Here, we used a ten-point Likert scale ranging from values 1 (not helpful at all) to 10 (extremely helpful). Finally, we asked all participants to give additional comments on the experiment.

14.2.4 Performance and scalability tests

To evaluate the scalability of the application modules in EMF Refactor, we conducted several performance tests. We performed these tests on a Lenovo ThinkPad W500, Intel Centrino vPro 2.8GHz, 4MB RAM.

Metrics calculation For evaluating the scalability of the metrics cal-

culation module we calculated a selected set of ten UML2 metrics on class models having 100, 500, 1.000, 5.000. 10.000, 50.000 and 100.000

elements and measured the duration of the calculation. Table 14.3 summarizes the selected metrics:

UML2 Metric Description

TNME Total number of elements in the model.

MaxDIT Maximum of all depths of inheritance trees (con- text: model).

MaxHAgg Maximum of aggregation trees (context: model). DNH Depth in the nesting hierarchy (context: package). NATIP Number of inherited attributes in classes within

the package.

NOPIP Number of inherited operations in classes within the package.

HAgg Length of the longest path to the leaves in the aggregation hierarchy (context: class).

MaxDITC Depth of Inheritance Tree (maximum due to mul- tiple inheritance; context: class).

NSUBC2 Number of all children of the class. NSUPC2 Total number of ancestors of the class.

Table 14.3: UML2 metrics used for performance and scalability testing

We selected these metrics to cover inheritance and nesting issues. Note that they are calculated on different context types (model, pack- age, and class). We consider UML2 models only due to the variety of implemented metrics (see Table14.2on pagee157).

Model instances are created using a basic model similar to the ex- ample class model shown in Figure 6.1 on page 56. To provide an adequate model instance of required size we duplicated respectively nested the root package. Doing this, we assure that the number of calculated metrics grows nearly linearly compared to the model size. For each model size, we repeated the metrics calculation ten times and reported the average time needed for metrics calculation.

Smell detection For evaluating the scalability of the smell detection

module we analyzed UML2 class models with 100, 500, 1.000, 5.000. 10.000, 50.000 and 100.000 elements with respect to a set of seven selected smells for UML2 models and measured the time needed for smell detection. The selected smells are:

1. _{Concrete Superclass} - The model contains an abstract class with a concrete superclass.

2. Equal Attributes in Sibling Classes - Each sibling class of a common parent contains an equal attribute.

3. _{Specialization Aggregation} - The model contains a general- ization hierarchy between associations.

4. _{Speculative Generality (Abstract Class)} - The model con- tains an abstract class that is inherited by one single class only. 5. Speculative Generality (Interface)- The model contains an

interface that is implemented by one single class only.

6. Unused Class - The model contains a class that has no child or parent classes, that is not associated to any other classes, and that is not used as attribute or parameter type.

7. Unused Interface- The model contains an interface that is not specialized by another interface, and not realized or used by any classes.

We selected the model smells with respect to their influence on quality aspect confinement (see Section4.2). Smells 1 to 3 use consis- tent language concepts being more complex than necessary. Smells 4 and 6 can also be found in the example case presented in Section6.1. Finally, smells 5 and 7 are similar to smells 4 an 6 but consider inter- faces instead of classes. Again, we consider UML2 model smells only due to the same reasons mentioned above.

The model instances are constructed in the same way as in the metrics calculation case. For each model size, we repeated the smell detection ten times and reported the average time needed for smell detection.

Refactoring execution For evaluating the scalability of the refac-

toring execution module we applied 7 pretty complex UML2 refactor- ings on models with a larger refactoring context instead of large-scale models. Table 14.4 summarizes these refactorings and describes the context each refactoring has been applied on.

We measured the time in-between committing the refactoring (i.e., after parameter input) and finishing the corresponding model change. Moreover, we repeated each refactoring application ten times to ad- dress potential side effects and reported the average time needed for refactoring execution.

14.3 e va l uat i o n r e s u lt s

This section summarizes the results and observations made during the evaluation tasks described in the previous section. The following discussions are guided by the order of goals and hypotheses defined in Section14.1and refer to the corresponding task if appropriate.

Refactoring Context

Extract Class Refactoring application on a class having 10 at- tributes and 10 operations.

Extract Sub- class

Refactoring application on a class having 10 at- tributes and 10 operations. The selected class has 10child classes already. Each child class has 10 at- tributes and 10 operations.

Extract Super- class

Refactoring application on 10 classes having 10 equal attributes and 10 equal operations.

Inline Class Refactoring application on a class having 10 at- tributes and 10 operations.

Introduce Parameter Object

Refactoring application on 9 parameters of an op- eration with 10 input parameters. The owning class has altogether 10 operations with 10 parame- ters each. Each operation has parameters equal to the selected ones.

Merge States Refactoring application on a state with 5 incoming transitions. The parameter state has entry, doAc- tion, and exit behaviour. The prameter state has 5 incoming transitions equal to the selected state. The owning region has 20 further states.

Remove Superclass

Refactoring application on a class having 10 at- tributes and 10 operations. The selected class has 10child classes already. Each child class has 10 at- tributes and 10 operations.

Table 14.4: UML2 refactorings used for performance and scalability testing

14.3.1 Suitability

The goals and hypotheses defined with respect to the suitability of EMF Refactor are evaluated as follows.

G1 – General suitability, correctness, and efficiency of application modules

The functionalities described in Chapters 12 and 13 as well as the entries in Table11.2 on page124show that EMF Refactor fulfills the requirements defined in Chapter 9 and Section 11.1. All basic func- tionalities (metrics calculation, smell detection, and refactoring appli- cation) are provided as well as further functionalities such as con- figurability and metrics reporting. Furthermore, the proof-of-concept implementations summarized in Section14.2.1show that EMF Refac-

tor supports metrics calculation, smell detection, and refactoring of EMF-based models to a high extent.

To analyze whether using the tools prevent from incorrectly per- forming quality assurance tasks we partially use the results of exper- imentEx_App. We start with a summary on the personal skills of the study participants.

Figure 14.1: Personal skills of the participants in experimentEx_App

Figure14.1shows the results of the pre-study questionnaire of exper- imentEx_Appfor both groupsMandT. The left diagram illustrates the average score of given estimations (ranging from1(beginner) to5(ex- pert)) in the context of the participant’s experience in software mod- eling, metrics calculation, model smell detection, and model refactor- ing. The diagram on the right-hand-side illustrates the corresponding median score. Both diagrams show that the participants of experi- ment Ex_App are obviously inexperienced in UML modeling (group M: average and median 2.5 each; groupT: average 2.1 and median 2). Especially, they are inexperienced in applying model quality assur- ance techniques. Here, both average and median scores are less than 2. Moreover, the participants of group T seem to be slightly less ex- perienced than those of group M. This means that the results of the following analysis of the returned forms and Eclipse projects are not influenced by different personal skills of both groups.

Figure 14.2 shows the percentages of correct results of the per- formed tasks Ex_App_M, Ex_App_S, and Ex_App_R. Task Ex_App_S is subdivided into two parts: one part addresses the correctness of the numbers of detected smells for each smell type, the other part ad- dresses the correctness with respect to the number of model smell occurrences. Furthermore, we illustrate four kinds of correctness. For each group M and T we provide the correctness of the returned re- sults concerning both the number of the submitted results and the maximal number of results.

Figure 14.2: Percentages of correct results concerning experimentEx_App

The results provided by groupTare highly correct. In fact, 599 out of 600 submitted (and also maximal possible) metrics and all sub- mitted model smells are correct as well as 49 out of 50 performed refactorings. For group M, only 135 out of the 320 submitted results are correct (42.2% respectively 22.5% according to the maximal num- ber of results). For smell detection and refactoring, these values are slightly better nevertheless far from perfect. In summary, the results show that the use of EMF Refactor (groupT) prevent from incorrectly performing quality assurance tasks as partially done by groupM.

To analyze whether the application tools in EMF Refactor provide results more quickly compared to manually performing the corre- sponding task we also partially use the results of experimentEx_App. Again, we start with an analysis of the returned forms and Eclipse projects.

Figure14.3shows the percentages of the performed tasksEx_App_M,

Ex_App_S, and Ex_App_R. From altogether 600 metrics that should be maximally calculated the participants of group M calculated 320 (53.3%) whereas those of group T calculated each metric. Further- more, in taskEx_App_Sthe participants of groupMsearched for 83 out of 100 possible smell types (83%). Again, the participants of group T searched for each smell type. Finally, the participants of group M performed 49 out of 60 demanded model changes (83.7%). Here, the participants of groupTperformed 59 out of 60 changes (98.3%).

The results show that during the 20 minutes time slot the partic- ipants of group T (which are even less experienced with the topic of the experiment according to the pre-study questionnaire, see discus- sion above) calculated above 87% more metrics than those of group M. Here, the speed-ups for smell detection and refactoring execution are approximately 20% each. However, these values might be even more distinctive if more time would be provided as the results of the inter-study questionnaire of experiment Ex_App show. Here, 5 out of 8 data items (62.5%) provided by the participants of group M claimed that the provided 20 minutes time slots for each task were too short respectively much too short. On the other hand, 6 out of 10 data items (60%) provided by the participants of groupTclaimed that these slots were too long respectively much too long.

G₂ – Suitability of supported specification technologies

The proof-of-concept implementations summarized in Section14.2.1 show that each supported specification approach is suited to specify metrics, smells, and refactorings which can then be used by the corre- sponding application modules. Our experiences in using the various specification approaches show that using Java has been the favorite approach for implementing specifications, especially for implement- ing refactoring specifications. In fact, this may be due to the pref- erences of the designer and the progress of supported approaches by the corresponding tool. Independent of the preferred specification language, we feel confident that OCL is particularly suited for spec- ifying metrics which can be directly deduced from the contextual model element using adequate meta attributes respectively references. Henshin transformations have been proven well-suited especially for specifying the model change part of a refactoring.

H₁ – Guided model quality assurance

To evaluate whether EMF Refactor guides inexperienced students along their way towards improved model quality we consider the results of experiment Ex_App. More concretely, we relate the number

of (correct) applied quality assurance techniques to the personal skills and the noticed severity of the performed tasks.

As shown in the diagram in Figure 14.1 on page 164, the partic- ipants of experiment Ex_App are obviously inexperienced in UML modeling, especially in applying model quality assurance techniques (see discussion on goal G1).

Figure 14.4: Difficulty scores for the tasks in experimentEx_App

Despite of this inexperiences the entries in the inter-study question- naire showed that the difficulties of the tasks in this experiment are at least rated as simple in general (see box plots in Figure14.4). The average score for groupT on the five-point Likert scale ranging from values 1 (very simple) to5 (very difficult) is 1.48 and the median is 1. Here, the analysis tasks are scored as being slightly easier to per- form (average 1.43 and median 1 for taskEx_App_M; average 1.5 and median 1.5 for task Ex_App_S) in relation to task Ex_App_R(average 1.6 and median 1.5). However, the scores of the manual groupM are significantly higher than those of groupT. Here, the average score is 3.13 and the median is 2.5.

These ratings are reflected in the submitted results of the tasks Ex_App_M,Ex_App_S, andEx_App_R(forms and Eclipse projects). We ana- lyze the results in detail in the discussions on goal G1 with respect to

correctness and efficiency. From this discussions, we can summarize that the participants of groupT

• correctly calculated all metrics demanded in taskEx_App_M, • correctly detected all smells demanded in taskEx_App_S, and

• successfully applied 98% of the refactorings demanded in task Ex_App_R.

Finally, in the inter-study questionnaire nearly all ratings (29 out of 30) returned by the participants of group T claimed that the func- tionality of EMF Refactor is either helpful or very helpful to perform the tasks of this experiment. Moreover, more than half of the ratings (16 out of 30) returned by the participants of group M claimed that during the experiment they often respectively always thought that the modeling environment should provide such a functionality (metrics calculation, smell detection, and refactoring).

In summary, the results presented above show that even inexpe- rienced modelers are able to apply a huge number of model qual- ity assurance techniques within a relatively short period of time (60 metric calculations, 23 smell detections, and 5 refactorings within 60 minutes) when using EMF Refactor.

H₂ – Guided specification process and usefulness of code generation

To evaluate whether the code generation facilities provided by EMF Refactor guide students on specifying new metrics, smells, and refac- torings for EMF-based modeling languages and allows to concentrate on the essential specification part only we consider the results of ex- perimentEx_Spec. We start with relating the number of (correct) im- plemented quality assurance techniques to the personal skills and the noticed difficulties of the performed tasks.

Figure 14.5: Personal skills of the participants in experimentEx_Spec

Figure 14.5 shows the results of the pre-study questionnaire of ex- periment Ex_Spec. The diagram illustrates the number of given es- timations per score (ranging from 1 (beginner) to 10(expert)) in the context of the proficiency with Java and OCL and the experience with EMF. It shows that the proficiency with Java is pretty diverse among the participants of the study (average 6.28; median 6). However, the proficiency with OCL is very low (1.88 on average; median 2) and the participants have little to moderate experience with EMF only (aver- age score 3.13; median 3).

Despite of this rather low proficiencies respectively inexperiences the entries in the inter-study questionnaire showed that the difficulties of the tasks in this experiment are rated as moderate in common. The average score on the ten-point Likert scale ranging from values 1 (very simple) to10(very difficult) is 4.5 and the median is 4. Here, the analysis tasks are scored as being easier (average 3.25 and median 3.5 for task Ex_Spec_M; average 3.38 and median 2 for task Ex_Spec_S;) whereasEx_Spec_R is scored as being the most difficult one (average 5.63 and median 6) as expected due to the complexity of refactoring specifications as repeatedly mentioned throughout this thesis.

The returned Eclipse projects containing the specified techniques af- ter finishing tasksEx_Spec_M,Ex_Spec_S, andEx_Spec_R lead to the following results:

• During the 40 minutes time slot of task Ex_Spec_M the partic- ipants implemented 3.6 metrics on average (11.2 minutes per metric on average).

• In 76% of the implemented metrics the participants used Java as specification language, in 12% they used OCL. The possibility to combine two metrics to more complex ones was used by every second participant.

• Just over half (52%) of the implemented metrics were specified correctly. Most of the erroneous implementations are caused by conceptional misunderstandings (8 cases) respectively incorrect usage of Java and OCL (3 cases). However, in 4 cases the code generation module lead to lost code due to overwriting files