As described in the previous two sections, writing the ATL transformations involved making a number of distinctions between elements in the source model and the tar- get model, leaving considerable scope for error. Therefore, we tested our transfor- mations on a number of UML class diagrams and Java programs and, in this section, we describe the approach used to test the transformations and discuss the coverage measures used to ensure that the test suites were adequate.
5.5.1
ATL Coverage Criteria
An ATL transformation includes rules to match elements in the source model as well as instructions to build the elements in the target model. By analogy with a standard grammar-based model such as that used in a yacc/bison file, the ATL transformations include both the rules to match the input and actions to build the output. Based on this analogy, we posit that concepts of coverage in an ATL trans- formation should include rule coverage for the matching part, and traditional code coverage for the generation part. Thus we define:
Rule Coverage. This is analogous to rule coverage in a grammar [Pur72]: it is simply the percentage of rules that were executed at least once during a trans- formation.
Instruction Coverage. This is analogous to line coverage in a high-level language [Bei90], with the additional benefit that formatting and layout do not effect the totals. The instruction coverage for a set of transformations is the percentage of instructions that were executed at least once during the transformation.
Branch Coverage. This measures, for each branch in a program, whether the true and false paths were taken. Thus each branch contributes a figure of 0, 1 or 2 to the total coverage, depending on whether neither branch, just one branch or both branches were taken at any stage during the transformation.
In order to calculate the rule, instruction and branch coverage of an ATL trans- formation it is necessary to profile the operation of each ATL rule as the transfor- mation takes place. Fortunately the design of the ATL system provides two useful features that facilitate this. First, the ATL rules are actually executed on top of a special-purpose virtual machine [JA06]. Second, it is possible to run the ATL sys- tem in debug mode which prints out a step-by-step execution log of instructions on this virtual machine.
The ATL virtual machine is similar in concept to the Java Virtual Machine (JVM) which greatly eases comprehension. It has instructions to access and cre- ate model elements, to manipulate data on the stack, and control instructions for selection, iteration and method calls. Thus, to measure coverage of ATL transfor- mations we have implemented a program or Coverage Analyser that works in two phases:
1. First, we process the file of compiled ATL instructions (represented in XML format) to extract information about the rules or operations, instructions and branch locations and targets
2. Second, we run the transformation and process the resulting log file to record the actual coverage data for that transformation.
We now describe how the three coverage criteria are measured from the log file. Each transformation rule is represented as an operation in the compiled ATL file and is executed on the ATL virtual machine. Therefore, implementing rule coverage involved tracking and recording the calls to the operation corresponding to each rule. The log file produced by running the transformation lists each instruction as it is executed, so it is relatively straightforward to measure this coverage as each instruction corresponds to a line in the ATL transformation file. Branches in an ATL transformation are represented byIFandITERATEinstructions in the compiled ATL
file, and whether they evaluated to true or false can be determined from the log file. For theITERATEinstruction, evaluation to true or false corresponds to iterating over an empty or non-empty collection respectively.
Coverage ATL Standalone Analyser Results ATL formation maximum coverage? Coverage Model formation Software Model Trans− Model Ecore Ecore Measure− ment Metamodel Instances no no yes constraints violated? yes yes no Test Oracle metric values? correct Tested Trans− by−step Execution Log Step−
Figure 5.6: Overview of the approach used to test our model transformation. The input to
the approach is the set of source models and the model transformation and the output is a tested model transformation.
These coverage criteria can then be used to evaluate the adequacy of any test suite used when testing model transformations. Since our measurement approach is concerned with transforming software models to instances of the measurement metamodel, in the next section we outline a strategy for testing model transfor- mations where the target model is a measurement metamodel instance and use the coverage criteria outlined here to ensure that the test suite used is adequate.
5.5.2
Testing Strategy
An overview of the strategy to testing model transformations is depicted in Figure 5.6. Each source model, in our case a UML class diagram or Java program, is used as input to the standalone ATL tool. A successful transformation produces two out- puts, an output model which is an instance of the measurement metamodel in Ecore format and an execution log which is processed to extract coverage information.
For the transformation to be correct the following two criteria must be satisfied:
1. The output model should conform to the measurement metamodel.
2. The calculated metrics for the output model should be the same as the metrics for the source model.
Step 1: Checking Metamodel Conformance:. After a set of input models or test suite are put through the transformation the output models that are produced are checked to ensure they are well-formed according to the OCL constraints for the measurement metamodel. If any constraints are violated then these constraints are inspected and the information is used to identify errors in the transformation. These errors are then corrected and the input models are put through the corrected transformation. This process is repeated until all output models conform to the measurement metamodel.
Step 2: Coverage Analysis:. When all output models conform to the measure- ment metamodel constraints the next step is to examine the transformation cover- age. The execution log produced during the transformation execution is used as input to our Coverage Analyser tool which determines and records the Rule, In- struction and Branch coverage for the test-suite as described earlier in this section. If full coverage has not been achieved then the output from the coverage analyser is used to determine the parts of the transformation that have not been exercised and this information is used to create further source models to cover these items. These models are added to the test suite and the testing process returns to Step 1. How- ever, if full coverage has been achieved then the second correctness criteria can now be checked; the metric values produced for the output models must be the same as those for the input models.
Step 3: Test Oracle Construction:. This step involves the construction of a test oracle for the measurement tool. The test oracle consists of a set of known metric values for each of the input models in the test suite. This test oracle is constructed by calculating and recording the correct metric values for each input model. The results produced by the measurement tool for the output models are then compared with the correct results for the equivalent input model. At present this is all done manually; this is feasible as long as the input models, in this case the class diagrams and Java programs are kept to a reasonable size.