To validate our research work we decided to apply our approach on a telecommunications case study. The target system was a Mobile Switch- ing Server (MSS), which acted as the system under test (SUT). In survey on model-based testing approaches, Neto et. al. [15] point out that hardly anyone is integrating MBT approaches with a software development process and that most MBT approaches lack empirical evaluation for an industrial environment. Hence, the reason for selecting the MSS as a SUT was that we wanted to evaluate our approach on a fairly complex system that is used in an industrial environment and also for investigating to what degree UML system models can be used for test generation. Particularly, we wanted to investigate the following aspects:
• How do we create UML models for test generation with as much reuse as possible?
• How are requirements traced from model to tests? • How can requirements be traces from test to models?
An MSS is a telecommunication networks core element and it is re- sponsible for controlling the rest of the network elements [38]. The MSS is connected to its surrounding elements via several different interfaces. Figure 2.16 shows an example of the topology of a telecommunications network.
In the figure, the central element is the MSS. It is connected to several other sub-systems e.g., Base Station Sub-system (BSS) and Radio Network Sub-system (RNS). The BSS is a 2G telecommunications sub-system and consists of several Base Station Controllers (BTC), which in turn controls several Base Transceiver Stations (BTS) [38]. The BTS is a radio tower
!" "# "# $% $% $% #&'$ #&'$ #&'$ $ $
Figure 2.16: Telecommunications network architecture.
equipped with 2G (GSM) technologies for sending and receiving radio signal from various telecommunications devices. Similarly to BSS, the RNS is a 3G telecommunications sub-system and consists of several Radio Network Controllers (RNC), which in turn controls several Node-B’s [39]. A Node-B is a radio tower equipped with 3G (UMTS) technologies for sending and receiving radio signal from various telecommunications devices. Most often the Node-B and the BTS are located on the same physical radio tower but are controlled by different controllers. The Media Gate Way (MGW) is a network element responsible for converting data formats from one network type to another, while the Home Location Register (HLR) is a database containing details about various mobile devices that are authorized to use the network.
The main features of the MSS that we investigated was the location up- date, voice call, and handover procedures. The location updating procedure enables devices to inform the network when they move from one location area to the next. A location area is a small geographical region covered by one radio tower. The voice call procedure is the method by which the network connects two mobile device together that wish to make a call. Han- dover is the procedure that enables mobile devices to move from one location area to another during an ongoing call, without disconnecting the call
Figure 2.17 presented the tool chain that was used in the case study. MagicDraw was the editor used for creating and validating our system mod- els. The MATERA transformation module was used to export our system
models to QML notation. As a test generator, we used Conformiqs Qtronic. Using a test renderer, the generated test were exported to EAST scripts. Finally, we used Nethawk’s EAST tool as a test execution platform.
! " # $%&'(%
Figure 2.17: Tool chain overview.
Modeling. As mentioned above, the MSS had 3 main features, location update, voice call, and handover. These features should be supported both in a 2G network as well as in a 3G network. Figure 2.18 depicts the MSS feature diagram.
For each decomposed feature we created a requirements diagram. Figure 2.19 depicts one of the three requirements diagrams created.
Figure 2.18: MSS feature diagram.
Figure 2.20: MSS use case diagram.
Figure 2.22: MSS domain model.
Once the requirements have been identified and modeled we created use cases and sequence diagrams for each feature. Figure 2.20 shows a use case diagram for the identified use cases, while Figure 2.21 shows a sequence diagram for the location update procedure. Similar diagrams where created for the remaining use cases.
In the next stage, we created a domain model, data models, and state machine models. The domain model shows a static view of the system under test and how it is connected to its environment. The diagram also shows what interfaces it uses for communication with the environment. Figure 2.22 shows an overview of the domain model.
For each interface in the domain, we created a data model. The data model describes the structure and parameter of the messages sent and re- ceived by the system under test. Figure 2.23 depict the data model for mobility management (MM) interface.
Figure 2.23: Data model for the Mobility Management (MM) interface. By overlapping sequence diagram with each other we obtained a set of state machines. In total, we obtained 19 state machine diagrams that to- gether describe the dynamic behavior of the MSS. Figure 2.24 show the state machine diagram for the Location update procedure. The state ma- chine also contains sub-state machines that, for example, specify behavior for sub-routines during the location updating procedure.
Figure 2.24: State machine diagram for Location update procedure.
That last type of diagram type we created was the test configuration diagram. This diagram represent an instantiation of the domain diagram. For example, Figure 2.25 shows a configuration with two test components (TC) connected to one test system (MSS). The two main test components can in turn inherit parameter values from other test components.
In total we created 40 different diagrams to represent the SUT. To give a sense of scope of the modeling effort, table 2.1 below shows a list of created artifacts. Type Amount Requirements 35 Use Cases 5 Sequence diagrams 9 Interfaces 10
State machine diagrams 19
Data Messages 69
States 88
Transitions 136
Table 2.1: List of modeled artefacts
Model Transformation. The entire collection of models represent- ing the SUT from different perspectives was then transformed into QML, a modeling notation understood by the test generator tool Qtronic. Trans- forming the models to QML took around 1 minute. The generated QML state machine were of the equivalent size as the UML state machine, i.e., 88 state and 136 transitions. The generated JAVA part for QML comprised of approximately 700 lines of code.
Test Generation. From the resulting QML models a total of 114 test cases [40] were generated in 30 minutes. Figure 2.26 shows a picture of the Qtronic test generation tool. After the test cases have been generated, the test generation tool can determine the generation order of test cases based on the annotated probability and priority values. For each generated test case, a weighted probability is calculated based on an algorithm implemented by the test generation tool. The weighted probability is calculated from both the use case probability and the requirement priority and determines the sequence in which test cases are ordered.
The generated test cases were rendered into executable EAST [33] test scripts using a scripting backend. The scripting backend is a piece of adapter code for translating abstract test cases into executable test scripts. The scripting backend had to be implemented manually and consisted of roughly 2000 lines of code. The executable tests were of the size 200-400 lines of code, depending on the amount of test step in each test. This clearly shows the
Figure 2.26: Screenshot of the Qtronic test generation tool. benefit of MBT over having to manually write tests.
Test Execution. The test were executed using NetHawk’s Environment for Automated System Testing (EAST) platform. While EAST is executing tests, the output of each test execution is stored in EAST test execution logs. These logs contain detailed information of every test step, executed method, covered requirements, etc. We use this information to keep track of which test that failed during execution and the requirements that were linked to such tests.
Requirements Re-Propagation. The information gathered from the test execution logs is used to trace requirements back to UML models and to update the priority value of those requirements were left uncovered during test execution. In other words, requirements that were linked to test cases that failed during testing. The update procedure is done by incrementally increasing the priority of requirements associated with the failed test cases, such that they will counterbalance the effect that the probabilities of the use cases have on the ordering of tests. As the process is iterated several times, the priority (importance) of requirements will change according to how much they have been tested. This means that, priority values for requirements that need to be tested more thoroughly in a subsequent test iteration are incremented with a predefined coefficient and automatically updated.
In this case study, the executed test cases reveled no errors. This is mainly because, in this case study, the generated test cases was executed on
production Mobile Switch Server (MSS) and we were not expected to find any errors. With this in mind, it is problematic to show the how require- ments are trace back to models when all test were successfully executed. However, after purposely manipulating test execution logs to make them look like tests had failed, we could gather information from failed test cases. Figure 2.27 depict tracing a requirements attached to a failed test case back to UML models.
Figure 2.27: Example of tracing uncovered requirements back to models.