MARCO REGULATORIO DE SOCIEDADES SUBSIDIARIAS Generación

There exist a plethora of commercial performance testing tools. In the fol- lowing, we briefly enumerate a couple of popular performance testing tools.

FABAN is an open source framework for developing and running multi-tier server benchmarks [85]. FABAN has a distributed architecture meaning load can be generated from multiple machines. The tool has three main components: A harness - for automating the process of a benchmark run and providing a container for the benchmark driver code, a Driver frame- work - provides an API for people to develop load drivers, and an Analysis tool - to provide comprehensive analysis of the data gathers for a test. Load is generated by running multiple scripts in parallel.

JMeter [67] is an open source tool for load testing and measuring per- formance, with the focus on web applications. JMeter can be set up in a distributed fashion and load is generated from manually created scenarios that are run in parallel. Httperf [86] is a tool for measuring the performance of web servers. Its aim is to facilitate the construction of both micro and macro-level benchmarks. Httperf can be set up to run on multiple machines and load is generated from pre-defined scripts.

LoadRunner [87] is a performance testing tool from Hewlett-Packard for examining system behavior and performance. The tool can be run in a dis- tributed fashion and load is generated from pre-recorded scenarios. Recently several authors have focused on using models for performance analysis and estimation, as well as for load generation.

Little research has been published in the area of generation of workload models using log data. However, when is comes to log data, some research exists in user clustering and pattern detection.

Kathuria et al. proposed an approach for clustering users into groups based on the intent of the web query or the search string [88]. The proposed approach clusters web queries into one of the three categories based on a K-means algorithm. Vaarandi [89] proposes a Simple Logfile Clustering Tool consequently called SLCT. SLCT uses a clustering algorithm that detects frequent patterns in system event logs. The event logs typically contain log data in various formats from a wide range of devices, such as printers, scanners, routers, etc. The approach is using data mining and clustering techniques to detect normal and anomalous log file lines. Shi [90] presents an approach for clustering user’s interest in web pages using the K-means algorithm. The approach uses fuzzy linguistic variables that describe the time duration that users spend on web pages as the classification variable. These approaches are different from ours in the sense that we assume to know the logging format and we classify users into group based on the request pattern rather that the intent, meaning, or time spent of a web page.

The solutions proposed by Mannila et al. [91] and Ma and Hellerstein [92] are targeted towards discovering temporal patterns from event logs us- ing data mining techniques and various association rules. Both approaches assume a common logging format. Although association rules algorithms are powerful in detecting temporal associations between events, they do not

focus on user classification and workload modeling for performance testing. Another approach is presented by Anastasiou and Knottenbelt [93]. Here, the authors propose a tool, PEPPERCORN, that will infer a performance model from a set of log files containing raw location tracking traces. From the data, the tool will automatically create a Petri Net Performance Model (PNPM). The resulting PNPM is used to make an analysis of the system per- formance, identify bottlenecks, and to compute end-to-end response times by simulating the model. The approach differs from ours in the sense that it operates on different structured data and that the resulting Petri Net model is used for making a performance analysis of the system and not for load generation. In addition, we construct probabilistic time automata (PTA) model from which we later on generate synthetic load.

Lutteroth and Weber describe a performance testing process similar to ours [94]. Load is generated from a stochastic model represented by a form chart. The main differences between their and our approach is that we use different types of models and that we automatically infer our models from log data while they create the models manually.

3.5

Conclusions

In this chapter, we have presented our modeling approach for modeling for performance testing. We have presented the main contributions and explained how they have been validated. We have also shown how our research has been validated in the terms of the tools that have been built and the empirical validation that was made in different experiments. For example, we showed how modeling can raise the abstraction level and hide implementation details. We have also showed that fairly complex models can been generated in a short amount time from small amounts of data and that large amount our unique traces can be generated from very simple models. These results together serve as a basis for answering the questions of how create models from load generation and how to ensure that load generated from the models stays close the load generated by real users.

Chapter 4

Conclusions

In this thesis we have presented an approach for using models for test gen- eration. The presented approach target both functional testing as well as performance testing. We have also sought solutions to the problems ad- dressed Chapter in 1, i.e. that most model-based testing approaches are not integrated with a software development process and that most MBT approaches lack empirical evaluation for an industrial environment. Also, a considerable amount of the reviewed literature lack or only offer partial tool support. To address these issues, we offer tool support that are inte- grated with well-known software development tools or work as stand-alone tools. We also demonstrate the applicability of our approach with empirical evaluations taken from the industry.

For model-based functional testing, we presented the MATERA ap- proach. The MATERA approach include several steps: model creation, model transformation, test generation, test execution. On top of this, re- quirements are traced through all of these steps. The main contribution is the actual modeling approach. Here, we show how the SUT models are built in a stepwise manner and how requirements and statistical information are included and traced to different model elements. This allowed requirement information to be propagated to test generation tools and be used as test generation goals and even test case ordering. Another important contri- bution is tool support. We presented the MATERA tool-set, and showed how it integrates with MagicDraw, one of the leading UML tools on the market. We discussed how the tool aid the tester in different phases, such as, model validation, model transformation, and requirements traceability. Finally, we demonstrated the applicability of our approach and tool support by validating it against an industrial case study. The goal of the case study was to investigate how to apply MBT in an industrial setting. The system under test was a mobile switching server. The main features of the mobile switching server that we investigated was the location update, voice call, and

handover procedures. In total, 40 different UML diagrams were created in order describe the static and dynamic parts the SUT. The UML diagrams contained, among other things, 35 requirements, 5 use cases, 9 sequence diagrams, 88 states, 136 transitions, and over a 100 different classes. This information was then translated into QML, using the MATERA transfor- mation module. From the resulting QML models, a total of 114 test cases were generated. The generated test cases reveled no errors. This was mainly because, in this case study, the generated test cases was executed on pro- duction Mobile Switch Server (MSS) where no errors were expected to be present.

To address the problems of model-based performance testing, we intro- duced MBPeT. MBPeT is a load testing process with assisted tool support. We described how our load testing process make use of workload models described as probabilistic timed automata (PTA). Due to their probabilis- tic nature, PTA models are suitable for representing user behavior. One of the main contribution is the process of creating workload models. We presented three different approaches for creating workload models: (a) au- tomatic derivation of PTA models from UML sequence charts, (b) automatic derivation of PTA models from log data, (c) manual systematic creation of PTA models from system specifications. Another important contribution is our process-assisted tool support. Here, we presented the MBPeT perfor- mance testing tool. MBPeT is a cloud-based performance testing tool that generates load using is distributed architecture. During load generation, the tool measure several key performance indicators, such as, response time, CPU utilization, memory usage, etc. At the end of the test run, the tool also presents a test report with the aggregated data.

Finally, we showed how MBPeT can be used in an industrial setting. We demonstrated the applicability of the tool on an auctioning web service and made a comparison with the well known-load generator JMeter. The comparison showed that MBPeT is on par with JMeter and even sometimes faster. Since the comparison was made, several improvements have been done to MBPeT. We also demonstrated the applicability of the tool support for creating workload models from log data. The experiments showed that PTA models can be generated in a matter of minutes, or even seconds. This is a huge speed-up compared to other manual approaches that requires up many hours in order to produce a workload model that accurately describes user behavior.

4.1

Discussion

In the next section, we acknowledge the limitations in the work presented in this thesis.

One of the most difficult problems to tackle in design science and research in general is the ability to generalize the results. In our research work, we acknowledge the fact that the results of our experiments and finding has been derived from one single case study where only one test system was used. To be able to further generalize our results, our approach would need to be applied on other systems in other domains. However, in an academic setting, developing a methodology and testing it thoroughly, simply lies beyond the capacity and focus of this thesis. Further, obtaining additional interesting and complex industrial case studies for verifying the results of our proposed approach also require the willingness of industrial partner to collaborate in a desirable fashion. We recognize that this is not always easy to achieve. We also recognize that our experiments, in the context of model-based functional testing and performance testing, were performed on different systems under test. Ideally the same test system should have been used. The reason for this is that the two approaches were developed in different stages and different projects were different industrial partners had their own needs and systems.

The work done in the context of model-based functional testing and MATERA was attributed to one particular tool-chain. We note the fact that in order to be able to fully generalize our results, other test genera- tion and execution tools should also be targeted. However, in the context of our work, we have tried to target and integrate our work with leading tools for each part in the testing process. During the development of the process, a lot of effort was put into tool integration. Despite the fact that the models were described using standardized modeling languages, such as UML and SysML, it was not possible to create a UML model using one tool and open it with another tool. Transforming the models between the tools can be used to circumnavigate this problem. Consequently, transfor- mations will be performed frequently, and hence there would be a tool-chain induced performance penalty. In addition, implementing a number of model transformations increases the costs of the tool-chain. The tools used in our tool-chain were all commercial and industry leading tools. Creating adapta- tion for an entire new tool-chain was simply out of the scope for this thesis. The MBPeT tool described in chapter 3 has been developed and tested in a private cloud with only 4 computing nodes. Public clouds, unlike private clouds, are usually preferred by organisations due to their availability and on-demand scalability. Due to the choice of targeting a private cloud, the MBPeT tool has not yet been tested to its limits. For a wider adoption of our research work regarding model-based performance testing, we acknowledge

the fact that we would need to target public clouds as well, for example Amazon EC2 [95]. However, we are limited to only certain clouds, due to that choice of using Python [96] as the implementation language of the MBPeT tool.

The work in the context of model-based performance testing and MBPeT was validate only against http-based systems. To be able to generalize the results in the context of MBPeT, we should also target other protocols and other systems in other domains. We recognize the difficulty in finding collab- oration parters with interesting test systems and performance requirements that meet our needs. For this reason, we decided to implement our own test system for use in our experiments.