The overall objective of this dissertation was to investigate and improve the current processes and mechanisms that support the automation of quality control for web applications in the client‐side. In order to detail the main contributions of this work I will review the list of detailed objectives which were identified at the beginning of the dissertation, and will explain how the contributions of this thesis address these objectives. ‐ To propose a complete methodology for the automation of software quality control for web applications. In the scope of this piece of research, quality control is understood as the activities aimed to assess the quality of a software system by means of (dynamic) testing and (static) analysis. The proposed methodology in this dissertation provides a generic process to achieve the automation of quality control by automating the navigation of web applications. The first step in this process is to define the correct navigation of the web application under test. This step is necessary since the final aim of quality control is to assess a piece of software, and for this evaluation it is a must to make comparisons. Thus, the definition of the navigation can be seen as the pre‐condition of the approach. I have provided the following ways to model the web navigation for testers: UML models, R&P scripts, and XML files. Regarding UML models, I have selected uses cases, activity diagrams, and presentation diagrams to provide a description for the navigation of a web application. This kind of modelling should be the candidate to automatically assess web applications in which UML is employed in the analysis and design development phases. Regarding R&P scripts, I have selected Selenium IDE scripts. This kind of modelling is useful when web applications are ready, for example in the operation phase.
Finally, the XML files are based on a self‐defined XSD schema, and this kind of modelling is also useful in the analysis and design phases, but this syntax provides more possibilities than UML (for example, test data and oracles can be defined in the XML files).
The structural coverture criterion defined to traverse the navigation has been the all‐paths criterion. Concretely, the all‐links criterion has been employed. In order to find all the paths in the modelled navigation, the web site under test is modelled as a (multi)digraph. Some algorithms and method has been studied to perform this word. Finally, the selected way of traversing a (multi)digraph is the Chinese Postman Tour (CPT). This tour has a strong condition: the digraph should be strongly connected (every node should be reachable from every other node). Due to the fact that a digraph from web navigation can meet or not this condition, the input digraphs are transformed to strongly connected by connecting its leaf nodes to the initial node, which represented the starting point of the navigation. These virtual links are interpreted later as a reset in the navigation, i.e. a way of returning to the initial node to traverse another path.
The final part of the methodology is composed by a study and selection of the quality attributes to be assessed by the approach. The functional and non‐functional requirements of the web under test will be evaluated. Regarding non‐functional requirements, the following quality attributes has been selected: performance, security, compatibility, usability and accessibility.
‐ To analyse the challenges and potential problems of the automated software testing for
web applications.
Automated software testing does not perform magic, and the deployment of such methods demands some kind of input. In this dissertation, the proposed input for the automated testing method is the navigational models as described before (UML, R&P or XML). The first step made to perform automated testing for web applications is deciding which one of the previously selected quality attributes will be ensured by means of testing. These attributes will be functionality, performance, and security.
Regarding functionality, the proposed automated testing approach employs both white‐box and black‐box testing. In the white‐box approach, the correct navigational structure is ensured. Each different state in the navigation is traversed automatically using a real browser. In the black‐box approach, the outcome of the defined oracles in the navigational models is assessed. Regarding performance and security, I employ a black‐box approach. The automated security testing is based on the generation of different attacks from the client to the server, observing the outcome. The automated performance testing approach is based on the simulation of a big number of virtual users which throws concurrent requests to the server. Performance figures such as response time or throughput are calculated. Both automated testing method follows the navigational structure defined before. ‐ Propose a detailed model to perform automated analysis for web applications.
The way in which (static) analysis is employed in this dissertation is by means of automated software analysis. This approach is based on patterns (best practices, patterns, assumptions, bad smell and fault description) to evaluate the system under analysis. In this dissertation, this
kind of analysis method has been employed to assess the compatibility, usability and accessibility of web applications.
These quality attributes are analysed following the general approach of the dissertation: the analysis of each web state in the paths of the navigation. While the traversing of the system under test and analysis is performed, the HTTP traffic is captured. Therefore, the source code (HTML and CSS) of the web pages and the web session is captured. These artefacts are used to perform static analysis by means of specific tools in terms of compatibility, usability and accessibility.
‐ Validate the feasibility of the research approach by means of developing reference
architecture of the proposed methodology.
A reference architecture based on the proposed approach has been defined. As a result, a reference implementation has been created. This implementation has been named Automatic Testing Platform (ATP), and has been released as open‐source using the Apache 2.0 license. In order to create this framework, several existing open‐source libraries and tools has been reused, for example JUnit, Ant, Selenium, JMeter, Wapiti, JTidy, CSSValidator, A‐Checker, or WebSat. In the context of the ICT‐Romulus project, an extension of this tool was created. This framework (also released as open‐source) has been named ATP4Romulus (Automatic Testing Platform for Romulus). The target of ATP4Romulus is web applications created using the Roma Framework. ATP4Romulus’ intellectual property rights have been approved on the 14th March 2011 in the Community of Madrid Registry with the file number M‐001760/2010. ATP’s intellectual property right is currently pending of approval.
Both frameworks have been employed to perform several experiments and case studies which have been used to validate the proposal. The ATP family of tools has proven to be an effective way of automating the quality control of web applications with a minimum human intervention. The target of these experiments has been web application of heterogeneous nature in the client‐side: Struts/JSP (Factur@), XHTML/CSS (EUProjectManager) AJAX/Echo2 (Cornelius), and JSP templates (Scrooge). In the case of Factur@, the testing effort has been compared (since this application is a finished application), and the quality control using ATP reduces the effort in a 96.67%, i.e. 1 PM in the traditional testing approach vs. 0.033 PM with ATP. In addition the amount and nature of the tests and analysis is broader with ATP since it is assessed the functionality, performance, security, compatibility, usability and accessibility. On top of that, these original contributions have been disseminated within European research projects, mainly in the ICT‐ROMULUS and ITEA‐MOSIS deliverables. In addition, the results of this work have been disseminated in several relevant national and international conferences and workshops, as well as international journal and book chapters.
In order to summarize the proposal of this dissertation, Figure 74 is provided. This picture shows a schematic view of the proposed approach to automated quality control (testing and analysis) for web applications in the client‐side. The automation is performed in several steps on the generic testing approach: requirements, design, implementation, and execution. Each part of the steps of this picture has been briefly described in the contributions by objectives depicted before. In addition, this picture shows three components in which human intervention is needed, namely: i) pre‐automation; ii) configuration; iii) post‐automation. The
first step is the definition of the correct navigation structure by using UML, R&P, or XML. Second, some configuration can be carried out in order to customize the approach. In this stage, for example it can selected the browser in which functional testing is performed, the number of concurrent users in performance testing, the attack database to be used in security testing, and the guidelines employed to assess compatibility, usability, and accessibility analysis. In addition, in this step it can be tuned the dictionary of data employed by ATP to generate random data when no test data is defined in the input models. Finally, the post‐ automation step is used to add new input and output data in the Excel spread‐sheets which stored this data.
Figure 74. Dissertation Summary