El principio rebus sic estantibus

Although a necessary amount of data has been collected to validate the benefits of CadaML, a limited amount of time (i.e., 1 hour) was given to the participants to develop the experiment tasks. As a result, the participants who were assigned to the manual approach could not accomplish the experiment tasks in an hour of time. This may lead to the threat that the participants assigned to the manual approach may thought the necessity to complete the experiment tasks in the allocated time without guaranteeing the validity of their code. Regardless the time constraint, the experienced participants stated that it would take them 2–4 hours to manually implement the tasks.

Another threat is that external factors such as participants’ knowledge and ex- perience in cloud applications and data storage implementation might also influence the experiment outcome. This threat is alleviated by the fact that the participants were as fairly allocated as possible for both approaches without considering their expertise level. Therefore, we can conclude that with CadaML less time is spent to implement the data layer of the given use case, and with less errors appearing in the generated application code.

5.11

Summary

This chapter evaluatedCadaMLagainst manual code re-factoring, and quantified the benefits of the modeling language. As a use case for the evaluation, an industrial business process analyzing application is evolved by deploying the application data to a combination of di↵erent cloud storage services. The evaluation is conducted following the controlled experiment design with the task analysis technique.

For the experiment, a number of developers with varying familiarity with Java, cloud application and data layer implementation has been recruited. The devel- opers are allocated to implement the experiment task either exploiting CadaML or

manually. During the evaluation, the productivity of developers, reliability of the generated code, and usability of the modeling language are assessed. The productiv- ity is estimated by calculating the completion rate of the evaluation tasks, and time required to model/implement the data architecture. The reliability is measured by debugging and testing the generated/written code against several test cases. The usability is analyzed through an interview and open-ended questions at the end of the evaluation.

Through the evaluation, we have demonstrated that exploiting CadaML can sig- nificantly reduce the time and e↵ort – by a factor of 4–7 – to implement the data architecture, and decrease number of errors in the application code. We also dis- cussed limitations that are gathered through the exit interview, and considered threats to our study.

Discussions & Conclusions

In this chapter, we summarize the thesis, describe the significance of our findings, discuss limitations of CadaML, and present new insights that emerged as a result of this research.

6.1

Thesis Summary

The aim of this thesis is to facilitate implementation of multi-tenancy in the data layer of cloud applications using MDE techniques. The research, to achieve this aim, is developed following the three research phases: 1) literature review of the past and current approaches that are geared towards addressing multi-tenancy concerns at the data layer; 2) development of CadaML, its graphical editor and modeling environment; and 3)evaluationof CadaMLin order to identify its application benefits. The purpose of the literature review phase is to analyze information gathered by identifying gaps and limitations in the related research, and formulating require- ments for CadaML. Chapter 2 examines the existing literature and identifies a set of (i) challenges of developing multi-tenancy at the data layer; (ii) current cloud modeling languages and their important features; and (iii) their weaknesses and limitations.

The outcome of the literature review is expanded in development phase to con- verge on requirements for CadaML and to build these requirements into an actual modeling language. Chapter 3, firstly, specifies the requirements along with as- sociated rationale for concepts that need to be included in CadaML, and for a meta-modeling language to implement the language. The chapter, then, identi- fies a methodology to develop CadaML through analyzing the existing literature on DSL development. Following the methodology, the specified requirements are trans- formed into a modeling language with a graphical editor. This was accomplished by analyzing di↵erent cloud storage solutions o↵ered at the PaaS provisioning level by four major cloud service providers, designing a meta-model based on the do-

main knowledge gathered through the analysis, and implementing CadaML and its modeling environment.

The evaluation phase is reflected in Chapters 4 and 5, where the modeling lan- guage is investigated through (1) a qualitative evaluation based on a case study; and (2) an experimental evaluation which involved a controlled user study and an in-depth interview with developers who have exploited CadaML.

The evaluation is performed on evolution of an industrial web application to adopt multi-tenancy, and further evolution of the data layer to store application data in di↵erent cloud data storage. The qualitative evaluation explores the appli- cation of CadaMLto a real-world industrial application, and demonstrates adequacy of multi-tenancy implementation. While the experimental evaluation provides quan- titative assessment of the benefits of CadaML in terms of well usability of the lan- guage, increased productivity of developers, and improved reliability of the data access layer code. Hence, these chapters present evidence that CadaML e↵ectively facilitates implementation of multi-tenancy at the data layer by engaging develop- ers with diverse programming and cloud application development background, and o↵ering concepts and notations that enable developers to build a multi-tenant data architecture in a cloud provider agnostic way.