El Diálogo

The research onion methodology defines several layers representing different methods and stances for scientific research1. The first layer is the philosophical stance. Defining

A.2 The Research Onion Methodology

a philosophical stance at the beginning of a research is important to explicit what is considered as acceptable evidence [71]. Examples of philosophical stances are positivism, realism and constructivism. The second layer is the approach used to devise a theory, which might be from practice to theory (inductive) or the opposite (deductive) [213].

The third layer refers to the strategy to investigate the hypothesis. Several empirical methods can be used, such as experiment, survey and case study [212]. It is important that the strategy choice is driven by the research objective and other research onion layers [71]. For instance, experiments are the common strategy for positivism using a deductive approach. The fourth layer is the choice regarding the methods to be used. Whereas quantitative methods use numerical data and statistical methods to analyse findings, qualitative methods use raw data (e.g., text and images) and do not rely on precise measures to draw conclusions [212].

The next layer refers to time horizons. They vary according to time of study, which might be short-term (cross-sectional) or long (longitudinal). Whereas in the former groups are evaluated at the same point in time, in the latter different groups are evaluated along the time [126]. Finally, the sixth layer covers techniques and procedures for data collection (e.g., forms) and analysis (e.g., non-parametric statistics). These techniques and procedures vary significantly according to the previous choices [256].

Appendix B

Experimentation in Software

Engineering

This appendix briefly summarises fundamental concepts of experimentation in Software Engineering (SE). These concepts are extensively used in Chapters 3, 4 and 5 to ad- dress the research objectives presented in Section 1.2. A (controlled) experiment is an empirical method to determine whether a cause-effect relationship exists [71]. A quasi-experiment is an experiment whereby treatments are not randomly assigned to participants [121]. As Section 2.4.1 explains, experimentation is a common strategy to identify factors that impact a particular QA [7, 21, 38, 40, 112]. Therefore, this is the strategy used in this research to identify factors impacting on cloud application portability.

There are several guidelines for conducting and reporting experiments in SE [118, 128, 131, 214]. This research used the framework proposed by Wohlin et al. [256] as it covers the experiment preparation, execution and result analysis in the depth required by this research. In addition, we use the Goal/Question/Metric (GQM) template of Basili & Rombach [27] to define the goal of each experiment. Figure B.1 illustrates the main elements of the experimentation framework of Wohlin et al..

An experiment investigates a Hypothesis that defines a causal relationship between Dependent and Independent variables. Usually, an independent variable (also called Factor ) consists of two or more values, called Treatments. For example, stating that a Java application (Treatment 1) is easier to debug (Dependent variable) than a C++ application (Treatment 2) is a valid Hypothesis.

Figure B.1: SE experimentation framework of Wohlin et al. [256].

A Variable must be measurable. There are different types of measures, but the most common are categorical and ratio. In our example, our Independent variable is measured in a categorical scale since we have two possible values: Java and C++. Although the Dependent variable is informally specified (“easier to debug”), at some point it must be formally specified. For instance, let’s assume that the ease of debugging an application is measured in terms of the amount of time that a developer takes to debug a particular problem. “Amount of time” is a data type that varies from zero to a number, which means that this is a ratio variable. Thus, one can objectively compare the time for debugging the two applications.

A Hypothesis can be classified into Null or Alternative. Usually, the Null Hypothesis states that there is no difference between Treatments whereas the Alternative Hypothesis states that there is some difference. For instance, the Null Hypothesis could state that the amount of time that a developer takes to debug a Java application is equals to a C++ application whereas the Alternative Hypothesis could state that the Java application takes less time. Note that the Alternative Hypothesis is what we are trying to show, according to the initial hypothesis defined in the previous paragraph. This is the case in most experiments.

Some Data Analysis techniques are used to evaluate the Null Hypothesis and define whether it should be rejected or accepted. In the case of the Null Hypothesis rejection, the Alternative Hypothesis is accepted as truth. The Data Analysis uses Data produced during the execution of a Task. The Task is used to test the Null Hypothesis. For

B.1 Experiment Preparation and Execution

instance, identifying the causes of incorrect results in an classification algorithm is an example of debugging task that can be used to test the anecdotal hypothesis introduced in this section. In this case, the Data could be the time that each Participant took to debug Java and C++ applications.

Participant is the individual that perform a Task in an experiment. Along with the Task, Participant is an important element to increase the realism of SE experiments [214]. For instance, using professional software developers rather than students could increase the realism of the debugging experiment although using students is quite com- mon in SE experiments [215]. Participant uses some Instruments to perform a particular Task, such as tools for software debugging. Training material and data analysis tools can also be considered as Instruments. Finally, Object is manipulated by Participant during the Task execution. For instance, the Java and C++ application used in our anecdotal experiment are example of Objects.

B.1

Experiment Preparation and Execution

Ideally, an experiment is carried out in four steps [118, 128, 214, 256]. Firstly, a protocol that details the experiment plan and execution is written. Secondly, the experiment is executed and data is collected. Thirdly, data is analysed and conclusions are drawn. Finally, results are reported and made available to the research community.

Preparing an experimental protocol consists of:

• Devising research questions and hypotheses;

• Identifying treatments;

• Identifying, investigating and selecting instruments and objects;

• Defining the strategy for recruiting participants;

• Preparing training and experimental material;

• Carrying out a pilot experiment to test assumptions and experiment material; and,

• Writing down the protocol.

The experiment execution consists of instantiating decisions defined in the experi- mental protocol. Ko, LaToza & Burnett [131] define eight steps for executing an exper- iment in SE. They can be summarised into:

• Recruiting, selecting, informing and characterising participants;

• Assigning participants to treatments;

• Training participants;

• Performing experiment tasks; and

• Debriefing, receiving feedback and compensating participants.