Análisis del microentorno - Mapa de actores

The experiment execution is the process where the plan discussed in the previous sec- tion is instantiated. The same experiment plan can be instantiated in several different ways, due to local constraints. For instance, both the number of subjects and their exact background may differ, from experiment to experiment, despite sharing a com- mon plan. It is important to document these peculiarities, as they help grasping the “field” constraints of the experiment, in complement to those previously defined in the experiment plan.

Figure 3.16: Experiment data collection

Collection clearance

When the data to be used in the experimental work belongs to individuals, or organiza- tions, collection clearance must be obtained before starting the data collection process. This is particularly important if the data being collected is considered sensitive by its owners. Granting anonymity of participants and organizations is an alternative com- monly used by researchers, with the agreement of data owners.

Motivation of participants

The difficulties of recruiting professional practitioners to participate in experiments often lead to the usage of students as surrogates for those practitioners. Sjøberg et al.’s systematic review of controlled experiments in Software Engineering [Sjøberg 05] shows that out of 5488 participants in 113 experiments reported on the main software engineering journals and conferences from 1993 to 2002, only 9.4% were professional practitioners, while 86.8% of the participants were students. The remaining partici- pants were either faculty members and post-docs, or of a background not disclosed in the papers reporting the experiment. Although the about 1/3 of the students has an unknown background, one can estimate from the remaining subjects that over 80% of the students were undergraduates.

In general, experiments are framed within a wider context. With professionals, experiments can be performed in the context of a real development project (e.g. a project being used as a pilot for the introduction of a new development tool) or as part of a training course. In most situations, professionals participate in the experiments as part of their job.

In [Benestad 05] Benestad et al. discuss the problems concerning the recruitment of professional participants for experiments, and conclude that:

• Practical constraints have to be taken into consideration when defining the target populations of experiments. These include geographic constraints, the organiza- tional profile, and the individual profile of participants.

• The participant organizations and individuals have to be offered flexibility and

added value, so that adequate samples of organizations and individuals can be recruited. The flexibility is a facilitator characteristic to ensure that the experi- ment is as non-intrusive as possible, so that it is not viewed as a burden by the participants. The experiment should also guarantee some form of added value

for the participants. This can range from direct payment (unfeasible, in most situations), to knowledge transfer from researchers to the practitioners, through training sessions, and seminars to share the results of the experiment internally, before making them publicly available. If the experiment shows an opportunity for improving some part of the software process within the organization, this can also be perceived as an added value.

• High professional and ethical standards must be achieved, if a continuing co- operation is sought. In the long run, successful experimental work that is found useful both for the researchers and the practitioners involved in it creates oppor- tunities for a continuing collaboration. A typical ethical concern is to ensure the anonymity of participants when some discrimination is to be made with respect to their qualitative assessment.

In experiments with students, the experimental work is usually carried out within a course being followed by the students. Students often have rewards of an academic nature, such as part of the course grade, or extra credits for the student’s degree, be- sides the didactic objectives that the participation on the experiment should have (e.g. the experiment participation involves the practical usage of concepts acquired during the course).

Data collection

The process of data collection corresponds to the actual enactment of the experiment. Experimenters should record information such as the schedule and effort used in the experiment by participants, so that this information can confronted with what was previously planned. Any problems detected on the experiment package should also be registered, so that it can be improved in further replications of the experiment.

Special events concerning the experiment, such as subject’s mortality (subjects that are removed from the experiment - in the case of human participants, this happens when a prospective participant ends up not participating) must be recorded for fur- ther analysis. Subjects’ mortality is important, at least from two perspectives: on the one hand, understanding the motives that lead to mortality of subjects may help im- proving the experimental design in future replications of the experiment; on the other hand, the potential impact of mortality on the experiments results should also be as- sessed. Patterns on the mortality of subjects may help uncovering factors which are important to the experiment but are not addressed by the followed experimental de- sign. These factors should be considered, when analyzing the threats to the validity of the experiment.

Data validation

This process aims at ensuring that the experiment data has been collected correctly. Problems with data collection can result from, for instance, erroneous performance (or usage) of collection tools, misinterpretation of data collection forms by the participants in the experiment, or deviations from the planned experimental protocol. The data quality is essential so that adequate inferences can be made from data. This validation process may involve not only the researchers conducting the experiment, but also the participants. The latter can help clarifying data that is found likely to be incomplete, or incorrect.

Problem reporting

All deviations from the original plan should be recorded. Detailed as an experiment plan may be, there are details that may not have been considered while planning, or were insufficiently dealt with at that phase. Identifying those problems and how they were dealt with by the experimenters is an enabling condition for experimental repli- cability, as well as an important step toward identifying potential threats to the validity of the experiment.