MARCO LEGAL

In a systematic secondary study, the implicit context is that one is looking for research evidence, that is, scientific or scholarly studies duly reported that bear on the subject at hand (collectively calledprimary studies). In the behavioral sciences, which are the most relevant for this study, a number of research methods have become standard; they are conventionally classified into thequantitativeand the qualitative.2

There are a number of qualitative methods, includingcase study(Yin 2009; Runeson et al. 2012),content analysisandthematic analysis(see e. g. Vaismoradi et al. 2013),grounded theory(Glaser and Strauss 1967),ethnography(see e. g. Crabtree et al. 2009; Morrison et al. 2010), andaction research(see e. g. Avison et al. 1999). Common to all of them is a focus on the particulars of a specific situation and attempting to achieve a deep understanding of it, and sometimes a beneficial change in it, instead of generalization into putatively universal laws. Commonly, the situation is looked at from the point of view of the participants instead of 2 _{Vessey, Ramesh, et al. (2005) developed a classification of, among other things, research} methods in computing, partially based on Alavi and Carlson (1992). I find these classifica- tions not very useful, as they do not define their terms very clearly (see Section 6.1).

the point of view of an outside observer. It should be noted that the mere use of qualitative data (such as interviews) does not make a study qualitative in nature. In quantitative research the goal is typically to estimate the effect of one or moretreatments(the choice of treatment, including perhaps their absence, form theconditions, also referred to as the values of theindependent variables) on one or more quantities of interest, thedependent variables, with the goal of testing theories consisting of (qualified) universal laws and asserting a causal connection between the independent and dependent variables. The methods are broadly categorized (see e. g. Whitley et al. 2013, p. 36–45) into theexperimentalapproach, in which the researchers control to various degrees the circumstances and conduct of the research, and thecorrelational, in which the researchers observe real-life phenom- ena without exerting control over them.

Experimental studies have, according to Whitley et al. (2013, p. 242), three defining characteristics: “manipulation of the independent variable”, “holding all other variables in the research situation constant”, and “ensuring that partici- pants in the experimental and control conditions have equivalent personal char- acteristics and are equivalent with respect to the dependent variable before they take part in the experiment”. They can bebetween-subjectsdesigns, in which the various treatments and perhaps their absence are assigned to different people (formingexperimental groupsand acontrol group, the latter being given no treat- ment or a control treatment), and the result is obtained by examining the differ- ence in the dependent variable values between the groups (Whitley et al. 2013, p. 252–255). Alternatively, they can be within-subjects(or repeated measures) de- signs, in which each participant is sequentially subjected to each of the experi- mental treatments and the control treatment in turn, and the result is obtained by considering the change in the dependent variables; within-subjects designs can becounterbalanced, in which the participants are divided into several groups, each getting the treatments in a different sequence (Whitley et al. 2013, p. 255– 259). Factorial designscan be used to measure the effect of several independent variables in the same experiment (Whitley et al. 2013, p. 264–255), in which case the experiment may be within subjects for some variables and between subjects for others.

Campbell and Stanley (1963) further classified experimental study designs into three categories:pre-experimentaldesigns,true experimentaldesigns, andquasi- experimentaldesigns. True experiments they defined to be experiments following all contemporary recommendations on experiment design, particularly the use of a control group for which the treatment is absent, and assignment of partici- pants to the groups by a random process. Pre-experimental designs predate the establishment of these standards and generally fall short of them, though they can be successful in limited circumstances. Quasi-experiments are experimental studies that lack one or more of the requirements imposed on true experiments due to circumstances of the experiment that preclude their employment (one sup- poses that if a design fails to meet the criteria for some reason attributable not to the circumstances but to the researchers, the study would be classified as pre- experimental and not quasi-experimental). They include within-subjects designs

(even counterbalanced ones) in the quasi-experimental category.

In this mapping study, I will assign experiments into three categories. The most broad category is that of experiments: studies in which the researchers at- tempt to influence one or more independent variables in order to cause changes in one or more dependent variables. The next category is that ofcontrolled ex- periments: experiments in which the experimental subjects (which, if human, are calledparticipants), are assigned into groups based on which treatment (or their absence) they are subjected to and in which sequence. I further require that in controlled experiments the groups cover all treatments (including their absence, if no control treatment is used) and all the possible sequences in which they are administered. Thus, I categorize a within-subjects experiment as controlled only if it is completely counterbalanced. The third category is that ofrandomized con- trolled experiments(also often calledrandomized controlled trials): controlled experi- ments in which subjects are assigned into groups by a random process. Note that there are studies that I categorize as experiments that Whitley et al. (2013) would not; further, while I believe all the Campbell and Stanley (1963) true experiments qualify as randomized controlled experiments, not all randomized controlled ex- periments are true experiments.