Descripción de propuesta

In this mapping study, I approach evidence without a preconceived hierarchy in mind (I do, however, use an evidence hierarchy in the analysis and synthesis of the included studies). The main criterion I use is whether a study provides scien- tific empirical evidence on a relevant question. This approach, however, requires me to confront the question of what evidence actually is. This is a question of epistemology; although a proper study of the relevant questions is beyond the scope of this study, I will sketch the main argument.

First, I must dismiss a number of historical epistemological stances. First is the idea that a series of successful empirical tests of a theory confirms a theory; the second is the idea, due to Popper (1980), that the only thing we can say of a theory is that it has or has not been falsified. The untenability of the former is well known (see e. g. Russell 1983, p. 35). The idea of falsification fails as well, for two separate reasons: the status of not-yet-falsified is absolutely useless when one must choose among several such theories, and as Quine (1951) noted, it is always possible to react to an empirical refutation of a theory by tweaking the theory (and in many cases this is even the right choice). These arguments are introductory-textbook material in the philosophy of science (see e. g. Bird 1998; Godfrey-Smith 2003).

In the social sciences, there are two major epistemological traditions of re- search. Each generally (though not universally) dismisses the other, sometimes with strong harsh words. One tradition, self-labelled asantipositivism, has given the other the (often pejoratively intended) labelpositivism(this label is, however, historically inaccurate, see Mackenzie 2011) and regards it as a decades earlier thoroughly discredited research paradigm (for a recent antipositivist formulation, see St. Pierre 2012). The antipositivists themselves divide into several sub-camps each having a label, proudly worn by its members, such ascritical theory,feminism, andconstructivism(for an overview, see Guba and Lincoln 1994).

On the other side of the divide, the researchers who are given the positivist label do not typically use that (or any other) label of themselves; they merely see their approach as good scientific practice and regard the antipositivist ap- proaches as unscientific or worse (for a recent strongly worded formulation, see Colquhoun 2011, p. 336–339), and often just ignore them.

epistemological, and axiological views which results in different and perhaps even incompatible methodology and standards of good research (for a summary written by antipositivists, see Lincoln et al. 2011). The antipositivists generally avoid quantitative methods, while the other tradition embraces them; hence, these two traditions are often (somewhat incorrectly) called the qualitative and thequantitativeparadigms, respectively.

In this study, I do not wish to take a firm stand for or against either ap- proach; however, the very fact that I am working within an evidence-based par- adigm (as well as my methodology here generally) does bias this study against the antipositivists somewhat (see e. g. Suri 2013). Instead, I have attempted to formulate an epistemological position that is reasonably agnostic on this issue.

There are, in recent philosophy of science, two main approaches to episte- mology. One isinference to the best explanation(see e. g. Lipton 2004), and the other isBayesianism (see e. g. Howson and Urbach 2006; Jeffrey 2004). Some authors (such as Godfrey-Smith 2003) are of the opinion that they are incompatible, but like Lipton (2004), I believe them to be compatible. In any case, for the purposes of this mapping study, the Bayesian approach is more instructive.

The central idea of Bayesian epistemology is that the proper way to assess a claim is to assign it a probability. A probability assignment based on the totality of current knowledge about the claim is called aprior probabilityor just aprior; when a new piece of knowledge is added, the prior is transformed into a new probability, theposterior probabilityor just theposterior. When another new piece of knowledge is about to be added, the old posterior becomes the new prior, and the new piece creates a new posterior.

Some Bayesians (e. g. Jeffrey 2004) posit that Bayesianism is about the ideal rational person, the Bayesian agent, defined as having the following characteris- tics: if it were to place bets based on its beliefs, it would not be vulnerable to a Dutch book – a set of bets which is certain to result in a net loss, such as betting against the ordinary mathematical statement 1+1=2, but usually more complex – and it reacts to observations by adopting the posterior probability suggested by a Bayesian analysis as its new prior. Others (e. g. Howson and Urbach 2006) re- gard the Bayesian theory of probability as alogic of induction, on a par with the more familiar logics of deduction (such as elementary first-order logic); it does not define a rational being, merely what it means to be rational.

From this point of view, the meaning of “evidence” becomes plain. First, evidence is an observation, something external to the observer that the observer becomes aware of. Second, evidence requires interpretation. Third, evidence never exists in isolation, rather all evidence is evidence about some proposition. In sum, evidence about a proposition is an observation that a rational person interprets as changing their confidence in that proposition. In other words:

Definition 4.Evidencecomprises reported observations about the contingent aspects of the world. Evidence isabouta claim if it has the potential to affect a rational person’s confidence in the claim. Evidence isscientificif it has been honestly, systematically and deliberately collected for a research purpose. Plain assertions, descriptions of function- ality, anecdotes, expert opinions, personal experience reports by the researchers, and formal proofs are not scientific empirical evidence.

This may be just an artifact of Bayesianism but I adopt it here: evidence is inherently empirical. A logically valid or contradictory proposition has, for all Bayesian agents, probability of one or zero, respectively, which no observation can possibly change within the Bayesian logic. Only contingent propositions can have evidence.

Definition 5. A proposition iscontingentif it there are possible worlds where it is true and possible worlds where it is false. In other words, a contingent proposition is not a logical tautology nor a logical contradiction; a Bayesian agent would know its truth value a priori.

Note that this definition sidesteps the notorious problem of old evidence often attributed to Bayesianism: how can an observation known to a Bayesian agent be first dismissed as irrelevant but later be recognized as evidence, which is something that happens in actuality? My definition does not require a person to be a Bayesian agent, it merely speaks of a hypothetical “rational person”, which is a Bayesian agent. When one recognizes an old observation as evidence, one is essentially realizing that a Bayesian agent would change its confidence in the proposition at hand upon observing it.

I have used these definitions in the mapping study as an aid to decide when a study provides (empirical) evidence and when it does not. The following chap- ter details the actual process, both for making those decisions and for other as- pects of this study.

This thesis reports a systematic mapping study; this chapter explains the map- ping process used. A high-level view to the process is shown in Figure 1.

SCIENTIFIC LITERATURE Systematic documented search POOL OF POTENTIALLYRELEVANT STUDIES Systematic documented selection SELECTED STUDIES Identification of relevant quotes “. . . ”_{“. . . ”“. . . ”} “. . . ” Coding DEBUGGINGEFFORT CASESTUDY CONDITIONALS

STATIC

E

Thematic Synthesis THEMATIC

MODEL

FIGURE 1 A high-level representation of the mapping process. This diagram omits many details.

First, I wrote a protocol document that described the planned process before any actual work commenced. As the study progressed, I revised the protocol several times. My supervisors reviewed the original protocol document and all

revisions before I started following them. I did not request an external review of the protocol. The protocol and all its revisions are available from me by request.

Throughout the study, I maintained a record on the process, including all the intermediate data generated during the process. The records form a text-based, both machine- and human-readable database under version control. Appendix 1 details the database and the tools I used.

I searched for candidate studies by manual and automatic search of various venues and databases and in several iterations, as discussed in Section 4.1. I then proceeded to decide which of the potential studies should be included in three phases: Phase I was a preliminary selection phase, in which only the most obvious cases were excluded, and the rest were retained for more careful checking in Phase II; I considered only on-line metadata in these two phases. In Phase III, I obtained the full text of all studies that survived the previous phases, and made my provisionally final decisions. I also conducted a single iteration of snowball search on studies that had provisionally been selected for inclusion; this yielded additional candidate studies that I then subjected to selection Phases I through III. After a selection evaluation exercise, the decisions were finalized. The selection process is discussed in Section 4.2.

After final selection decisions, I conducted a four-stage thematic synthesis process, as discussed in Section 4.3. I first read all studies selected for inclusion. Then, I extracted from the studies direct quotes that appeared relevant to the research questions. I then developed a coding scheme, and applied it to these quotes. I finally created a thematic model of the included studies.

4.1 Searching for candidate studies

The search for candidate studies consisted of three phases: manual search, auto- matic search, and snowball search. This process is summarized in Table 1.1

The first iteration of manual and automatic searches took place from De- cember 2010 to September 2011. A second iteration of manual and automatic searches was conducted in December 2012 and January 2013, to update the set of candidate studies to include studies published up to 2012. The single iteration of snowball search took place between February and April 2013.