CLASIFICACION DE LOS ARCHIVOS

Outlier identification and appraisal of their impact on findings is a challenging and complex matter (Hunter & Schmidt, 2004), and an outlier sensitivity analysis is strongly recommended to estimate the extent to which outliers

Page | 103

affect findings (e.g., Borenstein et al., 2009; Geyskens et al., 2009). Despite the importance accorded to it by researchers, a large majority of meta- analyses conducted in management do not report outlier sensitivity analyses (Geyskens et al., 2009).

Outlier sensitivity analyses in the present study were conducted in adherence

to the guidelines offered by meta-analysts (e.g., Borenstein et al., 2009;

Geyskens et al., 2009; Huber, 1980; Lipsey & Wilson, 2001; Tukey, 1960). The analyses entailed a direct comparison of results obtained with the entire dataset and those obtained with the dataset without outliers. Following the recommendations of Tukey (1960) and Huber (1980), top and bottom five percent of correlations (in terms of magnitude) were dropped from the dataset and a meta-analysis was performed on the remaining correlations. Thus, 10 percent of the correlations were identified as outliers in the current dataset. Hence, the summary effect size obtained with the removal of six correlations (i.e., 10 percent of the dataset values) comprising three correlations (i.e., five percent) each from the top and bottom ends, was compared with the summary effect size generated by the entire dataset.

Other types of sensitivity analyses also involved a comparison of the summary effect sizes obtained with the methods actually adopted, and the summary effect sizes from alternative meta-analytic decisions. The sensitivity analyses undertaken in this study (in addition to the outlier analysis) were: 1. Comparison of the summary effect sizes yielded with deployment of RE and FE models;

2. Comparison of the summary effect sizes obtained with and without making corrections for measurement errors, and

3. Comparison of summary effect sizes with and without the assignment of adjustment factors to individual correlations.

These types of sensitivity analyses are considered by meta-analysis experts as desirable (e.g., Borenstein et al., 2009; Cooper, 2010); however, the sensitivity analysis listed third in the list is unique to the current study and

Page | 104

unprecedented. The results of all sensitivity analyses are reported in the next Chapter.

It should be noted that a sensitivity analysis concerning sample size outliers (i.e., studies with extremely large sample sizes) was excluded in the current study as advised by Geyskens et al. (2009). This is because the studies with large firm samples are expected to provide a superior estimate of the true (construct-level) effect size, and their removal from the data-analysis can generally not be justified (Geyskens et al., 2009).

In addition to the sensitivity analyses, meta-analysts strongly recommend an estimation of publication bias that may distort meta-analytic results and thereby detract from their reliability (e.g., Aguinis et al., 2011; Geyskens et al.,

2009; McDaniel, Rothstein & Whetzel, 2006; Rosenthal, 1995). The methods

adopted for its estimation in the current study are outlined next.

4.8. THE ASSESSMENT OF PUBLICATION BIAS

To enhance the reliability of meta-analytic results, it is vital to detect the presence of, and estimate, publication bias (Aguinis et al., 2011; Hunter & Schmidt, 1990; Rosenthal, 1979; 1995). However, only a small proportion of meta-analyses undertaken in management have estimated publication bias (Geyskens et al., 2009). Publication bias occurs due to the general propensity of peer-review process to favour studies reporting statistically significant results for publication, rather than studies reporting non-significant results (Cooper, 2010). This potentially creates a biased representation of prior research in a meta-analysis. In other words, published studies are unlikely to accurately represent the entire population of studies (i.e., both published and unpublished) conducted in the past (Hunter & Schmidt, 1990; Rosenthal, 1979; 1995), and indeed, are likely to over-estimate the true effect size.

There are several approaches to detecting and quantifying publication bias

Page | 105

Orwin‘s fail-safe N (see Orwin, 1983), Begg and Mazumdar‘s rank correlation

(see Begg & Mazumdar, 1994), Egger‘s regression intercept (see Egger,

Smith, Schneider & Minder, 1997), and the Duval and Tweedie‘s trim-and-fill

method (see Duval & Tweedie, 2000a; 2000b; McDaniel et al., 2006). However, the two most commonly employed approaches are the Trim-and-fill method and Rosenthal‘s file drawer analysis.

Rosenthal‘s file drawer analysis enables the computation of the number of unpublished studies reporting null (i.e., statistically non-significant) results (Hunter & Schmidt, 2004; Rosenthal, 1979; 1995). The number of such

unpublished studies is referred to as the Rosenthal‘s fail-safe N (Rosenthal,

1979). Despite the acceptance of fail-safe N in the literature, the Duval and

Tweedie‘s Trim-and-fill method is often regarded as a superior method by

many researchers (e.g., Aguinis et al., 2011; Borenstein et al., 2009; Cooper,

2010; Geyskens et al., 2009), as it has the following advantages:

1) The Trim-and-fill method enables the calculation of a missing studies- adjusted summary effect size, and determination of the magnitude of difference between the observed and adjusted summary effect sizes (Aguinis et al., 2011). Thus, the method addresses a significant question, ―what is our best estimate of the unbiased effect size [emphasis in

original]?” (Borenstein et al., 2009: 286), and

2) Through a funnel plot (a salient feature of this method), the researcher can visually assess (albeit somewhat subjectively) the extent of publication bias in a dataset (Borenstein et al., 2009).

While Duval and Tweedie‘s Trim-and-fill method potentially possesses

advantages over the Rosenthal‘s file drawer analysis, the latter is very popular

and historically important (Borensteinet al., 2009). Thus, both the approaches

were adopted, also because they provided insights about the file drawer problem from somewhat different perspectives, and with different statistics,

Page | 106

The analysis and results of the file drawer problem (using the two methods outlined here) are presented in the next Chapter.

To assess the file drawer problem and perform computations as discussed in this Chapter (with the exception of the weighting scheme), a software program was procured. The selection of an appropriate software program for the current meta-analysis is now discussed, and this was the final decision concerning the study design and methodology.