A systematic review requires a series of protocols be developed for identifying the most appropriate research available to address specific research queries (Hemingway 2001: Torgerson 2003). This may include published research articles, unpublished materials, conference proceedings, book chapters, print articles, etc.
The protocol consists of several steps that commences with developing the research queries, followed by constructing and implementing research parameters for data searches, including identifying key words to use to complete searches, identifying appropriate target samples, including sample sizes, for inclusion, as well as identifying the appropriate research designs and analyses for inclusion. This is followed by conducting the extensive literature searches. Reading article abstracts; followed by reading relevant articles, eliminating data that is not appropriate for the study and collating the data that is. Data extraction is completed, appropriate analyses are undertaken and report is completed detail the procedures and findings. Additionally, in medical reviews, however, this may apply to a psychological review, a schedule is prepared for follow-searches, and additions may be made to the review at regular intervals. This is to ensure the systematic review remains relevant over time.
4.6.1 Types of analyses available
Systematic reviews are often accompanied by meta analyses of the data, though they are not an essential feature of a systematic review (Torgerson, 2003). The purpose of a meta-analysis is to synthesise and examine the findings from several studies statically to investigate “validity generalization” of the data in an unbiased way (Anastasi and Urbina, 1997, p 125).
Page | 4-92 To complete a meta-analysis effectively all sources of heterogeneity should be systematically and consequently statistically be accounted for, if possible. There are several possible sources of heterogeneity that need to be considered; some of which are easier to systematically account for statistically than others. According to Gagnier et al. (2012) there are three types of heterogeneity that need to be considered:
Methodological heterogeneity
Essentially how the studies differ in design and implementation
Clinical heterogeneity
Differences resulting from participant characteristics such as sex, age, presence of disorder/illness, comorbidities.
Statistical heterogeneity is the consequence of methodological and clinical heterogeneity (Gagnier et al. 2012). Statistical heterogeneity can alter the meta-analysis substantially resulting in inaccurate summary effects, flawed conclusions, and as a consequence of bias, the studies will not be measuring the same effects. Complex statistical analyses can be added to a meta-analysis or other subtypes of meta-analyses can be employed to mitigate some of these affects this may include subgroup analyses, and meta-regression (Gagnier et al. 2012). The difficulty in implementing these features, however, is that they require substantial statistical expertise to know when and how best to employ some of these analyses. Also, these recommendations generally apply to issues surrounding methodological heterogeneity; clinical heterogeneity is may be more challenging to address as there are not currently standardised procedures for addressing such (Gagnier et al. 2012).
There are numerous sources of heterogeneity when considering how to synthesise the studies of the PPI, PPI-R, and PPI-SF. An obvious issue is that of the assessments themselves. Frequently it has been assumed that since the PPI-R and PPI-SF are derived from the PPI, they are measuring the same thing, in the same way. Consequently, numerous studies have reported validity testing for PPI and apply to
Page | 4-93 PPI-SF() or suggest that if the PPI is valid and reliable so to must its derivatives be. It is imperative that researchers not assume these tools are interchangeable and that the changes made to the PPI-R and PPI-SF have not altered the assessment. This needs to be examined more thoroughly empirically, though there are some examples within the systematic review that explore such, more work needs to be done to either independently verify each tool’s merits with the construct of psychopathy as well as their correlations with each other. Another substantial issue is whether it is best practice to combine different samples. Specifically is it appropriate to combine the results form studies that focus on clinical or forensic samples with results from studies with ‘healthy’ participants. Further, is it appropriate to compare those formally diagnosed as psychopathic with those who have demonstrated key traits associated with the disorder but would not meet the criteria for a formal diagnosis and as well as comorbidity with other disorders.
Because there is so much heterogeneity within the data, combining it statistically via meta-analysis seems less than ideal without substantial statistical expertise, particularly when there is no standard procedure for how best to synthesis data across different samples (Gagnier et al., 2012). Some go so far as to argue that researchers are not to combine statistical data that lacks homogeneity for the purposes of meta- analysis Hemingway (2001). Whilst this recommendation seems excessive as there are statistical tools available, particularly that can be applied to methodological heterogeneity, there is another issue to consider that does make combining the data via meta-analysis inappropriate, in this instance. The objective of a meta-analysis is to explore data in an unbiased way, statistically. After careful consideration of the heterogeneity within the data, and that the bulk of the research has focused on ensuring that all measures of psychopathic traits, including measures of normal
Page | 4-94 personality when used for the purposes of assessing psychopathic traits, as well as the PPI and derivative measures correlate with the PCL-R, conducting a meta-analysis does not seem appropriate based on the concerns raised by Skeem and Cooke (2010). That research and theory have not remained consistent with the nomological network originally proposed that is generally accepted, which is the Cleckley/Hare model. Finally, the heterogeneous nature of psychopathy itself poses a challenge. By combining the data statistically, the variations that may present across studies and not others may be lost or downplayed when perhaps they should not. Therefore, a narrative empirical synthesis has been conducted. This includes a summary table and tabulation of data along with a critical review of the existing literature and discussion of future areas of research.
To reduce the potential for further bias that often occurs in the course of a narrative empirical synthesis, this review has incorporated some of Miller and Lynam’s (2012) protocols and the majority of research studies for inclusion to ensure that the study selection, and procedures for synthesising the data are as consistent as possible with the procedures for completing a meta-analysis of the data.
A narrative empirical synthesis, like a meta-analysis, is intended to combine sets of empirical data to establish validity (Hemingway, 2001). Where it diverts from the meta- analysis is that it does not include a complex statistical analysis of the data. Research is compared and contrasted by the researcher using a summary table of the studies, tabulations and concludes with a critical review. Like a meta-analysis it should include thorough and precise protocols for the literature searches, the selection process for the inclusion/exclusion should be to the same standard as that of a meta-analysis to reduce the chance of bias. Narrative empirical syntheses are less often used because they are believed to be more prone to researcher bias (Torgerson 2003) particularly when considering study inclusion, however, considering the existing bias in the
Page | 4-95 psychopathy literature toward the PCL-R derived nomological network, and that a meta-analysis of biased data would not be appropriate; the narrative empirical synthesis was selected. In an effort to overcome potential article selection/inclusion all of the studies included in Miller and Lynam’s (2012) meta analyses were included in the narrative empirical synthesis providing they fit within the research protocols outlined. This resulted in two studies being excluded as the pre-dated the data included in this analysis as they were published before 2005 and the date parameters for this systematic review were data published between 2005 and August 2012.