Because not all online communities are supervised by health practitioners, health- related interventions and behaviours suggested by the discussions do not necessarily reflect the current state of scientific medical knowledge. This can be seen as an information integrity problem—recommendations that are in agreement and are ultimately based on the current state of scientific medical knowledge can be described as information of high integrity. Other recommendations can be seen as information lacking integrity or as information of uncertain integrity.
In healthcare, recommendations known to be based on the current state of scientific medical knowledge are described as evidence-based medicine (Rosenberg & Donald, 1995). Recommendations that are not based on the current state of scientific medical knowledge are seen as anecdotal; physicians are expected to base their practice on evidence-based medicine.
To establish recommendations that can be described as evidence-based medicine, systematic reviews of medical research literature are conducted on an ongoing basis, and the results are published as databases. Describing and comparing all of the existing methodologies for such systematic reviews is beyond the scope of this thesis; instead, I briefly describe the methodology developed by Cochrane Collaboration (The Cochrane Collaboration, 2013)—an influential international organization promoting systematic reviews of medical research literature. In New Zealand, the database of systematic reviews maintained by the Cochrane Collaboration
(http://www.thecochranelibrary.com) was made available to all New Zealanders free of charge in 2006 (Jordan, Jeffery, & Farquhar, 2006) as a result of a national license purchased by the New Zealand Government. (The database remained available at the time of writing, March 2012.)
Cochrane Collaboration publishes the "Cochrane Handbook for Systematic Reviews of Interventions" detailing the methodology (Higgins & Green, 2008), henceforth in this section referred to as the Handbook. The role of the authors of systematic reviews is, in some respects, similar to the role of coders in content analysis. The issue of who could be a suitable author is not addressed in the Handbook. Nonetheless, all the reviews I
could access in the Cochrane Collaboration database were authored by individuals affiliated with medical institutions, even though the editorial procedures (Higgins & Green, 2008, p. 11) do not explicitly impose any restrictions on who can be a reviewer.
The Handbook does suggest that prospective review authors undergo training, and at the time of writing the Cochrane Collaboration web site provided extensive information on training opportunities.
The unit of analysis in a systematic review is a research publication (a study). The Handbook discusses in detail the approach to selecting the studies for analysis (a procedure corresponding to sampling in content analysis). The studies are selected based on the focus of the review (the questions the review intends to address), the study participants, the interventions, the outcomes for the participants, and the research design. The recommended sources of studies are primarily online bibliographic databases, although other sources (such as medical trial registers) are also suggested.
Once the studies are selected, specific information (to be analysed in the following stage) is extracted and presented in a uniform way, including information about the methods, the participants, the interventions, and the outcomes. This step is similar to extracting coding units from sampling units in content analysis.
Once the information is extracted and, using the vocabulary of content analysis, coding units are obtained, they are assessed for bias (a stage similar to coding in content analysis), such as selection, performance, attrition, detection, and reporting bias (I do not describe what these biases constitute, because that would require me to introduce the details of experimental research methodology, which is clearly beyond the scope of this literature review).
Similarly to content analysis, the Cochrane methodology employs inter-rater reliability as evidence of the validity of the finding. Nonetheless, comparing to the approach to content analysis by Krippendorff (2004), there is much less emphasis on inter-rater reliability. Reviews that do not involve the assessment of inter-rater reliability (e.g., with a single author) are allowed. Inter-rater reliability is assessed only for the most important data (such as the key outcomes), and insufficient inter-rater reliability does not preclude the review from being published. Although Krippendorff suggests that in
case of problems with inter-rater reliability the coding scheme should be updated and the data should be recoded using different coders, the Cochrane methodology
emphasizes carefully recording any disagreements. The Cochrane methodology does not preclude the same individuals who designed the study from assessing the studies reviewed for bias (the activity corresponding to coding in content analysis). Thus, any inter-rater reliability values obtained in a Cochrane review are likely to reflect the shared understanding obtained in designing the study.
The above comparison between the Cochrane methodology and the approach to content analysis suggested by Krippendorff (2004) is based on how the methodologies are formulated, not on the comparison of the actual practice. The actual practice may be closer than the stated methodologies. In particular, it is likely that Cochrane reviews with multiple authors and with inter-rater reliability assessed are easier to publish; moreover, as can be seen from the review of content analysis practice in studies of knowledge construction presented in section 2.6, content analysis studies do not always reach the inter-rater reliability benchmarks suggested by Krippendorff. It would be of interest to study and to compare the actual practice, but this is clearly out of the scope of the present study.
The last stage in the Cochrane methodology involves presenting a summary of results in tabular form, which is also a common way to present the results on content analysis (Krippendorff, 2004, p. 192). The Handbook suggests specific formats in which the results are to be summarized. The ultimate outcome of a review is a compact
presentation of the studies included in the analysis, and, for each study, an assessment of possible sources of bias. Studies that are sufficiently unbiased are seen as presenting evidence-based results.
To ensure that recommendations suggested in online health support group discussions are evidence based, a moderator would need to assess them by evaluating them against the databases of medical knowledge such as the databases provided by Cochrane Collaboration. A less expensive option would be to have a medically qualified participant occasionally present, who would assess the integrity of the information based on her expertise. It remains an open question, though, if the regular participants themselves, in discussions conducted at high levels of knowledge construction, would
be able to challenge recommendations that are not evidence based and thus maintain the integrity of information in the discussion.