In the experiment, the participants were asked to rate several aspects of a piece of information. These aspects were trust, readability, familiarity, information quantity, writing style, and structure. Answers could be provided on 7-point Likert scales. In fact, we were only interested in trust and familiarity. However, these aspects were boxed in by the other questions to avoid overemphasis on trust. Rated familiarity was used as manipulation check for the familiarity manipulation.
A 2 (source cues) × 2 (familiarity) × 2 (quality) design was used. The availability of source cues was a between-subjects factor whereas the other two manipulations were within- subject. The order of familiarity and quality was counter-balanced to control for order effects.
The availability of source cues was manipulated by presenting information from Wikipedia in the layout of Wikipedia or in a standardized layout. In this case, the standard layout of Microsoft Word 2003 was selected. By using the Word layout, the participants were not able to see that the information came from Wikipedia or any other source. All elements of the original articles were included in the Word-versions. Images and tables were presented in line with the text on the same locations and in the same sizes as on Wikipedia. The articles were presented on a 17” CRT-screen in either Microsoft Internet Explorer or Microsoft Word 2003.
The topics of the articles used were selected such that half of them were assumed to be relatively familiar to the participants and the other half relatively unfamiliar. Table 1 shows
88 |
Chapter 5which articles were selected as familiar and unfamiliar. Familiarity was assumed based on the age (about 20 years), culture (Western Europe), and education (college education in Psychology) of the participants.
Article quality was manipulated following the quality assessment ratings of the Wikipedia Editorial Team (Wikipedia: Version 1.0 editorial team, n.d.). Half of the articles were of high quality (featured articles and good articles), whereas the other half of the articles were of low quality (stub articles and start articles). High and low quality were evenly distributed in the familiar and unfamiliar condition. Table 1 shows the familiar and unfamiliar articles used in the high and low quality conditions.
2.3 Procedure
Participants were welcomed and asked to sign informed consent. Task instructions were given verbally, based on a standard protocol, and in writing. After this, a questionnaire on various demographics was filled out, after which the experiment started. Each participant performed the task eight times, with a questionnaire on the dependent variables presented after each trial. The experiment ended with a few control questions on the manipulations. The duration of the experiment was about sixty minutes.
3. Results
3.1 Familiarity Manipulation Check
The questionnaires after each article confirmed the manipulation of familiarity. On a 7-point Likert scale the participants rated their prior knowledge on familiar topics (M =
Table 1: Selected topics. All articles were obtained from the English Wikipedia on January 1st, 2011.
Topic Familiarity Quality
The Simpsons Familiar High
Facebook Familiar High
Classical Conditioning Familiar Low
Birthday Familiar Low
The retreat of glaciers since 1850 Unfamiliar High
Agartala Unfamiliar High
Sovereignty Unfamiliar Low
4.88; SD = 0.96) higher than their prior knowledge on unfamiliar topics (M = 2.40; SD = 1.15); t(40) = 13.65; p < .001.
3.2 Trust
Table 2 shows the level of trust in the various conditions of the experiment. A repeated- measures ANOVA with quality and familiarity as within-subject factors and the availability of source cues as between-subjects factor was performed to test the hypothesis. A main effect of quality on trust was found; high-quality articles were trusted more than low- quality articles; F(1, 39) = 27.66, p < .001. No main effects were found for familiarity, F(1 ,39) = 0.38, p = .54, and the availability of source cues, F(1, 39) = 1.23, p = .28. Hence, no support was found for hypothesis 1.
A significant interaction between the availability of source cues and familiarity with the topic was found, F(1, 39) = 5.07, p < .05. Only trust of familiar users was influenced by the availability of source cues; they trusted information less when they knew it came from Wikipedia. However, an additional three-way interaction with quality revealed that this effect was only visible with low-quality information, F(1 ,39) = 5.72, p < .05. A (two- way) interaction between the availability of source cues and familiarity was hypothesized. However, we expected an inverse effect of what the results showed. Therefore, the second hypothesis has to be rejected.
No other significant interaction effects were found. Therefore, our final hypothesis also has to be rejected, as no interaction between the availability of source cues and information quality on trust was found; F(1, 39) = 0.25, p = .62.
Table 2: Trust on 7-point Likert scales in all conditions. Standard deviations are given in parentheses.
With source cues Without source cues
HQ LQ Total HQ LQ Total Familiar 5.35 (1.59) 4.13 (1.49) 4.74 (1.32) 5.60 (0.80) 5.02 (1.07) 5.31 (0.75) Unfamiliar 4.33 (1.50) 4.48 (1.18) 4.90 (1.25) 5.60 (0.83) 4.45 (1.27) 5.02 (0.81) Total 5.34 (1.52) 4.30 (1.25) 4.82 (1.26) 5.60 (0.72) 4.74 (1.03) 5.17 (0.68)
90 |
Chapter 54. Discussion
In this study, we investigated the influence of the availability of source cues on credibility evaluations by users who are familiar or unfamiliar with the topic at hand. No support for our stated hypotheses could be found in the conducted experiment. First, no main effect of the availability of source cues on trust was found. Second, concerning the role of familiarity, an effect opposite to our hypothesis was found: only familiar users were influenced by the availability of source cues; unfamiliar users were not. A more detailed analysis revealed that familiar users were only influenced by source cues when viewing low-quality articles. Finally, no biasing effect of source cues on the influence of information quality on credibility was found as high quality information could be distinguished from low quality information in both conditions.
A few explanations can be given for the lack of an effect of the availability of source cues on trust of unfamiliar users. Eastin (2001) argued that in his experiment, no effects were found due to the overall high level of trust in online information. The same could be argued in the current study with Wikipedia as an information source, although the overall level of trust seems to be slightly lower than in Eastin’s study. A second explanation could be that knowing that information comes from Wikipedia has a polarizing effect on trust. As was demonstrated by Lucassen and Schraagen (2011b), the source (Wikipedia) was used often (about 30 percent of the cases) as a motivation to both trust and distrust information. It could thus be the case that about half of our participants trusted the information less because it came from Wikipedia, whereas the other half trusted the information more. However, although the standard deviation is slightly larger in the ´Wikipedia-condition´, visual inspection of the distribution of trust scores did not reveal such a polarization effect. A perhaps more plausible explanation may be derived from the reputation of Wikipedia, or the lack thereof. As shown in earlier studies (e.g., Lucassen et al., 2013), college students are well aware of open-editing model behind Wikipedia. This could mean that our participants knew that they could not attribute any value to the authors of the articles they read. If this is the case, then the condition without source cues was in fact very similar to the condition with source cues; no inferences were made about the source of the information. Hence, evaluation behavior and trust of our participants was the same on Wikipedia and in the condition without cues on the source of information. So it could be argued that for our (unfamiliar) participants, Wikipedia has no reputation at all.
The finding that users familiar with the topic at hand, as opposed to unfamiliar users, were influenced by the presence of source cues, is contradictory to our hypothesis. Our expectations were based on a study by Eastin (2001). He hypothesized the same effect as we hypothesized in this study, but also did not find corroborating results. However, individual tests of both groups (i.e., familiar and unfamiliar) indicated some validity of his original hypothesis.
An explanation for the contradictory findings in this experiment can be found in the difference in our conceptualization of source cues compared to Eastin’s study. He manipulated source expertise by presenting the same message from persons who could be expected to vary in their level of knowledge on the topic. In the present study, the source was manipulated by taking articles from Wikipedia, and removing any cues of this source to create a condition without source cues. However, the credibility of Wikipedia is ambiguous, with evidence in favor (e.g., Giles, 2005) and against (e.g., Denning, Horning, Parnas, & Weinstein, 2005) its credibility, both likely to be caused by its open-editing model. Hence, the fact that the information came from Wikipedia did not make the information more credible per se. In fact, the open editing model behind Wikipedia may be an explanation for the observation that users familiar with the topic at hand had less trust in the information when they knew it came from Wikipedia. Since only very little is known about the authors of Wikipedia articles (anyone can contribute), familiar users may have doubted the expertise of these authors (one of the two elements of credibility; Fogg & Tseng, 1999; Hovland, Janis, & Kelley, 1982) in comparison with their own knowledge on the topic. Unfamiliar users do not possess such knowledge, which may have led them to believe that the authors of Wikipedia at least know more about the topic than they do themselves. In their case, this may not have raised trust, but at least it was not diminished. This explanation seems to be at odds with the study by Chesney (2006), who found that experts had more trust in Wikipedia than novices. However, the experts in that study were actual renowned domain experts on the topics of the articles. This means that those participants were actually able to recognize that information was correct or incorrect. In this study, the ‘expert’ participants were merely familiar with the topic at hand. This means that they could not solely rely on their own expertise and had to consider other cues as well. One of these cues is the presumed credibility of the authors of information.
As can be seen in the results, the interaction effect between familiarity and source cues on trust was only present in low-quality articles. An explanation for this observation can
92 |
Chapter 5again be found in the ambiguity of the credibility of the information. The number of cues in the high-quality articles indicating high credibility may have been so overwhelming to our participants, that the lack of information on its source could not diminish their (positive) opinion. In contrast, the credibility of low-quality articles was much more ambiguous. In these articles, cues indicating both high credibility (e.g., no large errors) and low credibility (e.g., short text) could be found. In this case, it is much more likely that the opinion of the participants was also influenced by the source. In other words: source cues are much more influential when the credibility of the information is ambiguous in itself.
No biasing effect of source cues on credibility evaluation was visible in terms of difference in trust in high and low quality information. This means that our participants were always able to distinguish high and low quality, regardless of its source. While it has been shown before that college students (such as our participants) are able to make such distinctions (Lucassen & Schraagen, 2010), it does not necessarily mean that they also make such distinctions in actual search behavior online. For instance Walraven, Brand-Gruwel, and Boshuizen (2009) have shown large differences between reported and actual evaluation behavior of students. Moreover, not all Internet users are expected to have similarly advanced information skills as college students, who receive specific training in this task (Lucassen & Schraagen, 2011b).
The results of this experiment demonstrate an important aspect of the 3S-model (Lucassen & Schraagen, 2011; Lucassen et al., 2013). The influence of the three proposed user characteristics (domain expertise, information skills, and source experience) on credibility evaluation have been shown. However, the current experiment has also shown that these characteristics are not isolated from each other. In fact, specific user characteristics do not only influence which information characteristics are incorporated in credibility evaluation, but also have an effect on the other user characteristics. In this case, two potential influences can be observed.
First, we argued that the information skills of our participants (college students) may have been the reason that source cues indicating that the information came from Wikipedia did not raise or lower trust. In an earlier online study, such an effect was found (Lucassen & Schraagen, 2011b), but it can be argued that the information skills of these participants (a broad sample of general Internet users) were overall less advanced. This may mean that they were less informed about the risks of using an open-editing encyclopedia. Hence, the
overall higher level of information skills of the participants in the current study influenced their perception of the source at hand (Wikipedia) in the sense that no authority was attributed to this particular source.
Second, we have shown that topic familiarity (a weaker form of domain expertise) influenced the importance of source cues. Familiar users were more skeptical of Wikipedia, presumably because of the assumed knowledge of the authors behind this source. It can thus be argued that users that are familiar with a particular topic place less trust in a source if the source is known to be a collaborative repository rather than completely unknown. In conclusion, this study provides additional corroboration of the user characteristics as proposed in the 3S-model (Lucassen & Schraagen, 2011b). In addition, we also demonstrate an interdependency of these characteristics. Both domain expertise and information skills influence source experience, and thus the use of source cues in credibility evaluation.
4.1 Future Research
This study has begun to clarify the relationship between source cues, topic familiarity, and trust in online information. Many of the findings are likely to have been caused by the fact that a website featuring user-generated content was used as a case study. Similar experiments with other (authoritative) source could confirm our explanations for these findings. Moreover, future experiments should also focus on other Internet users than college students, since their knowledge on and experience with Wikipedia is expected to be extensive. Other users, who are less familiar with this source may value it differently. Finally, we also suggest to try to design future experiments to be as realistic as possible, in order to ensure external validity. While in this study, we attempted to move some emphasis away from the concept of trust, the imposed task was still quite artificial. Moreover, the participants were very well aware of the fact that they were participating in a lab experiment, which may for instance lead to socially desirable behavior. Online studies for example, could diminish such effects (cf. Lucassen & Schraagen, 2011b).
5. Conclusion
The main contribution of this study is that we have shown that the relationship between topic familiarity and source cues in credibility evaluation is not as straightforward as has been suggested previously. In the case of collaborative, open editing, websites such as Wikipedia, the presence of source cues can have a negative effect on trust of familiar users,
94 |
Chapter 5whereas in general, a positive relationship between source cues and familiarity may be expected. Future research should aim at further clarification of the relationship between these concepts. We have also demonstrated that knowledge of the source of information has a much larger influence on trust when the credibility of the information in itself is ambiguous.
Acknowledgments
The authors wish to thank Jerfy ter Bekke, Esther Brink, Marlyn Bijvank, and Niels van de Sande for their help in gathering data.
Factual Accuracy and Trust in Information: The Role of Expertise
The Role of Expertise
Credibility evaluation has become a daily task in the current world of online information that varies in quality. The way this task is performed has been a topic of research for some time now. In this study, we aim to extend this research by proposing an integrated layer model of trust. According to this model, trust in information is influenced by trust in its source. Moreover, source trust is influenced by trust in the medium, which in turn is influenced by a more general propensity to trust. We provide an initial validation of the proposed model by means of an online quasi-experiment (n=152) in which participants rated the credibility of Wikipedia articles. Additionally, the results suggest that the participants were more likely to have too little trust in Wikipedia than too much trust.