We designed an experiment in the context of citizen science using artificial stimuli. Citizen science is a “partnership between volunteers and scientists to address research questions” (Crall et al., 2011, p. 433) usually culminating in citizens assisting with data collection and/or analysis, defining the research question, or even designing a study while gaining scientific knowledge through their involvement in the research. Wiggins et al. (2011, p. 17) argue that “most [citizen science] projects show greater concern over the lack of contributor expertise than the lack of analysis methods suited to the type of data generated in citizen science.” Many citizen science projects, therefore, seek
knowledgeable contributors and can train contributors to acquire the desired level of knowledge as a means of ensuring data quality.
The target and distractor artificial stimuli used in this study are called tyrans and non-tyrans, respectively. These stimuli were designed following Kloos and Sloutsky’s
(2008) artificial stimuli. Tyrans are a class (species) of artificial insects whose members meet an inclusion rule (a set of attributes and values of these attributes). Stimuli that do not meet this rule are non-tyrans. The inclusion rule is that tyrans have a short tail, two or three buttons on their light blue bodies, blue wings, and either one or two rings on each blue wing. Non-tyrans may look like tyrans but will fail at least one of these
requirements. Each image was presented to participants in Powerpoint Slides. Figure 3.2 shows a sample tyran and a sample non-tyran used in the experiment.
We tested the materials with 12 students from the Department of Biology who are familiar with observing, classifying, and reporting organisms. We tested for the suitability of the prompt to elicit unbiased responses from contributors. We found that asking contributors a non-leading question like “what do you see?” was less biasing than asking contributors to identify the entity they have observed. So we used the prompt “What do you see?” in this study. We also tested for the complexity of the task and the ease of learning the inclusion rule. We carried out another pretest to examine the effect of changes made based on our initial pretest. The participants in the second pretest were fifteen business students who participated for course credit. All participants recorded their sightings on an answer sheet. Based on our findings from the pretests described in Appendix A, we set the display time for each image presented to participants to forty seconds. We also modified the inclusion rule to consist of five of the seven attributes of the target entity. The complete experimental material is available in Appendix B.
Tyran. Follows the inclusion rule: Two blue wings, short tail, light blue body, two or three buttons on the
light blue body, and one or two rings on each blue wing
Non-tyran because it has three rings on each wing. The number of legs is not diagnostic
Several variations of tyrans and non-tyrans were created to test each of the three hypotheses specifically. We presented sixteen images (a mixture of tyrans and non-tyrans) to participants. All sixteen images test the capacity of contributors to report accurate and diverse information. However, six images were selected to be examined for variations of diagnostic attributes and non-diagnostic attributes – three for each attribute type. For example, the antennae, even though non-diagnostic, are shorter in some of the images of tyrans presented than the ones presented in the training/orientation phase of the experiment. The presence of patterns on the wings of some of the tyrans, the number and shape of antennae, and the number of legs on the insect are additional manipulations present in the images.
Tyran (two-lobed antennae and shorter wings) Here a change to a non-diagnostic attribute has occurred
Tyran (shorter wings and different coloured tail) Here a change to a diagnostic attribute has occurred
Four slides containing catch items were placed intermittently among the test item slides (tyran and non-tyran insects) to check if participants paid attention and were alert to the experiment. The catch items were different shapes/coloured stimuli that are not insects, and the participant was expected to correctly report them as non-tyrans or provide specific descriptions of their attributes. The image slides were presented in a non-randomized order to all groups.
3.3.1 Participants
After approval from the University’s Ethics Review Board, 93 participants recruited for the study were assigned randomly to 3 groups: untrained, implicitly trained, and explicitly trained groups. Upon preliminary examination, one report was excluded from the implicitly trained group for incompleteness and another report for inaccurate reporting of catch item. Two other reports were excluded from the untrained group for inaccurate reporting of catch items and one for illegibility. To make the number of reports equal across the groups, we excluded the last report from the implicitly trained group and the last three reports from the explicitly trained group, leaving a total of 84 participants across the three groups whose reports were used for our analysis.
Consequently, each group had 28 participants who were all students of Memorial University of Newfoundland. Fourteen of the students participated for donations to their class graduation. In addition to these fourteen students, ten students participated solely for the chance to win one of two $100 gift cards. Sixty students participated for course credit. Thirty-six of the participants were male, and forty-eight were female.
Participants in the explicitly trained group were provided with an inclusion rule with which to classify stimuli as either tyrans or non-tyrans. They went through a training phase in which they were taught the rule and shown five sample tyrans to allow them to become familiar with applying the criteria in the classification task. Participants were also tested on their knowledge and received feedback on their ability to identify tyrans. This was achieved by presenting them with images and verbally inquiring if they thought it was a tyran or not, and why. After they provided their answers, we showed them the correct response and explained how they satisfied the inclusion rule.
The implicitly trained group was briefed on the task to be performed and shown the same five target stimuli used to teach the explicitly trained group, to allow them to infer the inclusion criteria. However, we did not provide explicit rules to this group, nor did we give them feedback on their ability to determine if a stimulus is a tyran or not. Also, we did not show the Untrained Group any sample images. However, like the other groups, they were informed that we were interested in examining how people report things. More information about the experimental procedure is presented in Appendix B.