There are many conclusions to be drawn from the work presented in this thesis. Here we walk through many of the salient conclusions organized into a "path" for crowdsourcing system designers to follow. Our articulation of the path itself is arguably a contribution to the crowdsourcing literature, but our intent is simply to walk the reader
thesis. Borrowing from military command and control doctrine, we use the OODA Loop--Observe, Orient, Decide, Act (Boyd, 1987)--as a mental model for this path, except we include a fifth step: Leverage.
6.1.1 Observe: A mix of expertise exists in the crowd.
As the starting point on the path for building crowdsourcing systems to leverage mixed expertise, crowdsourcing system designers must first "observe" that a mix of expertise even exists in the crowd. The literature review in Chapter 2 made it clear that human expertise likely exists on a continuum. The work in this thesis has given us more reason to believe this is true. Kurator taught us that there exists in the crowd some latent expertise for predicting how parents make decisions about sentimental audio recordings. If the "crowd" is extended to include family members, a mix of expertise is plain to see in the combination of "expert" parents, less expert family and friends, crowd workers with some expertise, and crowd workers with little expertise. Question Finding further developed this belief by showing us there are specialized subcrowds with various levels of scientific expertise, specifically within the topics of DNA, the universe, the solar system, and space and biology, in general.
6.1.2 Orient: Choose the "right" topic and type of expertise.
Knowing there is a mix of expertise is only helpful if designers know how to "orient" the expertise topics and types that their system will use. Our notion of "right" means there are some topics and types that are better choices than others, but there is likely not a single correct topic or type of expertise. The study of Question Finding illuminated the importance of scoping a topic appropriately, paying careful attention to the breadth of the topic, the widespread accessibility of information, and if there is a
sufficient depth of information on the topic. As well, throughout this thesis, we have seen varying types of expertise, from fact-based (Escalier) to opinion-based (Kurator). A combination of the two (Question Finding) proved useful, too. Designers ought to consider how well-structured or ill-structured their domain is (Voss and Wiley, 2006), which will guide their choice of expertise type (see Chapter 2 for more details).
6.1.3 Decide: There are ways to identify some of that expertise.
Equipped with a chosen expertise topic and type, designers must decide how they will identify expertise in that topic. This thesis partially worked out several ways to identify a mix of expertise in the crowd. Kurator tracked the crowd's expertise as a performance-based measure, possibly to be used later in weighted voting. The Question Finding study borrowed a validated survey instrument, added to it, and used the
subcomponents of the survey for topic-based expertise measures. That study also suggested it might be useful to crowd-source subjective judgments about question complexity. Designers can use other established metrics, similar to how Escalier used social network analysis (e.g., Zhang et al. 2007) and web browsing-based discount expertise metrics (Hung and Ackerman, 2015).
6.1.4 Act: Build an expertise-aware system without having expertise at the start. Designers may find themselves desiring to build a system to incorporate a topic and/or type of expertise that has not already been established. This thesis demonstrates at least one way to "act" on that desire to build without the need to use expertise right away. Expertise layering is a useful guiding concept for system builders, and we demonstrate its effectiveness with Escalier and Kurator. Expertise layering avoids "expertise cold start",
Escalier is an example of a social navigation system that can leverage mixed expertise when the expertise becomes available, and Kurator is similarly built to be effective "out of the box" but can improve as experts contribute to the tasks.
6.1.5 Leverage. Expertise is additive
Finally, even with a workable crowdsourcing solution that successfully accounts for mixed expertise, designers should remember there is more benefit to be gained if they continually "leverage" new data, new expertise, and new expertise measurement
algorithms or approaches. This is possible because expertise is additive. The literature on expertise finder systems taught us that, as systems evolve over time, more and more data and algorithms were added into systems to continually refine the expertise measures. Thus, where data is available and relevant, designers should continue to leverage it.
This path of system design considerations for incorporating mixed expertise is surely not the only path. However, the work in this thesis has demonstrated this path is workable, and we offer it as one way we believe is helpful to crowdsourcing system designers.
6.2 Limitations
There are many limitations to the studies presented in this thesis. Here we draw the reader's attention to two salient ones.
First, the systems and studies in this thesis primarily used Mechanical Turk, which means our results are generalizable only to paid microtasking environments. Paid microtasking environments are designed for micro tasks. If a task is really trivial, then maybe mixed expertise does not matter. Perhaps mixed expertise only matters for tasks
expertise in paid microtasking environments will likely have to face a tension between breaking a problem into trivial tasks while also maintaining enough complexity to
maximize the value of using human input. We did not examine this tension, so we do not know the effects it will have on mixed expertise crowdsourcing system design.
Second, in our study of Kurator, we speculate about the benefits of using weighted crowd voting. Tracking crowd workers' expertise is only helpful if the expertise can be leveraged, and weighting inputs from known13 workers is a straightforward way to account for their expertise. Although we assume this is a workable approach, we do not examine this idea of weighted voting. Doing so, and showing it is indeed workable, would bolster the strength of our argument that leveraging mixed expertise is worthwhile.