OPORTUNIDAD DE LA SENTENCIA:

When combined, the studies and models making up this dissertation lead to several major implications for research at the intersection between cognitive science and computer science, team cognition and collective intelligence, and multi-agent systems and prediction markets.

10.2.1

We should not limit our understanding of human-

machine teamwork to the principles of human team-

work

Given the technical limitations that have historically affected the develop- ment of AI agents, the research community’s understanding of the difference between human teamwork and human-machine teamwork has been severely limited. This dis- sertation is the first attempt at using reinforcement learning to study human-machine teamwork by leveraging a platform like NeoCITIES, which has been validated to be an effective tool for analysis for human teamwork. As mentioned previously, the differences in scores and situational awareness between the two types of human- machine teams in Study 3 (human-machine-machine and human-human-machine)

suggest that their superior performance compared to human-only teams cannot be attributed merely to the presence of the agents, but rather emerge from dynamics not yet fully understood. Given the long history of team cognition effectively pre- dicting human teamwork in NeoCITIES, the fact that the team cognition model is not a meaningful predictor of human-machine performance undermines the notion that human-machine teamwork can be fully understood through the lenses of human teamwork.

Furthermore, this dissertation as a whole makes a compelling case that there are far better models to understand human-machine teamwork: multi-agent systems and collective intelligence. The former is useful to understand the pivotal role in- centives and game theoretical dynamics play in cooperation between humans and machines. Specifically, Study 2 shows that that most principles of human team- work, even something that is usually thought of as critical as communication, are poor reference points in understanding how humans and machines can work together to accomplish goals. However, Study 1 and Study 2 also show the limitations of the multi-agent system model, because the humans and the RL agents in the experiments often managed to escape the mutually harmful Nash Equilibrium to successfully coop- erate despite the incentive not to do so. In turn, these results show that game theory alone cannot fully explain human-machine teamwork, which would otherwise be the case if human-machine teamwork could be entirely reduced in terms of a multi-agent systems model.

Study 4 on the other hand shows how collective intelligence is a useful model to understand the impact of human-machine interactions on the intelligence of a prediction market. The results also show several different ways in which collective intelligence emerges through different types of relationships between humans and machines: horizontal, in the case of the machine fulfilling a highly differentiated role

as a trading bot, and vertical, in the case of the AI as the aggregator combining the data generated by the prediction market and separately calibrating predictions. However, just like with the multi-agent system model, the collective intelligence model also has limitations in its ability to effectively account for human-machine teamwork. Specifically, the methodology used in Study 4 shows how having an AI play the role of aggregator enables the emergence of collective intelligence in the prediction market despite substantial overlap (50%) in participant knowledge, which runs contrary to the prior literature’s traditional understanding of how collective intelligence emerges in crowds. Indeed, this dissertation shows how the introduction of AI can enable humans to transcend many of the behavioral biases that plague prediction markets, thereby demonstrating how collective intelligence is not tied the marginal trader hypothesis. Thus, it may be more productive to move away from researching human- machine teamwork through analogies to human teamwork, and instead engage in a more interdisciplinary approach that leverages technical advances to not just explore the connections between machine-machine teamwork and human-machine teamwork, but also between multi-agent systems and prediction markets.

10.2.2

Team cognition and collective intelligence are analo-

gous phenomena happening at different scales

Another major contribution of this dissertation is the thorough analysis of the team cognition and collective intelligence literature to classify similarities and differences. Specifically, this dissertation uses the human-machine teamwork setting to show how both models speak to a similar emergent phenomenon albeit a differ- ent scales. While team cognition explains the emergence of a shared mental model as teammates respond to a physical situation, collective intelligence speaks to the

emergence of higher-level information processing as a crowd navigates a probabilistic setting. In both cases, a new and different cognitive process arises in the team that cannot be accounted for at the individual level, but that share many of the same properties that result in higher degrees of performance.

Overall, despite their differences, team cognition and collective intelligence have many core features in common. This dissertation builds upon this insight by showing how the Superforecasting research can help us reframe team cognition and collective intelligence as manifestations of the same cognitive phenomenon. Specifi- cally, researching Superforecasters shows how a large crowd is no longer necessary to have collective intelligence as long as teams are trained through particular cognitive strategies when engaging in forecasting. Additionally, this dissertation’s analysis of Superforecasting opens up a new research landscape for team cognition by showing how team cognition is not limited to perception, military settings, and responses to physical situations, but can also occur in prediction, forecasting, and probabilistic decision-making.

In essence, this dissertation suggests that a new type of intelligence can be unlocked through teamwork that goes beyond what the team cognition literature has been demonstrating. By identifying the analogous nature of the two phenomena, collective intelligence expands ways of thinking about team cognition beyond the situational response setting, and team cognition deconstructs the prevailing notion that collective intelligence is merely the by-product of the law of large numbers in a sample of random guesses. Specifically, as a team of Superforecasters displays both team cognition and collective intelligence, the two emergent phenomena, thought to be separate up until that point, can be unified as complementary aspects of a higher-level form of information processing that in turn can become the foundation for human-centered technology designed to enhance teamwork.

10.2.3

Thinking of prediction markets as human-machine teams

improves collective intelligence by enabling the effec-

tive use of AI

As mentioned previously, the current literature on prediction markets is almost entirely focused on a mechanistic view. To that end, most literature on the design of prediction markets is focused on improving the efficiency of the matching algorithm, on structuring different types of auctions, or on the most effective way to model real world events. This overly technical perspective comes at the expense of asking deeper questions about the nature of collective intelligence and the dynamics underlying its emergence in prediction markets.

This dissertation stands in sharp contrast to that approach, and shows how bridging the gap between the communities of cognitive science and computer science is critical in order to design more effective prediction markets. Specifically, Study 4 shows through both its methodology and its results how to conduct more robust experiments with prediction markets, which in turn supports new research validating teamwork approaches to the design of new and better prediction markets. Through this framework, a prediction market thus becomes a larger form of a human-machine team where multiple humans and multiple machines work together to generate the data necessary for the aggregator to improve the system’s collective intelligence.

In essence, by thinking of collective intelligence as a macro-form of team cog- nition, we were able to design a new type of prediction market that leverages both human as well as machine intelligence to aggregate local knowledge effectively to pro- duce very accurate estimates of probabilistic outcomes. Our design shows that AI can play a role in enhancing collective intelligence by both participating and trading alongside human participants to nudge them towards more effective forecasting, and

also as the overall aggregator of the information generated by the prediction market in order to calibrate its predictions and thus offset many of the behavioral biases identified in the literature.