The second part of this chapter will describe the methodological paradigm that this thesis followed: Naturalistic Decision Making (NDM). It will describe how Cognitive Task Analysis (CTA) can facilitate the collection of real-world data, and specifically describe the Critical Decision Method (CDM) interview technique that
35 was used to interview commanders from the emergency services. NDM seeks to understand the cognitive processes associated with choice implementation ‘in the wild’ (Gore, Banks, Millward, & Kyriakidou, 2006; McAndrew & Gore, 2013). Rather than impose artificially constrained experimental methodologies on research, NDM emphasises the importance of context, domain specific expertise and macrocognition (i.e. complete cognition) in team-based environments (Stanton, Wong, Gore, Sevdalis, & Strub, 2011). An overreliance on ecologically invalid lab based settings is inappropriate for problem solving research (Schneider & Shanteau, 2003) as the real-world is characterised by ill-structured problems, uncertainty, poorly defined goals, multiple feedback loops, time constraints, high stakes, multiple players and conflict between personal ideals and contextual requirements (Orasanu & Connolly, 1993). For instace, the influence of social interaction on team-based choice cannot be replicated in randomised experiments with participants who are unknown to one another. NDM strives to achieve meaningful, grounded and detailed conclusions by acknowledging the importance of personal experience (Klein, 2008) and social dynamics (Allwood & Hedekin, 2005).
NDM is based on the assumption that cognitive processing in the real-world varies, situation assessment is critical, mental imagery is important, the decision making context must be specified, decision making is dynamic, and research should focus on how decision makers actually function rather than how they ought to function (Lipshitz, 1993). Researchers must be flexible and open-minded when exploring their data, which in turn facilitates the management of large and ‘messy’ data sets to enable the discovery of novel or previously missed psychological phenomena. For example, analysis of observational data in the real world has facilitated a greater understanding of how human behaviour progresses over time and interrelates with social relationships (Faraone & Dorfman, 1997; O’Connor, 1999). The ‘cognitive interviewing technique’ has helped to improve knowledge on the cognitive processes associated with expertise and ‘Recognition Primed Decision Making’ (Klein, 1998; 2008). Klein (1998; 2008) utilised flexible NDM methods to discover how expert firefighters did not compare options to find the best response when faced with difficult decisions, but instead evaluated options in rapid singular succession to find one that was good enough. NDM methods have been used successfully with the emergency services, and are an appropriate paradigm to follow. Furthermore, as NDM methods are flexible and so allow researchers to develop and
36 test theories during the course of data collection, by for example, asking interviewees about developing theories as they emerge, this facilitated the generation of theory during the initial interviews that were conducted for this thesis.
Cognitive Task Analysis (CTA) is the key method used by NDM researchers to “systematically identify key cognitive drivers” (p.4; Crandall, et al., 2006) that influence decision making in naturalistic settings. It is an approach to real-world research that has been successfully applied to a variety of domains including the military (Cannon-Bowers, Bowers, Stout, Ricci & Hildabrand, 2013; Drury & Darling, 2007), aviation (Keller, Leiden & Small, 2003), intelligence analysis (Tecuci, Boicu, Ayers & Cammons, 2005) and emergency service response (Prasanna, Yang & King, 2009; Wong, Sallis & O’Hare, 1997). CTA is systematic approach to facilitate research on the expertise and ‘macrocognition’ (i.e. expertise in natural contexts) that individuals use to solve problems in cognitively complex, dynamic and uncertain environments (Gordon, 1995). It focuses on context rich descriptions of decision processes rather than decision outcomes (Zsambok, 1997). Importantly, CTA is an approach to research that includes a variety of different methods. The ‘Critical Decision Method’ (CDM) interview technique is a one type of CTA method used to collect rich and detailed data on the cognitive processes used by experts when responding to challenging events (Crandall, et al., 2008).
A primary goal of CTA is to develop an understanding of an experts’ mental models, perceptual skills (including subtle cues), sense of typicality, routines and declarative knowledge (Klein & Militello, 2004), whilst ensuring that findings have applied value to practitioners (Klein, Calderwood & McGregor, 1989). As this thesis sought to explore the concept of decision inertia in emergency response settings, the CTA process was an appropriate approach to follow. CTA moves beyond description of the steps required to perform a task and towards a greater understanding of the knowledge, skills and strategies used by experts (Klein & Militello, 2001). The CTA researcher is encouraged to immerse themselves in the practitioners’ world, creating an alternative mode of enquiry to laboratory-based research that often fails to embrace data source complexity (Gore, et al., 2006). CTA is ‘cognitive’ as its focus is on thinking and reasoning; it is concerned with ‘tasks’ as it is interested in the desired outcome of thought processing; and it is a type of ‘analysis’ that seeks understanding for how the component parts of cognition relate with task outcome
37 (Crandall, et al., 2006). This chapter will briefly outline the three key phases to conducting a CTA.
2.3.1 Phase one: Knowledge Elicitation (i.e. What do people know? How do they know it?)
The first phase to conducting a CTA is ‘knowledge elicitation’. This is when the researcher identifies the type of data that will be collected from participants (Crandall, et al., 2006). For example, is the research interested in eliciting knowledge through interviews, observations or questionnaires, and will data focus on descriptions of complex procedures or challenges to decision making? The knowledge elicitation phase starts during study design, as the researcher considers the type of data they are interested in and the most appropriate methods they can use to access this data. For example, a researcher interested in understanding how emergency department doctors expertly adapt to unanticipated medical emergencies would learn little from interviewing inexperienced medical students in the emergency department, or experienced doctors working on general wards. Knowledge elicitation is central to early CTA planning. Table 2.1 outlines the ‘framing questions’ (Crandall, et al., 2006) and answers that were used to develop knowledge elicitation aims for this thesis.
Table 2.1: Framing questions to guide the aims of knowledge elicitation Framing Question Answer
1) What issue or need do you plan to address?
To explore the potential causes to decision inertia in emergency response contexts.
2) What will you deliver at the end of the project?
An outline of the main impediments to strategic decision making and contributory causes to decision inertia, with recommendations for how to overcome them.
3) What sorts of people can tell you about this issue?
Experienced commanders from the Police, Fire and Rescue and Ambulance services, who are qualified to command at ‘tactical’ (or ‘silver’) level and upwards. 4) What aspects of
expertise or types of cognition do you need to know about?
The cognitive barriers to decision making that participants’ have found difficult to deal with in the past, along with the expert knowledge/techniques they used to cope with them.
5) What type of situation will tell you the most about the issue you are exploring?
Incidents that are high-stakes, complex, and with irreversible challenging consequences.
38 Once the aim of knowledge elicitation is established, it’s important to identify the process that could achieve these aims. Once more, self-reflective questions can help to facilitate this process (Crandall, et al., 2006). Specifically, researchers should think the data content that they are interested in (i.e. retrospective recall, real-time verbalisation or hypothetical scenarios; simulated or realistic incidents; easy or challenging events) and what data collection method is therefore most suitable (i.e. interviews, self-report, observation, think-aloud). Table 2.2 outlines the questions and answers that helped guide the knowledge elicitation process for this thesis’ two main CTA studies.
Table 2.2: How the planning for ‘knowledge elicitation’ was structured in this thesis
Research Question (RQ) Data Collection Data Content RQ1: What are the main
challenges to decision making as experienced by command level decision makers in the emergency services? Interview (Critical Decision Method) Retrospective Realistic Challenging
RQ2: Are the challenges to decision making as identified by RQ1 evident during a real-time simulation exercise involving command level decision makers from the emergency services?
Self-report (Questionnaires about simulated exercise) Real-time Simulated Challenging
2.3.2 Phase two: Data Analysis (i.e. structuring the data, identifying findings, discovering meaning)
Data analysis is the second phase to a CTA study. This is where the researcher starts to structure, identify themes and discover the meaning from their data (Crandall, et al., 2006). Analyses in CTA projects are predominantly qualitative and can include techniques such as thematic analysis (Braun & Clarke, 2006); grounded theory (Glaser & Strauss, 1965); and content analysis (Vaismoradi, et al., 2013). As traditional qualitative techniques tend to focus on social processing, it is important that the CTA researcher does not lose their focus on cognition (Crandall, et al., 2006). The data analysis phase of a CTA includes preparing the data for analysis (i.e. transcribing data); data structuring and initial coding (i.e. annotate transcripts;
39 catalogue nodes); discovering meaning (i.e. refine codes; identify themes); and finally linking findings (i.e. link to relevant theories; relate to practitioner) (Crandall, et al., 2006; Liamputtong, 2009). Depending upon the focus of the research questions identified in phase one, analyses will follow a structured approach to test for existing theories (i.e. deductive) or an explorative approach to discover emergent themes (i.e. inductive) (Wong, 2003). This thesis sought to develop a theoretical understanding of the main challenges to emergency commanding that may contribute to decision inertia; thus it favoured an inductive grounded theory approach to analysis, which is discussed in more detail below.
2.3.3 Phase three: Knowledge Representation (i.e. displaying data, presenting findings, communicating meaning)
Knowledge representation is the final phase to a CTA. It describes the means by which the researcher communicates their findings to the wider academic and practitioner communities. It is central to CTA, as a key strength to NDM research is that it can provide practitioner-based recommendations (Wong, 2003). The inherently qualitative nature of CTA projects means that there are often voluminous and text-heavy descriptions of cognitions and behaviour that are derived from the data. It is contingent upon the skills of the researcher to reduce and clearly convey the main message to be learnt from the CTA project (Liamputtong, 2006). For example, the translation of findings in the current thesis into visual models was important to translate dense and text-heavy chapters. Possible ways to present CTA data include the use of narratives that describe the story of how knowledge is derived; chronologies to communicate the structure of knowledge through time; or decision requirements tables to show the elements that interact with specific decisions (Crandall et al., 2006). Another popular method for knowledge representation is the use of concept maps. This involves the presentation of different ‘concepts’ in node format, which can then be visually linked to one another in order to explain the relationship between different concepts (Crandall, et al., 2006). Indeed concept maps can be used during earlier data analyses phases in order to facilitate the structuring of data into themes and codes (Bazerley & Jackson, 2013). Thus, as a whole, the CTA method facilitates the collection of data from real-world environments as it offers a systematic and scientific framework to structure a research project whilst, importantly, maintaining a context-rich and meaningful
40 approach to conveying real-world findings. In order to provide a worked example of how CTA can facilitate real-world research, a detailed description of the CDM interview protocol that was used for this thesis will now be discussed.