According to Ericsson and Simon (1993), making a video recording enhances a researcher's ability to treat verbal reports as hard data. However, care has to be taken to ensure that the verbalisations are coded accurately and interpreted within a
theoretical structure. Moreover, participants are more empowered by the research when they feel they are contributing to the study rather than being perceived as simply research objects (Nielsen, Clemmensen, & Yssing, 2002).
Using video playback to prompt prior thoughts and actions can be likened to the “mindtape” conceived by Nielsen and Christiansen (2000). These authors theorised
that as users think faster than they can verbalise, using video enables participants to interact with their dialogue after the event. In this way, researchers are able to gain greater insights into respondents’ cognitive processes than by exclusive use of concurrent verbalisation. Similarly a participant’s personal knowledge and skills can be augmented through appropriation of shared cognitive processes with one or more other participants.
Chapter 6 – Multi-crew Verbalisation – M, F, D – Mar 2017 (VL)
6.2.4.3 Transcription and analysis
Data are normally elicited from verbalisation sessions by means of transcription, which is then subject to some form of systematic analysis. This could be a form of coding or qualitative conjecture. Recordings from this study were initially transcribed by a research team member and checked by the author, who completed missing words and interpreted aviation jargon. The corrected transcripts were then made available in table format for ease of analysis.
Both quantitative and qualitative analysis was used to interpret data from
transcripts. Quantitative analysis of the transcripts recorded the number of utterances and length of discussion at each phase of flight. Qualitative analysis was completed by examining the transcripts in full in order to gain insight and understanding into participants’ thought processes and how these related to cognitive load.
In the first instance, the analysis aimed to establish a generalised view of the process and central themes that had emerged. The second (and most important) analysis coded the verbalisations. Using a grounded theory technique, encoding was achieved by merging data to produce categories rather than using a priori determined coding categories,. This grounded empirical approach, initially proposed by Glaser and Strauss (1967), allowed the research team to develop theories connected to the data collection and analysis process.
Of particular interest in this study were participants’ evaluations of the effects of
verbalisation in periods of low, medium, and high workload levels, respectively. The flight phases that were said to be comparable to the different workloads were: HWL – take-off, descent and landing; MWL – pre-taxi, taxi (pre- and post-flight) and
approach; LWL – flight planning at the gate prior to taxi and cruise flight. Figure 6.1 demonstrates the cognitive workload levels at different phases of flight.
The researchers hypothesised that participants’ workload would be highest at take- off and landing and during approach phases of flight, during which times they were expected to make relatively fewer verbal utterances. Conversely, during lower workload phases participants were expected to verbalise more. In other words, the higher the workload the higher the cognitive load and therefore the lower the anticipated ability to verbalise.
Figure 6.1 Adapted from Human factors for aviation basic handbook (Transport Canada, 2010)
Chapter 6 – Multi-crew Verbalisation – M, F, D – Mar 2017 (VL)
6.2.5 Findings and conversations
Smagorinsky (1998) made the assumption that as no-one really knows what is going on in someone else’s head, studies relying on verbalisation are really data- driven inferences rather than empirical evidence. However, Ericsson and Simon (1980, 1984) disagreed, suggesting that characterising a verbal exchange as a verbal report can provide an accurate account of thought processes. Whilst some may argue that the very act of verbalisation alters the task process, Ericsson and Simon argued that at worst it would slow down task performance.
The current study sought to ascertain whether eliciting retrospective accounts of observation immediately after a simulator event to contextualise the process would make it possible to identify participants’ inner thoughts during task analysis. This appeared to be successful as several participants commented that they recalled their thoughts as the scenes of the simulator events were played.
In studies of specialist areas such as aviation, the researcher’s knowledge and experience together with highly-localised analysis required to understand specialised concepts has been found to be critical to understanding how participants construct a task. For example, Wertsch (1985) stressed that activity settings were determined by those who worked within them and argued that this explained how in each
specialisation an understanding of a task is critical and culturally mediated in terms of both personal situations and professional histories. Study development, according to Westsch, thereby reflected the situational concepts of both subject and researcher.
The weakness of this theory is that it can produce a problem of reactivity whereby researchers anticipate a certain response and participants attempt to meet their
resulting from specific questioning, participants in this study were only asked general questions requiring them to express their actions and corresponding thoughts in their own words.
During this study, two kinds of verbalisations were demonstrated – those that were made for and integrated with, task performance and those that were extraneous to task performance. The first type of verbalisation was normal discourse during a flight, including normal talk pilots typically use to maintain flight safety, such as adhering to SOP. Pilots are also trained in CRM, which is aimed at communication within the cockpit and with the multi-disciplinary crew and involves discussing decisions and speaking up if errors are identified. Furthermore, pilots also engage in check and response processes to maintain safety during flight. These include repeating checklists between crew, discussing weather conditions, and entering data into the FMS. Correct CRM procedures should ensure that all communication is appropriate and completed as prescribed and that all crew members operate in a standardised and safe manner regardless of who they are flying with or their degree of familiarity with the FO.
The second type of verbalisation was comprised of any discourse not described above. This kind of communication may occur when crew members are comfortable talking aloud or have a preference for verbalising their thoughts. In the study, this type of verbalisation was primarily used to ascertain whether talking aloud improved or diminished pilot performance. However, on occasion verbalisation was requested by the researchers in order to determine whether it interfered with task performance at any of the three workload phases. This kind of verbalisation is known as verbalisation about rather than for task performance.
Chapter 6 – Multi-crew Verbalisation – M, F, D – Mar 2017 (VL)
The findings described below are based on the video transcripts and the follow-up debriefings. The intention is to provide examples of verbalisations and to identify any possible interference with task performance.
6.2.5.1 Observation 1
In contrast with LWL or MWL situations, in HWL situations no (or little) additional verbalisation was observed.
In normal situations, the term primary artefact is used to describe language as an action (e.g., a teacher might ask a student to complete a particular task, such as “write down this sentence”). In such a situation, choosing which words to use does not
require consciousness or reflection (Wartofsky, 1979). Conversely, as described by Roth and Lee (2007), language can also be about action. In this case, discussion of a task would render language a secondary artefact.
At an operational level (e.g., flying an aircraft) language can be viewed as an action when pilots are asked to verbalise their intentions for the benefit of the other person in the cockpit. This then becomes an explicit tool as they are reflecting on tasks, situations etc. As in many specialised areas, aviation has its own language within its own social context so that a verbaliser will draw on a specific set of words and meanings that are familiar within that environment. These are known as
mediating artefacts. In this case the actors anticipate certain actions verbalised by their use of language.
By comparing low-, medium-, and high-workload phases, the level of verbalisation made by participants according to the relative ease of the different phases of flight can be demonstrated. The transcript segments reported below reflect normal conversation deemed necessary by SOP interspersed with verbalisations over and above normal
communication between the crew expected to complete the task. In low- and to some extent medium-workload flight phases, participants were sometimes observed
verbalising outside standard phraseology. In such instances, they sometimes verbalised thought processes that were task relevant but not required by SOP.
MWL Scenario from Day 1
The excerpt 6.1 below, which is from a MWL (after take-off) scenario from Day 1,