Because workload is a multi-dimensional phenomenon, which is related to performance (e.g. changes in both primary and secondary tasks), to subjective experiences (e.g. mental effort and time pressure), and to physiological states (e.g. stress and effort), measuring/assessing workload involves various methods including performance criteria (e.g., the quantity and quality of performance), and subjective evaluation (e.g., the ratings of effort level) (Schvaneveldt et al., 1998) and physiological measurements (e.g. heart rate and alert level). An overview of all these three categories of measurements is given in this section, with some of them being described in more detail.
2.1.6.1 Self-Report on Subjective Experience
Assessing workload based on self-reported experience is one of the most commonly used methods. According to Kahneman (Kahneman, 1973), the definition of mental workload is directly related to the proportion of the capacity that an operator can spend on the task in hand, and the measurement of mental workload is the specification of that proportion (deWaard & Brookhuis, 1997). The workload changes sometimes are not easily observed in driving
performance measurements because drivers are inclined to actively cope with the increasing task demands by adapting their behaviour (Cnossen, Brookhuis & Meijman, 1997). However, the increased workload can be apparent in self-reports of the drivers due to the individual experience when dealing with the task, for example, on time pressure, stress or more
information processing required. As de Waard suggested (de Waard, 1996b), no one is able to provide a more accurate judgement with respect to the experienced workload, especially mental workload, than the persons themselves. Since they are the closest to the essence of workload, Sheridan cited in Wickens, (Wickens, 1984b), considered self-reports as the best measure for workload, although de waard (de Waard, 1996b) suggested that most self-report measures are not sensitive in the A2 region in Figure 2.4. The commonly used self-report methods are summarised in the following sections.
2.1.6.1.1 RSME (Rating Scale Mental Effort)
RSME is developed by Zijlstra (Zijlstra, 1993), including a scale from 0 to 150 mm, with a cross line in every 10 mm, and the higher effort is indicated by higher scale. Along the line, from 1 to 120 mm, 9 levels of statements are given, from “absolute no effort” to “extreme effort”. The amount of effort invested into a task has to be indicated at one point along the line. In this scale, the effort is considered as a whole, i.e., there is no separate evaluation for different types of workload, e.g. the physical and mental workload, known as uni-dimensional scale.
2.1.6.1.2 Activation Scale
The activation scale is comparable to RSME, i.e. a uni-dimensional scale which requires
subjects to mark on the line. However, instead of the effort level statements, on Activation Scale, the reference comments along the line are daily activities like “I’m reading a newspaper”, and the marking for subjects should be relative workload compared to the workload from the stated activities. The scale has a range from 0 to 270 and the measures are taken by the distance from the point 0.
A similar ratings system, knows as Dual-Task Activation Rating, was proposed and used effectively in a previous research on distractions during driving (Angell et al., 2006). In the Dual-Task Activation Rating, ratings are collected immediately after each secondary task. Drivers are asked to rate the workload caused by a secondary task by comparing the experience of completing this task with a reference task (for example turning on the radio), which has a fixed value of 100. Such Activation Ratings should be an optimal method for gaining feedback from drivers of their experience due to the minimised delay in subjective rating, and it can be easily collected during driving test as ratings are only given by the driver vocally, and are recorded by the experimenter on a questionnaire-style sheet. An example of Dual-Task Activation Rating is provided in Appendix B.
2.1.6.1.3 NASA-Task Load Index (TLX)
NASA-TLX is a multi-dimensional workload assessment scale (Hart & Staveland, 1988), which has been employed in numerous studies over the past 20 years. It was developed by the Human Performance Group at NASA Ames Research Centre. NASA-TLX was originally designed for assess pilot workload in the aviation domain, and later was used largely in ergonomics research. NASA-TLX provides an overall workload score based on a weighted average of rating on six subscales: mental demands, physical demands, temporal demands, the own performance, effort, and frustration.
2.1.6.1.4 Driving Activity Load Index (DALI)
Driving Activity Load Index (DALI) is a revised vision of NASA-TLX, which was especially designed for the IVIS-related workload measure in driving circumstance, especially when assess the workload due to using in-vehicle systems (Pauzie & Pachiaudi, 1997). In DALI subjective rating scale, there is a scale rating procedure for six pre-defined factors: the effort of attention, visual demand, auditory demand, temporal demand, interference, and the situational stress. The rating on these factors is followed by a weighting procedure in order to combine the six
2.1.6.1.5 SWAT (Subjective Workload Assessment Technique)
The SWAT workload assessment is a multi-dimensional Subjective Workload Assessment Technique (Reid & Nygren, 1988). It is a subjective rating technique that uses three levels: low, medium, and high, for each of three dimensions of time load, mental effort load, and
psychological stress load to assess workload. It uses conjoint measurement and scaling techniques to develop a single, global rating scale with interval properties. The use of SWAT entails three steps: the first step is called scale development. All possible combinations of three levels of each of the three dimensions are contained in 27 cards. Each operator sorts the cards into the rank order that reflects his or her perception of increasing workload. The second step is the event-scoring, that is the actual rating of workload for a given task or mission segment. In the third step, each three-dimension rating is converted into numeric scores between 0 and 100 using the interval scale developed in the first step.
2.1.6.1.6 Workload Profile (WP)
WP was developed by Tsang and Velazquez (Tsang & Velazquez, 1996) based on the Multiple Resource Theory, MRT (Wickens, 1987b). The Workload Profile asks the subjects to provide the proportion of attentional resources used after they had experienced all of the tasks. The tasks to be rated are listed in a random order down the column and eight workload dimensions are listed across the page. The workload dimensions used in this technique were defined by the resource dimensions hypothesised in the MRT: perceptual/central processing, response selection and execution, spatial, processing, verbal processing, visual processing, auditory processing, manual output, and speech output. The definition of each dimension is given to subjects when they are required to rate on the tasks. In each cell on the rating sheet, subjects provide a number between 0 and 1 to represent the proportion of attentional resources used in a particular
dimension for a particular task. A rating of “0” means that the task placed no demand on the dimension being rated; a rating of “1” means that the task required maximum attention. The ratings on the individual dimensions are later summed for each task to provide an overall workload rating.
2.1.6.1.7 Findings from Self-Reporting
Generally, mental workload assessed by self-report reflect the task demand level, where ratings in control conditions (i.e. with out secondary tasks) are lower than that in dual-tasking
(Lansdown, Brook-Carter, and Kersloot, 2002; Gopher, 1990), lower for simpler tasks than for more complex tasks (Lee et al., 2001), and lower for less resource-conflicting tasks (e.g. auditory tasks) than for high conflicting ones (e.g. visual tasks). In addition, greater time pressure was reported to be significant while undertaking the dual task conditions as opposed to
the primary task alone (Lansdown, Brook-Carter, and Kersloot, 2002). The results from drivers’ self-reported performance changes also suggest that drivers are able to recognise their impaired driving performance when conducting secondary tasks (Anttila & Luoma, 2005; Blanco et al., 2006; Jamson & Merat, 2005; Ma & Kaber, 2005). However, there is also evidence from self-reports to suggest that drivers tend to underestimate the potential severity of these
distractions and continue to multi-task (Anttila & Luoma, 2005; Blanco et al., 2006; Liu, 2001).
2.1.6.2 Task Performance Measures
2.1.6.2.1 Secondary Task Performance Measurements
Secondary task performance can be used to indicate workload level because human beings only have a limited amount of information processing capacity, and after drivers have managed to perform the primary task, the “unused” capacity or Spare capacity (Brown & Poulton, 1961b) is largely available for secondary tasks. Therefore the performance of secondary tasks will reflect changes in the spare capacity and total workload. When interacting with the in-vehicle systems, driving performance is likely to degrade, as the driver devotes greater attention to use the device and less to the other task, the secondary task performance will decrease. The commonly used measurements are: secondary task completion, accuracy (i.e. percentage of correct responses or errors), reaction time, etc. Measurements of secondary task performance have shown to be sensitive to task complexity, task type, and interface design. For example: Some drivers
voluntarily skipping secondary tasks or having the experimenter request to skip them because of safety concerns when performing complicated tasks (Blanco et al., 2006). Drivers shown to be more accurate when responding to message content for messages presented in the synthetic speech (concatenate) compared to the recorded human voice (Harbluk & Lalande, 2005), suggesting one caused less workload than the other. Anttila and Luoma 's (Anttila & Luoma, 2005) found in a field study that, both in visual (arrow) and in auditory continuous memory (cognitive) task, the percentage of correct responses was the highest for the static situation and the lowest when conducted in the urban environment comparing to rural and motorway. The percentage of correct responses is decreased with the increasing task difficulty. These results suggested that the secondary task performance could be an effective measurement for drivers’ workload.
2.1.6.2.2 Peripheral Detection Task (PDT) Measurement
PDT is another commonly used method for workload evaluating from the secondary task performance measure family. It has been confirmed to be valid as an objective measurement for workload or “spare capacity”. It uses various visual secondary tasks (e.g. flashing lights
requires continuous visual information processing, this method reflects the spare visual capacity, and was therefore applied for the safety evaluation of IVISs. Apart from the visual demand itself, some research also found that PDT is also sensitive to cognitive load, as it was found that
mental tasks can affect drivers’ eye movement patterns and peripheral vision (Miura, 1986). Crundall and colleagues showed in various studies (Crundall, Underwood & Chapman, 1999a; Crundall et al., 1999b) that drivers’ PDT task performance decreases when there were traffic hazards presented on videotapes.
However, PDT tasks sometimes still show the biased sensitivity of visual workload over mental or other sense modalities. It was found by Harms and Pattern (Harms & Patter, 2003) that only navigation messages including visual demand, i.e. visual navigation messages and combined visual and verbal messages, had a statistically significant effect on PDT performance. Thus, it is questionable to use PDT-task as the only standard measures for the workload increasing. Apparently, a method which is less dependent on one specific mode (visual) would be more reliable and more general for measuring workload. In addition, the extra visual distraction caused by the PDT task limited the application of PDT in the real-road driving condition.
2.1.6.3 Physiological Measures
Sometimes, when workload change is not easily shown in either secondary task or driving performance, since drivers can actively cope with the task demands by adapting their driving behaviour, they can be apparent in physiological measures (Cnossen, Brookhuis & Meijman, 1997), for example, heart rate or eye movement. Different physiological measures have been found to be differentially sensitive to either a global level or a specific stage in information processing. One of the advantages of physiological measures is that they do not require the drivers’ response, also they can be collected continuously, and most of them are not invasive to task performing. Historically, the workload can be measured by the Cardiac Functions,
Background Electroencephalogram (EEG), Blood Pressure, Respiration, Skin Resistance Response and Skin Conduction Response, Hormone levels, and Electromyogram, etc.
(O'Donnell & Eggemeier, 1986). In this section, only the eye movement measure is described.
The eye movement can actually be considered as both performance and physiological measures (de Waard, 1996b). Many researches have used eye activity measures that correlated with cognitive demands to measure the real-time workload (Ahlstrom & Friedman-Berg, 2005; Van Orden, Jung & Makeig, 2000; van Orden et al., 2001; Brookings, Wilson, & Swain, 1996; Wilson, 2001; Wilson & Caldwell, 2002). Findings indicate that the blink rate, blink duration, and saccade duration (saccade: rapid eye movement) decrease while the pupil diameter, number
of saccades, and frequency of long fixations are increased with the increased visual workload (Iqbal et al., 2005; Rognin et al., 2004; Van Orden, Jung & Makeig, 2000; Veltman & Gaillard, 1998; Zeghal et al., 2002). However, for some eye movement measurements, the mental workload may have the opposite effects as that in the visual workload (Recarte & Nunes, 2003; Recarte et al., 2008). There is a need for careful explanation of the eye movements in tasks as complex as driving, where multidimensional workload is intertwined. Same as other
physiological measures, eye movement measurements are constitutions and not invasive, which provide the opportunity of assess workload in real-time. A brief overview of some eye
movement parameters can be used for workload evaluation is presented below, and the visual functions, measurements, and correlation with workload and performance are discussed in more detail in Section 2.2.
2.1.6.3.1 Fixation Measurements
Fixation is the maintaining of the visual gaze on a single location. Eyes typically alternate
between saccades and visual fixations. Therefore, fixation is normally defined as the pauses intervals between discrete jumps of eye movement (i.e. saccades) during a specific visual task. It is assumed that during these pauses the eyes are to be relatively stable, i.e., “fixated” (Inhoff et al., 2008), and it is during fixation when the visual information is being extracted. Fixation duration and frequency are two commonly used parameters. In researches of observing eye movement patterns (Wilson & Eggemeier, 1991), the frequency of fixation was found to be related to the importance of viewing of instruments (i.e. the more important information being viewed more often), while the length of fixation was generally believed to be related to the difficulty of information extraction (i.e. the more difficult it is, the longer it takes to extract useful information).
2.1.6.3.2 Pupil Diameter Measurement
Pupil diameter increases as a result of Sympathetic Nervous System (SNS) -- innervated muscle groups cause pupil dilation. Kahneman (Kahneman, 1973) suggested that the increased task demands and increased resource investment were reflected in the increases of pupil diameter. The same relationship between workload and pupil diameter was also found by other
researchers (Recarte et al., 2008). However, the largest changes in pupil diameter can be caused by other reasons, e.g. changes in illumination. It is therefore suggested that the pupil diameter is suitable only in laboratory situations (Kramer, 1991).
2.1.6.3.3 Eye Blink Measurements
Blink is endogenous eye movement behaviour. The blink rate and blink duration are studied as sensitive measure for workload changes (Recarte et al., 2008; Velichkovsky et al., 2002). Hancock used the eye-blink frequency as the secondary task measures and found that turn sequences were associated with greater demands on central attentional capacity than straight driving (Hancock et al., 1990). Eye movement measures used to be considered only useful in the assessment of visual demands (Kramer, 1991), in recent years, some researchers have provided evidence that blink can also be a sensitive measure to auditory or cognitive demand situations (Recarte et al., 2008; Yang et al., 2009).
2.1.6.3.4 Eye Saccade Measurements
Saccade is rapid eye movement. Larger saccade amplitude is related to higher visual searching demand. Huestegge (Huestegge et al., 2010) found that saccade amplitude is significantly higher when viewing images with high hazard than those with medium hazard. A recent research found that the correlation between peak of saccade velocity and subjective mental workload during risky behaviour (Di Stasi et al., 2009).