This section discusses potential risks and pitfalls of human-automation interaction. It is important to understand that perfectly working automation that works exactly as supposed to and never fails or misinterprets a situation is not the source of problems with automation that is discussed below. We assume imperfect automation that will not act all the time objectively correctly. Very similar to human information processing (cf. Fig. 2.5), automated control systems alsosense,interpretandact, which is called aclosed control loop(cf. Fig. 2.9).
Controller Process Measurement Command input Manipulated variable Controlled output Disturbance input Feedback signal
Figure 2.9: A closed control loop consisting of the controller (act), processor (interpret) and measurement device (sense). Redrawn from [61].
Unfortunately, each of those steps is subject to potential errors. Sensory measurements of the environment can be noisy, inaccurate or even incorrect. This can lead to performing an inappropriate action or not performing an appropriate action. As a technical system consisting of hardware and software, it is probably not possible to guarantee 100% perfection. Sensor measurement errors, actuation failure, undefined system states, hardware defects, incorrect or outdated digital maps, locating inaccuracies, etc. can be minimised at most, but not entirely avoided. We use the termautomation failurefor any incorrect system behaviour in consequence of any errors occurring in the information processing chain.
Delegating a previously manually executed task to an automated system changes the role of the operator and the nature of work from active control to passivesupervisory control[125]. The operator is no longer part of the control loop, is out-of-the-loop, but monitors the actions executed by automation. If the automated control does not work as supposed to, the operator must take over control and correct the automated system. Nearly all automation involves humans supervising [125, 144, 148]. This shift of roles can lead to a number of risks, manifesting itself in lower operation performance. Many authors have documented those out-of-the-loop performance problems (e.g. [11, 172, 176]). Endsley and Kiris identified theloss of situational
awareness of the state and processes of the system as one of the major issues associated
with automation [51]. Situational Awareness (SA) means "the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the
projection of their status in the future" ( [51], p. 2). This particularly involves the perception, understanding and anticipation of system state variables. Degraded to a passive observer, the situational awareness of the operator is likely to decrease due to (1) insufficient supervision, (2) a change of feedback type, (3) insufficient automation transparency, and (4) complexity [104]. With correctly working automation this is not too problematic. But it can be as soon as manual intervention becomes necessary and when the operator does not possess a complete mental model of the situation and does not know the state of the system. With those fears in mind the expected reduction of workload can be foiled. Billings refers to observations that automation may decrease workload when it is already on comparatively low level, and that workload may increase by automation when it is high anyway [11].
Another potential problem with automation is the loss of manual skills. With a task au- tomated, motor or cognitive skills are likely to deteriorate through the lack of continuous manual training (deskilling). At the same time, the operator is expected to take over control in case of automation failure and be able to perform the task perfectly to the full extent. Wiener and Curry report the fear of aircraft personnel to lose their skills through extensive use of automation and to be inefficient and inaccurate in case of need [176]. Applied to automated driving, the use of automated longitudinal or lateral control systems over a long time may lead to potentially inaccurate control actions in the first moments, when the control is taken over by the driver. However, theories say that the better the operator mentally reproduces the actions executed by the supervised automated system, the lower is the effect of deskilling [104].
The most decisive factor in human-automation interaction congruently is the trust in the
automation. Trust is tightly tied to thereliability of an automated system. A system that often
fails will lead to less trust than a system that very rarely fails. Parasuraman and Riley [125] describe the influence of attitude towards automation in terms of different uses. Low trust in automation will probably lead to underutilization of an automation system (disuse). In contrast to that, overreliance on the automation can lead to less supervision (misuse). Using an automated system for a different purpose than it was built for isabuse.
Errors resulting from inappropriate reliance on automation can be expressed in two types of errors: following an incorrect decision directive of an automated system is calledcommission
error, whereas not noticing a problem that is also not detected by an automated system is
an omission error, cf. [77, 113]). To be precise, the error is always made by the human by
relying on the imperfect automated system. In this work, we will use the terms commission and omission errors more liberally also for the description of the errors made by automation itself (cf. chapter 4).
The establishment of trust in an automated system is mainly influenced by frequency, characteristics and comprehensibility of automation failure [104]. If it is obvious to the operator, why the automation failed in a certain situation, the operator rather accepts it, which thus does not necessarily lead to a decrease of automation trust. However, if a system frequently fails without the cause becoming apparent, the trust in this system is very likely to be low.
Acceptance of automation failure also depends on the usefulness of the system. Whereas underreliance on automated systems (and therefore more manual work) only misses the desired effects of workload reduction and increase of efficiency, overreliance (also referred to as
complacency, cf. [37, 112]) can result in serious safety issues. Complacency usually involves
an insufficient supervision behaviour which potentially results in overlooking an automation failure. Oakley et al. point out an inverse relationship between reliability and detection rate of automation failures [121]. The more performant, reliable and "better" the automation, the more undesired effects like overreliance and complacency are likely to occur. Ironically, the better the automation (still less than 100% reliable), i.e. the less likely an automation failure, the less likely an automation failure will be noticed [5]. In other words, automation will fail, when it is most needed [118]. Wiener and Curry state that "the question is no longer whether one or another function can be automated, but, rather, whether it should be, due to the various human factor questions that are raised" [176], p. 1. So, human error follows automation error [117, 135].
Repeatedly there are reports on aircraft pilots so confiding in their autopilot system that they neglect their monitoring duties [114]. Deviating hundreds of miles off course and even not noticing warning messages from air traffic control [163, 164], the impact of inappropriate automation trust becomes clear. Disregarding supervision can also result from a lack of
vigilance. "Vigilance refers to the ability of organisms to maintain their focus of attention and
to remain alert to stimuli over prolonged periods of time" ( [171], p. 1, from [36]). A lack of stimuli in a task can lead to monotony and boredom [63], and eventually to fatigue [39]. In order to avoid hypovigilance, tasks must be activating. Yerkes and Dodson were the first to describe the relationship between arousal (or activation) and performance [181]. According to their theory, performance increases with arousal, but only to a certain point. With even more arousal, performance decreases again. This relationship can be depicted in an inverted parabola (cf. Fig. 2.10). Galley [57] takes the view that there is not a single optimal point where performance is best, but there is a broader range of good performance, if the arousal level is not too low and not too high. They agree that too low activation (monotony, fatigue) and too high activation (stress, overload) affect performance negatively.
Wiener and Curry [176] come to a similar relationship (cf. Fig. 2.11) and point out the independence of automation control and monitoring functions. They also locate fatigue at the other end of the scale, resulting from long-term high workload.
There seems to be consensus in the whole research community that the level of automa- tion should be intermediate, in order to mitigate side effects. In the context of automated driving, Hancock et al. argue in favour of a driver-in-the-loop architecture, that should be flexible enough to match the wide range of abilities and skills of drivers [69]. But even in a shared control setting, perhaps with dynamically changing LOAs, confusion over authority, i.e. who is responsible for what, is likely to happen [147]. Sheridan and Parasuraman [145] propose a formalised methodology based on expected-value analysis in order to decide whether a human or the machine is most suitable for responding to failures. Another issue potentially resulting from shared control with changing levels of control is mode confusion [140, 141]. Due to a
P e rf o rm a n c e Arousal
low medium high
Yerkes-Dodson
Galley
Figure 2.10: Relationship between arousal and performance. Graphs according to Yerkes and Dodson [181] and Galley [57].
Control Functions Auto Manual Monitor Functions Manual Auto Computer Monitoring Pilot Controlling Computer Controlling Pilot Monitoring - High Workload - Fatigue - Boredom - Complacency - Erosion of competence
Figure 2.11: Relationship between control and monitoring functions and the corresponding implications for the human operator (redrawn from [176]).
lack ofmode awareness, the operator is not clearly aware of the system mode and the tasks this mode implies for him to execute, or the operator assumes the system to be in a different mode it actually is. This problem could also be shown by Petermann [129] in an experiment with changing automation modes in an automated driving scenario.
Norman sees the main source of many problems with automation in inappropriate feed- back rather than over-automation [118]. However, Woods warns againstclumsyautomation, that prompts the human operator with system feedback during a high workload situation [179]. Gao et al. promote the sharing of automation- related information, which may improve reliance and promote the cooperation between human and machine. Also the sharing of reliance information based on the current context could contribute to a working human-automation interaction [4].