Besides the verification of the autonomous vehicle system in simulations, the runtime monitoring is also used during the validation of autonomous vehicle systems in the real world (cf. right side of Fig. 4.1). Similar to the system verification in simulations, the runtime monitoring framework access the system data via the identical data interfaces as in the simulations and processes an abstract representation for each processing cycle of the system. The abstract function evaluates the correctness of the system part under monitoring based on the identical properties as they have been used in the simulations (cf. operation qualitative monitoring in Fig. 4.1). The qualitative runtime monitoring during operation enables the identification and mitigation of faulty system behavior during operation — even in situations which have not been considered and verified in the development of the system. The quantitative runtime monitoring uses the set of simulated situations as the reference set for the monitoring of situations during operation in the real world. During operation, the reference set is compared to the encountered abstract
situations and any abstract situations which have not yet been considered in the system
development are recorded (cf. operation quantitative monitoring in Fig. 4.1). Neither the set of simulated situations nor the set of encountered situations during operation will fully include all situations which are possible in the real world (cf. Section 3.8.1). The knowledge about unconsidered abstract situations and faulty system behavior can be used for the improvements of the specification, design, implementation, and verification for autonomous vehicle systems in further developments. An extensive V&V represents a key factor for the successful development of correct and safe autonomous vehicle systems. Our engineering approach supports the testing of autonomous vehicle systems
4.3. Seamless Development and Operation of Autonomous Vehicles
in simulations by enabling the definition of new test scenario and test cases from the recorded abstract situations during operation in the real world. The new test scenarios and test case are envisaged to increase the test coverage of the autonomous vehicle systems by verifying the behavior of these systems in the unknown traffic situations.
Test scenarios describe timed sequences of scenery changes and maneuvers by dynamic
objects in the environment of the autonomous vehicle system. These sequences are identified in the recorded traces of abstract situations from the runtime monitoring of the autonomous vehicle system during operation in the real world.
Definition 4.5 (Test Scenario). A test scenario defines a timed sequence of scenery
changes and maneuvers by dynamic objects in the vicinity of the system under test (SUT) for simulation-based tests.
Test cases are derived from test scenarios by parametrization. Parameters of a test
scenario still correspond to the abstract situations and do not correlate to the required concrete parameters of simulation frameworkssimulation framework. For example, abstract
situations may only reason about the relative velocity between two vehicles whereas
the simulation framework requires the actual velocity of each vehicle for its simulations (cf. Chapter 8). The parametrization estimates missing concrete parameters for the simulations from the available parameters of abstract situations in the test scenarios. Novel abstract situations in the simulations of these new test cases extend the reference set for the runtime monitoring of autonomous vehicle system during operation in the real world.
Definition 4.6 (Test Case). A test case represents the parametrization of a test
scenario in which all necessary parameters for the simulation have been defined.
The analysis of the operation of autonomous vehicle systems in the real world with by the runtime monitoring framework and the usage of runtime monitoring results for the evolution of autonomous vehicle systems are described in more detail in Chapter 7.
5. Monitoring Architecture
Runtime monitors represent the crucial part of the holistic engineering approach for autonomous vehicle systems (cf. Section 4.3). The runtime monitoring framework enables to qualitatively and quantitatively evaluate the behavior of autonomous vehicle systems and to record corresponding data for the seamless transfer between system development and system operation. Nevertheless, the seamless transfer of operational data requires a consistent architecture of the autonomous vehicle systems and the runtime monitoring framework throughout the complete system life-cycle. The architecture of the corresponding runtime monitoring framework consists of five layers (cf. Fig. 5.1).The two bottom layers address the basic structure of autonomous vehicle systems and their integration into simulation frameworks for their verification (cf. Section 3.6). The upper three layers represent the runtime monitoring framework.
Many runtime monitoring techniques exists with their individual benefits and drawbacks (cf. Section 2.3). The runtime monitoring framework in this work represents an external runtime monitoring, that does not require any instrumentation of the systems’ source codes. The framework non-intrusively monitors the behavior of the autonomous vehicle systems solely based on the operational data that is available by defined interfaces from these systems. The autonomous vehicle systems present themselves as black-boxes to the runtime monitoring framework — no knowledge about the internal structure of these systems nor any internal data and control flows is available for the runtime monitoring framework. Autonomous vehicle systems have to provide the interfaces consistently throughout the complete life-cycle (cf. Fig. 2.2) in order to enable the runtime monitoring framework to access the systems’ runtime data persistently. The correctness and safety of autonomous vehicle systems are qualitatively and quantitatively evaluated by the runtime monitoring framework based on the abstraction of the provided runtime data (cf. Fig. 5.1). In the following sections, we will describe each layer of the runtime monitoring architecture in more detail. We commence with the description of the system layer.
The runtime monitoring is executed alongside the autonomous vehicle systems (cf. Fig. 5.1). Runtime monitoring and systems may share the same hardware components, like processors, memory, and bus communication systems. For the validity of the runtime monitoring, the runtime monitors must not interfere with the operation of the autonomous vehicle systems. Otherwise, the behavior of the monitored autonomous vehicle systems may diverge from the behavior these systems will exhibit during operation without the runtime monitoring. In such a case, results from the runtime monitoring are not applicable to the general operation of these systems without the runtime monitoring. Following [GP10], four properties are defined for the runtime monitoring framework:
Situation Monitor Input Abstraction Output Abstraction Oracle Abstract Function Tested Situation Knowledge Oracle System Layer Qualitative Monitoring Layer Quantitative Monitoring
Layer SituationMonitor
Simulation
Layer Simulation Framework
Simulated World Conform /Unconform Tested / Untested Abstraction Layer Log Unconform Situation Knowledge
Preprocessing Function Postprocessing
World World Untested Situation Knowledge Log Situation Oracle Conformity Oracle
Figure 5.1.: Monitoring architecture for simulations at design time.
Dependability The dependability of the autonomous vehicle system is equal or greater
compared to the dependability of the systems alone. The runtime monitoring must not introduce any faults that will diminish the overall dependability.
Functionality The runtime monitoring must not introduce, change, or remove any
functionality of the monitored autonomous vehicle systems.
Schedulability The runtime monitoring must not lead to violations of real-time guaran-
tees for the monitored systems.
Certifiability The runtime monitoring does not require instrumentation or other modifi-
cation for the source code of the autonomous vehicle systems.
The only context in which the runtime monitoring is allowed to violate these properties is a specification violation by the monitored autonomous vehicle systems. In case the runtime monitoring detects incorrect and unsafe system behavior, safety measures have to be initiated in order to reduce the emerging safety risks by the faulty system. As the original behavior of the system is already compromised, the transfer of the corrupted systems into a safe state with acceptable risk has the highest priority (cf. Definition 2.8) Safety measures are considered in this thesis as part of the runtime monitoring, but their selection and execution are not explicitly addressed. The selection and execution of appropriated safety measures in critical situations are too complicated in order to sufficiently address it in this work. The reader is referred to other work in this field, i.e., [Hör11; RM15].
5.1. System Layer
The system layer represents the basic structure of the autonomous vehicle systems (cf. Section 2.1.3). As control systems, autonomous vehicle systems have to continuously
5.1. System Layer Process Output System Boundary Actuator Actuator Function Postprocessor Postprocessor Postprocessor Function Function Preprocessor Preprocessor Preprocessor Sensor Sensor Input World
Figure 5.2.: The segmentation of autonomous systems following the IPO model.
adapt the behavior of in reaction to changes by the ego vehicle and the world, e.g., other dynamic vehicles (cf. Fig. 5.2). Feedback loops allow these systems to consider changes of other vehicles in the environment in their processing (cf. Section 2.1.5). Autonomous vehicle systems manipulate the position and pose of vehicles in the world based on information about the environment from the vehicle sensors. Changes of the autonomous vehicle systems for the position and pose of the vehicles in the environment are executed by the vehicles’ actuators, e.g., brakes, steering, and engine. Sensors and actuators represent the interfaces between the physical world and the digital systems. As shown in Fig. 5.2, autonomous vehicle systems can be split into three segments — the input, process, and output segment. This segmentation originates from the input-processing-output (IPO) model (cf. [Goe10]) correlates to the architecture of control systems (cf. Fig. 2.5).
The input segment addresses the perception of the own vehicle and the real world by proprioceptive and exteroceptive sensors. The perception includes the preprocessing of sensor data into a comprehensive representation which acts as the input to the process segment. For the lane change assistant (cf. Section 3.1), the input segment correlates to the environment perception (cf. Section 3.3.1.1).
The process segment resembles the main control algorithms for the behavior planning and decision making of the autonomous vehicle systems. Based on the input from the
input segment, changes for the behavior of the vehicle are processed and forwarded to
the output segment. For the lane change assistant, the process segment correlates to the
situation assessment (cf. Section 3.3.1.2) and behavior planning (cf. Section 3.3.1.3).
The output segment addresses the execution of planned behavior by vehicle actuators. The intended behavior from the process segment is preprocessed into inputs for the different actuators, e.g., brakes, gearbox. The execution of actuators leads to a change of the vehicle state in detail and the world state in general — including states of surrounding dynamic objects. These state changes require the autonomous vehicle systems to revise their assumptions about the world state reiteratively and to adapt the vehicles’ behaviors. For the lane change assistant, the output segment correlates to the modules of the vehicle stabilization and hardware actuators (cf. the postprocessing of target points in Section 3.3.1.3).
Though the segmentation of autonomous vehicle systems may vary between different system instances (cf. Fig. 5.2), the three segments — input, process, and output — will exists for any autonomous vehicle system. Sensors — input — and actuators — output —
are mandatory for the interaction of the systems with the real world. For the vehicle control, a function — process — has to process targets for the vehicle actuators based on the inputs from the vehicle sensors.
Autonomous vehicle systems consist of multiple software and hardware components (cf. Fig. 2.7). Each component implements a portion of the system’s functionalities.
Components integrated with each other into a component hierarchy for the autonomous vehicle systems (cf. Section 3.3). Hardware parts, e.g., ECUs, are assembled from multiple other hardware parts, e.g., CPU and memory (cf. Section 3.3.2). Software components are either atomic or composites of other interacting software components (cf. Section 3.3.1). Each component of autonomous vehicle systems can be assign to one of the three segments — input, process, and output.
The following section describes the simulation layer and its substitutions in the verification of autonomous vehicle systems.
5.2. Simulation Layer
The simulation layer addresses the integration of autonomous vehicle systems into simulations. Software-based system simulations are used in the automotive domain for the verification of autonomous vehicles systems. Simulation frameworks imitate the environments of these systems — the world — including their sceneries and dynamic objects (cf. Section 2.2).
The integration into the simulation segments the autonomous vehicle systems into three parts (cf. Fig. 5.3); the part of the original system whose functionality is verified in the simulation and two parts of the system that are simulated by the simulation framework. The segmentation depends on the system functionality which is verified. The verified system part is also denoted as system under test (SUT) (cf. Definition 2.16). All system parts which are simulated by the simulation framework, e.g., sensors and actuators, have to be verified and validated by other V&V methods.
The segmentation of the autonomous vehicle system in the simulation defines the input and output interfaces for the interaction between system and simulation framework. For tests, the SUT is stimulated by input data from the current state of the virtual world. The output of the SUT is manually and automatically evaluated for a verdict about the system’s correctness (cf Section 2.2). In closed-loop simulations, the state of the virtual world is also adapted to the output of the SUT. The interfaces between the system part and simulation framework define the maximal extension of the system segment that can be monitored by the runtime monitoring (cf. Fig. 5.3).
The integration of autonomous vehicle systems and the virtual world requires simulation frameworks to substitute and simulate systems components by models (cf. Fig. 5.3). The extent of simulated components depends on the segmentation of the autonomous vehicle system in the simulation. Simulations impose different requirements for the availability of real software and hardware components (cf. Section 2.2). Models substitute sensor and actuator components in software-based simulations because they rely on hardware
5.2. Simulation Layer Process Function Function Function Process Output System Boundary Actuator Actuator Function Postprocessor Postprocessor Postprocessor Function Function Preprocessor Preprocessor Preprocessor Sensor Sensor Input Simulation Framework (Virtual) World Sensor (Model) Sensor (Model) Preprocessor Actuator (Model) Actuator (Model) Postprocessor Output Input Postprocessor Postprocessor Preprocessor Preprocessor World
Figure 5.3.: Integration of autonomous vehicle system and simulation framework.
and real-world objects. Hardware components are commonly verified independently in HIL tests (cf. Section 2.2.2.4).
Sensors, e.g., RADAR and LIDAR sensors, require the reflection of real-world objects for their physical measurements. However, objects in software-based system simulations are mostly virtual (cf. Section 2.2). The real propagation of radio waves and light beams is impossible to be included in these simulations. These sensors have to be simulated by corresponding sensor models. Sensor models with different degrees of accuracy can be included in simulations (cf. Section 2.2.1.2). Perfect sensor models use the parameters of objects in the virtual world as outputs of the sensor components and direct input for the SUT. Intermediate sensor models introduce measurement noise for the parameters of the virtual objects in the input for the SUT. Complex models may use techniques, e.g., ray-tracing (cf. [GGD12]), to model the propagation of radio waves and light beams in the virtual world.
The integration of actuators, e.g., brakes and engines, into software-based simulations, require extensive hardware setups. In closed-loop simulations, the physical behavior of actuators has to be interwoven with the virtual behavior of the vehicles in these simulations. Additional hardware has to physically represent the virtual world of the simulation frameworks to these actuators. For example, virtual road parameters, e.g., slope, would have to be applied as resistance on the crankshaft of the real engine for a realistic engine behavior. Virtual parameters of objects in the simulations have to be incorporated into the physical behavior of real hardware components. The costs for such complex hardware setups would neglect the benefit of software-based system simulations in comparison to real-world tests. Therefore, models substitute the real sensors, actuators, vehicle dynamics, and road characteristics in software-based simulations.
Inaccurate simulations restrict the validity of their results for the operation of autonomous vehicle systems in the real world. The system behavior in the real world has to match
the behavior of autonomous vehicle systems in simulations. The overall accuracies of simulations predominantly depend on the accuracy of the simulation models to represent the physical properties of real-world objects and real-world processes in the simulations. The output of simulation models must not significantly diverge from the output of the components during operation in the real world (cf. [HK16; Ste+15]). The accuracy of simulation models has to be verified by comparing the outputs of the models with their corresponding real components for the same real-world inputs.
The impact of insufficient simulation accuracy for the engineering approach is limited. The approach expects engineers to evaluate the results of the simulations before the knowledge from these simulations is used in the runtime monitoring. In case, simulations have not been accurate for the real world, or the behavior of autonomous vehicle systems has not been correct and safe, data from these simulations must not be used as knowledge for the runtime monitoring during operation. The qualitative runtime monitoring can be used as test oracle in simulations without any concurrent evaluation engineers as long as the monitors have been successfully verified.
The following sections describe the layers of the runtime monitoring framework in detail. We commence with the abstraction layer as the interface of the runtime monitoring to the autonomous vehicle systems.
5.3. Abstraction Layer
The qualitative and quantitative monitoring of autonomous vehicle systems require the consistent access of system data throughout system development and system operation. The abstraction layer (cf. Fig. 5.1) depicts the interface between the autonomous vehicle systems and the runtime monitoring framework. The layer consists of two components — the input abstraction and output abstraction. These components access the runtime data of the autonomous vehicle systems via defined interfaces and transform the runtime data into the abstract representations of the runtime monitoring. Types and values of the abstract representation are defined from the specification, requirements, and safety criteria of the autonomous vehicle systems (cf. Chapter 6). The correctness and safety of autonomous vehicle systems is evaluated by the qualitative and quantitative monitoring layers based on this abstract representation (cf. Fig. 5.1).
The part of the autonomous vehicle systems which is supervised by the runtime monitoring framework is denoted as function (cf. Fig. 5.1) or as system (part) under monitoring. Engineers define the function during the system development in relation to a specific system functionality. The function need not correlate with the process of autonomous vehicle systems (cf. Section 5.1) and, therefore, may vary between different autonomous vehicle systems. For one system, the function may incorporate components of the system’s
input and output parts while the runtime monitoring of another system solely focuses on
a subset of components within this process par as the function.
The runtime monitoring framework depicts the function and its components as a black-box. The function is solely monitored based on its external observable behavior without any consideration of its internal structure. The hierarchical architectures of autonomous
5.3. Abstraction Layer
vehicle systems have no impact on the runtime monitoring as long as the required input and output interfaces of the respective function are provided.
The processing of autonomous vehicle systems before the function is depicted as prepro-
cessing and mostly consists of the sensor perception and the processing of the physical
sensor data. The system components in the system’s processing chain following the
function is encompassed by the postprocessing. The postprocessing addresses the execution
of the function’s output by actuators in the real world. The correctness and safety of