Campus Monterrey
School of Engineering and Sciences
Software architecture proposal for navigation and decisions of autonomous ground vehicles in controlled environments with
unavoidable collision scenarios
A thesis presented by
Mario Alberto Valdes Saucedo
Submitted to the
School of Engineering and Sciences
in partial fulfillment of the requirements for the degree of Master of Science
in Engineering
Monterrey, Nuevo Le´on, June 15, 2020
To my mother.
iii
I would like to express my deepest gratitude to the Tecnol´ogico de Monterrey for the support on tuition and CONACyT for the economic support for living. Finally, I want to thank Dr. Oscar Eleno Carbajal Espinosa and Dr. Carlos Renato V´azquez Topete for being my advisors.
iv
of autonomous ground vehicles in controlled environments with unavoidable collision scenarios
by
Mario Alberto Valdes Saucedo Abstract
In recent years the ethic of autonomous vehicles has become one of the main topics of discus- sion in the field of autonomous driving. On this thesis the problem of the ethics of autonomous vehicles will not be resolved, at least not in its totality, but an approach to the problem and to a possible solution will be presented. In order to address this problem a software architecture is proposed, such architecture will be in charge of the navigation of the vehicle and obstacle avoidance, furthermore, it will provide the necessary information to take decisions in case of unavoidable collisions. Using a fuzzy logic implementation, the objective is to generate a set of outputs that, in theory, should allow a secondary system to take a decision in a case of unavoidable collision. Nevertheless, such secondary system is not present in this paper.
The results obtained were validated using statistical data. As a whole, the system shows a competent ability to determine the situation and generate the outputs.
v
2.1 Visual representation of the RGB and HSL color space [50, 17] . . . 10
2.2 Color mask [19] . . . 11
2.3 2D Convolution [2] . . . 12
2.4 Gaussian filter, 5x5 kernel example [25] . . . 13
2.5 Gaussian blur result [19] . . . 13
2.6 Sobel kernel 3x3 vertical and horizontal examples [26] . . . 14
2.7 Non-maximum Suppression [52] . . . 14
2.8 Hysteresis Thresholding [52] . . . 15
2.9 Canny edge result [19] . . . 15
2.10 Hough Transform sinusoid example [51] . . . 16
2.11 Hough Transform intersect example [51] . . . 17
2.12 YOLO [34] . . . 18
2.13 RRT, EXTEND [38] . . . 19
2.14 RRT Star, EXTEND [31] . . . 21
2.15 Potential field [21] . . . 22
2.16 Potential field gradient [15] . . . 23
2.17 Classical set [45] . . . 24
2.18 Fuzzy set [45] . . . 24
2.19 Classical set domain [45] . . . 24
2.20 Fuzzy set domain [45] . . . 25
2.21 Sharp-edged membership function [45] . . . 25
2.22 Continuous membership function [45] . . . 26
2.23 Truth tables [45] . . . 27
2.24 Truth tables graphs [45] . . . 27
2.25 Fuzzy Logic example [46] . . . 29
2.26 Trolley problem [1] . . . 30
3.1 Layers . . . 33
vi
3.4 Reference . . . 38
3.5 Unavoidable collision . . . 39
3.6 Steering angle . . . 40
3.7 Velocity . . . 40
3.8 Proximity . . . 40
3.9 Delta steering . . . 41
3.10 Delta velocity . . . 41
3.11 Severity . . . 42
4.1 Angle . . . 45
4.2 Distance . . . 45
4.3 Lane Detector steps . . . 46
4.4 Filtered angle . . . 47
4.5 Filtered distance . . . 47
4.6 YOLO misdetections . . . 48
4.7 Test environment . . . 49
4.8 Second test environment . . . 50
4.9 Steering angle . . . 51
4.10 Velocity . . . 52
4.11 Proximity . . . 52
4.12 Delta steering . . . 53
4.13 Delta velocity . . . 53
4.14 Severity . . . 54
4.15 Delta steering surface . . . 54
4.16 Delta velocity surface . . . 55
4.17 Severity surface with proximity = far . . . 55
4.18 Severity surface with proximity = near . . . 56
4.19 Severity surface with proximity = dangerously near . . . 56
4.20 Pedestrian fatality risk as a function of the impact speed of a car from the WHO [64] . . . 57
4.21 Pedestrian fatality risk as a function of the impact speed of a car from the PAHO [53] . . . 57
4.22 Pedestrian fatality risk as a function of the impact speed of a car from the European Commission [24] . . . 58
vii
45% . . . 59 4.24 Test 2. Person A: 40.3%, person B: 21.6%, from WHO person A: ≈ 35%,
from PAHO person A: ≈ 40%, from the European Commission person A: ≈ 15% . . . 59 4.25 Test 3. Person A: 13.8%, person B: 13.4%, from WHO person A: ≈ 10%,
from PAHO person A: ≈ 10%, from the European Commission person A: ≈ 5% . . . 60 A.1 RRT test 1. From point A (32, 13) to point B (525, 97). Path evaluation: Good 64 A.2 RRT test 2. From point A (32, 13) to point B (108, 695). Path evaluation: Poor 65 A.3 RRT test 3. From point A (32, 13) to point B (554, 531). Path evaluation: Good 66 A.4 RRT test 4. From point A (32, 13) to point B (295, 260). Path evaluation: Poor 67 A.5 RRT test 5. From point A (32, 13) to point B (864, 891). Path evaluation: Poor 68 A.6 RRT* test 1. From point A (32, 13) to point B (525, 97). Path evaluation: Good 69 A.7 RRT* test 2.From point A (32, 13) to point B (108, 695). Path evaluation: Good 70 A.8 RRT* test 3. From point A (32, 13) to point B (554, 531). Path evaluation:
Good . . . 71 A.9 RRT* test 4. From point A (32, 13) to point B (295, 260). Path evaluation:
Good . . . 72 A.10 RRT* test 5. From point A (32, 13) to point B (864, 891). Path evaluation:
Good . . . 73 A.11 Artificial Potential Field test 1. From point A (32, 13) to point B (525, 97).
Path evaluation: Average . . . 74 A.12 Artificial Potential Field test 2. From point A (32, 13) to point B (108, 695).
Path evaluation: Average . . . 75 A.13 RRT + Artificial Potential Field test 1. From point A (32, 13) to point B (525,
97). Path evaluation: Good . . . 76 A.14 RRT + Artificial Potential Field test 2. From point A (32, 13) to point B (108,
695). Path evaluation: Good . . . 77 A.15 RRT + Artificial Potential Field test 3. From point A (32, 13) to point B (554,
531). Path evaluation: Good . . . 78 A.16 RRT + Artificial Potential Field test 4. From point A (32, 13) to point B (295,
260). Path evaluation: Poor . . . 79
viii
A.18 RRT* + Artificial Potential Field test 1. From point A (32, 13) to point B (525, 97). Path evaluation: Good . . . 81 A.19 RRT* + Artificial Potential Field test 2. From point A (32, 13) to point B
(108, 695). Path evaluation: Good . . . 82 A.20 RRT* + Artificial Potential Field test 3. From point A (32, 13) to point B
(554, 531). Path evaluation: Good . . . 83 A.21 RRT* + Artificial Potential Field test 4. From point A (32, 13) to point B
(295, 260). Path evaluation: Good . . . 84 A.22 RRT* + Artificial Potential Field test 5. From point A (32, 13) to point B
(864, 891). Path evaluation: Good . . . 85
ix
3.1 Delta Steering Rules . . . 42
3.2 Delta Velocity Rules . . . 42
3.3 Severity Rules with Proximity: Far . . . 43
3.4 Severity Rules with Proximity: Near . . . 43
3.5 Severity Rules with Proximity: Dangerously Near . . . 43
4.1 Single-algorithms . . . 49
4.2 Multi-algorithms . . . 49
4.3 Single-algorithms per test results . . . 50
4.4 Multi-algorithms per test results . . . 50
4.5 Comparation between the decision node results and the statistical data . . . . 58
x
Abstract v
List of Figures ix
List of Tables x
1 Introduction 1
1.1 Problem Statement . . . 3
1.2 State Of The Art . . . 4
1.3 Proposed Solution . . . 7
1.4 Objectives . . . 7
1.5 Motivation . . . 7
2 Theoretical Framework 9 2.1 Computer vision . . . 9
2.1.1 Color space . . . 10
2.1.2 Masks . . . 11
2.1.3 Filters . . . 12
2.1.4 Canny Edge Detection . . . 12
2.1.5 Hough Transform . . . 16
2.1.6 You Only Look Once (YOLO) Algorithm . . . 17
2.2 Navigation . . . 18
2.2.1 Rapidly-exploring Random Tree (RRT) . . . 18
2.2.2 Rapidly-exploring Random Tree Star (RRT*) . . . 20
2.2.3 Artificial Potential Field . . . 22
2.3 Data Fusion . . . 23
2.3.1 Fuzzy Logic . . . 23
2.4 Ethical Framework . . . 29
2.5 Software . . . 32 xi
3.1.1 Mean Filter . . . 36
3.2 Object Detection . . . 36
3.3 Path Planning . . . 37
3.3.1 Trajectory Estimator . . . 38
3.4 Decision node . . . 38
3.4.1 Inputs . . . 39
3.4.2 Outputs . . . 41
3.4.3 Rules . . . 42
4 Results 44 4.1 Lane Detector . . . 44
4.1.1 Mean Filter . . . 46
4.2 Object Detection . . . 47
4.3 Path Planning . . . 48
4.3.1 Trajectory Estimator . . . 51
4.4 Decision node . . . 51
4.4.1 Inputs . . . 51
4.4.2 Outputs . . . 52
4.4.3 Rules . . . 54
4.4.4 Tests . . . 56
5 Conclusions 61 5.1 Future Work . . . 62
A Path Planing Test Results 63 A.1 RRT . . . 63
A.2 RRT* . . . 68
A.3 Artificial Potential Field . . . 73
A.4 RRT + Artificial Potential Field . . . 75
A.5 RRT* + Artificial Potential Field . . . 80
Bibliography 91
xii
Introduction
In 1769, Nicolas-Joseph Cugnot, developed the first automobile, a huge, heavy, steam-powered machine able to carry up to four people for twenty minutes at 3.6 Km per hour. Nowadays it could seem nothing impressive, but it was the dawn of one of the biggest technological race of history [18]. Since Cugnot’s invention, the automobile has suffered huge changes, such as the development of the internal-combustion engine, the introduction of the first electric cars and, in recent years, the pursuit for the autonomous vehicle.
One of the first persons to postulate the idea of an autonomous vehicle was John Mc- Carthy, in 1969, in his essay called “Computer Controlled Cars”, he developed the idea of an
“automatic chauffeur”, which could be able of navigating in public roads using a television camera input [22].
Nowadays the race for the autonomous vehicle is present in all scales, not only big com- panies, like Google, Tesla, Mercedes, etc. Also, universities and research centers are in the pursuit for it. However, with such a big number of companies, universities and individuals developing the same idea is usually to find the fact that not all of them are doing in the same way nor at the same level.
Thus, when we talk about autonomous vehicles, we usually use the Society of Au- tomotive Engineers (SAE) autonomy scale to determine the different levels of autonomous capability, which are,[60]:
• Level 0: No automation. Warning or intervention systems.
• Level 1: Driver assistance. Driver assistance system of either steering or accelera- tion/deceleration.
1
• Level 2: Partial Automation. One or more driver assistance systems of both steering and acceleration/deceleration.
• Level 3: Conditional Automation. Automated driving system of all aspects of the dy- namic driving task with the expectation that the human driver will respond appropriately to a request to intervene.
• Level 4: High Automation. Automated driving system of all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request to intervene.
• Level 5: Full Automation. Automated driving system of all aspects of the dynamic driving task under all roadway and environmental conditions that can be managed by a human driver.
Currently there are some companies that offers their autonomous vehicles for sale to the general public, as Tesla Autopilot (level 2), Volvo Pilot Assist (level 2), Mercedes-Benz Drive Pilot (level 2), Cadillac Super Cruise (level 2), Audi A8 (level 3), among others [30].
However, one of the vehicles who has reached one of the higher SAE levels is the Waymo for Google, currently autonomy level 4, but, in Google’s words, is pointing to be fully au- tonomous, level 5 [22].
Nevertheless, the pursuit for the autonomous car has found some impediments in recent years, with the development of a new technology always comes the development of a new legal framework, this with the objective of ensuring the security of the users and the other persons involve in the use of this technology, this has become the security a “must” in the development of autonomous vehicles.
In addition, the need for the development of and ethical framework has also arisen, and with it a serial of ethical and technological issues. The matter of ethics of autonomous ve- hicles as become an important concern in the implementation of this technologic. Even so multiple governmental institution, research centers and individuals are working in the devel- opment of such ethical framework, there doesn’t seem to be a definitive answer [43, 23, 6].
1.1 Problem Statement
Even though commercial systems are already operating, they still present accidents that could have been prevented with the improvement of the technology. An example is what happened in Florida, where a Tesla Model S in self-driving Autopilot mode hits an 18-wheel tractor- trailer, killing the occupant inside the vehicle [22].
This and many other cases have attracted attention to the debate about autonomous ve- hicles and some of the technical and ethical issues surrounding them. At the same time, have generated awareness of the scope of this technology and its presence in the market.
The development of new and more sophisticated controls systems has become a ne- cessity in order to avoid future incidents, however, some of these systems are developed by companies, which in order to protect their interests do not publish their findings. In such a competitive market, it is to be expected that most of this information will be kept secret in order to achieve the maximum income from it or to generate a strategic advantage over other competitors.
Even some universities, which are developing similar technologies, have chosen to sup- port this model, either by receiving financing from the leading companies in the market or by developing their own companies in order to market their research. One of the clearest exam- ples of this would be the company Oxbotica, launched by Oxford University in 2014, to drive the commercialization of their driverless car technology [57].
The lack of information from the manufacturers, about what may be one of the most sensitives topics in this matter, the ethic of their vehicles, is by far one of the most important problems.
It is not quite a surprise the why this topic is avoided by most of the manufacturers and even some of the others involve with this technology, in general the topic can be seen has a “minefield”, in current years there are few topic as complex to talk about as the ethics in autonomous vehicles.
Certainly on this paper the problem of the ethics of autonomous vehicles will not be re- solved, at least not in its totality, but an approach to the problem of integrating an architecture that considers the navigation, external factors and a decision process will be presented.
1.2 State Of The Art
In the design and development of navigation systems for autonomous vehicles, a great variety of algorithms and techniques are combined in order to develop better systems, most of them are divided into 3 categories, visual odometry, object detection and path planning.
More in detail, one of the bases for the autonomous navigation is the ability of the vehi- cle to locate itself in the space, one of the most used techniques for this is the visual odometry.
This technique has been used in multiple research with positive results [48].
In the paper of M.A. Brubaker, A. Geiger and R. Urtasun, in [10], (2016), a probabilis- tic model was proposed in which an efficient approximate inference algorithm is derived, it uses odometry computed from two video cameras and road maps as the inputs. In a similar way in [49], T. Mouats, N. Aouf, A.D. Sappa, C. Aguilera and R. Toledo, (2015), proposed a multispectral stereo odometry system for ground vehicles, in the implementation the cameras are localized as an stereo setup rather than two monocular cameras of different spectrums, techniques as cosine similarity function, pyramidal Lucas-Kanade tracker, Gauss-Newton op- timization and others, are also used.
Other authors as D. Scaramuzza and R. Siegwart in [58], (2008), proposed an appearance- guided monocular omnidirectional visual odometry for outdoor ground vehicles, using a real- time algorithm for computing the ego-motion of a vehicle relative to the road, which uses as input the images provided by a single omnidirectional camera. On the other hand, in [62], P. Sun, X. Zhao, Z. Xu, R. Wang and H. Min, (2019), proposed a 3D LiDAR data-based dedicated road boundary detection algorithm for autonomous vehicles, in the implementation the ground points are separated from the pre-processed point cloud data, then, the candidate points are searched along the predicted trajectory, finally, a spline fit model is applied.
Furthermore, authors as Y. Zhang, H. Zhang, G. Wang, J. Yang and J. Hwang in [69], (2020), proposed a bundle adjustment for monocular visual odometry based on detections of traffic signs, in the paper, a traffic sign feature-based joint bundle adjustment module is intro- duced to eliminate and relieve the incrementally accumulated pose errors. The experimental results show an improve in vehicular localization accuracy compared with the state-of-the-art baseline visual odometry method.
However, the visual odometry is just a part of this systems, in order to ensure the secu- rity of the user, the system must be able to detect persons, signals and other objects, some of the research that addresses this topic are presented below [4].
In the paper of X. Wang, L. Xu, H. Sun, J. Xin and N. Zheng, in [63], (2016), is pro- posed an on-road vehicle detection and tracking using MMW radar and monovision fusion.
The experimental results presented in the paper show that the proposed system is able to de- tect on-road vehicles with 92.36% detection rate and with 0% false alarm rate, and within ten frames (0.16 s) for the detection and tracking of each vehicle. In a similar way in [3], A. Al- magambetov, S. Velipasalar and M. Casares, (2015), proposed a robust and computationally lightweight autonomous tracking of vehicle taillights and signal detection by embedded smart cameras, the proposed system is able to detect and track vehicle taillights, recognize common alert signals, and count the cars passing on both sides of the vehicle.
Other authors as T. Chen, R. Wang, B. Dai, D. Liu and J. Song in [13], (2016), proposed a likelihood-field-model-based dynamic vehicle detection and tracking for self-driving, using a Velodyne LIDAR and a GPS-aid inertial navigation system as the inputs. In addition in [12], L. Chen, X. Hu, T. Xu, H. Kuang and Q. Li, (2017), proposed a turn signal detection during nighttime by CNN detector and perceptual hashing tracking, using technics as the Nakagami- image-based method, convolutional neural networks, softmax classifiers, among others. On the other hand, in [68], Y. Zeng, Y. Hu, S. Liu, J. Ye, Y. Han, X. Li and N. Sun, (2018), proposed a real-time 3-D vehicle detection in LiDAR point cloud for autonomous driving, in the implementation a pre-RoIpooling convolution technique and a pose-sensitive feature map design are used. The results of the experiments on the KITTI benchmark dataset show com- petitive detection accuracy with the capability of completes detection within 0.09 s.
Furthermore, authors as B. Leibe, K. Schindler, N. Cornelis and L. Van Gool in [39], (2008), proposed a coupled object detection and tracking from static cameras and moving vehicles, the approach is formulated in a minimum description length hypothesis selection framework. The experimental results demonstrate its performance on surveillance-type sce- narios and scenarios from inside a moving vehicle passing through crowded city areas.
Finally, in most cases, both, the visual odometry and the object detection, are used in order to allow the vehicle to follow a previously planned trajectory, commonly now as path planning this technique can be found in multiple research.
In the paper of D. Devaurs, T. Sim´eon and J. Cort´es,in [20], (2016), is proposed an optimal path planning in complex cost spaces with sampling-based algorithms, in the imple- mentation two algorithm, T-RRT* and AT-RRT, are used. The results show that the presented algorithms converge faster than RRT* toward the optimal path. In a similar way in [41], H.
Liu, C. Jin, B. Yang and A. Zhou, (2017), propose a formalization of the K Shortest Paths with Diversity (KSPD) problem, the framework includes the use of two judiciously designed lower bounds, one is dependent on and the other one is independent on the chosen path similarity metric.
Other authors as X. Liu, D. Zhang, H. Yan, Y. Cui and L. Chen in [42], (2019), propose a new algorithm of the best path selection based on machine learning, the implementation use the prior knowledge applied reinforcement learning strategy, and an A* shortest path algo- rithm. On the other hand, in [7], F. Belkhouche, (2009), propose a reactive path planning in a dynamic environment, in the implementation the concept of the virtual plane is introduced and used to create reactive kinematic-based navigation laws. The results present simulations to illustrate collision detection and path planning.
Finally, with regard to the ethics of autonomous vehicles, some of the works that have sought a solution to the matter are:
The German Ethics Code for Automated and Connected Driving in [43, 23], from Christoph Luetge. In July 2016, the German Federal Minister of Transport and Digital In- frastructure appointed a national ethics committee for automated and connected driving. In June 2017, this committee presented a code of ethics. In [43, 23], the 20 ethical guidelines proposed by this committee, alongside the main ethical topics of these guidelines and the discussions that lay behind them are presented. In addition, some authors like Y. Bigman and K. Gray in [8], “Life and death decisions of autonomous vehicles”, and H. Lim and A.
Taeihagh in [40], “Algorithmic decision-making in AVs: Understanding ethical and technical concerns for smart cities”, had make reference to this work. In a similar way it has been done by [27, 5, 44].
Furthermore, in the work from E. Awad, S. Dsouza, R. Kim, J. Schulz, J. Henrich, A.
Shariff, J. Bonnefon and I. Rahwan, The Moral Machine experiment, in [6], a multilingual on- line ‘serious game’ is used to collect large-scale data on how citizens would want autonomous vehicles to solve moral dilemmas in the context of unavoidable accidents.
1.3 Proposed Solution
In order to address this problem a software architecture is proposed, such architecture will be in charge of the navigation of the vehicle and obstacle avoidance, furthermore, it will provide the information necessary, based on The German Ethics Code for Automated and Connected Driving, to take decisions in case of unavoidable collisions.
1.4 Objectives
Design a software architecture proposal for navigation and decisions of autonomous ground vehicles in controlled environments with unavoidable collision scenarios. More specifically:
• Design and develop a navigations system for autonomous ground vehicles based on free code libraries, easy to export to different development platforms and modular or blockwise structure.
• Design and develop a basis for the implementation of a decision system in environments of unavoidable collisions, determine the input and output parameters and generate the set of rules to link such parameters, based on The German Ethics Code for Automated and Connected Driving.
1.5 Motivation
The ethic of autonomous vehicles has always been hard topic to discuss, the broad range of opinion and ideologies in the matter make difficult to reach a conclusive answer. Nevertheless, the progress of the technology has arisen the need to determine it, and with it has emerged the necessity for system able to follow it.
The proposed software architecture is not intended to be the final answer to this problem, but merely an approach to what may be a solution to some of the issues in the matter. The development of this project could represent a contribution to the state of the art. Furthermore, it stablishes a basis for future works in the field and for the ones who decide to continue with
the project. In addition, it could throw new light over current problems, helping to generate new solutions and allowing others to develop better systems or improve the existing ones.
Theoretical Framework
In this chapter we can find an overview of the algorithms and techniques used in the devel- opment of this work, alongside a brief explanation of their functioning. All this with the objective of allow a better understanding of the work presented in this paper.
The algorithms and techniques will be divided into 3 categories, computer vision, in this category are shown the algorithms or techniques used to enhance or extract information from an image; in the category navigation, are presented the algorithms used in order to generate paths from a map; finally in data fusion, the commonly used to merch data.
In addition, in the following section, Ethical Framework, an overview of the ethics of autonomous vehicles can be found. Furthermore, the software and libraries used in order to implement such algorithms and techniques will also be presented in the last section, Software.
2.1 Computer vision
The algorithms on this section have two main purposes, one is to enhance specific information from an image (colors, shapes, objects, etc.), like is the case with color space, masks and fil- ters, ant the other is to extract such information, in which case we use Canny Edge Detection, Hough Transform or YOLO.
9
2.1.1 Color space
In vision most of the images come in RGB (Red, Green & Blue), where red, green and blue ranges from 0 to 255. Even so is possible to work with these images, most of the time chang- ing them to another color space facilitates the task of feature extraction, for example the model HSL (Hue, Saturation & Lightness) [17].
(a) RGB
(b) HSL
Figure 2.1: Visual representation of the RGB and HSL color space [50, 17]
In the HSL color space, hue is the color type and ranges from 0 to 360◦ , saturation is the variation of the color depending on the lightness and ranges from 0 to 100%, and lightness ranges from 0 to 100% (from black to white) [17]. The conversion of RGB to HSL can be done as follows, [50, 54] :
1. Convert the RGB values to the range 0-1.
R0 = R/255 (2.1)
G0 = G/255 (2.2)
B0 = B/255 (2.3)
2. Find the minimum and maximum values of R’, G’ and B’.
Cmax = max(R0, G0, B0) (2.4)
Cmin = min(R0, G0, B0) (2.5)
3. Calculate the Lightness value.
L = (Cmax+ Cmin)/2 (2.6)
4. Find the Saturation.
∆ = Cmax− Cmin (2.7)
S =
0 , ∆ = 0
∆
1−|2×L–1| , ∆ 6= 0
(2.8)
5. Calculate the Hue.
H =
0◦ , ∆ = 0
60◦× (G0−B∆ 0 × mod6) , Cmax = R0 60◦× (B0∆−R0 + 2) , Cmax = G0 60◦× (R0−G∆ 0 + 4) , Cmax = B0
(2.9)
2.1.2 Masks
Once the image is the desired color space one of the technique most commonly used in order to extract features are masks, this are binary images, which can be merged using bitwise op- erations (and, or, & not).
Two of the most used masks are the color masks, used in order to extract regions of an specific color or range of colors, and the region of interest (ROI) masks, used to extract one or more specific sections of the images.
Figure 2.2: Color mask [19]
2.1.3 Filters
After applying one or more masks to an image, is common to use image filtering in order to enhance the extracted features as well as to clean the noise present in the image. Image filtering changes the range (i.e. the pixel values) of an image, so the colors of the image are altered without changing the pixel positions [32].
In vision, most filters consist of a matrix, commonly know as kernel, which is applied to the image trough 2D convolution. The output image is produced by sliding the kernel over the input image [32].
g [x, y] = w [x, y] ∗ f [x, y] =
a
X
i=−a b
X
j=−b
w [i, j] · f [x + i, y + j] (2.10)
Figure 2.3: 2D Convolution [2]
2.1.4 Canny Edge Detection
Some algorithms as the Canny Edge Detection, developed by John F. Canny in 1986, make use of filters and other methods in order to detect edges in an image. The steps in order to implemented it are presented below [52].
Noise Reduction
Since it is susceptible to noise, the first step is to remove the noise in the image with a 5x5 Gaussian filter, also known as Gaussian blur [52].
Figure 2.4: Gaussian filter, 5x5 kernel example [25]
Figure 2.5: Gaussian blur result [19]
Intensity Gradient of the Image
The image is filtered with a Sobel kernel, in both horizontal and vertical direction, to get the first derivative in the horizontal direction (Gx) and the vertical direction (Gy), then the edge gradient and direction for each pixel is calculates as follows, [52] :
Edge.gradient(G) =q
G2x+ G2y (2.11)
Angle(θ) = tan−1(Gx
Gy) (2.12)
Figure 2.6: Sobel kernel 3x3 vertical and horizontal examples [26]
Non-maximum Suppression
The image is scanned to remove any pixels which may not constitute the edge, for it, every pixel is checked if it is a local maximum in its neighborhood in the direction of the gradient.
The result is a binary image with “thin edges” [52].
Figure 2.7: Non-maximum Suppression [52]
Point A is on the edge ( in vertical direction). Gradient direction is normal to the edge.
Point B and C are in gradient directions. So point A is checked with point B and C to see if it forms a local maximum. If so, it is considered for next stage, otherwise, it is suppressed ( put to zero) [52].
Hysteresis Thresholding
This stage decides which are really edges and which are not. In order to do so, two threshold values are used, minVal and maxVal. Any edges with intensity gradient greater than maxVal
are sure to be edges and those below minVal are sure to be non-edges, so discarded. Those who lie in between are classified edges or non-edges based on their connectivity. If they are connected to “sure-edge” pixels, they are part of edges, otherwise, they are also discarded.
This step also removes small pixels noises on the assumption that edges are long lines [52].
Figure 2.8: Hysteresis Thresholding [52]
The edge A is above maxVal, so it is considered as “sure-edge”. Although edge C is below maxVal, it is connected to edge A, so it is also considered as edges and we get that full curve. But edge B, even though it is above minVal and is in the same region as edge C, it is not connected to any “sure-edge”, so it is discarded. The result are the strong edges in the image [52].
Figure 2.9: Canny edge result [19]
2.1.5 Hough Transform
Once we have an image with defined edges, in most cases we want to extract the lines which represent such edges, in order to do so, we use the Hough Transform, which follows the idea that if, [51] :
1. For each point (x0, y0), we can define the family of lines that goes through that point as:
rθ = x0· cos θ + y0· sin θ (2.13)
2. If for a given (x0, y0) we plot the family of lines that goes through it in a plane θ − rθ we get a sinusoid.
Figure 2.10: Hough Transform sinusoid example [51]
3. If the curves of two different points intersect in the plane θ − rθ, that means that both points belong to a same line.
Figure 2.11: Hough Transform intersect example [51]
4. A line can be detected by finding the number of intersections between curves. The more curves intersecting means that the line represented by that intersection have more points.
In other words, the Hough Transform keeps track of the intersection between curves of every point in the image. If the number of intersections is above some threshold, then it de- clares it as a line with the parameters (θ, rθ) of the intersection point [51].
2.1.6 You Only Look Once (YOLO) Algorithm
There are techniques which allow to detect not only edges but objects as a whole, such is the case of YOLO (You Only Look Once), it consist of a single CNN (Convolutional Neural Network) which split the image in a grid while simultaneously generates bounding boxes and class probabilities for those bounding boxes [11, 34].
Figure 2.12: YOLO [34]
However, the algorithm is susceptible to perturbation (environmental light, partially ob- structed or overlapped objects, etc.), these perturbations may not avoid the object detection or the correct delimitation of the bounding boxes, but in most cases, they will decrease the class probabilities [11, 34].
2.2 Navigation
In this section are presented the algorithms that have as main purpose to generate a set of paths or routes which must allow a determined robot to navigate from a given point to another in a known map.
2.2.1 Rapidly-exploring Random Tree (RRT)
In navigation one of the algorithms most commonly uses for path planning is the Rapidly- exploring Random Tree (RRT), this algorithm is designed for efficiently searching nonconvex high-dimensional spaces, and ca be implemented as follows: [38, 37]
RRT starts from an initial state xinit. During each iteration, a random state x is selected from the configuration space. If the random sample x lies in an obstacle free region, then the nearest node to x is searched in the tree and denoted as xnear. If x is accessible to xnear according to a predefined step size, ε, then the tree is expanded by connecting x and xnear. Otherwise, it returns a new node xnew, which is at a distance ε from xnear in the direction of x, then expands the tree by connecting xnewwith xnear [31, 38, 37].
Algorithm 1 Rapidly-exploring Random Tree (RRT) BUILD RRT(xinit)
1 T .init(xinit) 2 for k = 1 to K do
3 xrand ← RAND CONF() 4 EXTEND(T, xrand) 5 Return T
Algorithm 2 RRT EXTEND EXTEND(T, x)
1 xnear ← NEAREST NEIGHBOR(x, T ) 2 if NEW STATE(x, xnear, xnew, unew) then 3 T .add vertex(xnew)
4 T .add edge(xnear, xnew, unew)) 5 if xnew= x then
6 Return Reached
7 else
8 Return Advanced
9 Return T rapped
Figure 2.13: RRT, EXTEND [38]
Nevertheless, in some cases an RRT is insufficient to solve a planning problem, mainly because it does not ensure finding the optimal path, in such cases is preferable to use some of
its variants as the Rapidly-exploring Random Tree Star (RRT*).
2.2.2 Rapidly-exploring Random Tree Star (RRT*)
Similar to the RRT in its implementation, but the EXTEND function now involves a “best parent” and a “rewiring” step, in the first step the algorithm search for the vertex with the lowest “cost” to which the new vertex can be connected, in most cases the cost function is the accumulated distance of each vertex relative to its parent vertex [14].
In the second step the vertices previously connected are check to see if connecting them to the new vertex will decrease its “cost”, if so this are connected to the new vertex, such allowing a progressive optimization of the path [14].
Algorithm 3 RRT* EXTEND BEST PARENT(xnew, T )
1 xnear ← NEAREST NEIGHBOR(x, T ) 2 if NEW STATE(x, xnear, xnew, unew) then 3 T .add vertex(xnew)
4 xbest ← BEST PARENT(xnew, T ) 5 T .add edge(xbest, xnew, unew)) 6 T .add cost(xnew, unew)) 7 REWIRING(xnew, T ) 8 if xnew= x then
9 Return Reached
10 else
11 Return Advanced
12 Return T rapped
Algorithm 4 RRT* BEST PARENT BEST PARENT(xnew, T )
1 for k = 1 to K do
2 if INSIDE RADIUS(x, xk) then
3 if COST(x, xk) <COST(x, xlowest) then
4 xlower ← xk
5 Return xlowest
Algorithm 5 RRT* REWIRING REWIRING(x, T )
1 for k = 1 to K do
2 if INSIDE RADIUS(x, xk) then 3 if LOWER COST(x, xk) then 4 T .add edge(x, xk, u)) 5 T .add cost(xk, uk)) 6 Return T
(a) Find near neighbor (b) Select best parent
(c) Check cost for rewiring (d) Rewired tree Figure 2.14: RRT Star, EXTEND [31]
However, the “rewiring” step can significantly increase the computational cost of the algorithm, other algorithms as Artificial Potential Field require less computational time and can find optimal-like paths.
2.2.3 Artificial Potential Field
The philosophy behind the algorithm of Artificial Potential Field is simple, the robot moves in a field of forces, where the position to be reached is an attractive pole and the obstacles are repulsive surfaces [35, 15].
Under that philosophy the general formula goes as follows, [35, 15] :
U (q) = Ugoal(q) + Uobstacle(q) (2.14) where the total potential U is the sum of an attractive potential Ugoaland a repulsive potential Uobstacle.
Figure 2.15: Potential field [21]
Energy is minimized by following the negative gradient of the potential energy function, [35, 15] :
F (q) = −∇U (q) = −DU (q)T = − ∂U
∂q1
(q), ..., ∂U
∂qm
(q)
T
(2.15) at every point in time, the robot computes the vector F at the current point and goes in that direction.
Figure 2.16: Potential field gradient [15]
2.3 Data Fusion
This section introduces the data fusion algorithms, these algorithms are used to merch big amounts of data into more manageable variables. This results extremely useful when using a big amount of sensor or redundant systems.
2.3.1 Fuzzy Logic
In general, data fusion is done in function of the sensor or data that the system have, neverthe- less, some techniques allow for a higher level of abstraction. Such is the case of Fuzzy Logic, this method intends to model logical reasoning, based on vague or imprecise statements [16].
Fuzzy Sets
Fuzzy logic has as its base the theory of fuzzy sets. A fuzzy set is a set without a crisp, clearly defined boundary. It can contain elements with only a partial degree of membership [65, 45].
Figure 2.17: Classical set [45]
Figure 2.18: Fuzzy set [45]
Any statement can be fuzzy. The biggest advantage that fuzzy reasoning offers is the ability to reply to a true-false question with a not-quite-true-or-false answer. In fuzzy logic, the truth of any statement becomes a matter of degree [45].
Figure 2.19: Classical set domain [45]
Figure 2.20: Fuzzy set domain [45]
Fuzzy logic then can assign degrees of truth to propositions, where 0 represents “totally false” and 1 represents “totally true”. The functions to do so are called membership functions [16, 45].
Membership Functions
A membership function is a curve that defines how each point in the input space is mapped to a membership value (or degree of membership) between 0 and 1 [67, 45].
Figure 2.21: Sharp-edged membership function [45]
Figure 2.22: Continuous membership function [45]
The output axis is the membership value, a number between 0 and 1. The curve is known as a membership function, often designated as µ. This curve defines the transition from not tall to tall. Both people are tall to some degree, but one is taller than the other [67, 45].
The only condition a membership function must satisfy is that it must vary between 0 and 1. The function itself can be any arbitrary curve whose shape can be defined as a function.
It must seek to ensure simplicity, convenience, speed, and efficiency [45].
Logical Operations
An important thing to notice about fuzzy logical reasoning is the fact that it is a superset of standard Boolean logic. In other words, if you keep the fuzzy values at their extremes, 1 and 0, standard logical operations will remain [45].
Now, because in fuzzy logic the truth of any statement is a matter of degree, is necessary to seek a function that preserves the results of the AND truth table, for example, and also extend to all real numbers between 0 and 1 [45].
One way to do so is as follows, we can resolve the statement A AND B by using the function min(A, B). In addition, the OR operation can be replaced with the max function, max(A, B). Finally, the operation NOT A becomes equivalent to the operation 1 − A [45].
Figure 2.23: Truth tables [45]
Another way to see it, is converting the truth table to a plot of two fuzzy sets applied together in order to create one fuzzy set [45].
Figure 2.24: Truth tables graphs [45]
The upper part presents the plots corresponding to the two-valued truth tables, while the lower part shows how the operations work over a continuously varying range of truth values A and B according to the fuzzy operations previously defined [45].
If-Then Rules
Fuzzy sets and fuzzy operators are the subjects and verbs of fuzzy logic. These if-then rule statements are used to formulate the conditional statements that comprise fuzzy logic. A fuzzy
if-then rule assumes the form, [66, 45] :
If x is A, then y is B
Where A and B are linguistic values defined by fuzzy sets on the ranges X and Y, respec- tively. The if-part of the rule ”xisA” is called the antecedent or premise, while the then-part of the rule ”yisB” is called the consequent or conclusion [66, 45].
Usually, the input to an if-then rule is the current value for the input variable and the output is an entire fuzzy set. This set will be defuzzified, in order to assign one value to the output. This step is also known as implication [45].
For an if-then rule, the antecedent, p, implies the consequent, q. In binary logic, if p is true, then q is also true (p → q). In fuzzy logic, if p is true to some degree of membership, then q is also true to the same degree (0.5p → 0.5q). In both cases, if p is false, then the value of q is undetermined [66, 45].
The consequent determines the fuzzy set that will be assigned to the output. The im- plication function then modifies that fuzzy set to the degree specified by the antecedent. The most common ways to modify the output fuzzy set are truncation using the min function or scaling using the prod function [66, 45].
Figure 2.25: Fuzzy Logic example [46]
2.4 Ethical Framework
Multiples techniques has been presented till this point, together, this algorithm and methods should allow a vehicle to “see”, “plan”, and “think”. Nevertheless, the question that remains is what such machine thinks, in most cases such question is irrelevant, since most decision surrounding the autonomous driving task are trivial. But in real world scenarios, this question takes a different meaning when the situation involves the live of human beings, when it be- comes an ethical decision. This by itself has become one of the most complicate topics in the field of autonomous vehicles [36, 43, 23, 6].
When the dilemma implies to choose between two human lives, there is not easy an- swers. Such is the case of the classic trolley problem, an ethical problem who presents the next decision: [29]
”You see a runaway trolley speeding down the tracks, about to hit and kill five people.
You have access to a lever that could switch the trolley to a different track, where a different person would meet an untimely demise. Should you pull the lever and end one life to spare five?” [29].
Figure 2.26: Trolley problem [1]
From this problem arises the experiment The Moral Machine, this experiment is de- signed to test nine different comparisons: should a self-driving car prioritize humans over pets, passengers over pedestrians, more lives over fewer, women over men, young over old, fit over sickly, higher social status over lower, law-abiders over law-benders. And finally, should the car swerve (take action) or stay on course (inaction)? [29, 6].
The experiment found that the taken decisions differ widely between countries, culture and economics. This rise the following questions, what ethics should an autonomous vehicle follow?, what factors should it take in consideration at the moment of deciding?, how do we measure the value of a life? [29, 6].
Certainly, there is not a conclusive answer for any of these questions, but one thing seems clear and is that autonomous vehicles need their own ethic. In the following chapters, this thesis will propose the elements that such ethic should take in consideration at the mo- ment of a decision. Nevertheless, such decision is still an open question.
In order to select such elements, we are going to use The German Ethics Code for Au- tomated and Connected Driving as a base, more specific, we are going to use the ethical guidelines oriented to unavoidable dilemma situations (5-9), except for the 6 which is not rel- evant for our work: [43, 23]
Ethical Guideline 5
“Automated and connected technology should prevent accidents wherever this is practically possible. Based on the state of the art, the technology must be designed in such a way that crit- ical situations do not arise in the first place. These include dilemma situations, in other words a situation in which an automated vehicle has to “decide” which of two evils, between which there can be no trade-off, it necessarily has to perform. In this context, the entire spectrum of technological options—for instance from limiting the scope of application to controllable traffic environments, vehicle sensors, and braking performance, signals for persons at risk, right up to preventing hazards by means of “intelligent” road infrastructure—should be used and continuously evolved. The significant enhancement of road safety is the objective of de- velopment and regulation, starting with the design and programming of the vehicles such that they drive in a defensive and anticipatory manner, posing as little risk as possible to vulnerable road users.” [43, 23].
Ethical Guideline 7
“In hazardous situations that prove to be unavoidable, despite all technological precautions being taken, the protection of human life enjoys top priority in a balancing of legally pro- tected interests. Thus, within the constraints of what is technologically feasible, the systems must be programmed to accept damage to animals or property in a conflict if this means that personal injury can be prevented.” [43, 23].
Ethical Guideline 8
“Genuine dilemmatic decisions, such as a decision between one human life and another, de- pend on the actual specific situation, incorporating “unpredictable” behavior by parties af- fected. They can thus not be clearly standardized, nor can they be programmed such that they are ethically unquestionable. Technological systems must be designed to avoid accidents.
However, they cannot be standardized to a complex or intuitive assessment of the impacts of an accident in such a way that they can replace or anticipate the decision of a responsible driver with the moral capacity to make correct judgements. It is true that a human driver would be acting unlawfully if he killed a person in an emergency to save the lives of one or more other persons, but he would not necessarily be acting culpably. Such legal judgements, made in retrospect and taking special circumstances into account, cannot readily be transformed
into abstract/general ex ante appraisals and thus also not into corresponding programming ac- tivities. For this reason, perhaps more than any other, it would be desirable for an independent public sector agency (for instance, a Federal Bureau for the Investigation of Accidents Involv- ing Automated Transport Systems or a Federal Office for Safety in Automated and Connected Transport) to systematically process the lessons learned.” [43, 23].
Ethical Guideline 9
“In the event of unavoidable accident situations, any distinction based on personal features (age, gender, physical, or mental constitution) is strictly prohibited. It is also prohibited to offset victims against one another. General programming to reduce the number of personal injuries may be justifiable. Those parties involved in the generation of mobility risks must not sacrifice non-involved parties.” [43, 23].
2.5 Software
In order to develop this work multiple libraries and software were used. The list of such software and libraries is presented below:
• OpenCV, an open source computer vision and machine learning software library [9].
• ROS (Robot Operating System), a collection of tools, libraries, and conventions that aim to simplify the task of creating complex and robust robot behavior across a wide variety of robotic platforms [61].
• MATLAB, a desktop environment tuned for iterative analysis and design processes with a programming language that expresses matrix and array mathematics directly [47].
Development
In this chapter, the implementation of the proposed software architecture is presented. The project as a whole consists in three layers, figure 3.1, vehicle layer, in this layer are all the components directly related with the vehicle (steering wheel, brake pedal, etc.); hardware layer, this layer contains most of the components used in order to interact whit the ones in the previous layer (motors, microcontrollers, etc.); and robotic layer.
Figure 3.1: Layers
The robotic layer is where the proposed software architecture can be found, figure 3.2.
Making use of ROS a series of nodes were implemented. The nodes are also connected through standard messages, so they can be changed or replaced in an easy manner, allowing to test different algorithm with ease. In the following sections a brief overview of each node is presented.
33
Figure 3.2: Robotic layer
3.1 Lane Detector
This node takes as input an image in order to generate a set of references for the track of a lane in real-time. In this case, the references consist in the deviation of the vehicle with respect the lane (a distance) and the difference between the orientation of the vehicle and the lane (an angle). The node basically consists in a series of visual algorithms that were implemented using OpenCV [28]. The whole process goes as follows:
Input Image
Color space change to HSL
Color mask for white, pixels that have a Lightness value
greater than 190.
Color mask for yellow, pixels with a Hue value between 10 and 50 and a high Saturation value.
ROI mask, region of the image who contain the lane
Bitwise operation:
(White mask OR Yellow mask) AND ROI mask
Canny Edge
Hough Transform
Computation of the lane lines using the slopes and intercepts
Estimation of the deviation of the vehicle
and the difference in the orientation
References
Figure 3.3: Lane Detector flux diagram
3.1.1 Mean Filter
In this node the last 5 inputs received from the previous node, Lane Detector in this case, are filtered through a mean filter, the result is then sent as the output. This process helps to mitigate the effect of situations were the lanes are not correctly defined. Since lanes normally tend to change slowly this process as only a small effect over the accuracy.
Algorithm 6 Mean filter MEAN(xnew)
1 L.deleteOldest() 2 L.addData(xnew) 3 for i = 1 to 5 do
4 xsum ← xsum+ L.get(i) 5 xmean ← xsum/5
6 Return xmean
3.2 Object Detection
The node was also implemented trough OpenCV, using the YOLOv3 object detector pre- trained on the COCO dataset, [55, 56]. It takes as input an image and outputs the bounding boxes and class labels for those bounding boxes, it also outputs the estimated distance be- tween the detected object and the vehicle.
Is important to mention that the class labels are determined with the class probabilities and a threshold, in other words, if the class probability is below the threshold the detected object is discarded, this in order to reduce the effect of perturbations and misdetections.
The threshold was selected based in how much is expected the object to change between one and another, for example, traffic signs tend to be very similar to each other, but humans by the most don’t, or at least not in the same degree.
Another of the consideration in the selection of the threshold was how dangerous will be to discard a correct detection, going back to the previous example, is more dangerous to ignore a human than a traffic sign. The principal objects to detect and the selected thresholds for them are presented below:
• Car ⇐⇒ 90%
• Truck ⇐⇒ 90%
• Person ⇐⇒ 60%
• Sign ⇐⇒ 95%
• Bike ⇐⇒ 80%
• Bus ⇐⇒ 90%
3.3 Path Planning
This node has 5 different implementations, this in order to taste the performance of the differ- ent algorithms. Nevertheless, all the implementations take as input a binary image of the map and, the start and goal coordinates. The output is also the same, the coordinate of the next point to reach. The algorithms implemented are:
• single-algorithms – RRT
– RRT*
– Artificial Potential Field
• multi-algorithms
– RRT + Artificial Potential Field – RRT* + Artificial Potential Field
The multi-algorithms consist in an implementation of a base algorithm (RRT or RRT*) in order to generate the general path, followed by an implementation of a second algorithm (Artificial Potential Field) used to compute the point to point path of the general path.
This can sound redundant, but the point of this implementations was to test if the use of a global planner followed by a local planner could ensure a sturdier planner without signifi- cantly increasing the computational time.
3.3.1 Trajectory Estimator
Determines the steering angle using the position of the vehicle and the target. In this node the inputs are a vector, which represents the current “goal”, and the current position of the vehicle, also a vector. Using both vectors the node estimates the trajectory to follow in order to reach the “goal” and outputs the reference to do so. In this case, the reference is the angle between the vehicle and the “goal.
θ = cos−1( u · v
kukkvk) (3.1)
Figure 3.4: Reference
3.4 Decision node
The node consists in a fuzzy logic implementation. The objective is to generate a set of out- puts that in theory should allow a secondary system to take a decision in a case of unavoidable collision. Nevertheless, such secondary system is not present in this paper.
Figure 3.5: Unavoidable collision
In order to stablish the input and outputs of this node the next consideration and assump- tions were made:
• All human lives wort the same, regardless of their gender, age, race or any other physical quality.
• Inputs and outputs need to have a level of abstraction, so they can be used regardless of the systems.
• The system will only be presented situation with two single individuals.
The membership functions for the inputs and outputs, and the rules to merge them are pre- sented below.
3.4.1 Inputs
Steering Angle
Steering angle estimated by the lane detector and the path planning, divided in highly safe, safe, risky, highly risky and dangerous. 5 fuzzy sets.
Figure 3.6: Steering angle
Velocity
Current velocity of the vehicle, divided in low, medium and high. 3 fuzzy sets.
Figure 3.7: Velocity
Proximity
Distance between the vehicle and the object, divided in far, near and dangerously near. 3 fuzzy sets.
Figure 3.8: Proximity
3.4.2 Outputs
Delta Steering
Divided in low, medium low, medium, medium high and high.
Figure 3.9: Delta steering
Delta Velocity
Divided in low, medium low, medium, medium high and high.
Figure 3.10: Delta velocity
Severity
Divided in S0, S1, S2 and S3.
Figure 3.11: Severity
3.4.3 Rules
Delta steering
Steering Angle — Proximity µf ar µnear µdangerously−near
µhighly−saf e µlow µlow µmedium−low
µsaf e µlow µmedium−low µmedium−low
µrisky µmedium−low µmedium µmedium
µhighly−risky µmedium µmedium−high µmedium−high
µdangerous µmedium−high µhigh µhigh
Table 3.1: Delta Steering Rules
Delta velocity
Velocity — Proximity µf ar µnear µdangerously−near
µlow µlow µmedium−low µmedium
µmedium µmedium−low µmedium µmedium−high
µhigh µmedium µmedium−high µhigh
Table 3.2: Delta Velocity Rules
Severity
Steering Angle — Velocity µlow µmedium µhigh
µhighly−saf e µS0 µS0 µS0
µsaf e µS0 µS0 µS0
µrisky µS0 µS0 µS1
µhighly−risky µS0 µS1 µS1
µdangerous µS1 µS1 µS2
Table 3.3: Severity Rules with Proximity: Far
Steering Angle — Velocity µlow µmedium µhigh
µhighly−saf e µS0 µS0 µS0
µsaf e µS0 µS0 µS1
µrisky µS0 µS1 µS1
µhighly−risky µS1 µS1 µS2
µdangerous µS1 µS2 µS3
Table 3.4: Severity Rules with Proximity: Near
Steering Angle — Velocity µlow µmedium µhigh
µhighly−saf e µS0 µS0 µS1
µsaf e µS0 µS1 µS1
µrisky µS1 µS2 µS2
µhighly−risky µS2 µS3 µS3
µdangerous µS3 µS3 µS3
Table 3.5: Severity Rules with Proximity: Dangerously Near
Results
In this chapter is presented a fast overview of the tests perform to the nodes along side the results obtained of such tests. Since the project is oriented to controlled environments, some of the test where realized in a real-world-like environment, while others were done using real world data. Nevertheless, for security reason this was not possible with all the nodes.
4.1 Lane Detector
This node was tested using a real-world footage of a road from the point of view of a camera mounted in a car. Below are presented the graphs of the references obtained through each frame of the video; in figure 4.1 is the deviation between the orientation of the vehicle with respect the road, and in figure 4.2 is the distance between the center of the vehicle and the center of the road.
44
Figure 4.1: Angle
Figure 4.2: Distance
The resulting images of each step of the lane detection algorithm are presented in figure 4.3; in (a) is presented the original frame, in the following steps (b-c) are shown the results of the used masks, in (d) we can see the output of the Canny edge, finally in (e) is presented the resulting lanes obtained with the Hough transform.
(a) Input image (b) Color mask
(c) ROI mask (d) Canny Edge
(e) Hough Transform Figure 4.3: Lane Detector steps
4.1.1 Mean Filter
The test of this node was made using as inputs the resulting references for the previous node (Lane Detector). In the following graphs we can see a comparation between the original references and the ones obtained after the filtering process; in figure 4.4 is the comparation between the original angle and the filtered one, in a similar way, in figure 4.5 is the compara- tion between the original distance and the filtered one.
Figure 4.4: Filtered angle
Figure 4.5: Filtered distance
4.2 Object Detection
In a similar way to previous nodes, this node was tested using a series of real-world footage of roads from different environments, urban zones as parking lots and highways, and rural zones as small towns outside of the city. Some of the cases with miss detections are presented
below in figure 4.6; in (a) is presented a road with only one car but two are detected, in (b) a building detected as a truck, and in (c) a garbage truck detected as a bus. Nevertheless, is important to mention, that most of these misdetections were eliminated by the thresholds proposed previously.
(a) Misdetected car (b) Misdetected truck
(c) Misdetected bus
Figure 4.6: YOLO misdetections
4.3 Path Planning
In order to test the different implementations of this node the algorithms were classified in 2 groups, single-algorithms (RRT, RRT* & Artificial Potential Field) and multi-algorithms (RRT + Artificial Potential Field & RRT* + Artificial Potential Field).
All the algorithms were tested using the same input parameters for its corresponding group. Nevertheless, both groups were tested in the same environment, in this case the test environment was a section of a parking lot, as presented in figure 4.7.
Figure 4.7: Test environment
In the test the algorithms needed to find a route between two point in the map. Each algorithm was tested 1000 times. The average computational time and path cost of all the tests is presented below.
Single-algorithms RRT RRT* APF
Time (s) 1.03 4.94 0.011
Path cost 1081 972 1060
Table 4.1: Single-algorithms
Multi-algorithms RRT + APF RRT* + APF
Time (s) 1.31 5.15
Path cost 1100 1012
Table 4.2: Multi-algorithms
Furthermore, the algorithms were texted in a bigger environment in order to test their sturdiness. In this case the test environment was a greater section of the same parking lot, as presented in figure 4.8.
Figure 4.8: Second test environment
In this test each algorithm was run 5 times, the path cost of each test is presented below.
However, not all the algorithms were able to find a route in this test, those who couldn’t have the label N.P. (no path).
Single-algorithms RRT RRT* APF
Test 1 577 499 693
Test 2 2728 776 1061
Test 3 1889 1675 N.P
Test 4 1151 449 N.P
Test 5 2451 1657 N.P
Table 4.3: Single-algorithms per test results
Multi-algorithms RRT + APF RRT* + APF
Test 1 734 659
Test 2 1096 949
Test 3 2335 2260
Test 4 1285 567
Test 5 2865 2244
Table 4.4: Multi-algorithms per test results
4.3.1 Trajectory Estimator
Since this node is computationally speaking really simple, it was tested using two random point and then the result was compared with the one given by MATLAB, this process was done 10 times, all of the tests gave the correct result.
4.4 Decision node
This node was implemented in MATLAB using its Fuzzy Logic Toolbox. In order to test it the first step was to stablish the ranges for the input and output parameters. In the following sections are presented the selected ranges and the results obtained from the tests.
4.4.1 Inputs
Steering Angle
Presented in figure 4.9. Ranges from 0-80 degrees, the range was determined based in our knowledge about the navigation system developed in previous nodes.
Figure 4.9: Steering angle
Velocity
The range was selected based in the standard ISO 26262, in [33], it goes from 0-50 Km/h, as presented in figure 4.10.
Figure 4.10: Velocity
Proximity
Based on the “Reglamento de la Ley de Movilidad y de Transporte del Estado de Jalisco”, in [59], the range was set from 0-10 m, as shown in figure 4.11.
Figure 4.11: Proximity
4.4.2 Outputs
Delta Steering
Presented in figure 4.12. Ranges from 0-1, can be understood as the percentage of force that the motor in charge of the steering needs to deliver.
Figure 4.12: Delta steering
Delta Velocity
Similar to the previous output, it ranges from 0-1, as presented in figure 4.13, and has the same implication but over the motor in charge of the brakes.
Figure 4.13: Delta velocity
Severity
Based in the standard ISO 26262, in [33], it can be seen as an estimation of the severity of the injuries the person would suffer if the maneuver success, is important to remember that the system is designed for unavoidable collisions; it ranges from S0 to S1, as shown in figure 4.14.
Figure 4.14: Severity
4.4.3 Rules
In the following sections, figure 4.15 to 4.19, are presented the surfaces that are obtained when an output variable is evaluated, using the rules presented in previous sections, with each com- bination of the input variables. In other words, these surfaces represent the relation between the input and output variables.
Delta steering
Figure 4.15: Delta steering surface
