Structure analysis through multispectral image processing from a UAV

Texto completo

(1)Instituto Tecnológico y de Estudios Superiores de Monterrey Campus Monterrey. School of Engineering and Sciences. Structure Analysis through Multispectral Image Processing from a UAV A thesis presented by. Ernesto Vega Prado Submitted to the School of Engineering and Sciences in partial fulfillment of the requirements for the degree of Master of Science in Intelligent Systems. Monterrey, Nuevo León, December 5th , 2017.

(2)

(3) Instituto Tecnológico y de Estudios Superiores de Monterrey Campus Monterrey School of Engineering and Sciences. The committee members, hereby, certify that have read the thesis presented by Ernesto Vega Prado and that it is fully adequate in scope and quality as a partial requirement for the degree of Master of Science in Intelligent Systems.. Dr. José Luis Gordillo Moscoso Tecnológico de Monterrey Principal Advisor. Dr. Andrés Hernández Gutiérrez Universidad de Monterrey Co-advisor. Dr. Diego Fabián Lozano Garcı́a Tecnológico de Monterrey Committee Member. Dr. Rubén Morales Menéndez Dean of Graduate Studies School of Engineering and Sciences Monterrey, Nuevo León, December 5th , 2017. i.

(4)

(5) Declaration of Authorship. I, Ernesto Vega Prado, declare that this thesis titled, ”Structure Analysis through Multispectral Image Processing from a UAV” and the work presented in it are my own. I confirm that: • This work was done wholly or mainly while in candidature for a research degree at this University. • Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated. • Where I have consulted the published work of others, this is always clearly attributed. • Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this dissertation is entirely my own work. • I have acknowledged all main sources of help. • Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself.. Ernesto Vega Prado Monterrey, Nuevo León, December 5th , 2017. c 2017 by Ernesto Vega Prado All Rights Reserved. iii.

(6)

(7) Dedication. This work was specially dedicated to my parents Virginia Prado and Ernesto Vega, my sister Alejandra Vega, my grandparents Emilio Vega, Inés Nanni, Virginia Benito and Armando Prado, and to all my family and friends. Thanks you all for these years of support and confidence. Thanks for the patience on the hardest moments and for encouraging me to pursue my dreams. You are my motivation to be better each day.. v.

(8)

(9) Acknowledgements. I would like to begin by thanking CONACyT and the ITESM for the support given over these two years through the scholarships granted and for supplying the resources to fulfill my studies and thesis work. I want to thank my advisor, P.h. D. José Luis Gordillo, for his tutoring during the development of the thesis and for the guidance to keep improving. Many thanks to P.h. D. Andrés Hernández Gutiérrez for the constant help during the learning phase and preparation of this work, and for the help on the experiments performed to validate the results. Also, special thanks to P.h. D. Fabián Lozano Garcı́a for the support and interest on this project, in addition to providing the equipment necessary for its realization. I would like to give my gratitude to the teachers and students at the Tecnológico de Monterrey, campus Monterrey, specially to the autonomous vehicles group, who accompanied and assisted me during the preparation of the thesis. A special thanks to Luis Eduardo Garza and Alan Samuel Orozco who taught me the operation and programming of the drone used for this project, and to the B.Sc. students Mayela Girón Carreón, Luis Andrés Castillo Hernández, José Ezequiel Castro Elizondo, Alejandro Elizondo Salinas and Mariel Aguirre Peña for the help during the preparation of both the vehicle and the tests for the validation of the thesis. My most sincere thanks to my family, specially to my aunt Yuriria Prado, my uncle Héctor Vázquez del M. and my grandmother Virginia Benito, for helping me during my stay in Monterrey. My deepest gratitude to my parents Virginia Prado and Ernesto Vega, along with my sister Alejandra Vega, for their unconditional love and support during all these years of study. Thanks to all my cousins and friends, who were always by my side at the hardest moments to cheer me up.. vii.

(10)

(11) Structure Analysis through Multispectral Image Processing from a UAV by Ernesto Vega Prado Abstract For the industry, the inspection of structures has become a tedious task which requires the investment of an important amount of time and resources, yet necessary to guarantee the proper safety measures demanded by laws. With each new improvement in the structural analysis, companies keep trying to find faster and more reliable ways to check their buildings without risking their employees or having to rely on expensive methods of inspection for accurate results. This graduate thesis focuses on the development of a platform for aerial inspection based on multispectral image processing. An unmanned aerial vehicle performs both the navigation and vision processes through an embedded microcomputer and a multispectral sensor that captures images of the structure. The drone is positioned over the target through semiautonomous commands before starting the analysis. Different spectral images of the structure are collected to extract features and use them along with the spectral signatures of the possible failures to classify each flaw on the inspected section. The results of the failure detection are sent to a ground control station where the operator is able to find should there be problems on the structure, hence reducing the time required for the inspection and the amount of resources and workforce required.. ix.

(12)

(13) List of Figures 1.1 1.2 1.3. Navigation process and online inspection over a structure. . . . . . . . . . . . Operation scheme for the navigation and vision processes. . . . . . . . . . . Architecture of the ground control station and the aerial vehicle. . . . . . . .. 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9. Hexacopter’s configurations and reference frames. . . . Hexacopter’s basic movements. . . . . . . . . . . . . . UAV’s structure and flight components. . . . . . . . . Power distribution diagram. . . . . . . . . . . . . . . . Control diagram. . . . . . . . . . . . . . . . . . . . . Communication diagram. . . . . . . . . . . . . . . . . Navigation process. . . . . . . . . . . . . . . . . . . . MAVLink packet’s structure. . . . . . . . . . . . . . . Orientation comparison of the UAV over the structure.. . . . . . . . . .. 8 9 10 12 13 14 14 15 18. 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9. Lens optics on a camera. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pinhole model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Types of radial distortion . . . . . . . . . . . . . . . . . . . . . . . . . . . . Comparison between a correct alignment and tangential distortion. . . . . . . Table of common image transformations and their degrees of freedom. . . . . Affine transformation examples. . . . . . . . . . . . . . . . . . . . . . . . . Comparison between unaligned image (left) and an aligned one (right). . . . . Graphical example of interpolation and function estimation. . . . . . . . . . Structure delimitation example. (a) Structure aerial sight. (b) Delimited area to be inspected. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Edge detection of different environments. . . . . . . . . . . . . . . . . . . . Canny edge detection process. (a) Original image. (b) Smoothed image through Gaussian filter. (c-d) Gradients of the image. (e) Non-maxima filtered image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Line representation on polar coordinates. . . . . . . . . . . . . . . . . . . . . . Hough transform. (a) Points tested on the line. (b) Lines representation on polar coordinates with the intersection that corresponds to the line to which the previous points belong. . . . . . . . . . . . . . . . . . . . . . . . . . . . Structure’s form and inspection section. (a) Examples of goal like structures. (b) Inspection section highlighted in red. . . . . . . . . . . . . . . . . . . . .. 22 23 24 25 26 26 27 29. 3.10 3.11. 3.12 3.13. 3.14. xi. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. 3 4 5. 30 30. 31 32. 33 35.

(14) 3.15 Line’s selection procedure. (a) Lines obtained from the structure, plotted in different colors. (b) Reduction of short lines using the length as restriction. (c) Filtering of non horizontal lines using angles as thresholds. (d) Region marked by joining the last pair of lines. . . . . . . . . . . . . . . . . . . . . 3.16 Reflection on materials. (a) Types of reflectors. (b) Diffuse reflection. . . . . 3.17 Construction of a spectral signature based on multispectral imaging. . . . . . 3.18 Graph with spectral signatures for different materials. . . . . . . . . . . . . . 3.19 Comparison between a multispectral image over a wall’s surface and the detected damage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.20 Example of the minimum distance classifier. . . . . . . . . . . . . . . . . . . 4.1 4.2 4.3 4.4 4.5. 4.6 4.7 4.8 4.9 4.10 4.11 4.12. 4.13. 4.14. 4.15 4.16. Takeoff tests. (a) Manual takeoff behavior. (b) Autonomous takeoff up to 7 meters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Comparison between manual flight test on Stabilize mode (left) and assisted flight test on Loiter mode (right). . . . . . . . . . . . . . . . . . . . . . . . . Landing tests. (a) Manual landing from Loiter mode. (b) Autonomous landing from Loiter mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Test of all the flight commands with three consecutive routines of takeoff, navigation on Loiter mode and landing. . . . . . . . . . . . . . . . . . . . . Test structure’s specifications. (a) Lateral view of the structure. (b) Frontal view of the structure, with the area to be inspected marked in red. (c) Real test structure (without base) with mock failures mounted. . . . . . . . . . . . Height threshold calculation. (a) Minimum distance conditions. (b) Maximum distance conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Diagram of connections between the microcomputer and the multispectral camera. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Camera distortion’s correction. (a) Distorted image. (b) Corrected image. . . Alignment test samples. (a) Test image at 5 meters. (b) Test image at 10 meters. (c) Test image at 15 meters. (d) Test image at 20 meters. . . . . . . . Test image at 30 meters. (a) Original BGR image. (b) Original NIR image. . Characteristic points (red points) in a spectral image for alignment verification. Structure detection process. (a) Real image of the object to be detected from camera 3. (b) Edges image produced after applying the Canny algorithm. (c) Lines obtained after using the Hough transform and the line selection criteria. (d) Final result of the detection which indicates the area that will be analyzed during the multispectral processing. . . . . . . . . . . . . . . . . . . . . . . The section of the structure to be inspected, with the mock failures as stripes all over it. (From left to right) The failures placed on it are green paper, grey tiziano paper, green fluorescent paint and grey tiziano paper. . . . . . . . . . Classification test. (a) Image of the section inspected using the BGR channels. (b) Image of the section inspected using the NIR channels. (c) Image of the area chosen for the classification based on the detection algorithm. (d) Result of the classification test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ground control station’s interface. . . . . . . . . . . . . . . . . . . . . . . . Final experiment setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii. 36 38 39 39 41 42 47 48 49 50. 51 52 53 55 56 57 57. 58. 59. 61 62 63.

(15) 4.17 Final experiment results at 7.2 meters. (a) Image of the section inspected using the BGR channels. (b) Result of the classification on the final experiments. . 4.18 Final experiment results at 11 meters. (a) Image of the section inspected using the BGR channels. (b) Result of the classification on the second online experiment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.1 Multispectral camera slaves order (Master is considered as camera zero). . . . B.1 Interpolation and estimation curves of the affine transformation from camera 0 to camera 3. . . . . B.2 Interpolation and estimation curves of the affine transformation from camera 1 to camera 3. . . . . B.3 Interpolation and estimation curves of the affine transformation from camera 2 to camera 3. . . . . B.4 Interpolation and estimation curves of the affine transformation from camera 4 to camera 3. . . . . B.5 Interpolation and estimation curves of the affine transformation from camera 5 to camera 3. . . . .. matrices . . . . . matrices . . . . . matrices . . . . . matrices . . . . . matrices . . . . .. obtained for the . . . . . . . . . . obtained for the . . . . . . . . . . obtained for the . . . . . . . . . . obtained for the . . . . . . . . . . obtained for the . . . . . . . . . .. C.1 Spectral signatures obtained though the test samples. . . . . . . . . . . . . . C.2 Spectral signatures obtained from the final experiment. . . . . . . . . . . . .. xiii. 64. 65 74 75 76 76 77 77 79 80.

(16)

(17) List of Tables 2.1 2.2. MAVLink packet’s sections. . . . . . . . . . . . . . . . . . . . . . . . . . . MAVLink flight modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 15 16. 4.1 4.2 4.3 4.4 4.5 4.6. Alignment’s mean errors calculated from the tests at 5 and 10 meters. Confusion matrix of the classification test. . . . . . . . . . . . . . . . Confusion matrix of the final experiment’s classification at 7.2 meters. Confusion matrix of the final experiment’s classification at 11 meters. Table of mean values from the results of the final experiments. . . . . Table of standard deviations from the results of the final experiments.. . . . . . .. . . . . . .. . . . . . .. . . . . . .. 57 61 64 65 66 66. A.1 A.2 A.3 A.4. Multispectral camera specifications. . . . . . . . . . . . . . . . . . Multispectral camera corresponding wavelengths and colors. . . . . Multispectral camera intrinsic parameters and distortion coefficients. Multispectral camera transformation matrices between cameras. . .. . . . .. . . . .. . . . .. . . . .. 73 73 73 74. . . . .. C.1 Mean values and standard deviations for all materials on the different channels for the test samples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.2 Mean values and standard deviations for all materials on the different channels for the final experiment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. xv. 80 81.

(18)

(19) Contents Abstract. ix. List of Figures. xiii. List of Tables. xv. 1. 2. 3. Introduction 1.1 Context and Background 1.2 Problem Statement . . . 1.3 Objective . . . . . . . . 1.4 Methodology . . . . . . 1.5 Scope and Contributions 1.6 Composition . . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. UAV operation and navigation 2.1 Hexacopter flight modelling . . . . . . . . 2.2 UAV’s structure and architecture . . . . . . 2.2.1 Components . . . . . . . . . . . . 2.2.2 UAV’s architecture . . . . . . . . . 2.3 UAV’s movement and operation . . . . . . 2.3.1 MAVLink protocol and flight modes 2.3.2 Autonomous takeoff . . . . . . . . 2.3.3 Semiautonomous movement . . . . 2.3.4 Autonomous Landing . . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. 1 1 2 2 3 5 6. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. 7 8 10 10 12 14 15 17 18 19. Image processing and multispectral analysis 3.1 Image capture and correction . . . . . . . . . . . . . . . 3.1.1 Pinhole model . . . . . . . . . . . . . . . . . . 3.1.2 Image calibration . . . . . . . . . . . . . . . . . 3.2 Multiple image alignment . . . . . . . . . . . . . . . . . 3.2.1 Affine transformation . . . . . . . . . . . . . . . 3.2.2 Matrices calculation, interpolation and estimation 3.3 Structure detection . . . . . . . . . . . . . . . . . . . . 3.3.1 Edges detection and Canny algorithm . . . . . . 3.3.2 Hough transform . . . . . . . . . . . . . . . . . 3.3.3 Line’s selection criteria . . . . . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . .. 21 22 23 24 25 26 27 29 30 32 35. xvii. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . ..

(20) 3.4. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. 37 37 38 39 40 42. Experiments 4.1 Unmanned aerial system design . . . . . . . . . . . . . 4.1.1 Multispectral camera specifications and montage 4.2 UAV’s flight experiments . . . . . . . . . . . . . . . . . 4.2.1 Command tests . . . . . . . . . . . . . . . . . . 4.2.2 Integrated flight test . . . . . . . . . . . . . . . 4.3 Vision process experiments . . . . . . . . . . . . . . . . 4.3.1 Location of the camera . . . . . . . . . . . . . . 4.3.2 Multispectral camera operation . . . . . . . . . . 4.3.3 Image correction and alignment . . . . . . . . . 4.3.4 Structure detection test . . . . . . . . . . . . . . 4.3.5 Multispectral characterization of failures . . . . 4.3.6 Classification test . . . . . . . . . . . . . . . . . 4.4 Integrated experiment . . . . . . . . . . . . . . . . . . . 4.4.1 Ground control station’s interface . . . . . . . . 4.4.2 Experimental setup . . . . . . . . . . . . . . . . 4.4.3 Failure detection results . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . .. 43 43 45 46 47 50 51 51 53 55 58 59 60 62 62 63 64. . . . .. 67 67 68 70 71. 3.5. 4. 5. Multispectral analysis . . . . . . . . . . . 3.4.1 Changes on material’s reflectance 3.4.2 Spectral signature . . . . . . . . . 3.4.3 Near-Infrared Spectrum . . . . . Failure detection and classification . . . . 3.5.1 Minimum distance classifier . . .. Conclusions 5.1 Flight analysis . . . . 5.2 Vision analysis . . . 5.3 Overall performance 5.4 Future Work . . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . . . .. . . . .. . . . . . .. . . . .. . . . . . .. . . . .. . . . . . .. . . . .. . . . . . .. . . . .. . . . . . .. . . . .. . . . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. A Multispectral camera specifications and calibration parameters. 73. B Estimated curves of affine matrices values. 75. C Graphs of spectral signatures. 79. Bibliography. 85. xviii.

(21) Chapter 1 Introduction 1.1. Context and Background. Present day, many industries are still in need to prepare periodical inspection tasks in order to fulfill the compulsory safety conditions demanded by laws and guarantee the security of their resources and employees. For a proper safeguarding review, it is necessary to perform structural investigations and visual verification of failures on periods between 5 and 10 years [BCA, 2012]. The process to check every structural component can be quite time-consuming and costs lots of resources to companies when applied on different buildings. Not only does the study and inspection of structures cost time, but also requires the implementation of specialized instruments and equipment to find possible problems and defects that usually are hard to detect [Chong et al., 2003]. Common cameras and sensors cannot cover all the possible sources of failures, and most of the time it is necessary to check inaccessible or potentially dangerous places. Because of occluded failures sources, it is useful to include new devices that can gather data from different sensors and adequate them on specialized machines during the inspection. To accomplish complex data-gathering tasks over large areas, it is possible to use different kinds of mechanisms and vehicles for remote inspection [Bishop, 2000]. The automation of processes has come along with the use of more independent devices that could effectuate repetitive activities that usually are tedious and time-consuming for people. In the case of vehicles and similar devices, they are instructed to perform tasks related to monitoring and inspecting according to their instruments. For this document, analysis on the near-infrared or NIR spectrum is considered as a way to inspect wireframe and tubular structures such as oil refineries or bridges; this through techniques that analyze the spectral signature of the structure’s material. The usage of a UAV (Unmanned aerial vehicle), has been proposed as an alternative to speed up the checking process and reduce costs because of the easiness to maneuver this kind of vehicles and the possibility to use the same equipment for the inspection of different structures [Lovelace, 2015]. The operation of an aircraft to gather images from inaccessible regions of a structure helps to perform a more complete and proper inspection of the facilities, and computer vision algorithms can be used to detect possible failures based on the images captured. 1.

(22) CHAPTER 1. INTRODUCTION. 1.2. 2. Problem Statement. The inspection process performed nowadays is changing to allow faster and more reliable methods for data acquisition on structures. The information gathered from the maintenance processes helps to identify possible problems that can become threats. However there are still risks in the way this procedure is done. During the inspection, it is necessary that an operator approaches to a specific section of the structure. Sometimes accessing this area can be difficult and dangerous for the workers, apart from the time and resources required for the whole process, which result to be expensive. UAVs are an alternative to optimize the monitoring and detection of possible failures on structures due to the easiness to operate them and the usage of different kind of sensors on the aircraft to receive all the required information from the construction, even if it is on an inaccessible spot. These platforms can stand on a stabilized pose during structure monitoring and perform takeoff and landing maneuvers on reduced spaces; however, the problem with the operation of drones is the way they are used and how results are extracted. Normally drones are used as flying sensors that gather data, but no process or inspection is done until the vehicles return and the data is extracted. This is a time-consuming process that can potentially require various iterations once a failure is detected, slowing down the maintenance inspection and ultimately becoming more costly to the industry.. 1.3. Objective. For the purpose of this work, we will understand online to the point a UAV has been deployed and is flying over a defined target. Offline will refer to the work done with the device on the ground. The general objective of the following investigation is to elaborate a system capable of online-inspection of a structure to detect possible failures using multispectral images for a deeper analysis of the damage and send the results to a ground station. Along with the previous point, some procedures are needed to perform the multispectral inspection and failure detection. The particular objectives of the work are: • Develop an integrated algorithm that allows to navigate a drone over a wireframe structure and inspect the desired location once positioned. • Locate and stabilize the aircraft over the target through autonomous and semiautonomous instructions before the inspection. • Correct and align the different channels of a multispectral camera prior to the image analysis. • Detect the wireframe structure and the area to be analyzed on it through lines detection. • Elaborate a multispectral image processing algorithm capable to perform the detection and classification of failures at the structure. • Mount the multispectral camera on an aerial vehicle to collect the images of the structure during the flight..

(23) CHAPTER 1. INTRODUCTION. 1.4. 3. Methodology. The navigation process consist on positioning the vehicle over the wireframe structure before starting the inspection and land once the classification finishes. It is needed both a human pilot to move the aircraft to the desired position and a flight controller to send commands and stabilize the vehicle in the desired position. The whole process for the navigation will be described on five steps: 1. Vertical Takeoff: The drone will start on the ground and a command will be sent from the control station to begin the takeoff. The drone will stop climbing once it reaches an altitude defined by the user. 2. Movement toward the target: The vehicle will move toward the area where the structure is located. This process is performed by the pilot while using a controlled flight mode to perform movements with greater precision. 3. Positioning over the structure: Once the UAV has reached the structure, it will be placed just over it using visual corroboration from the images received on the ground station. The drone will hover once it is properly positioned. 4. Vision process: The multispectral analysis will begin once the drone is hovering over the structure. The inspection will finish after all the classified failures are sent to the ground control station. 5. Landing: After the inspection finishes, the drone will be placed over a landing spot and descend to the ground. The proposed procedure is shown on Figure 1.1 step by step. The drone will approach to the wireframe structure to gather the data. Once it has been positioned over it, the multispectral analysis will begin. After the inspection finishes, the drone will move away from the structure and land. The expected result of the inspection consist on an image containing the failures found on the region, classified depending on the type of defect it corresponds to, helping the user identify them.. Figure 1.1: Navigation process and online inspection over a structure..

(24) CHAPTER 1. INTRODUCTION. 4. The vision process involves all the procedures required to gather the multispectral data, detect all the failures on the material and classify them on the respective kind of defect. The process can be divided into three steps, each with their own phases: 1. Camera operation: The camera is used over the structure to take a photograph of the section to be studied. All the multispectral images will be extracted from the camera and saved in the onboard computer for their processing. 2. Optical correction: All the pictures gathered are corrected and aligned before starting the image processing to reduce errors on the results. 3. Multispectral processing: This step uses information from the kinds of failures that can appear on the structure, as well as the corrected images from the previous point. The structure’s pictures will be analyzed to detect if a problem is present. In case a failure is found, it will be assigned to its corresponding class. The multispectral image processing will take time during the flight to detect all the failures before the drone can descend. When the images have been completely reviewed, the result is sent to the ground station . The complete operation scheme for the navigation and vision process can be seen on Figure 1.2, showing the order of the steps for each and when the inspection begins during the navigation.. Figure 1.2: Operation scheme for the navigation and vision processes. The problems that arise when analyzing data during flight are associated with the capacities of the drone to perform the inspection and navigation simultaneously. To allow online processing on the aircraft, embedded systems like microcomputers are required. The equipment used must be able to gather data during the flight and work with vision algorithms that require high computational resources [Colomina and Molina, 2014]. The proposed architecture for the experiment is shown in Figure 1.3. The multispectral camera is mounted on a UAV, which includes a flight controller and an embedded micro-.

(25) CHAPTER 1. INTRODUCTION. 5. computer to send instructions for navigation. Different modules of communication are used simultaneously for both command transmission to the vehicle and data reception on the ground station. The multispectral camera is used exclusively to gather information from the wireframe structure on flight, while the webcam allows seeing the drone’s position and orientation in real-time.. Figure 1.3: Architecture of the ground control station and the aerial vehicle.. 1.5. Scope and Contributions. This work was focused on the inspection of wireframe and tubular structures. This kind of shapes are common on oil refineries, bridges, pipelines, antennas and power towers which are strategic constructions. To validate the performance of the online inspection, a tubular structure with mock failures was proposed. The structure was made using PVC tubes to facilitate the assembly, with a section covered in grey paint where the mock failures were placed. The flaws consisted on stripes of three materials: grey Tiziano paper, green cardboard paper and green fluorescent paint. These failures have different spectral responses and were chosen to test the capability of the program to differentiate them even if similar at the visible spectrum. The inspection was performed from above the structure to facilitate the approximation and stabilization of the vehicle over it during the image capture. The failures were detected and simultaneously segmented according to their corresponding class. The navigation process was performed with autonomous commands and semiautonomous movements with the assistance of the flight controller for proper stabilization. Visual verification of the vehicle’s position and orientation was received at the ground station in real-time. The main contributions of this work are the following: • A multispectral image processing method and a program capable to detect and classify failures based the spectral signatures from different materials, using the mean and standard deviations values obtained from images gathered during offline tests. • An integrated vehicle capable of both aerial navigation and online multispectral analysis through the operation of a specialized camera onboard for data gathering..

(26) CHAPTER 1. INTRODUCTION. 1.6. 6. Composition. This thesis has been organized into four more chapters that explain the process followed to develop the proposed solution to the wireframe and tubular structure’s inspection; and the experiments developed to verify its performance. Chapter 2 explains the basics of unmanned aerial vehicles operation and navigation. The flight dynamics and basic movements of an hexacopter are explained. The architecture defined for the control and communication of the UAV is introduced, including all the components mounted on the drone. The commands for semiautonomous operation are mentioned, along with how these can be used by a pilot to guarantee a safe and stable navigation. Chapter 3 consists of the explanation of the multispectral image processing. The basic pinhole model is introduced, mentioning the intrinsic and extrinsic parameters of a camera. The corrections for the images are detailed, starting with the calibration to reduce possible distortions and continuing with the alignment between the different cameras. The structure’s detection algorithm is detailed, breaking down the steps required to delimit the inspection area from the structure. Multispectral analysis is fully explained, from the meaning of the spectral signature to the usage of near-infrared spectrum on structural inspection. Finally, the minimum distance classifier is described, mentioning the way to cluster a point from an image as a specific failure based on multispectral data. In Chapter 4, the experiments performed will be explained. All the elements of the UAV for the experiments are properly described on this section. The navigation tests are included, showing the behavior of the drone based on the instructions given. The offline tests for the vision process are divided to show the steps followed to develop the final classification algorithm. The results from an online fully integrated test are mentioned and explained, along with a comparison between the original images obtained from the camera and the results of the classification. Each test was validated to determine the performance, calculating the mean errors on the corrections and the percentages of samples successfully classified. Finally, in chapter 5, the conclusions of the work will be mentioned. A brief summary of the work is included along with the overall results obtained from the tests performed and some proposals to optimize the process developed on this thesis. In addition to the chapters, there are three appendixes with information from the tests prepared for the image correction, alignment and classification. Appendix A includes the multispectral sensor’s specifications and the intrinsic parameters, extrinsic parameters and transformation matrices obtained from the calibration of each camera. Appendix B consists on the graphs of the matrices used for the alignment between cameras and the models obtained from the approximations. Appendix C includes graphs with the spectral signatures of the mock failures and materials used on this work, along with tables of their mean values and standard deviations at each wavelength..

(27) Chapter 2 UAV operation and navigation An unmanned aerial vehicle, or UAV, is an aircraft without a pilot aboard that can be controlled remotely or through predefined instructions. The easiness on its manipulation has proven to be useful for research purposes [Floreano and Wood, 2015]. There are different tasks in which these vehicles are used like monitoring or surveillance [Addati and Prez, 2014], but the approach boarded in this thesis will be their application for wireframe and tubular structures inspection. Industries use drones to inspect over different kind of structures that can be hard to access or would require expensive equipment. Gas pipelines inspection is an example where drones are cost-efficient substitutes to the use of helicopters [Rathlev et al., 2012]. UAVs have also been used for remote oil fields inspection [McNeal, 2014], where the collected data is sent to technicians for further analysis, turning this into a fast, safe and accurate method to provide information and improve productivity. These aircrafts are useful for constant data gathering from a specific area thanks to the implementation of sensors that allow them to detect any change on the environment wherever they are deployed. The usage of specialized sensors improves the quality of the results and measures by reducing the margin of error. The real-time gathering and transmission of data helps to deliver applications and specific solutions for tasks that are hard to approach by common monitoring methods. These previous examples show how the inspection process can be improved with a proper platform like a UAV and the adequate instruments. However, it is important to know how the drone works. The chapter begins with the modelling of the UAV, including its reference frame and the basic movements it can perform. The following section mentions the components used to build a fully operational UAV and how these elements interact through control and communication architectures. At last, the different flight modes used by the controller are explained, along with how each one is used to perform instructions such as: taking off, aerial navigation and landing; and whether these could be done autonomously or semiautonomously.. 7.

(28) CHAPTER 2. UAV OPERATION AND NAVIGATION. 2.1. 8. Hexacopter flight modelling. UAVs can be classified depending on their size, autonomy or weight. Multirotor UAVs are quite popular because of their capability to perform vertical takeoff and landing, also the easiness in the maneuvering on flight that simplifies their operation [Nonami et al., 2010]. One of the most used drones within the multirotor UAVs are the quadrotors/quadcopters and hexacopters, UAVs with four and six motors respectively, which have higher stability on flight and, because of this, are commonly deployed for surveillance and monitoring tasks, along with proper sensors installed for easier navigation and inspection. For this chapter, an hexacopter model will be described, which was used for the experiments explained in the thesis work. The drone with six motors allows to carry higher weights without losing stability, which is useful when carrying heavy instruments for inspection. The hexacopter counts with six motors, three which spin clockwise and three counterclockwise to allow the elevation of the platform without losing stability and orientation [Artale et al., 2013]. Because of this, there are two main configurations for this kind of UAV. Figure 2.1 shows both of these examples, along with their proper reference frames. For the purpose of this thesis work, the configuration used on the UAV was the ”+” hexacopter, with a pair of motors on the X axis.. Figure 2.1: Hexacopter’s configurations and reference frames..

(29) CHAPTER 2. UAV OPERATION AND NAVIGATION. 9. The reference frame used normally to model a UAV is the NED frame, or North, East and Down frame, where the three axis are aligned toward these directions. The dynamics of the hexacopter are based on the changes between the position of the body frame of the vehicle and the reference frame, which can be separated in four basic movements, exemplified on figure 2.2: • Roll: It is the change of angular position around the X axis and enables the UAV to move from side to side (left to right). • Pitch: It consists of the change of the angular position around the Y axis. This rotation helps the drone to move forward or backward. • Yaw: It occurs when the angular position around the Z axis is changed. The drone spins over its own center to change its heading. • Throttle: Also called elevation, is the change of height of the drone and requires evenly reducing or increasing the speed of the motors.. Figure 2.2: Hexacopter’s basic movements..

(30) CHAPTER 2. UAV OPERATION AND NAVIGATION. 2.2. 10. UAV’s structure and architecture. To better understand how the UAV operates, the components that conform it will be fully explained. This section includes those elements that are required to fly the drone, along with some others that improve the performance, and the diagrams of the UAV’s architecture.. 2.2.1. Components. This section contains all the basic elements for the operation of the hexacopter. Figure 2.3 shows the UAV with the components included on this section.. Figure 2.3: UAV’s structure and flight components. • Hexacopter mechanical frame: It is the structure on which every other component is mounted. For this hexacopter, the frame follows a specific form, with two arms positioned over the X axis. The motors are located at the end of each arm, while the rest of the elements are mounted at the center of the frame. • Motors: The hexacopter uses six brushless phase motors to elevate. Each motor must be capable to reach a specific speed and power demand accordingly to the weight of the UAV and any additional load required. • Propellers: These are designed to provide enough trust to elevate the UAV. The speed of the motors, form and size of the propellers determine the maximum trust possible..

(31) CHAPTER 2. UAV OPERATION AND NAVIGATION. 11. • Electronic speed controllers: Also known as ESCs, these traduce the PWM signal from the flight controller to the three phase signal for the motors. • Energy source: A lithium potassium (LiPo) battery which gives energy to the motors and the rest of the components on the UAV. Usually only one battery is needed in order to reduce weight, unless some specific function demands an additional source. • Landing gear: It is the base of the drone, below the UAV’s frame. It can be retractable or fixed, and reduces the impact when the drone reaches the ground in order to let it land without damaging the structure. • Flight controller: It is the responsible of the lowest level instructions to the UAV in order to perform a proper operation during the flight. It usually has an inertial measurement unit (IMU) with an accelerometer, gyroscope and magnetometer, a barometer and I/O ports to connect other sensors and devices. • RF receptor: This allows the UAV to receive instructions from a remote controller via radio-frequency. Even with these basic components, there are other elements integrated to guarantee a proper performance of the drone. Some of these elements are: • GPS: It helps to get a more accurate position of the UAV through global coordinates and can be used along with a flight controller to move the drone based on GPS points. • Ultrasonic sensor: Used to estimate the the altitude of the UAV. The range of operation is commonly short, but it is useful to determine the remaining distance before reaching the ground while landing. • Onboard microcomputer: It can be connected to the flight controller and other elements on the UAV to allow more complex procedures and flight commands. The full extent of its use will be explained on the following chapters. • Wi-Fi and RF modules: Used with the onboard computer to communicate via Wi-Fi or radio-frequency from a ground station. • Vision sensors: It can be a camera or any other vision-based device used to capture images and gather information during the operation. For this thesis work, both a Multispectral camera and a webcam are used. • Gimbal: Useful for inspection tasks while carrying a camera on the drone. It allows mounting the camera on the platform and keeping it on the desired position and corrects the variations on the angular positions while flying..

(32) CHAPTER 2. UAV OPERATION AND NAVIGATION. 2.2.2. 12. UAV’s architecture. The different elements mentioned in the previous section allow the UAV to fly and work properly during its operation. However, it is necessary to design an adequate system with all of them to perform the desired inspection task efficiently. The hexacopter architecture proposed is divided into two main topics: Elements interconnection, which refers to the power distribution and connection between the components on the drone, and communication architecture, related to the way on which the hexacopter communicates with the user and ground station through different channels. Both of these points will be further explained in this section. Elements interconnection The components on the UAV are used for both the flight commands and the analysis process involved on the inspection task. The necessity of doing these procedures simultaneously requires the drone’s devices to be properly interconnected. There are two different schemes that explain how these elements interact: 1. Power distribution diagram: Represents how each elements receives energy to operate. Figure 2.4 shows how all the elements are connected to their respective power sources. 2. Control diagram: Represents how each component interacts and controls others in the UAV. Figure 2.5 shows these connections between devices on full control over one another; which means it controls the behavior of the second element completely, or partial control, when it only controls the component for certain tasks.. Figure 2.4: Power distribution diagram..

(33) CHAPTER 2. UAV OPERATION AND NAVIGATION. 13. Figure 2.5: Control diagram. Communication The UAV must be in constant communication with the user while flying. The elements involved in this process are divided depending whether these are on the ground station, where the user receives the data from the vehicle and controls it using commands and/or the radio controller, or the UAV, as modules to allow the data interchange. The way in which the user can send instructions and receive information from the UAV is separated into three different modules, which are exemplified on Figure 2.6: • Radio controller module: It consists of the radio-frequency receptor mounted on the UAV and the radio controller used by the user. This is the main method to control the vehicle manually, sending the set points to the flight controller to change the motors speed. For safety, it is always recommended to have a pilot to take control over the drone, even if it is working autonomously, in order to manipulate it in case of an emergency. • RF module: This channel consists of a RF receiver module connected to the onboard microcomputer on the UAV and a RF transmitter on the ground station. Through this way the user can send specific instructions for autonomous landing, takeoff or to perform a full routine. The onboard computer receives the corresponding command and sends the instructions directly to the flight controller. • Wi-Fi module: This channel requires a Wi-Fi adapter on the UAV and a modem on the ground station to allow the user to receive images from the drone and check the program status during the desired task in real-time. This is also useful when communicating with the onboard computer during the mission to start the program on the drone from the ground station. This is the only channel that can send and receive information as the the drone is offline or online. Both the flight controller and onboard microcomputer share information between them in order to perform the desired commands or send the current status to the ground station..

(34) CHAPTER 2. UAV OPERATION AND NAVIGATION. 14. Figure 2.6: Communication diagram.. 2.3. UAV’s movement and operation. The UAV’s navigation process is divided into five different steps as mentioned in Chapter 1. The complete process is shown in Figure 2.7, and each one of these steps require specific commands from the flight controller and the user to be performed correctly. The five parts of the navigation are: 1. Vertical takeoff: Process in which the drone elevates to a defined altitude. 2. Movement toward the objective: Displacement of the UAV until reaching the desired point of inspection. 3. Positioning over the structure: Adjustment of the vehicle over the place until the right angle and position are reached to start the analysis. 4. Vision process: The drone stands still until the image processing task finishes. 5. Landing: The drone moves toward a landing zone and approaches to the ground slowly. On this section, the different flight modes will be mentioned, including a brief description of the Mavlink protocol used by the flight controller to receive instructions and manipulate the drone. The commands used for each step will be explained, justifying the reason to perform each one autonomously or manually.. Figure 2.7: Navigation process..

(35) CHAPTER 2. UAV OPERATION AND NAVIGATION. 2.3.1. 15. MAVLink protocol and flight modes. MAVLink is a header-only message protocol extensively used for micro aerial vehicles. It is used to send and receive information between the drone and the ground station using packets through a serial channel. This protocol allows exchanging information with high efficiency and at a fast rate. For this thesis work, this protocol was used for the communication between the onboard microcomputer and the flight controller in order to accomplish two important tasks: Retrieving data from the drone’s location and sending it to the ground station, and giving specific instructions to the UAV during the flight through a previously prepared program on the onboard computer, enabling autonomous behavior. The packet structure is inspired by the CAN and SAE AS-4 standards, and has a defined length of a minimum of 8 bytes and a maximum of 263. Figure 2.8 shows the structure of the packet [QGroundControl, 2017], while Table 2.1 explains what each section of the packet represents. A packet in MAVLink is structured on XML, and can be created and translated using the onboard computer, allowing transmitting instructions to the UAV and receiving data from the embedded sensors in the flight controller. It is necessary to constantly send a heartbeat-like message (in a pulse manner) that would let the flight controller know that the system is responding. This message is usually sent at a rate of 1 Hz.. Figure 2.8: MAVLink packet’s structure. Byte Index 0 1 2 3 4 5 6 to (n+6) (n+7) to (n+8). Content Packet start sign Payload length Packet sequence System ID Component ID Message ID Data Checksum (LSB to MSB). Value v1.0: 0xFE (v0.9: 0x55) 0 - 255 0 - 255 1 - 255 0 - 255 0 - 255 (0 - 255) bytes ITU X.25/SAE AS-4, excluding start sign. Table 2.1: MAVLink packet’s sections. Once a message has been prepared, it can be sent to the flight controller to either receive specific data/measures from a process, or to perform a task in particular. The flight controller contains different kinds of modes and commands that can be executed depending on the model and firmware. The kind of commands that can be used depend also on the sensors and other devices integrated on the drone, even allowing the UAV to perform specific routines, save coordinates or change modes..

(36) CHAPTER 2. UAV OPERATION AND NAVIGATION. 16. The UAV can change into different modes predefined by the flight controller. Each one of these allows compensating errors on flight or executing a variety of movements that are harder to accomplish on manual operation. Because of this, it is recommended to implement these different modes to the routine according to the needs of each step. MAVLink includes a group of recommended flight modes: Stabilize, AltHold, Loiter, Return to Land (RTL) and Auto. Along with these, there are additional modes for more complex tasks. Table 2.2 mentions some of the most used flight modes [Ardupilot, 2016a]. Flight mode Stabilize. AltHold. Loiter. Return to Land (RTL). Auto. Land. Description Allows the user to fly the vehicle manually, but self-levels the roll and pitch axis. Because of this, the pilot would need to constantly hold a certain roll and pitch in case of wind moving the drone. The throttle changes accordingly to the pilot’s input. On this mode the UAV will maintain a constant altitude while allowing the pilot to change the roll, pitch and roll normally, as in Stabilize mode. While the throttle stick is between 40% and 60%, the drone will keep its altitude. If the stick is below 40%, the drone will descend depending on the stick’s deflection, and if the stick is over 60%, the drone will ascend depending on the stick’s deflection. This mode uses the measure of the barometer to determine altitude. This mode automatically attempts to maintain the current location, heading and altitude. The pilot may fly the vehicle as if it were in a manual flight mode, but when the sticks are released, the vehicle will slow to a stop and hold position. Horizontal position can be changed by the pilot at a defined maximum speed. Throttle can be manipulated as in AltHold mode. This mode requires the GPS and the IMU for the stabilization. Once this mode is activated, the drone will return to home location. First, the UAV will rise to a defined altitude (or stay in the current altitude if it is higher than the default one), then it will move toward the coordinates of the home location and hover over it. Home location is defined as the point where the vehicle was armed for the first time. This mode requires the barometer to determine altitude. On this mode the UAV will follow a set of predefined ordered coordinates, maintaining altitudes as in AltHold and Loiter. This mode requires GPS to confirm the position of the drone. This mode allows the UAV to approach to the ground at a designated speed. Above the default altitude threshold, the drone requires the barometer’s measure. Once below the threshold, it will use the ultrasonic sensor and descend at a default speed. Table 2.2: MAVLink flight modes..

(37) CHAPTER 2. UAV OPERATION AND NAVIGATION. 17. Using the previous flight mode, it is possible to develop a complete routine to operate the UAV during a structure’s inspection task.. 2.3.2. Autonomous takeoff. The first step on the flight routine consists of rising the UAV to a specified altitude where it can navigate over the structure. The altitude at which the drone will be moving depends on the height of the structure and the optimal distance between the section to be analyzed and the camera mounted on the vehicle. The exact value of this altitude will be explained in the following chapters, but it is important to consider that the drone must reach this value and keep it afterwards to evade errors during the vision process. Through MAVLink messages it is possible to instruct the UAV to take off autonomously using the onboard computer. The procedure to elevate the drone requires four steps: 1. Start the drone in Stabilize mode: This part can be done manually or by changing the mode through the onboard computer. It lets the UAV start at an easy-to-operate mode with fewer restrictions to start the operation. Other modes likeLoiter can be used, but are more restrictive. To do this, it is necessary to send a command long (ID 176) to change mode and set it to Stabilize. 2. Arm the UAV: This can be done manually by using the remote controller or autonomously by sending an armed message from the onboard computer. It is strongly suggested this is done manually to prevent the drone from working without the pilot’s consent. Once armed, the vehicle’s motors will react to the set points fixed by the pilot and the movement messages sent by the onboard computer. It is necessary to send a command long (ID 400) to toggle disarmed to armed. 3. Takeoff command: This will let the vehicle climb straight up from its current location to the specified altitude [Ardupilot, 2016b]. The command consists of seven parameters. However, only the last one (altitude) is required for it to work. It is necessary to be on Guided mode for the user to apply this command, so a command long (ID 176) must be sent first to set the flight mode to Guided, and then a second command long (ID 76) to execute nav takeoff (ID 22), assigning the desired value for the altitude to the seventh parameter on the message. 4. Override the RC channels: When the takeoff command is sent from the ground, it is necessary to increase the throttle a little for it to start. To make this procedure autonomous, it is essential to override the values from the radio controller. With a value of zero, the pilot will have priority over the UAV, otherwise the UAV will take the value established on the channel, which could be around 1000 and 1900, which are the valid values. This message will switch the current value to 1100 momentarily on the throttle channel to let the takeoff begin. After the drone starts working, the value is set to zero again and the pilot recovers the control. To change these values, it is necessary to send a RC channels override (ID 70) and assign a value of 1100 to the corresponding channel of the throttle, and repeat the process with a value of zero once the motors start moving..

(38) CHAPTER 2. UAV OPERATION AND NAVIGATION. 2.3.3. 18. Semiautonomous movement. To locate the UAV over the desired structure, both maintaining the adequate altitude for the inspection, and getting the proper image orientation for the vision process to work, are needed. In real life cases, it is difficult to manipulate a drone around buildings and structures with different dimensions. Moreover, to create a program that could adequately identify diverse kind on elements/surfaces on a structure. For this thesis work, a semiautonomous process for the movement toward the target was proposed as a way to guarantee both the altitude needed for the image capture process, and a precise approach to the structure as controlled by the user. The movement process consists of three steps: 1. Change flight mode to Loiter: To maintain the altitude defined for the inspection, the drone’s operation mode is adjusted, so the flight controller can compensate any disturbance. The Loiter mode is useful for this task because it allows the pilot to move the platform at a limited speed without worrying about altering the current height value. The Loiter mode is activated by sending a command long (ID 176) to change mode and set it to Loiter. 2. Move the drone toward the structure: For this step, the pilot will manually position the UAV over the target. The speed at which the vehicle moves will be restricted by the flight controller on Loiter mode, avoiding changes on altitude during the movement. A webcam can be used on the drone to receive a visual confirmation on the ground station of the current location of the vehicle over the structure until it reaches the desired position. 3. Orientation correction: Once the UAV is located over the target, it is important to orientate it properly. For this thesis work, the vision algorithm to detect the desired region of the structure requires an adequate orientation to delimit the area. Due to this, the pilot must use the images received from the ground station to ensure that the heading of the drone is correct. Figure 2.9 shows a comparison between a wrong position and a proper orientation of the drone over the structure.. Figure 2.9: Orientation comparison of the UAV over the structure. Once the UAV has been located over the structure, the vision process starts and the position remains without changes until the images have been gathered and the detection has finished..

(39) CHAPTER 2. UAV OPERATION AND NAVIGATION. 2.3.4. 19. Autonomous Landing. The final step for the movement operation consists of landing on the platform. Once the vision process finishes, the drone must move away from the structure and return to ground. A manual landing process at this point can be risky, especially while carrying sensitive equipment on the drone, so an autonomous process is suggested to let the drone approach to ground in a safe way. The MAVLink’s Land mode is used to let the UAV descend slowly, using data from the sensors to determine when it has reached the ground. The landing process has two simple steps to follow while descending depending on the altitude of the drone: 1. If the UAV is over 7 meters, the barometer’s value is used to determine the altitude and the drone will continue to descend at a constant speed. 2. If the UAV has descended below the 7 meters, or if it was below 7 meters when Land mode was activated, it will use the measure from the ultrasonic sensor to determine the altitude and reduce the rate of dissension. Once it reaches the ground and after a brief moment, the motors will stop. It is imperative to disarm the drone once the motors have stopped working. If the UAV is left armed, it is possible that the motors start moving again if the remote controller sends any value of throttle or is turned off. To disarm the drone, it is necessary to send a command long (ID 400) to toggle ”armed” to ”disarmed”. With the previous commands fulfilled, the UAV is able to finish the navigation routine, and ready for future tasks..

(40)

(41) Chapter 3 Image processing and multispectral analysis Structure inspection is a tedious process that can take a long time when performed by humans. The use of computer programs is crucial to speed up the inspection without compromising accuracy. There are examples on which vision-based models are used to inspect linear elements and structures like pipelines based on their features [Rathinam et al., 2008]. Image processing helps to identify characteristics of a structure. However, the visible spectrum is not enough to determine failures on the material. Although this kind of data cannot be acquired using common means, the analysis on a wider region of the spectrum helps to improve the failure detection. Multispectral sensors allow identifying flaws on structures through the collection of pictures corresponding to different wavelengths and the analysis of each image to detect changes on their response against its normal behavior. This process allows determining if a specific problem exists on the structure depending on the composition of the images collected [Henrickson et al., 2016]; for example, by using both infrared and RGB cameras to analyze and estimate deterioration on bridges to detect subsurface delamination [Khan et al., 2014]. The previous points are the basis of the vision process proposed to detect and classify the failures on the tubular structure through the equipment mounted on the UAV. This chapter begins with an explanation upon the pinhole camera model used to represent the image capture process and the image calibration and correction. Later, an affine transformation is proposed to align the images from each multispectral channel. Matrices interpolation and estimation are implemented to get the curves of displacement between channels as a way to reduce computing time. The structure’s detection process is fully explained, describing the steps to get an image of the interest region on the structure. The multispectral image analysis is fully explained, starting with the definition of spectral signature and explaining how it can be modified by different changes on the material, along with a description of the near infrared spectrum’s usage for the failure detection. Finally, the minimum distance classifier is described and proposed as a way to solve the failure detection problem using the multispectral data.. 21.

(42) CHAPTER 3. IMAGE PROCESSING AND MULTISPECTRAL ANALYSIS. 3.1. 22. Image capture and correction. The main purpose of the vision algorithms is to recognize and locate objects on the environment through image processing [Sucar and Gómez, 2011]. To accomplish this task, a sensor is required to capture the images from the surroundings to create data that can be managed on the computer as digital pictures. Figure 3.1 shows an example of how a camera actually captures a scene and the correlation between the real position and the pixel that represents it [Szeliski, 2010].. Figure 3.1: Lens optics on a camera. For the computer to understand and work with a picture, it is necessary to transform it into something than can be analyzed by a program. An image can be modelled as a matrix, where each element corresponds to a pixel [Nagabhushana, 2005], and each pixel has an intensity value usually represented by a byte’s value (0-255). Equation 3.1 shows an example of this, where M will represent the number of rows and N the number of columns that conform the image. For a color image, each pixel has a certain number of channels that represent the intensity of each color. For a BGR image, these channels will be blue, green and red. . a0,0 a1,0 . .. a0,1 a1,1 . ..   A=   aM −1,0 aM −1,1.  ... a0,N −1 ... a1,N −1    ... .   ... . ... aM −1,N −1. (3.1). In the following sections, the camera model will be explained to easily understand how a scene from the real world is converted into a 2D image for the computer to process. Furthermore, how changes on the lens and the position of the camera can affect the image and how to use these parameters and coefficients to correct the pictures taken, along with the description of the most common kind of distortions..

(43) CHAPTER 3. IMAGE PROCESSING AND MULTISPECTRAL ANALYSIS. 3.1.1. 23. Pinhole model. The modelling of the camera is usually approximated to a projection through a lens just as shown in the previous section. This projective model is known as the pinhole model and uses both extrinsic parameters, - that correlate the camera’s frame with the world’s frame -, and intrinsic parameters, - that represent the optic model - [Cyganek and Siebert, 2009]. Figure 3.2 shows the pinhole model and how a point on the world is translated to a pixel on the image taken [OpenMGV, 2013].. Figure 3.2: Pinhole model. Extrinsic parameters are associated with the rotation and translation between the camera’s frame and the world’s reference, while intrinsic ones consist of the position of the center of the lens on the camera and the focal length towards it. Equation 3.2 represents the projection matrix, which uses both intrinsic and extrinsic parameters to project a point from the world’s reference to the camera frame.    ui f /hx 0 ou f /hy ov   R3x3 Pc =  vi   0 wi 0 0 1.  tx ty  Pw tz. (3.2). In the previous equation, f represent the focal length of the camera, hx and hy represent scale factors that usually take the value of 1, Ox and Oy represent the center of the image, R is the rotation matrix from the world’s frame to the camera’s frame and tx , ty and tz are the translations from the world’s frame to the camera’s frame. Pc is the point in the camera’s frame, while Pw is the point in the world’s frame in homogeneous coordinates. Through this method, it is possible to represent the position and orientation of a place or object from the real world as an image arranged in a matrix that the computer can analyze..

(44) CHAPTER 3. IMAGE PROCESSING AND MULTISPECTRAL ANALYSIS. 3.1.2. 24. Image calibration. Even though the pinhole model accurately represents the way on which an image correlates with a real scenario, it does not consider the deformations provoked by the lens of the camera. The position and form of the lens can slightly modify the perception of the object or place in the world, and is because of these changes that distortions can appear on the images taken. There are two main sources of distortion for an image: Radial distortion and Tangential distortion. Radial distortion This kind of distortion is perceived as a slight curvature in the projection of straight lines. Radial distortion occurs because light blends more near the edges of the lens than at the optical center. Figure 3.3 shows the different kinds of curvatures that can be generated by this effect on the lens [MathWorks, 2017].. Figure 3.3: Types of radial distortion . The radial distortion can be easily compensated through the use of a simple low order polynomial. Equations 3.3, 3.4 and 3.5 show the polynomials used to represent the distortion produced on the image [Szeliski, 2010]. Xd = x ∗ (1 + k1 ∗ r2 + k2 ∗ r4 ). (3.3). Yd = y ∗ (1 + k1 ∗ r2 + k2 ∗ r4 ). (3.4). r 2 = x2 + y 2. (3.5). In the previous equations, Xd and Yd are the distorted coordinates on the image, while x and y are normalized pixel’s coordinates, which are calculated by translating the pixel coordinates to the optical center and dividing by the focal length. r, K1 and K2 are the radial distortion parameters, which are used to correct the image. Once radial distortion is fixed, the final coordinates can be obtained following the steps of Equations 3.6 and 3.7 [Szeliski, 2010]. X f = f ∗ x + cx. (3.6). Y f = f ∗ y + cy. (3.7).

(45) CHAPTER 3. IMAGE PROCESSING AND MULTISPECTRAL ANALYSIS. 25. Tangential distortion This distortion appears when the lens are not parallel to the to the camera’s sensor. Figure 3.4 exemplifies the situation in which this kind of error occurs [MathWorks, 2017].. Figure 3.4: Comparison between a correct alignment and tangential distortion. To compensate tangential distortion, a procedure similar to radial distortion compensation is followed. Equations 3.8 and 3.9 show the operations needed to calculate the distortion based on tangential distortion parameters P1 , P2 and r, where the value of r is the same shown in Equation 3.5 used for radial distortion compensation, x and y are normalized pixel’s coordinates, and the values of Xt and Yt are the distorted pixel’s coordinates [MathWorks, 2017]. Xt = x + [2 ∗ P1 ∗ x ∗ y + P2 ∗ (r2 + 2 ∗ x2 )]. (3.8). Yt = y + [2 ∗ P2 ∗ x ∗ y + P1 ∗ (r2 + 2 ∗ y 2 )]. (3.9). Once the parameters have been calculated, the final value for the undistorted coordinates can be obtained through Equations 3.6 and 3.7, - as used previously for radial distortion compensation -.. 3.2. Multiple image alignment. The usage of matrices to represent images is useful to correlate each pixel of a picture to a specific point. However, an image from a multispectral camera usually represents only one specific channel on a certain wavelength. To completely represent a larger portion of the spectrum, an image is required for each channel, and with this, the new issue to be solved is the alignment of all images from different channels to correlate one point to the same pixel position on each one. The mapping of the points from an image to a new position can be performed through various methods. Figure 3.4 shows some of the most common transformations for images [Szeliski, 2010]..

(46) CHAPTER 3. IMAGE PROCESSING AND MULTISPECTRAL ANALYSIS. 26. Figure 3.5: Table of common image transformations and their degrees of freedom.. 3.2.1. Affine transformation. From the transformations shown previously, the affine matrix allows modifying an image while keeping parallelism. This matrix can be used to warp pictures from different points of view to the same position, as long as perspective does not differ too much between them. The affine transformation uses a 2 x 3 matrix that applies both translation and linear mapping. Equation 3.10 shows the representation of an affine transformation from point Pa to Pb . Equation 3.11 and 3.12 show an algebraic representation of the same transformation. Figure 3.6 exemplifies different kinds of affine transformations of an image, where parallelism is kept on every kind of deformation.     Pax Pbx Pby  = a00 a01 tx Pay  (3.10) a10 a11 ty 1 1 Pbx = Pax ∗ a00 + Pay ∗ a01 + tx. (3.11). Pby = Pax ∗ a10 + Pay ∗ a11 + ty. (3.12). Figure 3.6: Affine transformation examples..

(47) CHAPTER 3. IMAGE PROCESSING AND MULTISPECTRAL ANALYSIS. 27. To properly align a group of images taken from different positions, both a reference point and an affine transformation for every channel to be moved toward this position are needed. It is useful to have one image set as the reference point, so the others must be transformed to be aligned over it. Figure 3.7 shows the comparison between an image with all their channels unaligned and a fully aligned picture. Notice how the aligned image has unaligned sections at the edges, which occurs because of the affine transformation that deforms each channel differently.. Figure 3.7: Comparison between unaligned image (left) and an aligned one (right).. 3.2.2. Matrices calculation, interpolation and estimation. For an appropriate alignment to be performed, it is necessary to obtain each affine matrix for every transformation from the cameras to the reference point. The parameters required for each matrix can be estimated by using the Enhanced Correlation Coefficient, or ECC, maximization. This procedure consist of an iterative method to reach the best parameters to transform a warped image, which corresponds to any of the cameras misaligned, to a target area, which consist on the image selected as fixed point. The algorithm uses the zero mean normalized cross-correlation as a base to the parametric alignment [Evangelidis and Psarakis, 2008]. The first step is to define a reference vector ir from the target area and a warped vector iw (P ), where P are the parameters used to map the warped image into the target area. These vectors consist on the intensities at k coordinates of the corresponding images as expressed on the Equations 3.13 and 3.14. ir = [Ir (x1 ), Ir (x2 ), ..., Ir (xk )]t. (3.13). iw (P ) = [Ir (y1 (P )), Ir (y2 (P )), ..., Ir (yk (P ))]t. (3.14). The previous vectors must be turned into zero mean vectors, which can be made by subtracting the mean value of the vector to each point as shown in Equations 3.15 and 3.16..

(48) CHAPTER 3. IMAGE PROCESSING AND MULTISPECTRAL ANALYSIS. zr = [(Ir (x1 ) − īr ), (Ir (x2 ) − īr ), ..., (Ir (xk ) − īr )]t. zw (P ) = [(Ir (y1 (P )) − īw (P )), (Ir (y2 (P )) − īw (P )), ..., (Ir (yk (P )) − īw (P ))]t. 28. (3.15). (3.16). For the normalization of these values, it is necessary to calculate the standard deviation of defined vectors. Equations 3.17 and 3.18 show the equivalences of these values based on the Euclidean norm. r 1 σir = ||zr || (3.17) k r 1 ||zw (P )|| (3.18) σiw (P ) = k The zero mean vectors and standard deviations are used to form the zero mean normalized cross-correlation, or ZNCC, as shown in Equation 19.. ECC(P ) =. 1 t z z (P ) k r w. σir ∗ σir. 1 t z z (P ) k r w. zr t zw (P ) q = = q ||zr || ∗ ||zw (P )|| ( k1 ||zr ||) ∗ ( k1 ||zw (P )||). (3.19). The Enhanced Correlation Coefficient allows concentrating on the geometric transformation because it is invariant to photometric distortions [Evangelidis and Psarakis, 2008]. The bigger the value of the coefficient, the more correlated both images are. However, the maximization requires nonlinear optimization through an iterative process based on gradient, which can be time consuming. Due to the time needed to calculate each matrix, this operation is not adequate for a flight inspection, where the vehicle may need to check other regions in the least time possible. A way to solve this problem is to have a set of matrices saved previously, so the vision program can access to them whenever it is necessary to do an alignment, but the amount of matrices vary depending on the position of the drone over the structure. For each new altitude, another set of matrices is required. A solution to this problem is to use matrix interpolation and the estimation of a function which models the behavior. Through the use of various matrices calculated previously at a defined set of distances, one is able to interpolate their values to obtain the intermediate approximated points. Once the affine matrices are calculated through interpolation, trend lines can be estimated based on the points calculated. The process of alignment can be done quite easily and much faster using functions that depend of the distance toward the target by only updating this value. Figure 3.8 shows an example of how a set of points can be used to approximate intermediate values by interpolation and follow with the estimation of the corresponding function. An estimated model for each channel that will be mapped during the alignment is necessary, and to have at least one channel designated as a reference point for the others..