This work is licensed under a Creative Commons Attribution-NonCommercial- NoDerivatives 4.0 International License.
This is a postprint version of the following published document:
Al-Kaff, A., Martín, D., García, F., de la Escalera, A. &
María Armingol, J. (2018). Survey of computer vision algorithms and applications for unmanned aerial vehicles.
Expert Systems with Applications, 92, 447–463.
DOI: 10.1016/j.eswa.2017.09.033
© 2017 Elsevier Ltd. All rights reserved.
Review
Survey of computer vision algorithms and applications for unmanned aerial vehicles
Abdulla Al-Kaff
∗, David Martín , Fernando García , Arturo de la Escalera , José María Armingol
Intelligent Systems Lab, Universidad Carlos III de Madrid Calle Butarque 15, Leganés 28911, Madrid, Spain
a rt i c l e i n f o
MSC:
00-01 99-00 Keywords:
UAV
Computer vision Navigation system Pose estimation Obstacle avoidance Visual servoing Vision-Based applications
a b s t r a c t
Thispaperpresents acompletereviewofcomputervisionalgorithmsandvision-basedintelligent ap- plications,thataredevelopedinthefieldoftheUnmannedAerialVehicles(UAVs)inthelatestdecade.
Duringthistime,the evolutionofrelevanttechnologiesforUAVs; suchascomponent miniaturization, theincreaseofcomputationalcapabilities,andtheevolutionofcomputervisiontechniqueshaveallowed animportantadvanceinthedevelopmentofUAVstechnologiesandapplications.Particularly,computer visiontechnologiesintegratedinUAVsallow todevelop cutting-edgetechnologiestocope withaerial perceptiondifficulties;suchasvisualnavigationalgorithms,obstacledetectionandavoidanceandaerial decision-making.AlltheseexperttechnologieshavedevelopedawidespectrumofapplicationforUAVs, beyondthe classicmilitaryand defensepurposes.UnmannedAerialVehiclesand ComputerVisionare commontopicsinexpertsystems,sothankstotherecentadvancesinperceptiontechnologies,modern intelligentapplicationsaredevelopedtoenhanceautonomousUAVpositioning,orautomaticalgorithms toavoidaerialcollisions,amongothers.Then,thepresentedsurveyisbasedonartificialperceptionap- plicationsthatrepresentimportantadvancesinthelatestyearsintheexpertsystemfieldrelatedtothe UnmannedAerialVehicles.Inthispaper,themostsignificantadvancesinthisfieldare presented,able tosolvefundamentaltechnicallimitations;suchasvisualodometry,obstacledetection,mappingandlo- calization,etcetera.Besides,theyhavebeenanalyzed basedontheircapabilities andpotential utility.
Moreover,theapplicationsandUAVsaredividedandcategorizedaccordingtodifferentcriteria.
1. Introduction
UnmannedAerialVehicle(UAV),RemotelyPilotedAerialSystem (RPAS)orwhatiscommonlyknownasadroneisthetermthatde- scribes theaircraftplatformwithouta humanpilotonboard.UAV can be either teleoperated remotely by the pilot in the Ground ControlStation(GCS)orautonomouslyusingtheonboardsensors mountedonit,followingpreprogrammedoperations.However,this terminologynotonly referstothe vehicleitself,butalsotoall of the supporting hardware and software including sensors, micro- controllers,groundstations,communicationprotocolsanduserin- terfaces(Beard&McLain,2012).
Therearemanyclassificationschemesthathavebeenpresented tocategorize theUAVs. Theseschemes arebasedonalargenum- ber of different characteristics; such as the mass, size, mission
∗ Corresponding author.
E-mail addresses: [email protected] (A. Al-Kaff), [email protected] (D.
Martín), [email protected] (F. García), [email protected] (A.d.l. Escalera), [email protected] (J. María Armingol).
range,operationaltitude,operationduration,MeanTakeoff Weight (MTOW), flying principle, propulsion mode, operation condition, capabilitiesorthecombinationofthesecharacteristics.
Fig.1showsthethreemainclassificationsandmodelsofUAVs basedonitsbodyshapeandflyingprinciples.
One of the detailedand widely used schemes has been pro- posed by van Blyenburgh (2006), as it is shown in Table 1. In which,theUAVsare classifiedbasedonthemass,range,altitude, and endurance. Moreover, another scheme based on MTOW and thegroundimpactrisk hasbeenproposedby Dalamagkidis,Vala- vanis,andPiegl(2012),asitisshowninTable2.
Although UAVs were designed and supported originally for defense and military purposes; such as aerial attacks or mili- tary air cover; to avoid the risk of human lives. Recently, with the developments in microelectronics and the increase of the computing efficiency, small and micro unmanned aerial vehicles (SUAVsandMAVs)haveencounteredasignificantfocusamongthe robotics research community. Furthermore,because of their abil- ityto operateinremote,dangerousanddullsituations, especially helicopters and Vertical Take-Off and Landing (VTOL) rotor-craft
Fig. 1. UAV Models.
Table 1
Classification of UAVs based on mass, range, altitude and endurance (SOURCE van Blyenburgh (2006) ).
Category Mass (kg) Range (km) Flight altitude (m) Endurance (h)
Micro < 5 < 10 250 1
Mini < 20/25/30/150 a < 10 150/250/300
Tactical
Close range (CR) 25–150 10–30 30 0 0 2–4
Short Range (SR) 50–250 30–70 30 0 0 3–6
Medium Range (MR) 150–500 70–200 50 0 0 6–10
MR endurance (MRE) 50 0–150 0 > 500 80 0 0 10–18
Low altitude deep penetration (LADP) 250–2500 > 250 50–90 0 0 0.5-1 Low altitude long endurance (LALE) 15–25 > 500 30 0 0 > 24 Medium altitude long endurance (MALE) 10 0 0–150 0 > 500 30 0 0 24–48 Strategic
High altitude long endurance (HALE) 250 0–50 0 0 > 20 0 0 20,0 0 0 24–48 Stratospheric (Strato) >2500 >20 0 0 > 20, 0 0 0 > 48
Exo-stratospheric (EXO) TBD TBD > 30, 500 TBD
Special task
Unmanned combat AV (UCAV) > 10 0 0 1500 12,0 0 0 2
Lethal (LET) TBD 300 40 0 0 3–4
Decoys (DEC) 150–250 0–500 50–50 0 0 < 4
a Varies with national legal restrictions.
Category
Number T GIa MTOW Name Note
0 10 2 Less than 1 kg Micro Most countries do not regulate this category since these vehicles pose minimal threat to human life or property
1 10 3 Up to 1 kg Mini These two categories roughly correspond to R/C model aircraft
2 10 4 Up to 13.5 kg Small
3 10 5 Up to 242 kg Light/ultralight Airworthiness certification for this category may be based either on ultralights (FAR bpart 103), LSA c(Order 8130), or even normal aircraft (FAR Part 23)
4 10 6 Up to 4332 kg Normal Based on MTOW these vehicles correspond to normal aircraft (FAR Part 23) 5 10 7 Over 4332 kg Large These vehicles best correspond to the transport category (FAR Part 25)
aT GIis the minimum time between ground impact accidents.
bFederal Aviation Regulations.
cLight Sport Aircraft.
systems; forexample quad/hexa/octo-rotorsare increasingly used in several civilian and scientific applications; such as surveying and mapping, rescue operation in disasters (Adams & Friedland, 2011; Erdos, Erdos, & Watkins, 2013), spatial information acqui- sition, buildings inspection (Choi & Kim, 2015a; Eschmann, Kuo, Kuo, & Boller, 2012),data collectionfrominaccessible areas,geo- physics exploration (Fraundorfer et al., 2012; Kamel, Santana, &
De Almeida,2010), traffic monitoring(Kanistras,Martins, Ruther- ford, & Valavanis, 2013), animalprotection1 (Xu, Solmaz,Rahma- tizadeh, Turgut, & Boloni, 2015), agricultural crops monitoring (Anthony, Elbaum, Lorenz, & Detweiler, 2014), manipulation and transportation2 (Michael,Fink, & Kumar,2011) ornavigation pur- poses (Bills, Chen, & Saxena, 2011; Blosch,Weiss, Scaramuzza, &
Siegwart,2010).
Withthecurrenttechnologyandthevarietyandthecomplexity ofthe tasks,modern UAVs aimathigher levelsof autonomyand performing flight stabilization. The main part of an autonomous UAV isthe navigationsystemand itssupporting subsystems. The autonomous navigation system utilizes information from various subsystems in order to achieve three essential tasks: to estimate the pose ofthe UAV (position and orientation)(Localization), to identify obstacles in the surrounding andact inconsequence; in ordertoavoidthem(Obstacledetectionandavoidance)andsend commandstostabilizetheattitudeandfollowguidanceobjectives (Controlloop).
The difficulty appears due to working with SUAVs or MAVs;
such as Ar.Drone Parrot3, DJI Pantom series4, AscTec Humming- bird5,Voyager36,3DRSOLO7orTALIH5008.Thatisbecauseofthe sizeofthesevehiclesisgettingsmaller(fewcentimeters)andthe weight isgettinglighter(fewgrams),which leadstoa significant limitationinthepayloadcapabilitiesandthepowerconsumption.
Therefore, with these properties,mounting onboard sensors that arehelpfulforthenavigationpurposesisconsideredachallenging problem.
In outdoor operations, most of the navigation systems are basedonGlobalPositioningSystem(GPS)(Hui,Xhiping,Shanjia,&
Shisong,1998;Kim,2004;Tao&Lei,2008)tolocatetheirposition.
In thesesystems, the precision depends directly on the number of satellites connected. However, GPS-based systems do not pro- videreliablesolutionsinGPS-deniedenvironments;suchasurban areas,forests,canyons orlow altitude flights that canreduce the
1http://www.bbc.com/news/business-28132521 .
2http://www.service-drone.com/en/production/logistics-and-transport .
3http://ardrone2.parrot.com/ .
4http://www.dji.com/products/phantom .
5http://www.asctec.de/en/uav- uas- drones- rpas- roav/asctec- hummingbird/ .
6http://www.walkera.com/en/products/aerialdrones/voyager3/ .
7https://3dr.com/solo-drone/ .
8http://www.walkera.com/en/products/aerialdrones/talih500/ .
satellite visibility.Furthermore,inother scenarios like indoor op- erations,GPS loses totallyitsefficiencybecauseofthe absenceof information.
UAVs need a robust positioning system to avoid catastrophic control actions, which can be caused by the errors in the posi- tionestimation,sothatdifferentapproachesareproposedtosolve thisproblem.UsingtheinformationprovidedbytheGPScombined with the data obtained by the Inertial Navigation System (INS) isone of themost popularapproaches, at whichthe dataof the INSandtheGPSarefusedtogethertominimizethepositionerror (Beardetal., 2005;Nakanishi, Kanata,& Sawaragi,2012;Soloviev, 2010;Yoo &Ahn, 2003). Twomain drawbacksappeared ofthese approacheswhich affect the localization process. First, the infor- mationarestilldependentontheexternalsatellitesignalsandthe numberofsatellitesdetected. Second,thelackofprecision ofthe IMUmeasurementsbecauseofseveralgeneratederrorsovertime.
Therefore,some specificsolutionshavebeenprovided;suchas using radar (Quist, 2015) or laser sensor (Bry, Bachrach, & Roy, 2012;Grzonka,Grisetti,& Burgard, 2012). However, thesesensors requiremorepayloadcapabilitiesandhigherpowerconsumption.
Owing to its capability to provide detailed information about thesurroundingenvironments,visualsensorsandcomputervision algorithms play a vital role as the main solution in indoor and outdoorscenarios(Bloschetal., 2010;Kameletal., 2010;Krajnik, Nitsche,Pedre,Preucil,&Mejail,2012;Mourikisetal.,2009).Inad- dition,visual sensorscanbe usedasstand-alonesensorsorcom- binedwithothersensors;toimprovetheaccuracyandrobustness ofthenavigationsystem.
Visual sensors, such as cameras, have the advantage of lightweight,lowpowerconsumptionandrelativelylow-cost.Inad- dition, they provide rich information of the environment, which canbe processed andapplied toreal-time applications.However, the accuracy of these approaches depends on different factors;
such as imagesresolution, capturing time, viewing angle,illumi- nation,differentstructuresofaerialimagesandreferencedata.
UAV vision-based systems have intelligent capabilities, which areanimportantbrandinExpertSystems,andtheapplicationsde- rivedfromthemhaveagreat potentialasaresearch andinnova- tionfield
ThisSurveypresentsaliteraturereviewoftheUAVapplications, the algorithms and the techniquesthat are mainly based on the computer vision applied on the UAVs. In addition, demonstrates the efficiencyof the visual devices as a main orcomplementary sensor that provides information about the environment for the purposesoftheUAVsnavigationsystems.
Moreover, fromthis literature andby studying several vision- based algorithms, the obtained information and data provided a solid backgroundtoproposed differentapproachesin thefield of autonomousUAVs andcomputer vision. Oneof theseapproaches is to present a vision-based system for infrastructure inspection
usinga UAV(Al-Kaff et al.,2017b).In addition,anotherapproach was presented to mimic the human behavior in detecting and avoidingfrontalobstaclesusingmonocularcamera(Al-Kaff,García, Martín,DeLaEscalera,&Armingol,2017a).
The remainder of this paper is organized as follows;
Section 2 introduces a review of the computer vision algorithms andtechniques that are used withthe UAVs. Section 3 presents the navigation systems and its subsystems (Pose estimation, Obstacle detectionand avoidance, and Visual servoing), followed byshowing differentautonomousapplication basedon computer visioninSection4.Finally,inSection5conclusionissummarized.
2. ComputervisionforUAVs
ComputervisionplaysavitalroleinthemostoftheUnmanned AerialVehicles(UAVs)applications.Theseapplicationsvaryfroma simpleaerial photography, to very complex tasks such as rescue operationsoraerialrefueling.Allofthemrequirehighlevelofac- curacyinordertoprovidereliabledecisionandmaneuvertasks.
Aerial imagery or aerial filming is considered one of the ba- sic and demanding application; such as filming sports games9, events10orevenweddings11.
Withthe advancesincomputer visionalgorithms andsensors, theconceptofusingaerialimagesjustforphotographyandfilming waschangedtobeusedwidelyinmorecomplexapplications;such asthematicandtopographicmappingoftheterrain(Ahmadetal., 2013;Cui,Lin,&Zhang,2007;Li&Yang,2012;Ma,Li,Tong,Wang,
&Cheng,2013;Tampubolon&Reinhardt,2014);explorationofun reachable areas such as islands (Ying-cheng et al., 2011), rivers (Rathinametal.,2007),forests(Cuietal.,2014;Yuan,Liu,&Zhang, 2015a) oroceans (Sujit, Sousa,&Pereira, 2009a;2009b); surveil- lancepurposes(Gengetal.,2014;Govindaraju,Leng,&Qian,2014;
Lilien et al., 2015; Semsch, Jakob, Pavlicek, & Pechoucek, 2009);
andsearchand rescueoperationsaftercatastrophes(Erdosetal., 2013;Kruijff etal.,2012;Waharte&Trigoni,2010).
Another widely demanded application that takes the advan- tages of the aerial imagery over the traditional sensing, is the trafficmonitoring(Kanistrasetal., 2013).Trafficmonitoring using UAVsincludes theestimationof thetraffic flowbehavior (Heintz, Rudol,&Doherty,2007;Kimetal.,2012),trafficspeed(Ke,Kim,Li,
&Wang,2015),roadsstate(Lin&Saripalli,2012;Zhou,Kong,Wei, Creighton,&Nahavandi,2015),inadditiontotheemergenciesand caraccidents(Puri,2005).
Furthermore, wide variety of different autonomous applica- tionshavebeenpresented;suchasautonomoustake-off andland- ing (Cabecinhas, Naldi, Marconi, Silvestre, & Cunha, 2010; Jung, Bang,& Lee,2015;Lee,Su,Yeah,Huang,&Chen,2014b;Sanchez- Lopez,Saripalli, Campoy,Pestana, & Fu,2013),autonomous aerial refueling (Aarti & Jimoh O, 2013; Campa, Napolitano, & Far- volini,2009; Xufeng,Xinmin, & Xingwei, 2013;Yin etal., 2016), autonomous tracking (Achtelik, Zhang, Kuhnlenz, & Buss, 2009;
Lin, Lum, Chen, & Lee, 2009; Martínez Luna, 2013; Zhao, Fei, &
Geng, 2013) or autonomous route planning (Govindaraju et al., 2014;Kothari,Postlethwaite,&Gu,2009;Sangyam,Laohapiengsak, Chongcharoen,&Nilkhamhang,2010;Yamasaki,Sakaida,Enomoto, Takano,& Baba, 2007; Yang, Qi,Xiao, & Yong,2014),where high levels of accuracy of localization, detection and tracking are re- quired.
Differentsurveysthatcoverdifferentcomputervisionconcepts, techniquesandapplicationsthatarerelatedtoUAVsarepresented inCampoyetal.(2009)(visualservoing),LiuandDai(2010)(aerial surveillance and multi-UAV cooperation), Adams and Friedland
9The Future of Sports Photography: Drones .
10Airborne camera makes concert scene .
11Camera in the Sky: Using Drones in Wedding Photography and Videos .
(2011) (disaster research),Kanistras etal.(2013) (trafficmonitor- ing),andYuan,Zhang,andLiu(2015b)(forestfiremonitoring).
Thissurveydiscussestheevaluationofvision-basedalgorithms, methods and techniques that are relatedto the UAVs navigation systemsinthelast decade.Inaddition,itpresentsthemostmod- ernanddemandedapplicationsthatarebasedoncomputervision.
3. UAVs’navigationsystems
Modern UAVs aimat higherlevels ofautonomywith accurate flight stabilization. The main part of an autonomous UAV is the navigation systemand its supporting subsystems. The navigation systemutilizesinformationfromvarioussensors, inordertoesti- matetheposeoftheUAVintermsofpositions(x,y,z)andorienta- tions(
φ
,θ
,ψ
).Othersupportingsystemssolverelevanttaskssuchasobstaclesdetectionandtracking(staticordynamic),orobstacle avoidance.
Withthis increase in thelevels of autonomyand flight stabi- lization,robustandefficientnavigationsystemsarerequired.Com- puter vision algorithms by means of monocular cameras can be helpful to enhance the navigation activities. As it is shown in Table 3,the navigationsystems are divided intothree main sub- systems:Poseestimationwhichaimstoestimatethepositionand the attitude of the UAV in two and three dimensional represen- tations,Obstacledetectionandavoidancethat detectsandfeeds backthepositionoftheobstacles thatare situatedinthepathof theUAV, andfinally theVisualservoing subsystemat whichthe maneuvercommandsaremanagedandsent inordertomaintain theflight stabilityandfollowingthe path.The followingSections (3.1,3.2and3.3)addressthesethreenavigationsubsystems.
3.1. Poseestimation
Pose estimation is the process of estimatingthe position and theorientationofthevehicleduringthemotion;basedonthein- formation generated by one ormore sensors; such as IMU, GPS, vision,laser, ultrasonic,etc. The informationcanbe generated by each sensor separately or by fusing the data fromdifferent sen- sors.Poseestimationisconsideredasafundamentalphaseforany navigationormappingprocesses.
3.1.1. Globalpositioningsystem(GPS)
GlobalPositioningSystem(GPS)(Kaplan&Hegarty,2006;Zogg, 2009)ortheSatellite-basedNavigationSystem(SNS)isconsidered as one of the most known approaches that is used with UGVs (Abbott & Powell, 1999; Amini, Vaghefi, de la Garza, & Buehrer, 2014; Wei, Cappelle, & Ruichek, 2011; YAGIMLI˘ & Varol, 2009;
Yoon,Park,&Kim,2006),UAVs(Choetal.,2007;Huietal.,1998;
Isaacs etal., 2014; Kim, 2004; Yun,Peng, & Chen, 2007) oreven Autonomous Underwater Vehicle (AUV) (Lee, Li, Hoang, & Lee, 2014a;Meldrum&Haddrell,1994;Taraldsen,Reinen,&Berg,2011) toprovidethe3Dpositionfornavigationpurposes.
In most cases, the GPS is used as the main sensor for lo- calization process to obtain the position of the vehicles. One of the earlier works that is based on the GPS for localizationwith UAVswaspresentedbyHuietal.(1998).Inthiswork,theauthors showedtheeffectofusingtheDifferentialGlobalPositioningSys- tem(DGPS);toreducetheerrors(satelliteclockerror,satellitepo- sitionerroranddelayerror)comparingto theuseoftheGPS re- ceiveralone.SimilarlyinChoetal.(2007),aDGPSisimplemented to asingleantenna receiver;inorder toincrease the accuracyof thepositioninginformation.
Inthesesystems,theprecisiondependsdirectlyonthenumber ofsatellites connected.Thisnumber canbe insufficient onurban environmentsdueto buildings,forestsor mountainsthat can re- duce the satellite visibility.Furthermore, inother scenarios; such
System Description Method Related work Localization Estimate the UAV 2D/3D
Position and Orientation
VO ( Fu et al., 2015; Nikolic et al., 2014; Omari et al., 2015; Warren & Upcroft, 2013 ), ( Grabe et al., 2012; Krajnik et al., 2012; Lim et al., 2012; Mouats et al., 2015 ),( Domínguez, Zalama, García-Bermejo, Worst, & Behnke, 2013; Romero et al., 2013; Zhang, Stahle, Gaschler, Buckl, & Knoll, 2012; Zhang et al., 2014 ),( Bloesch, Omari, Hutter, &
Siegwart, 2015; Chunhui, Rongzhi, Tianwu, & Quan, 2014; Willis & Brink, 2016 ) SLAM ( Ahrens et al., 2009; Bonin-Font et al., 2015; Mart et al., 2015 ), ( Blosch et al., 2010;
Kerl et al., 2013; Zeng et al., 2014; Zhang et al., 2015 ), ( Engel et al., 2014; Fu, Olivares-Mendez, Suarez-Fernandez, & Campoy, 2014; Meng, de Silva, & Zhang, 2014;
Weiss, Scaramuzza, & Siegwart, 2011 ),( Davison et al., 2007; Li, Aouf, & Nemra, 2012b; Milford et al., 2011 )
Obstacle Detection and Avoidance
Detecting the possible obstacles and collision zones and making the suitable avoidance decisions
Stereo ( Gao et al., 2011; Hrabar, 2008; Jian & Xiao-min, 2011; Na et al., 2011 ) ( Broggi et al., 2013; Gageik, Benz, & Montenegro, 2015; Hou et al., 2016; Odelga, Stegagno, &
Bülthoff, 2016 ) ( Majumder et al., 2015 )
Monocular ( Al-Kaff, Meng, Martín, de la Escalera, & Armingol, 2016; Bills et al., 2011; Ma et al., 2015; Saha et al., 2014 ) ( Lenz, Gemici, & Saxena, 2012; Mori & Scherer, 2013 ) Visual Servoing Maintain UAV stability and
flying maneuvers based on visual data
( Lee et al., 2012; Lyu et al., 2015; Neff, Lee, Chitrakaran, Dawson, & Burg, 2007;
Olivares-Mendez et al., 2015 ) ( Bošnak, Matko, & Blaži ˇc, 2012; Kurnaz, Cetin, &
Kaynak, 2010; Shang, Liu, Zhao, & Chen, 2016; Zhang, Fang, Liang, & Zhang, 2016 )
asindoorflying,GPSlosesitsefficiencybecauseoftheabsenceof the satellite signals. Therefore, some expensive external localiza- tionsystemsareused;such astheVICONsystems(Al Habsi,She- hada,Abdoon,Mashood,&Noura,2015;Bryetal.,2012;Mellinger, Michael, & Kumar,2012;Michaeletal., 2011) tocapturethe mo- tionoftheUAVinindoorenvironments.
3.1.2. GPS-aidedsystems
Althoughstand-alone GPSisusefultoestimate thepositionof thevehicles,italsogenerateserrorsbecauseofthedisabilitytore- ceivesatellitessignals,orbythejammingofthesignalsthat con- sequentlymayleadtolosenavigationinformation.
UAVs need a robust positioning system to avoid catastrophic control actions that can be caused by errors in position estima- tion, sothat differentapproachesare used tosolve thisproblem.
One example ofthese approachesis GPS-aidedsystems. In these approachesthegathereddatafromtheGPSarefusedwiththein- formation obtained fromother sensors, this multi-sensory fusion canbe oftwosensors(Beardetal., 2005;Qingbo, Nan,& Baokui, 2012;Tao&Lei,2008)ormorethantwosensors(Jiong,Lei,Jiang- ping, Rong, & Jianyu, 2010;Nakanishi et al., 2012; Vincenzo An- gelino,Baraniello,&Cicala,2013;Ziyang,Qiushi,Chen,&Ju,2014).
GPS/INSisoneofthemostpopularconfiguration,atwhichthe data from the INS and the GPS are fused together; to compen- satethegeneratederrorsfrombothsensorsandincreasethepre- cisionofthelocalizationprocess.InYooandAhn(2003),thedata frommultiple antennasGPS are fusedwiththeinformation from theonboardINSusinglinearKalmanfilter.However,thisalgorithm hasbeenimplementedtobeusedwithUAVs,althoughtheexper- imentshavebeenperformedonagroundvehicle.
Similarworkswerepresentedtoreducethepositionerrorusing ExtendedKalman Filter(EKF)(Barczyk& Lynch,2012),orby em- ployingtheKalman-Complementaryfiltering(Yun etal.,2007), or by fusionStrap-downInertialNavigation System(SINS) datawish theGPS(Qingboetal.,2012;Tao&Lei,2008).
In Vincenzo Angelino et al. (2013), an Unscented Kalman Fil- ter (UKF)wasimplementedtofusetheGPSdatawiththecamera information andthedata obtainedfromtheIMU in orderto im- provethelocalizationprocess.Thisfusionshowedimprovementin the results comparing to the resultof each sensor, However, the experimentswerelimitedtosimulations.
Moreover, in other works(Jiong etal., 2010; Nakanishiet al., 2012), an altitude sensor was added to the GPS/INS system; in order to improvethe reliability and increase the accuracy ofthe navigationbyenhancingtheaccuracyoftheGPSverticalmeasure-
ments.Butthesesystemsstillhaveinaccurateresults,especiallyif theUAV flies inlow altitudes; because the barometer is affected bythegroundeffectandestimatedaltitudeslowerthantheactual ones(Nakanishi,Kanata,&Sawaragi,2011).
Another multi-sensor fusion based system for multiple MAVs wasintroduced inWangandGe(2011).Atwhich,thedataofthe GPSarefusedwiththeinformationfromtheIdentificationFriend- or-Foe(IFF)radar systemforlocalization enchantment usingEKF.
Inthesimulations, ithas beenproved thatby usingtwo GPS re- ceiversbetterinformationisobtainedratherthanasingleGPSre- ceiver.
InIsaacsetal.(2014),aGPSlocalizationsystemisusedonLock- heedMartin’sSamariMAV.Atwhich,agreedysourceseekingalgo- rithmwasusedtotracktheradiofrequencysourcesbyestimating theangleofarrivaltothesourcewhileobservingtheGPSsignalto noiseratio;inordertokeepthequalityoftheGPSsignal.
Two main drawbacks appeared on these approaches, affect- ing the localization process. First, the information are still de- pendent on the external satellite signals. Second, the lack of precision of the IMU measurements. These difficulties favored the apparition of vision-based systems. These novel approaches enhance the localization by means of computer vision-based algorithms.
3.1.3. Vision-basedsystems
Owing to the limitations and drawbacks of the previous sys- tems,vision-based poseestimation approaches havebecome one ofthe maintopics inthe field ofintelligentvehicles applications andgainmorepopularitytobe developedforUGVs(Scaramuzza, Fraundorfer,& Siegwart,2009; Tardif, Pavlidis,& Daniilidis,2008;
Zhang,Singh,&Kantor,2014),AUV(Bonin-Font, Cosic,Negre,Sol- bach, & Oliver, 2015; Dunbabin, Corke, & Buskey, 2004; Kunz
& Singh, 2010; Mehmood, Choudhry, Anwar, Mahmood, & Khan, 2016),andUAVs(Caballero,Merino,Ferruz,&Ollero,2009;Fraun- dorferet al., 2012; Kneip, Chli, & Siegwart, 2011; Lindsten etal., 2010;Yoletal.,2014).
Regardless to the type of the vehicle and the purpose of the task,differentapproachesandmethodshavebeenproposed.These methods differ on the type of the visual information used; such as horizons detection (Dusha, Boles, & Walker, 2007; Grelsson, Linköpings universitet, & Institutionen för systemteknik, 2014), landmarkstracking(Amor-Martinez, Ruiz,Moreno-Noguer,& San- feliu,2014;Eberli,Scaramuzza, Weiss,&Siegwart,2011), oredges detection(Kim,2006; Wang,2011).Furthermore,theycan bedif- ferentiated based on the structure of the vision system: it can
be monocular (Milford, Schill, Corke, Mahony, & Wyeth, 2011;
Yang,Scherer,&Zell,2013;Zeng,Wang,Liu,Chen,& Deng,2014), binocular (Vetrella, Savvaris, Fasano, & Accardo, 2015; Warren, 2015),trinocular(Jeong,Mulligan,&Correll,2013;Martínez,Cam- poy, Mondragón, & Olivares Mendez, 2009), or omnidirectional (Amorós,Paya,Valiente,Gil, &Reinoso,2014;Killpack,Deyle,An- derson,&Kemp,2010;Scaramuzza&Siegwart,2008;Wang,Zhao, Davoine,&Zha,2012b)camerasystem.
Some of the early experimental works that use visual infor- mation;inorder toestimate the aircraftattitude were presented inTodorovic, Nechyba, and Ifju(2003), Cornall and Egan (2004), Cornall, Egan, and Price (2006), Thurrowgood, Soccol, Moore, Bland,andSrinivasan (2009). Theseapproaches arebased onthe skylinesegmentation usingforward-looking camera. In theseap- proaches, Bayesian segmentation model with a Hidden Markov Trees (HMT) model were used to identify the horizon based on the color intensities, andtexture clues; in order to estimate the rollangleorboth rollandpitchangles,asthework presentedin Dushaetal.(2007).Theseapproachesprovidesuccessfulresultsin highaltitudeswhere theprocess of skylinesegmentation is rela- tivelyeasy.Ontheotherhand,inlowaltitudesorindoorenviron- ments,thepossibilitytodetectthehorizonisverylow duetothe complexityofthiskindofenvironments.
Two famous philosophies have appeared to deal with the vision-basedposeestimationproblem;VisualSimultaneousLocal- izationAndMapping(VSLAM)andVisualOdometry(VO).
SLAM. SimultaneousLocalizationAndMapping(SLAM)algorithms (Bailey & Durrant-Whyte, 2006; Csorba, 1997; Durrant-Whyte &
Bailey,2006), ingeneralaimtoconstructaconsistentmap ofthe environment and simultaneously estimate the global position of therobotwithinthismap.
Approaches such as those that have been presented in Angeli, Filliat, Doncieux, and Meyer (2006), Ahrens, Levine, An- drews,andHow (2009), Bloschetal.(2010),introduced different camera basedalgorithms; such as ParallelTracking andMapping (PTAM) (Klein & Murray, 2007) and MonoSLAM (Davison, Reid, Molton,&Stasse,2007)toperformVSLAMonaerialvehicles.
Blosch et al. used a downward looking camera on the Hum- mingbirdquadcopter for a vision-based approach for localization (Bloschetal.,2010).TheposewasestimatedusingtheVSLAMal- gorithm,and thena Linear Quadratic Gaussian (LQG)control de- signwithLoopTransferRecovery(LTR)(LQG/LTR) applied;to sta- bilizethevehicleatadesiredsetpoints.
InMilfordetal.(2011),avision-basedSLAMwithvisualexpec- tationalgorithmwasintroduced. In thisapproach,a place recog- nitionalgorithmbasedon thepatchtrackingisused; toestimate theyaw angle andthe translation speed of the vehicle. In addi- tion,thevisualexpectationalgorithmisusedtoimprovetherecall process of the visited places. This is achieved by comparing the currentscenewiththelibraryofsavedtemplates.Finally,bothal- gorithmsare combined to a RatSLAM (Milford, Wyeth, & Rasser, 2004) for constructing the maps. However, this system loses its efficiencywiththenewscenes that arenot visitedbefore by the vehicle.
InKerl,Sturm,andCremers(2013),aSLAMapproachwithRGB- D cameras has been presented. In this approach, direct frame- to-frameregistrationmethod,withtheentropy-basedmodel,was usedtoreducethedrifterroroftheglobaltrajectory.
Another direct frame registration method hasbeen presented in Engel, Schöps, and Cremers (2014). In contrast to the RGB-D approach,thismethod implementeda monocular SLAM withthe advanceoftheabilitytoconstructlargescalemaps(Fig.2).
A laser-assistedsystemwaspresented inZenget al.(2014)to estimatetheattitudeoftheUAV.Atwhich,theposeoftheUAVis obtainedby alaser scan matchingbasedontheSumofGaussian
(SoG).Alaserspotiscapturedbya cameramountedontheUAV, andby using graycorrelation template matching model,the dis- tanceofthespotisobtained. Then, theposeofUAVisestimated byusingSoG. Inaddition,EKFisusedtocombinetheinertialin- formationtothevisualsystem;inordertoimprovethenavigation process.
AnotherVSLAM approachwaspresentedinZhang,Xian,Zhao, andZhang(2015)tocontrolananoquadcopter.Themotionofthe quadcopterhasbeenobtainedbasedonan opticalflowmodel.In addition,toeliminatethedrifterrorintheflight,aPTAMwasused.
Similarly, to theprevious work, a Kalman filterwasused tofuse thedatafromtheIMUandthebarometer,withthevisualinforma- tion; inordertoimprovethemotionestimation.The maindraw- backofthissystemisthedifficultytoachievethehovermodefor a long time, this is because ofthe limitation of the optical flow algorithm.
AlthoughSLAM, orin particularVSLAM, isconsidered to be a precise method forpose estimation purposes,the outliers in the detectionaffectthe consistently oftheconstructed map. Further- more, these algorithms are complex andcomputationally expen- sive.
Visual odometry. Visual Odometry (VO) algorithms (Nister, Naro- ditsky, & Bergen, 2004; Scaramuzza & Fraundorfer, 2011) handle theproblemofestimatingthe 3D positionandorientation ofthe vehicle.Theestimationprocessperformssequentialanalysis(frame afterframe)ofthecapturedscene;torecovertheposeofthevehi- cle. SimilartoVSLAM thisvisual informationcanbe gatheredus- ingmonocularcameras(Guizilini&Ramos,2011;Roger-Verdeguer, Mannberg, & Savvaris, 2012; Romero, Salazar, Santos, & Lozano, 2013; Wang,Wang, Liang, Chen, & Wu, 2012a) ormultiple cam- eras systems (Maimone, Cheng, & Matthies,2007; Mouats, Aouf, Chermak,&Richardson,2015;Warren&Upcroft,2013).
IncontrasttoVSLAM,VOalgorithmsdealtoestimateconsistent local trajectories,ineach instant oftime without maintaining all thepreviousposes.
VOfirstlyproposedbyNistér(Nisteretal.,2004;Nistér,Nar- oditsky,&Bergen, 2006),itwasinspired bythetraditionalwheel odometry,to estimatethemotionofgroundvehiclesusingstereo camera, incrementally by detecting the Harris corners (Harris &
Stephens,1988)ineachframe.Inthisapproach,theimagefeatures were matched between two frames, and linked into image tra- jectories,byimplementinga fullstructure-from-motion algorithm thattakesadvantage ofthe 5-pointalgorithmandRANdomSAm- pleConsensus(RANSAC)(Fischler&Bolles,1981).Fromhisexper- iments, it wasproved that the VO accuracy was better than the wheelodometrywithpositionerrorof[0.1–3%]ofthetotaltrajec- tory.
Within the NASA Mars Exploration Program (MER) (Cheng, Maimone, & Matthies, 2005; Maimone et al., 2007) a stereo VO algorithm based also on Harris corner detector has been implemented on the MER rover; to estimate its 3D pose in the terrain Mars (feature-poor terrain). Related works to em- ploy VO algorithms on the groundvehicles havebeen presented in Scaramuzza et al. (2009), Lin, Jiang, Pu, and Song (2010), FabianandClayton(2014)andSoltani,Taghirad,andRavari(2012). A hybrid model of visual-wheel odometry is presented in Zhangetal.(2014). Inthismodel,thepositionofthe groundve- hicle isestimated basedmainly onmonocular camera, then both oftherotation andtranslationarerecovered separately usingthe Ackermannsteeringmodel.
Recently differentmotionestimationschemes based onstereo VO algorithms are presented; tobe applied on theUAVs such as theworksinWarrenandUpcroft(2013),Omari,Bloesch,Gohl,and Siegwart(2015),Fu,Carrio,andCampoy(2015).
Fig. 2. Example of LSD-SLAM ( Engel et al., 2014 ).
InWarrenandUpcroft(2013),stereoVOsystemispresentedto enhance theinitialposeofthestereocamera.Thisinformationis basedonasequenceof8–10frames,insteadofusingasinglepair.
Althoughthis systemshowedgoodresultsinlarge-scale environ- ments, it cannot be used withthe MAVsbecause ofthe require- mentofabigsizestereocamerawithabaselineof78cm.
Ontheotherhand,inOmarietal.(2015),Fuetal.(2015),anew smallsizeRGB-DVI-sensor(Nikolicetal.,2014)hasbeenusedon theMAVs.Thefirstworkusedaprobabilisticmodeltoincorporate the RGB-D camera with the IMU, to estimate the motion of the UAV,andbuild3Dmodelsoftheenvironment.Thelaterwork,pre- sentedastereoVOalgorithmbasedonfeaturetrackingtechnique, where a combination of Features from Accelerated Segment Test (FAST) (Rosten & Drummond, 2006) and BinaryRobust Indepen- dentElementaryFeatures(BRIEF)(Calonder,Lepetit,Strecha,&Fua, 2010) are used for thefeature tracking step. However, this com- bination provides fast processing, but it cannot provide accurate datacomparedtootheralgorithms suchasScale-InvariantFeature Transform(SIFT)(Lowe,2004).
Furthermore,althoughthestereocamerausedinthesesystems issmallandlightweight,suitable tomountonasmallUAVs,the smallbaselinecausedasignificant limitationofthesysteminthe large-scaleenvironments.
Research in Grabe, Bulthoff, and Robuffo Giordano (2012) in- troducedan opticalflow vision-basedsystem, combinedwiththe onboard IMU to estimate the motion of the UAV. In this sys- tem,theShi-Tomas(Shi&Tomasi,1994) algorithmisusedforthe feature detection, then the pyramidal Lucas–Kanade (LK) model (Varghese, 2001) is used to trackthe detected feature. Then the obtainedvelocityfromIMUisusedtocompensatethevelocityer- ror estimatedby theopticalflow algorithm.The sameconceptof
usingthecombinationoftheoptical flow andIMUmodelis pre- sentedinLim,Lee,andKim(2012);forcontrollingthehoverflight modeofthequadcopter.Themainlimitationofthemodel,ispro- vidingan unbalancedrepresentationofthescene,whenthereare insufficientnumberorfeatures, orifthetracked features arenot distributedacrosstheimageplane.
Apositionestimationapproachofaerialvehicles,basedonline detectionandcornerextractionispresentedinKameletal.(2010). Inwhich,linesandcornersareextractedbyHoughtransformand Harris corners detection, then the rotation, translation andscale are estimated. Finally, a geometric model estimation is used to mapthe highresolution imageonto alow resolution,whichpro- videspositionestimation.
A monocular camera-based navigation system for an au- tonomous quadcopter was presented in Krajnik et al. (2012); to determineonlytheUAVyawandverticalspeed.Oneofthelimita- tionofthismethodisthattheUAVcanonlyoperatealongpathsit hastraveledduringahuman-guidedtrainingrun.Moreover,these paths can be composed only from straight-line segments witha limitedlength.
InSamadzadegan,Hahn, andSaeedi(2007),they presentedan approach of vision-based (2D–3D) pose estimation of UAVs. In which,thealgorithmaligns2D datafromaerialimage intoageo- referencedorthosatellite image3D basedonfuzzyreasoningsys- tem.
An euclidean homography method was presented in Kaiser, Gans, and Dixon (2006); to maintain the vehicle navi- gation, when GPS signals are not available. This system allows sets of feature points of a series of daisy-chained images to be related; such that the position and orientation can continuously beestimated. This methodwaslimitedto simulationresults,and
ithasthedisabilitytoestimatethedepthwherethereisachange inenvironmentplanes.
SimilarlyinMadison,Andrews, PaulDeBitetto, Rasmussen,and M.Bottkol(2007),avision-aided navigationsystemisusedto re- placeGPS whenit istemporarilydenied.Asinglecamera system detects,tracks,andgeo-locates3Dlandmarksobservedintheim- ages;toestimateabsolutepositionandvelocitydata.
Another multi-sensor data fusion model is introduced in SamadzadeganandAbdi(2012).Inwhich,thesystemusesanEKF tofuse the vision information whichprovides attitudeand posi- tionobservations,withthedata fromtheIMU motionmodel,for accuratelydeterminingtheposeparametersofthevehicle.
3.2.Visualobstacledetectionandavoidance
Obstacledetectionandavoidanceisafundamentalphaseinany autonomousnavigationsystem. Inaddition,thisprocessisconsid- eredasachallengingprocess,especiallyforvision-basedsystems.
In vision-based navigation systems, differentapproaches were presented to solve the problem of obstacle detection and avoid- ance. Approaches such as Beyeler,Zufferey, and Floreano (2007), Hrabar(2008),Gao,Ai, Rarity, andDahnoun (2011), Na,Han,and Jeong (2011), built a 3D model of the obstacle in the environ- ment.Otherworkscalculatethedepth(distance)oftheobstacles;
such as in Jian and Xiao-min (2011) and Saha, Natraj, and Wa- harte(2014).Alltheseapproacheshavethedisadvantagethatthey arecomputationallyexpensive.
Atechnique basedonstereocameras;inordertoestimatethe proximityoftheobstacles, wasintroduced inMajumder, Shankar, andPrasad(2015).Basedonthedisparityimagesandtheviewan- gle,thesystemdetectsthe sizeandthepositionoftheobstacles, andcalculatestherelationofthesizeanditsdistancetotheUAV.
Another stereo vision-basedobstacle detection forground ve- hicles is presented in Broggi, Cattani, Patander, Sabbatelli, and Zani (2013). In which, a Voxel map is reconstructed from the 3D point cloud provided by the stereo camera. Thereafter, a lin- ear Kalmanfilteris used todistinguish betweenthe moving and stationery obstacles. Finally, with the aid of the computed ego- motion,thesystemestimatesthepositionandthevelocity ofthe detectedobstacles.
Ontheother hand,bio-inspired (insect,animalorhumanlike) approachesestimatethepresenceoftheobstacleefficiently,with- out calculatingthe 3D model,e.g. using motionparallax (i.e. op- ticalflow) (Beyeler et al., 2007; Hrabar, Sukhatme, Corke, Usher,
&Roberts,2005;Merrell, Lee, &Beard, 2004) orperspectivecues (Billsetal.,2011;Celik,Chung,Clausman,&Somani,2009;Chavez
&Gustafson,2009).
In de Croon,de Weerdt,DeWagter, andRemes (2010), itwas presented an approach based on the textureand color variation cue;todetectobstaclesforindoorenvironments.However,thisap- proachonlyworkswithdetailedtextures.
Working with Hybrid MAVs, Green et al. proposed an optical flowbasedapproachforlateralcollisionavoidance,mimickingthe biologicalflyinginsects(Green&Oh,2008).
In Lee, Lee, Park,Im, andPark(2011), theSIFT descriptorand Multi-scaleOriented-Patches(MOPS)arecombinedtoshowthe3D informationof the obstacles. At which,the edges and corners of the object are extracted using MOPS by obtaining and matching theMOPSfeaturepointsofthecorners,thenthe3Dspatialinfor- mationoftheMOPSpointsisextracted.Afterthat,SIFTisusedto detecttheinternaloutlineinformation.
InBillsetal.(2011),itwasproposedanapproachforindooren- vironments,withauniformstructurecharacteristics.Inthiswork, HoughTransformisusedtodetecttheedgesthatareused;toclas- sifytheessenceofthescenebasedonatrainedclassifier.However, theirexperimentswerelimitedtocorridorsandstairsareas.
A saliencymethod based on DiscreteCosine Transform (DCT) ispresented inMa,Hu, Shen,Kong,andZhao(2015) forobstacle detection purposes. From the input images, the system assumed that the obstacle is a unique content in a repeated redundant background,thenbyapplyingamplitudespectrumsuppression,the methodcan removethe background.Finally,by usingtheInverse Discrete Cosine Transform(IDCT) anda threshold algorithm, the centeroftheobstacle isobtained.Furthermore,a pinholecamera modelisusedtoestimatetherelativeanglebetweentheUAVand theobstacle,thisangleisusedwithaPDcontrollertocontrolthe headingoftheUAVforobstacleavoidance.
In Saha et al. (2014), the authors presented an approach for measuringtherelativedistancetotheobstacle.Atwhich,thecam- erapositionisestimatedbasedontheEKFandtheIMUdata.Then the3Dpositionoftheobstaclecanbecalculatedbybackproject- ingthedetectedfeaturesoftheobstaclefromitsimages.
An expansion segmentation method was presented in Byrne and Taylor (2009). At which a conditional Markov Ran- dom Field(MRF) is usedto distinguish if the frontal objectmay representacollisionornot.Additionally,aninertialsystemisused to estimate the collision time. However, the experiments of this workwaslimitedtosimulations.
Anotherapproach presentedin Mori andScherer (2013), used featuredetectioninconjunctionwithtemplatematching;todetect the size expansionsof the obstacles . However, the experiments were limitedona tree-likeobstacles anddid notshow resultsof othershapes.
In Eresen, Ímamo˘glu, and Önder Efe (2012), an optical flow basedsystemhasbeenpresentedtodetecttheobstaclesandjunc- tions inoutdoor environments.This systemisbasedon theHorn
&Schunkmethod(Horn&Schunck,1992);inordertolookforthe collisionfreeareasandthejunctionsinapredefinedflightpath.In addition, a PID controller is used asa low-level control scheme.
However, all the experiments were limited to virtual flights in GoogleEarthsoftware.
Kimetal.presentedablock-basedmotionestimationapproach formovingobstacles(humans)(Kim&Do,2012).Inwhich,thein- putimageisdivided insmallerblocksandcomparingthemotion in each block through consecutive images. This approach works wellwithlargesizeobstacles(humans).
In addition, surveys of different approaches of UAVs guid- ance,navigationandcollisionavoidancemethodsarepresentedin AlbakerandRahim(2009)andKendoul(2012).
3.3. Visualservoing
Visual Servoing (VS) is the process of using the information that are obtained by the visual sensors as a feedback in the ve- hicle (UAV) control system. Different inner-loop control systems havebeenemployedtoachievethestabilizationoftheUAVs,such as PID (Golightly & Jones, 2005; Kada & Gazzawi, 2011; Mar- tin, 2012), optimal control (Suzuki, Ishii, Okada, Iizuka, & Kawa- mur, 2013), sliding mode (Lee, Ryan, & Kim, 2012), fuzzy logic (Limnaios & Tsourveloudis, 2012), and cascade control structure (Bergerman, Amidi, Miller, Vallidis, & Dudek, 2007). References;
suchasWagtendonk(2006),BeardandMcLain(2012),Elkaim,Lie, andGebre-Egziabher(2015)providedetailedinformationaboutthe principlesandtheoriesrelatedtotheUAVflightcontrolling.Onthe otherhand,higherlevelcontrolsystemscanbeusedforguidance purposes;suchaswaypointsrangorpathfollowing(Elkaimetal., 2015;Olivares-Méndez,Mondragón,Campoy,&Martínez,2010).
Acomparative studyhas beenintroduced in Altug, Ostrowski, andMahony(2002)toevaluate twocontrollers (mode-based feed- back linearizing and backstepping-like control) using visual feed- back. At which, an external camera and the onboard gyroscopes are used to estimate the UAV angles and position. From the
simulations, ithasbeenfoundthat thebackstopping controlleris betterthanfeedbackstabilization.
In Lee etal. (2012), an image-based visual servoing hasbeen described;to usethe 2D informationasan inputto theadaptive slidingmodecontrollerforautonomouslandingonamovingplat- form.
Avisualsystembasedontwocameras(externalcameralocated onthegroundandonboardcamera),waspresentedinisMinhand Ha (2010)forflightstabilizationpurposes inthehovermodes. At which, both cameraswere set to see each other, and a tracking algorithm wasused to trackcolor blobsthat are attachedto the cameras.Thereafter,theposeoftheUAVwasestimated.Finally,the LinearQuadraticTracking(LQT)controllerandoptimalLQGcontrol were usedwiththe visual feedbackinorder tostabilizethe atti- tude ofa quadcopter. However, the performance ofthe proposed controllerwasverifiedinsimulations.
Adesignoffuzzycontrolfortrackingandlandingonahelipad hasbeenpresentedinOlivares-Mendez,Kannan,andVoos(2015). Inthisapproach,fourfuzzycontrollerswereimplementedtocon- trol the longitudinal, lateral, vertical, and heading velocities to keeptheUAVinthecenterofthemoving helipad.Theestimation oftheUAVposeisbasedonavisionalgorithm.
An inertial-visual aided control system was presented in Baik,Shin,Ji,Shon,andPark(2011).Kanade–Lucas–Thomasi(KLT) featuretrackeralgorithmisusedtoestimatetheUAVattitude,then thevaluesaresent toaPIDcontrolsystem. However,thiscontrol system is lacking ofa filtering stage, resulting a significant drift error.
Recently, Lyu et al. proposed a visual servoing system that is basedoncooperativemappingcontrolframeworkofmultipleUAVs Lyu etal.(2015).Thisframeworkconsistsofa masterUAVwhich leads and controls multiple slave UAVs. Both master and slaves are equipped withdownward cameras to obtain rectifiedimages of the ground. The visual servoing is achieved by using themo- mentoftheSIFTfeatures.WheretheextractedSIFTfeaturesbythe masterUAV arematched withthe featuresextractedby theslave UAVs. Afterwards,themomentfeature isgenerated.Finally,based ontheobtainedinformation,avisualservoingcontrollerisapplied toguidetheslaveUAVstofollowthemasterUAV.However,allthe resultsareobtainedbysimulations.
4. Vision-basedautonomousapplicationsforUAVs
Thefieldsofcomputervisionandimageprocessinghaveshown a powerful tool in different applications for UAVs. Autonomous UAVsapplicationsareaninterestingarea,butatthesametimeare consideredasachallengingsubject.Amongtheseapplications,this literaturethrowslightontheautonomousapplicationsfortake-off andlanding,surveillance,aerialrefueling,andinspectionasshown inTable4.
4.1. Autonomouslanding
Autonomous Take-off and Landing is a fundamental task not only forVTOLvehicles (Cocchioni,Mancini,& Longhi,2014; Gau- tam, Sujit,&Saripalli,2014)butalsoforfixedwingsUAVs (Huan, Guoliang,&Jianqiang,2015;Keke,Qing,&Nong,2014).Forvision- basedtake-off andlanding,differentsolutionshavebeenproposed inordertodealwiththisproblem(Costa,Greati,Ribeiro,daSilva,
& Vieira, 2015; Herissé, Hamel, Mahony, & Russotto, 2012; Lee etal.,2014b;Xiang,Cao,&Wang,2012).
Wenzel et al. introduced a solution using Wii IR camera (Wenzel,Masselli,&Zell,2011).Theconceptofthisapproachisto detect fourlights LEDspatternsituatedon a mobile robot. How- everthesystemisabletotrackthelandingpattern,buttheuseof IRcamera hasseverallimitations;such asthat thesystemcannot
beapplicableinoutdoorflightsbecauseofthesensorsensibilityto thesunlight.Furthermore,thesystemhasmaximumdetectionre- gionupto2.5mbecauseofthelimitationoftheIRcameras. An- othervision-basedcooperationbetweena UAVandan Unmanned Ground Vehicle (UGV) hasbeen presented in Hui, Yousheng, Xi- aokun, and Shing (2013). At which, RGB camera is used to de- tectthelandmark.ThisapproachusedHoughTransformtodetect 20cmradiuscircularlandmarkattachedtothemobilerobot.Then thedetected circleisrestrictedby asquare shape inordertoes- timatethecenter.Finally,aclosed-loop PIDisappliedto perform thecontroloftheUAV.
Multi-scaleORB method (Rublee, Rabaud, Konolige,& Bradski, 2011) integrated wsith the SLAM map to detect the landing site hasbeenpresentedinYangetal.(2013).Althoughtheexperiments haveshown good results, thismethod is dependent on the map generatedfromthe SLAM, andconsequently losesits accuracy in thecaseoftheabsenceofthemap.
4.2.Autonomoussurveillance
Surveillancebased onaerialimaginary isone ofthe mainap- plicationsthattakesadvantagesofUAVsinbothmilitaryandcivil areas.Different methods andapproaches havebeen presented to optimizethe solution ofthe surveillance in terms oftime, num- berof UAVs,autonomy,etc. Freed, Harris,andShafto(2004) pre- sentedan evaluationapproach,comparingtheperformanceofthe methodsandalgorithmsthat employedthe UAVsforautonomous surveillancetaskswiththeguidanceofhumanoperators.Recently, a studybased on the nature of the tasks and thecapabilities of theUAVshasbeenpresentedinCusackandKhaleghparast(2015). Inthisevaluationstudy,aschemecomparingdifferentsmallUAVs hasbeenproposed;inordertoselecttheadequate UAVthatpro- videsthehighperformance andsafetytoimprovethetrafficcon- formanceintelligence.
A feature-based approach for detecting and tracking multi- ple moving targets from UAVs was presented in Siam and El- Helw (2012). First, the features are extracted usingHarris detec- tor,thenthepyramidalLKopticalflowmodelandtheLMEDSare usedinordertoclassifythemovementofthedetectedfeatures.Fi- nally,aKalmanfilterandatemplatematchingalgorithmareused totrackthedetectedtargets.
UAV–UGV cooperationscheme forautonomousindoor surveil- lance tasks has been presented in Saska, Krajník, and Pfeu- cil (2012). In this system, both vehicles are based on visual in- formationfor navigation,localizationandlanding (UAV).In addi- tiontothehelipadthatcarriestheUAV,theUGVisequippedwith thesensorsnecessaryforthesurveillancetasks.BasedonSpeeded UpRobustFeatures(SURF)detector,theUGV candetectandtrack thelandmarkfeatures fromthe inputimagesandestimate itslo- cation,thenmoveautonomouslyalongpredefinedwaypoints.Once theUGVreachestoaninaccessiblelocation,theUAVfliesfromthe UGV and startsthe aerial inspection task. Finally, by using color detectionalgorithms,theUAVlocatesthehelipadpatternandper- formstheautonomouslanding.
DealingwiththemissionplanningproblemformultipleUAVs, Geng,Zhang,Wang,Fuh,andTeo(2013)proposedanapproachthat provides continuous surveillance operations. This approach is di- videdintotwophases.Thefirstphaseaddressesthesearchofthe locationsofthecamerasto providethecomplete coverage ofthe targetsinthearea.Toachievethis,aGeneticAlgorithm(GA)isim- plementedtoobtaintheoptimalsolution.Thesecondphasedeals withdistributingtheselectedlocationsthatareobtainedfromGA overa numberofUAVs,andcreatingthe paths tobe followedin thesurveillance.AntColonySystem(ACS)algorithmisusedtofind the solution for the paths and endurance. However, the experi- mentshavebeenlimitedtosimulations.
Application Description Purpose Related work Autonomous
Landing
Take-off and Landing VTOL ( Costa et al., 2015; Huan et al., 2015; Jung et al., 2015 ) Herissé et al. (2012) ; Lee et al. (2014b) ; Yang et al. (2013) ( Beck et al., 2016; Casau, Cabecinhas, &
Silvestre, 2011; Wenzel et al., 2011 )
Fixed Wing ( Daibing, Xun, & Weiwei, 2012; Kim et al., 2013; Kong, Zhang, & Zhang, 2015; Kong et al., 2014; Muskardin et al., 2016; Pan, Hu, & Shen, 2015; Pouya & Saghafi, 2009 ) Using aerial imaginary for
monitoring and vigilance purposes
Traffic ( Cusack & Khaleghparast, 2015; Heintz et al., 2007; Ke et al., 2015; Kim et al., 2012 )
Autonomous Agricultural crop ( Anthony et al., 2014; Navia, Mondragon, Patino, & Colorado, 2016; Tokekar, Hook, Mulla, & Isler, 2016 )
Surveillance Animal protection ( Ward, Hensler, Alsalam, & Gonzalez, 2016; Xu et al., 2015 ) Other ( COE, 2016; Semsch et al., 2009 )
Aerial Refueling the aircrafts during the flight by using a tanker aircraft
Boom-and- Receptacle
( CHEN, JIA, & ZHANG, 2010; Mammarella et al., 2010; Williamson et al., 2009; Yuan, Yan, Qu, & Zhao, 2015c ),( Haibin Duan & Qifu Zhang, 2015; Yaohong, Jizhi, Qichuan, &
Jing, 2013; Yuan, Whidborne, & Xun, 2014 )
Refueling Probe-and-Drogue ( Bai, Wang, Yin, & Xu, 2014; Mati, Pollini, Lunghi, Innocenti, & Campa, 2006; Ruiz, Martin, & Ollero, 2015; Wu, Zhang, Xu, Zhou, & Luo, 2013 ) ( Martínez, Richardson, &
Campoy, 2013; Su, Wang, Shao, & Yao, 2015; Xufeng et al., 2013 ) Inspection Inspecting the damages and
collapses in the structures for monitoring and maintenance purposes
Buildings ( Choi & Kim, 2015b; Eschmann et al., 2012; Nikolic et al., 2013; Omari et al., 2014 )
Bridges ( Chan, Guan, Jo, & Blumenstein, 2015; Hammer, Dumoulin, Vozel, & Chehdi, 2007;
Metni & Hamel, 2007 )
Wind turbines ( Høglund, 2014; Stokkeland, 2014; Stokkeland et al., 2015 )
Power lines ( Araar & Aouf, 2014; Benitez, Bogado, Guerrero, & Arzamendia, 2016; Cao, Zhu, Han, Wang, & Du, 2013; Du & Tu, 2011 )
Search and Rescue
Gather information in disaster and hazardous sites
( Agcayazi, Cawi, Jurgenson, Ghassemi, & Cook, 2016; Naidoo, Stopforth, & Bright, 2011;
Tao & Jia, 2012 ) ( de Araujo, Almeida, Miranda, & de Barros Vidal, 2014; Erdos et al., 2013 )
Mapping Collecting topographical, thematic and geospatial data
( Fraundorfer et al., 2012; Hackney & Clayton, 2015; Tampubolon & Reinhardt, 2014 ) ( Cui et al., 2007; Gotovac, Gotovac, & Papi ´c, 2016; Li & Yang, 2012; Navia et al., 2016 ), ( Ahmad et al., 2013; Ma et al., 2013; Pérez-Ortiz et al., 2016; Ying-cheng et al., 2011 )
4.3.Autonomousaerialrefueling
AutonomousAerialRefueling(AAR)describestheprocessofair- to-airrefueling, or in other words,in-flight refueling. AAR is di- videdinto two main techniques (Li, Mu, & Wu, 2012a), the first oneisBoom-and-ReceptacleRefueling(BRR),inwhichasinglefly- ingtube(boom) ismovingfromthetankeraircraftforconnecting thereceptaclethat issituatedinthereceiver aircraft.The second techniqueisthe Probe-and-DrogueRefueling(PDR),inwhich,the receiverreleasesaflexiblehose(drogue)andthetankermaintains itspositiontoinserttherigidprobeintothisdrogue.Fig.3shows theconceptofthetwotypesoftheAARsystem.AARisverycrit- icaloperation andusuallythe tankerpilot hastobe well trained toperformthesecomplexoperations.Ontheotherhand,inUAVs, theremote controlling forAAR operation increases the complex- ityofthetask.DifferenttechniquesuseGPSandINStoobtainthe relative pose of the tanker with respect to the receiver aircraft.
However,thesetechniqueshavedrawbacks:First,incertain cases, theGPSdatacannot beobtained,especiallywhenthereceiverair- craftis biggerthanthe tanker,andpreventstheconnection with thesatellites.The second drawbackis theintegration driftofthe INS. On the other hand, the vision-based methods proposed an alternative or complementary solution forAAR. Different studies andsurveysofvision-basedmethodsandapproachesforAARthat areusedwithUAVs havebeenintroducedinMammarella,Campa, Napolitano, and Fravolini (2010), Li et al. (2012a), Aarti and Ji- mohO(2013).
In Xufeng et al. (2013), a machine vision approach has been presentedto providea solutionforPDR technique. Atwhich,the features are detected and extracted from Hue, Saturation, and Value(HSV)colorspaceimages.Then the leastsquareellipse fit- tingmodel isapplied to the detected features to find the center ofthedrogue.Fromtheirexperiments,ithasbeenshownthatthe
usingofHSVcolorspaceincreasestheaccuracyofthefeatureex- tractionstep.
Ontheother hand,Dengetal. developedasystemoftheBRR techniquefortheAARbasedonstereovision(Deng,Xian,&Duan, 2016).Atwhich,thetankerisprovidedwithabinocularcamerain orderto detectthe colorcharacteristics ofthe markers.Then the systemestimatesthepositionofthecontactingpointoftheboom tothereceptacle.Althoughthesystemshowedgoodresultsofthe markerdetectionphaseintheoutdoorexperimentswithdifferent light conditions,but also, it needs improvements in the binocu- larmeasurementstoincreasethestabilityandtheaccuracyofthe poseestimationofthereceptacleforthedockingphase.
Recently, a visual framework for AAR has been presented in Yinetal.(2016).Atwhich,twoclassifiershavebeencombinedfor thedetection andtrackingofthe drogue. TheD-classifier isused to detect the drogue from the input images. In addition, the T- classifier is used to track the detected drogue. Although the re- sults showed better performance, the systemhas a limitation in thetimeofcomputationwhichisnotsuitableforreal-timeopera- tions.
4.4. Autonomousinspection
AerialInspectionisone ofthemostrecentandindemandap- plications that takes the advances of the UAVs (especiallyrotor- crafts). Along withthe safety and the decreasing of human risk, UAVshastheadvantageofreducingoperationalcostsandtimeof theinspection.However, itisimportanttokeepthe imagestabil- ity against anykind ofmaneuver (Cho, Ban, & Kim, 2014). UAVs can perform inspection tasks in different terrainsand situations;
such asbuildings,bridges (Metni& Hamel,2007), windturbines, power plantboilers (Burri, Nikolic, Hurzeler, Caprari, & Siegwart, 2012),powerlines(Du&Tu,2011),andeventunnels.
Fig. 3. Aerial Refueling techniques.
An integrated visual-inertial SLAM sensor has been proposed in Nikolic et al. (2013) in order to be used with the UAVs for industrial facilities inspection purposes.This systemconsists ofa stereo camera and MicroElectroMechanical System(MEMS) gyro- scopesandaccelerometers.The UAVperforms autonomousflights followingpredefinedtrajectories.ThemotionoftheUAVismainly estimated by the inertial measurements; then it is refined using the visual information.From theexperiments,it hasbeen shown that thesystemsuffersfromadelaybetweentheinertialsensors andthe stereocamera. Thus, a calibrationprocess is required.In addition,theresultsshowedadrifterrorof10cminthedisplace- mentovertime.Anothervisual-inertialsensorhasbeenintroduced in Omari, Gohl, Burri, Achtelik,and Siegwart (2014). At which, a visual-inertial stereo camera is used to estimate the poseof the UAVaswellastobuild a3Dmap oftheindustrialinfrastructures whileinspection.
InAraarandAouf(2014),twovisual servoingapproacheswere presented for power line inspection purposes. Both approaches dealtwiththe problemofkeepingthe UAVwitha closeandde- terminate distance to the power lines while performing the in- spection.Inthefirstapproach,anIBVSformulationhasbeencom- bined withtheLinear QuadraticServo(LQS)to improvethe con- troldesignoftheUAV.Whileinthesecondapproach,thecontrol problemwassolvedusingthePartialPosedBased VisualServoing (PPBVS) model.Asithasbeenshownfromtheir experiments,the PPBVSismore efficientandmore robustthan theIBVS.However, PPBVSapproachisverysensitivetothecalibrationerrors.
Autonomous UAV for wind turbines inspection has been presented in Stokkeland, Klausen, and Johansen (2015), Høglund (2014). First, the Global Navigation Satellite System (GNSS) and altimeter are used for positioning the UAV in a de- terminate distance from the tower, then the UAV are rotated to face thehubusingthe visualinformation.Theseworksarebased onHoughTransformtodetectthetower,thehub,andtheblades.
The only difference isin thetracking phase wherein Stokkeland et al.(2015) the Kalman filter is used to trackthe center ofthe hub,whileinHøglund(2014),thetrackingisbasedonopticalflow algorithms,thenthemotiondirection,velocityanddistanceofthe hub andtheblades canbeestimated. Finally,theUAVflightsina preprogrammedpathinordertoperformtheinspectiontask.
5. Conclusions
In this paper, vision-based systems for UAVs have been re- viewed as a whole methodology to cope with cutting-edge UAV technology,where environmentperception hasbeen studiedasa complex and essential task for UAV navigation and obstacle de- tectionandavoidanceinthelast decade.The advantagesandim- provements ofcomputervision algorithms towardsthe presented reliablesolutions havebeenpresented throughreal resultsunder demanding circumstances, such as, poseestimation, aerialobsta- cleavoidance,andinfrastructureinspection.So,complextasksand applications havebeen analyzed and difficulties havebeen high- lighted,wherethetrustableperformanceofthevision-basedsolu- tions andthe improvementsin relationto theprevious worksof theliteratureareprovided.
The differentvision-based systemsmounted in an UAVrepre- sent actualapplications andhelpto overcomeclassical problems, such as research works performed by authors, like autonomous obstacle avoidance orautomatic infrastructure inspection, among others. So, the strengths of the presented computer vision algo- rithmsforUAVshavebeenclearlystatedinthemanuscript.How- ever, presented applications have specific drawbacks that should be taken intoaccount. That is, thevision-based systems are low cost sensor devices,which provides high amountof information, buthavethedrawbackofthehighsensitivitytolightingconditions (e.g.directsunlightmayleadtolackofinformation).Moreover,all thepresentedalgorithmsandapplicationsgive fullunderstanding andconvergencetothenextgenerationofUAVs.
Thepresentedsurveyprovidesafullreviewofthevision-based systemsinUAVsinthelastdecade,includingauthor’sworksinthis field,whichcontributestothefullunderstandingofnovelapplica- tionsderivedfromthem,andfostersthedevelopmentofoutstand- ingUAVsthatarecapableofthemostadvancedandmoderntasks inthemostchallengingscenarios.
Futurework will focuson mainly inoptimizing thecomputer visionalgorithms by intelligentmechanisms based onknowledge andrefineddata. Secondly,theimprovementof thereal-time ca- pabilities of the algorithms and on-board data fusion constitute the key point ofthe intelligentsystems in UnmannedAerial Ve- hicles.Thethirdfutureworkpassbytheinjectionofflyingknowl-
