Desarrollo de la fase experimental de la investigación

The sensor fusion approaches are demonstrated below in figures7.4 to7.7. The estimated

height and error are shown as a function of time with the dashed line indicating the time of fault occurrence. The upper plot illustrates the estimated height, while the lower plot gives the resultant error for the various sensor fusion approaches. Similar to the FDI demonstration, a variety of fault profiles are selected that are the most difficult to detect across the different sensor technologies.

0 50 100 150 200 250 300 350 Time (s) 0 200 400 600 800 1000 1200 Heigh t (m) 0 50 100 150 200 250 300 350 Time (s) 0 10 20 30 40 50 Error (m) Median Weight Optimal Truth

Figure 7.4: The estimated and actual height for a GPS bias fault versus time.

0 50 100 150 200 250 Time (s) 0 200 400 600 800 Heigh t (m) 0 50 100 150 200 250 Time (s) 0 10 20 30 40 50 Error (m) Median Weight Optimal Truth

0 20 40 60 80 100 120 140 Time (s) 0 100 200 300 400 500 600 Heigh t (m) 0 20 40 60 80 100 120 140 Time (s) 0 10 20 30 40 50 Error (m) Median Weight Optimal Truth

Figure 7.6: The estimated and actual height for a ILS noise fault versus time.

0 20 40 60 80 100 120 Time (s) 0 100 200 300 400 500 600 Heigh t (m) 0 20 40 60 80 100 120 Time (s) 0 10 20 30 40 50 Error (m) Median Weight Optimal Truth

Figure 7.7: The estimated and actual height for a RA jamming fault versus time.

The upper plots in figures 7.4-7.7 show no obvious deviations and estimate the aircraft

height in a robust manner. However, the lower plots of the actual height estimation error are concerning as none of the sensor fusion approaches can consistently and accurately estimate the aircraft height. None of the error plots drastically deviate after fault oc- currence except for the RA jamming error in figure 7.7, where the fault detection and isolation approach does not immediately identify the fault. Once the fault is identified, the weighted averaging sensor fusion immediately adjusts, while the optimal estimation approach has a smaller transient error as it deviates and corrects itself much slower. An interesting observation is that the error in the weighted average and median approaches

is negligible just before landing (highlighted by the dashed red rectangles), while the op- timal estimator performs poorly. This corresponds to the location where the aircraft is above the runway and the effect of the terrain database can be ignored as height and relative altitude is with reference to the runway. This proves that the performance of the median and weighted average sensor fusion methods is dependent on the accuracy of the measurements and terrain database.

7.5

Conclusion

The three sensor fusion architectures yield average accuracies between 5.1 m and 7.4 m,

with the 90th _{percentile error in a range of 14 m and 21.8 m. It is not conclusive which}

approach yields the best performance and therefore the merits and drawbacks are sum- marised instead. The weighted average approach has the smallest average error, but is dependent on the accuracy of the FDI architecture. The median is robust to any fault profile, but will always result in a sub-optimal height estimate as it has no regard for the various sensor accuracies. The optimal estimation has the largest average error due to the lack of an adequate model that can be used to predict future height estimates. It is suggested that further work should be done on incorporating the rate of change in height in the Kalman filter model as this should improve its performance. The uncertainty and accuracy of the terrain database should also be investigated further as the use of a more accurate terrain database will decrease the measurement uncertainty and improve the sensor fusion accuracy. The robust height estimator has met the research goal of fusing dissimilar sensor measurements in an manner that is robust to the failure in at most one sensor technology.

Chapter 8

Conclusion and Recommendations

This chapter presents a high-level summary of the work done in this project, highlights the most significant results and observations, and provides a summary of the contributions of this project to the research field. Recommendations are also made for future research on the topic of Robust Height Estimation.

8.1

Summary and Conclusion

This project developed and verified a fault-tolerant sensor fusion system that provides robust height estimates for commercial airliners. Commercial aircraft systems estimate the aircraft height by taking the median of a group of radio altimeter sensors. How- ever, this approach is not robust to the failure of the entire radio altimetry system. The proposed system uses technology redundancy by combining the measurements from the available aircraft sensors to improve the robustness of the height estimate. The robust

height estimation problem is divided into two major sub-problems: fault detection and

isolation and sensor fusion. Firstly, a faulty sensor is identified by the fault diagnostic system before the remaining healthy sensor measurements are fused in an optimal manner. The real dataset supplied by Airbus was not deemed rich enough to serve as compre- hensive training and testing data. A simulation model was therefore used to generate synthetic training and testing data. Mathematical models were established for the air- craft motion, the sensors, and the terrain. The structure and nominal parameters for the sensor models were based on information sourced from literature, and then the sensor pa- rameters were tuned to fit a real dataset provided by Airbus. Fault models for six types of sensor faults were also created. The simulation model was used to generate a large dataset of representative sensor measurements containing both “no-fault” and “fault” conditions. A variety of analytical redundancy approaches were investigated to address the fault detec- tion and isolation sub-problem. These included data-driven and model-based techniques. Two general approaches were considered: fault diagnosis using sensor measurements from a single time instant, and fault diagnosis using sensor measurements from a window of consecutive time instants.

The single time instant fault diagnostic system made use of binary and outlier detec- tion algorithms for fault detection. The multi-class classifiers were activated when a fault was detected and used to identify the faulty sensor. The data-driven outlier detection

algorithms that were considered include: Elliptic Envelope, Local Outlier Factor and Iso- lation Forest. The following binary and multi-class classifiers were considered: Logistic Regression, Gaussian Na¨ıve Bayes, k-Nearest Neighbors, Decision Tree and Support Vec- tor Machine. The use of outlier, binary and multi-class classifiers required establishing different sets of training data. The outlier detection algorithms were trained on a single class and used as novelty detectors. The binary classifiers were trained on equal per- centages of fault and no-fault data, while the multi-class classifiers were trained on equal percentages of fault data. Data preprocessing was introduced to improve class separa- bility and classification accuracy. The following techniques were considered: orthogonal transform, kernel transform and residual transform. Preprocessing was applied to all training and test datasets to ensure that the best combination of classification technique, and preprocessing technique was chosen. Learning curves were analysed using five-fold cross validation to ensure that algorithms were properly trained, and ROC curves were analysed to ensure that the optimal unbiased decision threshold was used. The fault detec-

tion accuracy ranged from 95.72 % (Support Vector Machine with kernel preprocessing) to

53.76 % (Local Outlier Factor with no preprocessing). The fault isolation accuracy ranged

from 94.16 % (k-Nearest Neighbors with residual preprocessing) to 85.04 % (Support Vec-

tor Machine with no preprocessing). The locations of all incorrect classifications for the top three performing single time instant approaches were analysed. This revealed that the single time instant approaches struggled to adapt to changes in the sensor accuracies. The top performing k-Nearest Neighbors with residual preprocessing had false alarms that were situated deep within the no-fault region. Therefore, the Gaussian Na¨ıve Bayes with residual preprocessing architecture was selected as the most reliable single time instant fault diagnostic system. The results were validated on actual aircraft landing data, and the fault detection and isolation performance was demonstrated on different fault profiles. A similar approach was taken to evaluating the consecutive time instant approaches. The following techniques were investigated: Robust Kalman filter, Bank of Kalman filters, Model Consensus, and Dynamic Principal Component Analysis. The single time instant Gaussian Na¨ıve Bayes with residual preprocessing was given fault detection memory to allow for an objective comparison. This significantly improved the isolation accuracy

of the Gaussian Na¨ıve Bayes architecture from 93.8 % (single time instant) to 97.65 %

(consecutive time instant). The fault detection accuracy ranged from 99.18 % (Robust

Kalman filter) to 68.02 % (Dynamic PCA). The fault isolation accuracy ranged from

99.15 % (Robust Kalman filter) to 92.98 % (Model Consensus). The locations of the in-

correct classifications were analysed for the top three consecutive time instant approaches. The top performing Robust Kalman filter had no incorrect isolations, four false alarms situated near the decision boundary, and the majority of its missed opportunities were situated in the no-fault region. Similar to the single time instant approaches, the results were validated on actual aircraft landing data, and the fault detection and isolation per- formance was demonstrated on different fault profiles.

Both the single and consecutive time instant approaches have their own unique attributes. The single time instant approach is adaptable in the sense that each sample is re-evaluated and classified accordingly with no bias shown towards previous sensor statuses. The draw- back to this approach is the large number of missed opportunities. It is impossible to label faulty data points within the no-fault region correctly without prior knowledge that a sensor fault is active. Likewise, the consecutive time instant approach incorporates prior knowledge in the form of previous predictions, trajectory data, or measurement history

to improve performance, but is more rigid and cannot instantly isolate a failure.

The knowledge of an active fault is used to ensure that the estimated height is robust to failure. The following sensor fusion methods were considered: median, weighted averag-

ing and optimal estimation. The 90th _{percentile for height estimate error ranged between}

14 m and 21.8 m for the optimal estimation and the median approach respectively. Sim-

ulated data was used to demonstrate the performance of the complete Robust Height Estimation architecture. It can be concluded that this project has successfully met the

objectives that were set in Chapter 1. A wide variety of methods and approaches were

considered that can be used to improve safety and reliability in the aviation industry by demonstrating their performance on the problem of aircraft height verification.