La Chispa

Two evaluation techniques were implemented to examine the various methods of background characterization described above. To scrutinize the data pool from which a given background was calculated, the Chi-Squared test for MVN was implemented without deviation from the method outlined in the literature. The rank ordered statistical (or Mahalanobis) distance of each pixel in the distribution was plotted against the expected value from the chi-squared distribution, and the goodness of fit was measured via equation 2.32.

To implement the formation of a ROC curve, a few simple modifications were made to tailor detection results for the experiment. As mentioned in the background section, the axes of the ROC curve plot were changed to detection rate versus false alarm rate. Points on the ROC curve were generated by counting the number of false alarms at each occurrence of a target pixel detect and dividing by the total number of background pixels. Due to the limited number of target pixels for certain targets, reporting a probability of detect and false alarm would be an over-generalization of the results.

In keeping with the rest of the experiment, all targets except the one being sought were discarded for ROC curve formation. This was a simple implementation given the numbering scheme of the truth maps (given in Table 3.1). Pixels on other targets were labeled as guard pixels in the truth maps, and thus automatically excluded from the ROC curve calculations. Again, this allowed for an isolation of each target in a natural background, and eliminated spatial dependencies in some of the results, but also eliminated the possibility

of studying the target discrimination power of background characterization techniques.

Like the goodness of fit metric in the Chi-Squared MVN test, the average false alarm rate (AFAR) metric was used to reduce a curve to a single number. AFAR is an approximation of the area above the ROC curve and was calculated by averaging all of the values along the x-axis of the curve. The ability of AFAR to provide a comparison between algorithms is linked to the type of false alarms occurring in the ROC curve. To illustrate this, Figure 3.28 shows ROC curves for two different algorithms detecting the same target.

ROC Curve AFAR Example

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1.E-07 1.E-06 1.E-05 1.E-04 1.E-03 1.E-02 1.E-01 1.E+00

False Alarm Rate

De te c ti o n Ra te A B

Figure 3.28 Example ROC Curve Illustrating the Need for a Partial AFAR

The AFAR for algorithm A is an order of magnitude less than for algorithm B, yet the ROC curve demonstrates that algorithm B is preferable for operation at false alarm rates lower than 1.E-03. The last few target pixels were more

difficult for algorithm B, driving up the final AFAR value in spite of good performance at low false alarm rates. A partial AFAR, averaging false alarms only to a certain rate of detection, could be calculated to show better performance for algorithm B. In this case, selection between AFAR and partial AFAR would reverse the decision of which detector is best. Not unlike the decisions surrounding the level of target exclusion, evaluation of a detection result with partial AFAR requires a fundamental decision about the type of application in which the detector will be employed. Algorithm A would be preferable for applications where some target pixels needed to be identified with the fewest possible false alarms. As an example, an automated target detection system might require low false alarm rate operation with a tolerance for not locating every pixel on a given target. Likewise, algorithm B would be preferable for applications where some false alarms are acceptable but false negatives (i.e. missed targets) are not. An application allowing for a high false alarm rate might be searching through large data sets to flag potential targets for further scrutiny by an analyst. Selecting the latter application, and for consistency in results, the full AFAR was calculated for each detection result in this experiment. Another practical reason for using the full AFAR metric was that the majority of high contrast target pixels were detected without false alarms using any of the methods. In order to compare the methods, the most difficult pixels on these relatively easy targets needed to be included. The use of a partial AFAR would have changed the relative ranking of algorithms in detecting some of the low contrast targets, but would have had minimal impact on the observed trends overall.

The full AFAR included the false alarms detected in locating every pixel on a given target in the image. To compare the performance of several background characterization techniques in finding sub pixel and fully resolved targets, a separate ROC curve and AFAR metric were calculated for the detection of only those pixels identified in the truth maps as sub or full pixel targets. Full pixels targets were generally easier to find than other types, and sub pixels were generally more difficult. The AFAR result for the all pixels category incorporated false alarms from the detection of all types of target pixels and therefore was always the greatest value. In cases where the target contained glare or shadow pixels, the full and sub AFAR results did not combine to form the all pixels result. Included in Appendix A are ROC curves for the all pixels case along with AFAR results for the all, full, and sub pixel targets on one plot. While inspection of the ROC curve remains the most informative evaluation method, a complete picture of detection performance can be summarized with these three AFAR results. For this reason, only the summary AFAR charts are given in the results section.