3.4.2.1 Spot-wise Condition Monitoring Accuracy
One of the key features of the AVL-Genius system is its ability to collect RSC data at specific locations or spots along the route being monitored. The first question of relevance for this investigation is therefore concerned with the accuracy of the system, i.e., how accurately can the system classify the RSC of each image? This performance is important as it reflects the system’s ability to identify the time of occurrence and the location of poor road surface conditions, using the GPS and timestamp information associated with each classified image.
In answering this question, we must first obtain the “ground truth” of the RSC in each image. As discussed previously, this was done by manually classifying all images. Table 3.5 shows the confusion matrix of the classification results by AVL-Genius using the manual classification results as the “ground-truth”. A total of 15,913 images collected by the data collection units over 20 events were manually classified and used for evaluating the spot-wise condition monitoring performance of the system.
Of all the images collected, a total of 10689 images (67%) were manually classified as bare. The AVL-Genius system correctly classified 82% of these bare condition images. Approximately 15% of these images were misclassified as Partly Snow Covered, which is somewhat expected considering that for some of the images there is only a small difference (in snow coverage) between bare and partly snow covered, especially in events of low precipitation. Three percent of bare conditions were misclassified as fully snow covered, which could be caused by the effect of glaring and residual salt as detailed in the later sections of this chapter.
A total of 4522 images (28%) were manually classified as Partly Snow Covered. Approximately 55% of these images were correctly classified by AVL-Genius, while 41% of them were classified as Bare and the remaining 4% as Fully Snow Covered. Interestingly, 30% of these images were associated with the lower end of the snow coverage scale (< 25%), which could account for a high number of partly snow covered conditions being misclassified as bare. The other main reason for misclassification is that dark coloured slush is not being accurately detected by the current image recognition algorithm.
The classification accuracy for Fully Snow Covered conditions was much lower. Of the 702 Fully Snow Covered images, approximately 38% were classified correctly, with 47% of them classified as
Partly Snow Covered and the remaining 15% as Bare. One of the main reasons for this problem was the high proportion of conditions showing wheel paths covered by slushy snow, appearing to be track-bare and thus classified as partly snow covered. A closer examination of the images automatically classified by AVL-Genius as fully snow covered shows 62% being manually classified as either 50%~75% snow covered or fully snow covered. This result indicates a possible overestimation of snow coverage by AVL-Genius, where if only one wheel track is clear of snow and ice, the resulting automatic classification is fully snow covered. A detailed discussion on the associated issues is provided the following section.
Table 3.5 - Confusion Matrix of AVL-Genius Classification Results
By number Manual Classification (Ground Truth) AVL-Genius Classification Total BP PS FS BP 8729 1575 385 10689 PS 1840 2484 198 4522 FS 106 330 266 702 Total 10675 4389 849 15913 By percentage Manual Classification (Ground Truth) AVL-Genius Classification Total BP PS FS BP 81.70% 14.70% 3.60% 100% PS 40.70% 54.90% 4.40% 100% FS 15.10% 47.00% 37.90% 100%
3.4.2.2 Route Level Condition Monitoring Accuracy
The previous section evaluates the performance of the AVL-Genius system in classifying the RSCs based on the point-wise observations or individual images taken at locations along the test route. AVL-Genius can also provide summary statistics at a route level in terms of proportion of individual RSC types detected along a route. These route level statistics could be used to assess the performance of the system in providing aggregate information on the overall conditions of a patrol route in accordance with the current practice and needs of MTO. This section compares AVL-Genius results against manual classification, patrol observations and MTO’s TRIP system.
3.4.2.3 AVL-Genius vs. Manual Classifications
Table 3.6 shows the summary statistics of the proportion of RSCs occurring over the route for two sample events. For each run, the proportions of RSCs are listed according to manual and automatic classification, with the single aggregated RSC class conforming to TAC definitions. It is shown that, while the proportions of individual RSC classes vary for each run, the single route-level RSC class matches perfectly with the manual class, with the exception of one run. This observation emphasizes that even though there is performance variation in the classification of individual images, system classification performance is satisfactory at the route level, which is representative of the current state of practice. These results remained consistent over the course of all remaining events.
Table 3.6 – Comparison of AVL-Genius and Manual Classifications for Route-level Conditions Feb 28th 2014
Run
Route-Level RSCs Single TAC RSC
Manual AVL-Genius Manual System
1
87% Partly Snow Covered 62% Partly Snow Covered
Partly Snow Covered
Partly Snow Covered
11% Bare 38% Bare
3% Fully Snow Covered
2
93% Partly Snow Covered 79% Partly Snow Covered
Partly Snow Covered
Partly Snow Covered
4% Bare 21% Bare
4% Fully Snow Covered
3
72% Bare 61% Partly Snow Covered
Partly Snow Covered
Partly Snow Covered 25% Partly Snow Covered 33% Bare
3% Fully Snow Covered 6% Fully Snow Covered
4
94% Bare 86% Bare
Bare
Partly Snow Covered 6% Partly Snow Covered 14% Partly Snow Covered
5
83% Bare 53% Bare Partly Snow
Covered
Partly Snow Covered 17% Partly Snow Covered 47% Partly Snow Covered
Mar 15th 2014
Run
Route-Level RSC Single TAC RSC
Manual AVL-Genius Manual System
1
86% Partly Snow Covered
Partly Snow Covered
Partly Snow Covered 100% Partly Snow Covered 4% Bare
10% Fully Snow Covered
2
75% Partly Snow Covered 71% Partly Snow Covered
Partly Snow Covered
Partly Snow Covered
21% Bare 25% Bare
4% Fully Snow Covered 4% Fully Snow Covered
3
72% Partly Snow Covered 50% Fully Snow Covered
Partly Snow Covered
Partly Snow Covered 30% Fully Snow Covered 44% Partly Snow Covered