In this experiment performance of global ranking using LDM is tested and analysed. Measures of Kendall’s τ and accuracy are utilized for performing the evaluations, this is because global rankings are generated here.
To demonstrate a full ranking with 120 configurations matching approach based on LDM is used. This is because according to evaluation performed LDM generated better accuracy com- pared to most other approaches evaluated i.e. BoV for smaller vocabularies with inverted file and BoV with the min-Hash data structure. Similar to previous testing voting files are con- structed using rankings from strong-5, weak-5 and all-10.
Kendall’s τ and accuracy for the three scenarios is presented in Table 5.13. It can be seen that coefficients for Kendall’s τ are nearly 0. This is an indication that output ranking which takes matching into account is approximately independent of input ranking (strong-5, weak-5 and all-10). Furthermore, complementary observations can be made from accuracy, where value
is below 0.5. This represents that matching does not perform well and is unable to return true matches in most instances. Due to this appropriate comparisons are not made in the voting input to Kemeny-Young.
measure strong-5 weak-5 all-10 Kendall’s τ -0.026 0.015 0.080 acc 0.360 0.365 0.353
Table 5.13: Kendall’s τ and accuracy averaged over five runs of random splits when matching using LDM.
5.3.5 Summary
An evaluation of image matching methods was performed using various measures including accuracy based on correctly ranked pairs and Kendall’s τ . From the performance analysis, it was found that matching using LDM and BoV with inverted file (with a higher number of 1M visual words) provided better results compared to other approaches. This is as expected because LDM uses the full set of feature descriptors from the training set for nearest neighbour search. And in the BoV model, an image is represented by a histogram of occurrences of visual words. These visual words are part of a vocabulary which is built by clustering the features of the training images and so a higher number of visual words should generally improve the performance.
In particular, BoV with min-Hash was shown to give an inferior performance with slight im- provements after the number of hash tables are increased. This needs to be done in conjunction with a minimum number of hash functions. The min-Hash algorithm works by using indepen- dently chosen visual words as global descriptors. A higher number of similar features in two images will lead to higher probability of them having the same min-Hash. However, in the context of the datasets used, the global nature of this algorithm is prohibitive because testing is based on image configurations. So this algorithm is more likely to be suitable for near duplicate image detection because differences between test and training images are minimum. Compared to the min-Hash algorithm, BoV with the inverted file data structure was shown to give better performance. An increment was particularly seen for larger vocabulary sizes of 500K and 1M.
5.4. Conclusions 81
On varying the parameters of the inverted file structure, such as, weighting of the histogram counts, normalization of the histograms and distance function, slight differences were seen in the performance.
Another interesting observation was made in the consistency of image-based and configuration- based absolute accuracy when evaluating. The highest values obtained for BoV with inverted file and LDM were consistent for the image and configuration-based accuracy. These were moreover obtained for strong-5 ranking in all the cases, cf. highlighted accuracy in the tables of results. However, accuracy obtained for the min-Hash algorithm was not consistent between image and configuration-based measures. A constant high scoring for strong-5 was also not seen. This was most likely due to lower performance of the min-Hash algorithm where the ac- curacy obtained was mostly near 0.5 (random value). In summary, better results were obtained when using LDM, that is why it was used for generating a global ranking.
5.4
Conclusions
An evaluation of matching approaches to achieve ranking was performed and it was established that LDM and BoV with inverted file (1M visual words) gave a better performance. Therefore matching from LDM was used to construct an overall configuration ranking. On testing this it was found that performance achieved was prohibitive in this context. This is most likely because although LDM generates better matches compared to other approaches, the quality of matching achieved is still not adequate. On comparing accuracy for global ranking from matching with that generated in Chapter 4 Section 4.3.2, matching based ranking shows weaker performance. For example, comparison of an accuracy of 0.94 for strong-5 with 10% error for attribute recognition from Section 4.3.2 with a value of 0.360 for strong-5 in Section 5.3.4. Therefore, performance of ranking i.e. accuracy obtained from Table 5.13 will be used as a basis to compare with the approach proposed in Chapter 6.
Chapter 6
Ranking images by applying graphical
modelling
In Chapter 3, the process of obtaining ranked lists of images from manual annotation based on comparative scoring was described. Rankings achieved from this can be utilized as reference sets for learning and evaluation. Given this data, a graphical model is trained to capture the attributes and relations between them. Using this model comparisons can be performed based on visual aesthetics when image pairs are presented at the input. The graph based approach alongside representation of the rankings using learning is first presented in Section 6.1. Next, an example based on the learning method is demonstrated in Section 6.2, which shows the attributes and their relation potentials for a toy example. Then, results from the experiments performed to test the graphical model and overall ranking which includes the recognition sys- tem estimates are discussed in Section 6.4. Finally, accuracy from the graphical modelling approach is compared with the baseline methods in Section 6.5.
6.1
Image model: learning and recognition
An overview of the approach for ranking images based on their aesthetic impression is first given. Next, procedure of how the global ranking is utilized to model various attributes and rating criteria is described. Finally, specifics of the model for usage within application of fashion assessment is explained.