5.1 Conclusions
This section presents our conclusions from the conducted research and experi- ments. We researched five proprietary property ranking algorithms in this study that utilize different approaches to ranking that vary from relatively simple term frequency counting up to complex ontology-based terminological analysis. Several ranking algo- rithms employed novel approaches to the ranking of properties from ontology class structures not studied before in this context. The conducted ranking algorithm exper- iments were described in full detail in this paper and we concluded each section with a comprehensive review of the respective ranking approach performance.
The performance characteristics of these ranking algorithms were measured objec- tively using an appropriate human designed ranking baseline extracted from Inbox templates. The usage of this ground truth set was defended in paragraph 2.4. We de- scribed how we created a test data set from this baseline for the purpose of ranking evaluation in Chapter 3.
A total of 19 ranking metrics were described in paragraphs 3.6, 3.7 and 3.8 that as- sisted us in measuring the performance of ranking algorithms using the test data set. The paired t-test used to compare ranking algorithms described in detail in paragraph 3.9 is a scientifically sound statistical approach. In Table 15 we present an outline of the overall results for the various conducted experiments. We have restricted the summary to the key rank correlation metrics as the evaluation across these dimensions was central in our research.
Algorithm Kendall
τ Spearman ρ Significant with respect to alphabetical?
Cohen correlation classification for ρ (rounded to 0.1)
Alphabetical 0.53 0.54 N/A Small
Word-frequency 0.61 0.65 Yes Moderate
Term similarity 0.51 0.51 No None
N-gram 0.61 0.64 Yes Moderate
Heuristic 0.65 0.68 Yes Moderate
Dynamic
terminological 0.55 0.55 No Small
A review of Table 15 shows that the largest rank correlation metric has a value of 0.68 (the average normalized Spearman ρ metric for heuristic ranking). The regular, that is, the non-normalized average Spearman ρ value is 0.36, computed using the simple conversion formula ( . Note that the regular Spearman ρ metric has a range from -1 up to 1. The top boundary (1) denotes a perfect matching of ranks; the low boundary (-1) indicates complete reverse rankings. A score of 0 denotes a complete abundance of association, that is, rankings that share no common- ality. A value between 0.30 and 0.50 is viewed by Cohen [40] as a “moderate” corre- lation. This classification from Cohen remains to hold even in the somewhat weaker case of the averaged Kendall τ score (0.65 normalized, 0.30 regular). Although the correlation is categorized as moderate, it must be noted that the amount of possible rank permutations is substantial. Hence, the term ‘moderate approximation’ must be viewed in perspective. We now have come to the point to derive a conclusive answer for the main research problem that we defined in paragraph 1.2:
To what extent can an ontology property ranking algorithm approximate a hu-
man-designed ranking to support automated view generation in Semantic Web browsers?
Based on our research the answer to the overall research question is that three of the studied ranking algorithms approximated a human designed ranking up to a mod-
erate extent. Two of these methods are terminology-driven and intrinsically NLP-
based. The discussed disambiguation issues are both challenging and limiting aspects of NLP-based approaches. However, these ranking algorithms are potentially applica- ble to domains other than the currently discussed Semantic Web context. Further- more, we established that the three ranking approaches outperformed alphabetical ranking in a statistically significant manner (see also Table 15). The techniques sur- pass alphabetic ranking significantly based on the averaged Kendall τ metrics and the averaged Spearman ρ metrics, both in English and in the multi-lingual evaluation context.
Rather than re-iterating over the individual experiment conclusions we give our view with respect to the most useful ranking based on the key lessons learned and the ranking algorithm evaluation data. The review of the key ranking algorithm perfor- mance metrics with respect to ranking ontology properties has led to the overall con- clusion that the heuristics-based algorithm is the best performing approach to ranking that we studied. In addition to the key performance characteristics we also looked at secondary dimensions such as the ranking algorithm implementation complexity, the speed of ranking and the lack of external dependencies. A holistic evaluation leads to the conclusion that the heuristic ranking approach is clearly the overall “winning” ranking method. The heuristics-based approach does not require a large amount of code to implement and the involved logic is quite modest. Furthermore, ranking speed is fast and predictable.
5.2 Future work
Our research reviewed several ranking algorithms that we developed through an empiric and holistic approach. The algorithms were inspired partly by past work, partly by gaps in the literature and partly by evolving insights. Future work may uti- lize machine learning techniques to discover new approaches to ranking. In particular, we speculate that it may be possible to predict the rank of specific properties based on the rank of comparable properties in other classes. It might be possible to find proper- ty interrelationships that can ‘predict’ the ranking position of other properties. This type of research would reverse engineer ranking patterns using techniques such as data mining and possibly also feature learning to discover recurring ranking patterns. The heuristic ranking algorithm in particular will benefit from this type of research. Another interesting future work research direction would involve a further strengthen- ing of the presented methods through inclusion of instance data sampling.
The evaluation framework, as well as the constructed test data set, may be used by future researchers to review alternative ranking algorithms. Our study results can act a solid foundation for future reference comparisons. The evaluation framework itself may be extended further with additional metrics to support the review of other aspects of rankings, as is seen fit. Future work may also involve a subjective (‘human’) re- view of our ranking algorithms to provide a different perspective on the performance of the studied ranking algorithms.
Another obvious way to further complement this research would involve the appli- cation of our algorithms within a semantic web browser to test-drive our algorithms in “real life” scenarios. An implementation based on the Fresnel Display Vocabulary for RDF53 is already being considered within the current institution’s research program.
Last, but not least, we wondered how the ranking information in Infoboxes evolved over time. Wikipedia is fully versioned which enables a thorough research of histori- cal ranking information. Are properties stable over time in terms of their rank posi- tion? What types of properties are typically reordered? The central research question would involve the identification of patterns that can be utilized to improve the heuris- tic ranking approach. Data mining techniques may again proof useful for this type of research.