We introduce several measurements that have been widely used in recommender sys- tems to evaluate recommendation/prediction performance. These include Precision,
Recall, Accuracy, F1 measure, RMSE, and AUC.
Precision, recall, accuracy and F1 measure are all defined in terms of a set
of retrieved entities (web documents, people, papers, or other entities) and a set of relevant entities. They can also be defined by using four traditional terms in classification task: true positive tp, true negative tn, false positive f p, and false negative f n. tp indicates the number of positive (relevant) entities that are also predicted as positive samples; f p is the number of entities that are actually negative (non-relevant) entities but are predicted as positive entities; f n indicates the number of entities that are actually positive but predicted as negative, and tn indicates the number of actually negative entities that are also correctly predicted as negative.
• precision: in traditional IR system, precision is the fraction of retrieved en- tities that are relevant to the search:
precision = krelevant entities ∩ retrieved entitiesk
kretrieved entitiesk (2.19)
or
precision = tp
tp + f p (2.20)
• recall: recall indicates the fraction of entities that are relevant to a query that are successfully retrieved.
recall = krelevant entities ∩ retrieved entitiesk
krelevant entitiesk (2.21)
or
recall = tp
tp + f n (2.22)
• F1 measure: F-measure (or F-score) is the harmonic mean of precision and
recall. In a general case, it can be computed as: Fβ = (1 + β2)
precision · recall
β2· precision + recall (2.23)
The most widely adopted F-score is the the F1 measure where β is set to be
1, indicating that precision and recall are evenly weighted. We have: F1 = 2 ·
precision · recall
precision + recall (2.24)
• Accuracy: Accuracy can be computed as: Accuracy = tp+tn+f p+f ntp+tn
Precision@k, Recall@k and Accuracy@k are adopted for computing the corre- sponding Precision, Recall and Accuracy values for the top-k returned entities. • RMSE (abbreviated for Root Mean Squared Error): RMSE is widely used for rating-related recommendations, such as user-item ratings or user-movie ratings prediction. It amplifies the contributions of the absolute error between the predicted values and real values. Suppose for a given user-item pair, the
real rating for user i to item j is rij, and the predicted value is ˆrij, then the
overall RMSE value for the recommender system is: RMSE = s 1 k T k X (i,j)∈T (ˆrij − rij)2 (2.25)
where T is the set of all user-item pairs whose ratings are predicted.
• AUC abbreviated for Area Under the ROC Curve: ROC curve is a two- dimensional depiction of a classifier’s performance, on which the true positive rate (tp+f ntp ) is plotted on the Y-axis and false positive rate (f p+tnf p ) is plotted on the X-axis. AUC indicates the actual area under the ROC curve, which can be computed as:
AUC = s0 − n0(n0+ 1)/2 n0n1
(2.26)
where n0 is the number of positive samples, n1 is the number of negative
samples, and s0 = PNi=1ri, ri is the rank of the ith positive sample in the
ranking list, given that we have N positive samples in total.
Other ranking based IR metrics such as MAP and NDCG can also be used as the evaluation metrics for recommendation tasks whose definitions have been introduced in detail for the task of expertise retrieval.
Chapter 3
Expert Ranking: Topic-Driven
Multi-Type Citation Network
Analysis
In this chapter, we present an enhanced integrated probabilistic model which com- bines both content-based and graph-based approaches for expert ranking in an aca- demic environment. We construct a heterogeneous academic network which con- sists of multiple types of academic entities. We introduce the application of Topical PageRank into link analysis over the academic network and propose a heteroge- neous PageRank-like algorithm into exploring the impact of weighting various fac- tors. Comparative experimental results based on data extracted from the ACM digital library show that 1) the multi-type academic network works better than the graphs integrating fewer types of entities, 2) the use of Topical PageRank can fur- ther improve performance, and 3) Heterogeneous PageRank with parameter tuning can work even better than Topical PageRank.
3.1
Introduction
Estimating researchers’ contributions or reputations is of great importance since it can offer support when making decisions about researchers’ job promotions, project funding approvals, and scientific award assignments. With the rapid development of academic digital libraries, the increasing volume of online scientific literature provides abundant sources of reputation evidence in terms of researchers’ (authors’) publications, as well as the citation relationships among these publications, both of which can be taken advantage of in evaluating researchers’ reputations.
In order to evaluate the reputation of a researcher, especially within one scientific domain, there are typically two basic approaches. One is called the content-based approach, in which relevant documents representing expertise of a researcher can be considered, and information retrieval models can be applied to evaluate the relevance of these documents and thus authors to the query topic [8, 50, 108]. Researchers’ publications in the academic digital libraries provides such good expertise resources. Another important approach, which is also our main focus in this chapter, is via social network analysis [182]. The citation network1 is one form of social network in
which scientific factors, like authors and papers, can be represented as nodes, and their mutual interactions such as citations, can be modeled as edges.
Citation network analysis has long been a popular mechanism to evaluate the importance of publications and authors. Initially, citation analysis mainly focused on counting the number of citations [58, 59]. Under this scheme, an author will have higher reputation if he can be cited by many other authors.
With the recent success of graph-theoretic approaches in ranking network en- tities, researchers have begun to introduce link analysis approaches like PageRank [132] and HITS [93] into citation network analysis. Further attention has also been paid to integrate different kinds of citation networks, including a coauthor network for authors and a citation reference network for papers and take advantage of their mutual reinforcement to improve reputation ranking performance. The assumption
1
We simply name this academic network as the citation network, since in this work, ‘citation’ is the primary relationship among scientific factors that we considered.
in this group of approaches is that more influential authors are more likely to pro- duce high quality and thus highly cited papers, and well-cited papers can bring greater prestige to their authors.
In spite of the constant improvement in citation network analysis, including combinations with content-based approach, integration of different kinds of citation works, there still remain some limitations. For example, current citation network analysis seldom goes beyond that of the citation relationship among authors or papers. PopRank [131] integrates conferences and journals, yet there are still some other useful and easily available information in the scientific literature, such as authors’ affiliations. In this chapter, we propose a novel probabilistic model which can integrate the citation between authors, papers, affiliations and publishing venues in a single model. Affiliation offers a good indication of authors’ expertise, since high quality organizations tend to hire researchers (authors) with higher reputation.
In order to explore the different impact among factors, we propose a heteroge- neous PageRank, permitting us to consider different propagation rates among fac- tors. Furthermore, one distinguished contribution of our work is that we introduce the topical link analysis, which has shown success in web page authority evaluation, into citation network analysis. In summary, our main contributions include:
1. Proposing a novel probabilistic model which combines content-based analy- sis with a multi-type citation network, integrating relationships of authors, papers, affiliations and publishing venues in one model. This model can be extended to include more types of social factors.
2. Proposing a heterogeneous PageRank random surfer model compared to the original uniform PageRank model, to reflect the impact among different fac- tors.
3. Introducing topical link analysis into citation network analysis. In particular, Topical PageRank [130] is adopted for citation link analysis.
4. A comparative study using ACM digital library data on various PageRank extensions as well as different complexity of citation networks.
The rest of this chapter is organized as follows. We introduce the multi-type citation network framework and the heterogeneous PageRank random surfer model in section 3.2. Section 3.3 introduces topical link analysis model. Experiments and results analysis are described in section 3.4. We review related work in section 3.5 and conclude this chapter in section 3.6.