Expert finding is the task of identifying a list of people who are knowledgeable about a given expertise topic, according to [BBA+07]. The expert finding task does not nec- essarily require the extraction of expertise topics because the user searches for experts by providing a query that describes the topic of interest. Even so, in this section we will show that expert finding results can be improved when expertise topics and the relations between them are automatically extracted from text.
Within an organisation or community, several competent people have to be ranked based on their relative expertise. Documents written by a person are used as an indi- rect evidence of expertise, assuming that an expert often mentions his areas of interest. Data-driven approaches such as the ones developed by the information retrieval com- munity [PC06, BdRA06, SH08], do not attempt to map domain knowledge to identify knowledgeable individuals. Instead, associations between people and expertise topics are uncovered based on co-occurrences between topics and people in the same context. Information retrieval approaches do not make use of performance indicators to measure expertise, as done in competency management, but rely instead on the broader notion of relevance [BdRA06].
Relevance-based approaches rely on exact matches of expertise and are less reliable for general expertise topics that are mentioned less often in text. If a person does not mention the name of a domain, they will not be considered an expert although they often discuss about related subfields. For example, although a person does not explicitly mention Semantic Web in their documents, if they frequently discuss about Linked Data, RDF data, ontologies and SPARQL they are likely to be Semantic Web experts. Although latent semantics methods such as LSA [DDL+90], pLSI [Hof99], LDA [WC06] address this problem, they are more appropriate for single-word terms and have to be extended to n-gram models to cover longer terms, which are computationally more expensive.
In this chapter we extend existing relevance-based measures of expertise, using topical hierarchies to introduce a semantically motivated measure of expertise. Again, we consider the tf-idf measure described in the previous section in Equation 4.9 to measure the relevance of a given expertise topic for a person. People are represented by an aggregated document that is constructed by concatenating all the documents
4.2 Applying topical hierarchies to Expertise Mining
authored by a person. Therefore, the relevance score R(i, t) that measures the interest of an individual i for a given topic t is defined as:
R(i, t) = tf idf (t, i) (4.11)
Expertise is closely related to the notion of experience. The assumption is that the more a person works on a topic, the more knowledgeable they are. This performance indicator is similar to the frequency indicator mentioned in [Paq07]. We estimate the experience of a person based on the number of documents that they wrote about an expertise topics. It is only those documents for which the expertise topic is extracted as a top ranked keyphrase that are considered. Let Di,t be the set of documents authored
by the individual i, that have the expertise topic t as a keyphrase. Then, the experience score E(i, t) is defined as:
E(i, t) = |Di,t| (4.12)
where |Di,t| is the cardinality, or the total number of documents, in the set of
documents Di,t. It can be argued that it is not only the number of documents that
indicates expertise, but the quality of those documents as well. For example, in a peer- review setting, the impact of a publication measured using citation counts is often used as an indicator of publication quality. Similarly, page rank can be used as a quality indicator for web pages, the number of comments for blogs, the number of retweets for tweets, the number of followers for users. But each of these indicators is specific to the content type and have to be investigated separately depending on the domain, therefore we leave the integration of document quality measures for future work.
Relevance and expertise measure different aspects of expertise and can be combined to take advantage of both features as follows:
RE(i, t) = R(i, t) · E(i, t) (4.13)
Both the relevance score and the experience score rely on query occurrences alone. A topical hierarchy, such as the one constructed in Section 4.1.3, can provide valuable information for improving expert finding results. When the subtopics of an expertise topic are known, we can evaluate the expertise of a person based on their knowledge of specialised fields. A previous study showed that experts have increased knowledge at more specific category levels than novices [TT91]. We introduce a novel measure for
4. CONSTRUCTING TOPICAL HIERARCHIES FOR EXPERTISE MINING
expertise called Area Coverage that measures whether an expert has in depth knowledge of an expertise topic. Let Desc(t) be the set of descendants of a node t, then the Area Coverage score C(i, t) is defined as:
C(i, t) = |{t
0∈ Desc(t) : t ∈ p(i)}|
|Desc(t)| (4.14)
where p(i) is the profile of an individual i constructed using the method presented in the previous section. In other words, Area Coverage is defined as the proportion of descendants of a query that appear in the profile of a person. Area coverage is larger than zero only for topics that have more than one descendant, therefore this measure does not contribute to finding experts for specialised topics that appear as leaves in a topical hierarchy.
Finally, the score REC(i, t) used to rank people for expert finding is defined as follows:
REC(i, t) = RE(i, t) · C(i, t) (4.15)
This score combines several performance indicators, measuring the expertise of a person based on the relevance of an expertise topic, the number of documents about the given topic, as well as his depth of knowledge of the field, also called Area Coverage.
4.3
Experimental setup
In this section we present the experimental setup used for an empirical investigation of research question RQ2. This research question is concerned with extracting and applying topical hierarchies to Expertise Mining. We discuss our choice of evaluation metrics and we present three datasets annotated for Expertise Mining in Section 4.3.1. The benchmarks used in our experiments are described in Section 4.3.2.