In all experiments only the ‘Match’ category is considered as a positive match for a synset. Related and part-of articles are considered as a ‘no match’. This
imposes strict standards for the evaluation since even closely related articles are not considered as correct mappings. Also at most one article is selected for each synset as the correct mapping, even though other close matches may exist.
Using this approach collapses the categorisation to a simple binary case of match or no-match. Once the categories are collapsed into the binary case this way, the annotated set can be considered to be an instantiation of the gold standard matchideal : synset → article function (defined in Section 3.1) for the 200 synsets in
the 200NS set.
The candidate articles generated from the methods in Section 3.3 are evaluated in terms of recall against the gold standard data. The mapping methods in Section 3.4 and Section 3.5 are evaluated using precision, recall, accuracy and F1 measure against the gold standard data. These are standard metrics which have been widely used in language processing evaluations (Olson and Delen, 2008). The specific method of application of the metrics for the mapping methods over the gold standard data is explained in this section.
4.2.1 Evaluating recall for candidate articles
The aim is for the candidate article sets retrieved using the methods of Section 3.3 to include the correct matching article (where there is one). The candidate sets are evaluated in terms of recall against the 200N S set. Let 200N Smatch be the set of
126 synsets where a matching article was found.
To help give a better comparison of the different candidate retrieval approaches, each output candidate set is ranked in order, with those most likely to be matches occurring at the start of the list. This results in an ordered sequence candseq rather
than the unordered set of articles cand for each synset. This then allows measurement of recall up to the ith value in each sequence, a standard metric for recall at n (Baeza- Yates et al., 1999).
recallseq(i) = P s∈200N Smatch scoreseq(s, i) |200N Smatch| = P s∈200N Smatch scoreseq(s, i) 126 (4.1)
where the scoreseq function is defined as:
scoreseq(s, i) =
1 if ∃x. x ≤ i ∧ candseq[x] = matchideal(s)
0 otherwise
(4.2)
where candseq[x] is the xth article in the sequence.
This recall metric can be illustrated using a miniature gold standard example, 5N S. Consider 5 synsets [p, q, r, s, t] with correct mappings respectively [a, b, c, d, null]. Given the following candidate sequences:
candseq z }| { Synset 1 2 3 Correct p x y a a q b x y b r x c y c s x d y d t x y z null recallseq(1) = 1
4 = 0.25 since only 1 correct match (b) is correctly found in column 1.
recallseq(2) =
3
4 = 0.75 because 3 matches (b, c, d) are found in columns 1 and 2.
recallseq(3) =
4
4 = 1 since all 4 matches (a, b, c, d) are found in the 3 columns. Here the null mapping for t is irrelevant since we are only looking at recall of positive matches.
4.2.2 Evaluating mappings
The aim of this evaluation is to compare automatically generated mappings against the gold standard. Where there is a good matching article, the mapping method should identify this correctly. The mapping method should also correctly identify when no mapping (or null) is found for a synset.
This section describes how standard evaluation metrics for a given match function are calculated over the 200N S sample set. Given a particular match function, the precision metric indicates what fraction of the labelled positive (i.e. non-null) mappings are correct. In contrast the recall metric indicates what fraction of the positive mappings from the gold standard data are correctly identified by the match function. The precision and recall metrics are averaged in the F1-measure to give an overall measure of performance. Finally the accuracy metric simply identifies what fraction of the mappings in the match function are correct.
Firstly we can identify those labelled positive by match within 200N S as matchpos:
matchpos = {s|s ∈ 200N S ∧ match(s) 6= null} (4.3)
Then we can identify the true positives using the gold standard matchideal
matchtp= {s|s ∈ matchpos∧ match(s) = matchideal(s)} (4.4)
This allows us to find the precision of the match mapping as the number of true positives divided by the number of those labelled positive by match:
precision = |matchtp| |matchpos|
(4.5) To find the recall we require the number of positive matches in the gold standard. We can reuse the 200N Smatch from the previous section, giving recall as the number
of true positives divided by the number of gold standard positives:
recall = |matchtp| |200N Smatch| =
|matchtp|
126 (4.6)
This is similar to the recall metric for the candidate articles, except there is at most one article present in the mapping, which simplifies the calculation. To combine precision and recall we can then use the standard F 1 measure:
F 1 = 2 · precision · recall
precision + recall (4.7)
Finally for accuracy we require the correct mappings found in 200N S (including null values). This is defined as:
matchcorrect= {s|s ∈ 200N S ∧ match(s) = matchideal(s)} (4.8)
This gives the accuracy as the number of correct mappings divided by the total number of synsets in 200N S:
accuracy = |matchcorrect| |200N S| =
|matchcorrect|
We can again illustrate these metrics using the simple miniature gold standard set 5N S, where synsets [p, q, r, s, t] map to [a, b, c, d, null] respectively. Consider the following mappings:
Synset Match Correct
p a a
q b b
r x c
s null d
t null null Then we have precision = |matchtp|
|matchpos| = 2
3 = 0.67, since there are 3 positive labels (a, b, x) of which two are correct.
Then recall = |matchtp| |5N Smatch| =
2
4 = 0.5, since there are 4 positive instances (a, b, c, d) in the gold standard set, of which 2 are correctly identified.
From this F 1 = 2 · precision · recall
precision + recall = 0.57.
Finally accuracy = |matchcorrect| |5N S| =
3
5 = 0.6, because there are 3 correct matches out of the 5 instances.