La prensa española tras la Revolución del

To assess the quality of a system several statistics can be collected. Suppose entity a belongs to class A and b does not belong to class A.

True positives. A true positive event is observed when the system classifies a correctly as to belonging to class A.

True negatives. A true negative event is observed when the system classifies a correctly as not belonging to class A.

False positives. A false positive event is observed when the system classifies b as to belonging to class A.

False negatives. A false negative event is observed when the system classifies a as not to belonging to class A.

relevant entities non relevant entities identified True Positives (TP) False Positives (FP) not identified False Negatives (FN) True Negatives (TN)

Table 2.28: Contingency matrix for the outcomes of a term identification system. This shows all possible outcomes of a system which has the task of identifying relevant entities in a text.

All outcomes of a term identification system can be assigned to one of the four possibilities in the contingency matrix in table 2.28. Any classification algorithm aims at maximizing the true positives and the true negatives.

Recall (or sensitivity) is the measure of the ability of an algorithm to extract all relevant entities.

recall = number of relevant entities extracted total number of relevant entities in the abstract =

T P T P + F N

A high recall indicates that most of the relevant entities were found. In contrast, a low recall indicates that the majority of the relevant entities were missed by the algorithm. Recall 100% means that every relevant entity was found. In theory, it is easy to achieve

good recall values: All entities for every document simply have to be returned. Recall by itself is not sufficient to characterize the effectiveness of an algorithm.

Precision measures of the ability of an algorithm to retrieve only relevant entities.

precision = number of relevant entities extracted total number of entities extracted =

T P T P + F P

Precision is a measure how well an algorithm performs in avoiding to return non relevant entities. A precision of 100% is reach when every entity extracted is relevant. There is no trivial way to achieve a 100% precision other than returning no term at all for every document.

F-measure combines recall and precision in a single efficiency measure.

F = 2 · P recision · Recall P recision + Recall It is the harmonic mean of precision and recall.

In Baldi et al. [17] the authors point out that precision and recall are not only properties of the evaluated system but also depend on the frequency of positive examples in the eval- uation corpus. Consider a corpus that contains 1000 positive examples and 1000 negative examples. A system assigns 90% of the positive examples and 10% of the negative examples as relevant. The system has a precision of 90% and a recall of 90% (TP=900, FP=100, TN=900, FN=100). Evaluating the system on a corpus with only 100 positive examples and 1900 negative examples the same system has a precision of only 32.1% and a recall of 90% (TP=90, FP=190, TN=1710, FN=10). Evaluating the system on a corpus with only 1900 positive examples and 100 negative examples the same system has a precision of 90% and a recall of 99.4% (TP=1710, FP=10, TN=90, FN=190). As a conclusion of this two systems can only be compared when they are evaluated on the same corpus.

To compare the efficiency of the algorithms, a recall-precision graph can be used, as described in the Text REtrieval Conference (TREC) quality benchmark [119].

Precision and recall are set-based measures. They evaluate the performance of an al- gorithm on an unordered set of retrieved items, e.g. terms in this case. However, they do not account for the quality of ranking the extracted terms. The better the algorithm is, the more relevant terms are among the top items in the hit list. The relevance of a ranking can be measured by computing the precision at different cutoff values, precision at n. In a list, where the first three terms are relevant and the fourth and fifth are non relevant, one gets a precision of 100% for the first three and a precision of 60% for all five terms.

Figure 2.14 shows an interpolated precision graph for a list of 20 terms extracted from a document. The first, second, third and fifteenth terms in the hit list are relevant terms. The graph is plotted at 11 standard recall levels r = (0.0, 0.1, ..., 1.0). The exact recall points are 0.25, 0.5, 0.75 and 1.0. That means that after retrieving 50% of the total number of relevant terms, two of four terms in this case, the precision is still 100%. At recall level 10 (90% of the relevant terms retrieved), the precision is 27%.

Recall-precision Graph is a way of displaying the precision at n, as one goes through the list of extracted terms. It plots average interpolated precision numbers against per- centage recall for all processed documents. Recall-precision graphs usually have a concave shape. Trying to increase the recall typically gives more and more false positives in the list,

Figure 2.14: Interpolated term extraction precisions for one article. The dotted line marks the interpolated values.

the precision decreases disproportionally. The area under the recall-precision graph is the average precision.

precision(recall = r) = max(precision(recall ≥ r))

Recall-precision averages table is computed by summing the interpolated precisions at specific recall levels taken from the interpolated precision graphs for all processed documents. The values of the recall-precision table are later used to draw the recall-precision graph.

Average precision and average recall do not measure the average of the precision and recall values at standard levels, they rather evaluate the averages of the precision and recall values, obtained after each relevant term is calculated.