Conclusiones

When designing and updating a summarization system, it is necessary to have a tool or method to track the performance of the system and the changes being made. For summarization, there are two main types of evaluations: Intrinsic and Extrinsic. Intrinsic evaluations are used for evaluating the quality of summaries. They may help in answering questions about a summary such as its coherence, grammar and whether it suggests incorrect information deductions from the original text, or redundant information within the summary itself. Extrinsic evaluation on the other hand helps in answering whether a summary meets the purpose it is generated for. For instance, it can help determine whether a summary is a good replacement of the original documents and conveys the most important information within it.

2.6.1 Intrinsic Evaluations

Intrinsic evaluations base their judgments on the output of the summaries. Human intrinsic evaluations measure the clarity, cohesion and informativeness of a summary [116]. Automatic intrinsic evaluations compare systems summaries with other reference summaries generated by humans. Among the main intrinsic tools used are Precision/Recall, BLEU, ROUGE and Pyramid. The Pyramid requires manual annotation of systems summaries and reference summaries before they can be automatically compared. Precision/Recall, BLEU and ROUGE on the other hand are fully automated and only require reference summaries.

Precision, Recall and F-Measure are among the simplest evaluation approaches available that measure the relevance of a summary by the relevance of the sentences it contains. Precision (P) is the number of sentences appearing in both the system summary and the reference summary divided by the number of sentences in system summary. Recall (R) is the number of sentences occurring in both system and reference summaries divided by the number of sentences in the reference summary. F-Score is a composite combining both P and R [12]. The F-Score can be computed with the following formula:

R P PR F   (1 ₂ 2)   2.2

Whereβ is a weighting variable that is adjustable to affect precision and recall.

The Precision/Recall measure is not without its limitations. Suppose that two persons were asked to choose the top two most important sentences from a document to represent as a summary. It is possible that the human judges could choose two different sentences that carry the same meanings. I can name these summaries A and B. If one of the two human summaries, say A, was used as a reference summary, a system summary producing sentences that exist in summary A and not B would rank highest. If another system produced a summary containing sentences from summary B, the first system would still rank higher.

Bilingual Language Evaluation Understudy (BLEU) [117] is an n-gram precision based system originally developed for machine translation. It works by finding the number of n- grams in the system summary that matches those in a reference summary. BLEU is a precision-based evaluation giving higher scores to system summaries with content that

include incorrect sentences in their summaries. On the other hand, it does not penalize systems that do not include the right sentences in their summaries.

As a response to address the limitations of the BLEU system mentioned earlier, the Recall Oriented Understudy for Gisting Evaluation (ROUGE) was proposed in 2003 [118] at the Information Science Institute. It is roughly based on BLEU but focuses on recall instead. Also, it measures words overlaps in sequences and was found to correlate better with human evaluations than many other systems.

Several variants of ROUGE have been proposed [119]:

 ROUGE-N: counts contiguos n-gram. N ranges from 1 to 4.  ROUGE-L: Longest Common Subsequence (LCS) based metric  ROUGE-W: Weighted LCS favouring sequential LCS

 ROUGE-S: uses skip-bigram: words pairs in sentence order, ignoring gaps  ROUGE-SU: uses skip-bigram and unigram.

In the Documents Understanding Conference (DUC) and Text Analysis Conference (TAC), ROUGE-SU and ROUGE-2 have been the standard metrics for summaries evaluation. With ROUGE-SU, sentences such as “The truck was bought by the company”

and “The company bought the truck” are found to be a match since the order of the terms

is not relevant as long as they are within the same sentence. With ROUGE-S, they would not be a match.

In contrast to ROUGE and BLEU, the Pyramid method [120] requires humans marking and grouping items within summaries. It attempts to address the issue that summaries generated by summarization systems may include information which are related to the topic but not necessarily included in the reference summary. Every reference summary is first annotated with Semantic Content Units (SCU) which are facts or important events at the clause level. The SCUs are then given weight based on their occurrence frequencies

within all the reference summaries and are organized in a Pyramid based on their weights. SCUs appearing near the top of the pyramid are of higher weights and appear more frequently in the reference summaries than SCUs near the bottom. The SCUs that appear in all the reference summaries will be in the top of the pyramid while those appearing in single reference summaries are in the bottom of the pyramid.

The advantage of using the Pyramid method is that, unlike n-gram based methods, it is based on the semantic content of the summaries. However, due to it being labour and time intensive, it is not being used frequently with many systems especially when the summarization system is in a continuous update stage incurring many changes at different stages in time.

For evaluating multi-lingual summaries, a framework that relies on similarity measures was suggested in [121] to evaluate English and Chinese summaries. The framework is capable of evaluating single, multi-document, abstractive and extractive summaries. As for speech summarization, custom measures such as the Summary Accuracy in [122] were proposed. Other measures such as precision, recall and ROUGE were also used in other systems for speech summarization.

2.6.2 Extrinsic Evaluations

Extrinsic evaluations [123] help show how well a summarization system performed in a specific task. They can help measure the relevancy of a summary to a topic or indicate a category for a document. When choosing and implementing a specific task, it is important that the task is clear enough for any person to attempt and answer with high level of confidence. If a task is not clear enough and persons can not agree on the answer, it is not

be used to test systems and evaluate their performance. An example for this is what was performed in the TIPSTER SUMMAC Text Summarization Evaluation [4] in which 16 systems participated. One of the main goals of the evaluation was to judge the usefulness and relevancy of the information presented in the participants summaries.