UNA NUEVA MIRADA SOBRE EL PODER

ysis

In this section we described a comparative investigation of the retrieval behaviour of di↵erent systems for an Internet video dataset. These experiments add to the evidence that segmentation of the content plays the main role in retrieving the relevant content. When the segments have high recall and precision, and the rest of the segment belongs to the same topic, all ranking methods tend to rank relevant items on the top of the list. Also, textual metadata can be useful when the segment (with high recall and non relevant content, or with low recall) would otherwise be ranked low in the retrieved list.

6.5

Summary

In this chapter we overviewed SCR experiments that serve as examples of trends across multiple types of content and retrieval tasks: precision and recall of the relevant content within the segmentation unit play an essential role in its efficient retrieval, no matter what ranking scheme is used as the IR model.

A detailed example on the AMI data that compares the changes in the ranking of a number of relevant segments showed that the ASR errors exerts more complicated influence on averaged retrieval behaviour than only decreasing the ranking of the relevant segments. As the segments containing nonrelevant content may be a↵ected

by ASR errors, they may be moved lower in the result list, thus allowing segments with relevant content that would otherwise be lower to move up within the list.

These findings define the strategies of the retrieval experiments that are intro- duced in the following chapters: to target higher ranking of relevant content through segmentation adjustement in terms of relevant content precision, to explore the seg- ment combination to improve ranking for each individual query, and to investigate the possibilities of document expansion that can help to deal with ASR transcript errors.

Chapter 7

Filtering of Overlapping Ranked

Results and Segment Boundary

Adjustment

Following on from the findings and analysis in the previous chapter, in this chapter we report experiments that aim to examine whether the use of overlapping segments combined with subsequent filtering approaches adjusted to each query, can improve overall ranking of relevant segments and proximity to their ideal jump-in points. We also investigate whether these boundaries can be further adjusted at a more detailed level using additional features such as the position of pauses or certain words within the sentences.

We run SCR experiments on two datasets: the NTCIR-9 SpokenDoc collection and our AMI corpus collection, since they both have ASR and manual transcripts that enable us to compare evaluation of the real case (ASR transcripts) and ideal case (manual transcripts). Both these datasets are structured in a way that enables us to carry out segmentation in units of fixed length with an overlap step.

The NTCIR collection is split into interpausal units (IPUs) (Akiba et al., 2011) that can be regarded as sentences for dataset segmentation, and we evaluate the results using IPU-based metrics (uMAP, pwMAP, fMAP), introduced in Section

5.2.1.

Since the AMI corpus collection is provided with a transcript that includes time stamps for words, but lacks sentence punctuation, we carry out the dataset seg- mentation in terms of temporal units. The presence of word boundary information permits further experiments that evaluate closeness to the actual jump-in point in the retrieved segment with imperfect start time. Thus we explore the use of pauses and energy peaks in the audio to help to identify potential ideal jump-in points for the relevant content. As lexical cohesion based methods are based on the actual con- tent of the transcripts, we investigate another approach to adjust the boundaries of segments of fixed length by using the boundaries for the segments in this region that are created using the lexical cohesion based segmentation methods. Since we base segmentation and boundary adjustment on the time information, at the evaluation stage we use not only MAP, Section 3.1, but also such time-based metrics as mGAP, introduced in Section 3.2.3, and MASP and MASDWP introduced in Section 5.2.3. An overview of filtering methods and boundary adjustment, that will be further described and discussed in this chapter, is shown in Table 7.1. The main idea is to start with di↵erent types of initial segmentation and to carry out post-processing of the retrieval results by: i) filtering the result list, in cases when initially the collection is represented with overlapping segments, and ii) adjusting the result seg- ment boundaries using various sources of information (other types of segmentation, acoustic information about pauses or loudness). Each line in Table 7.1 represents potential segment ranking and boundary adjustments that are implemented for the initial segments, e.g. ‘time overlap’ means that all the content is segmented in units of the same length in seconds, and ‘+’ in each column means that the results of the run are modified first by one of the filtering approaches using removal or combina- tion, and second using boundary adjustment.

Target: Target: to improve closeness to JP Initial to improve ranking (Boundary adjustment for JP) segmen Filtering Lexical Use of Use of

tation overlap overlap cohesion pauses loudness type removal combined based first longest (energy

“RemSeg” “CombSeg” peaks)

lexical cohesion – – – + + + based len – – + + + + len nsw – – + + + + time overlap + + + + + +

Table 7.1: General framework of the experiments that target ranking improvement via filtering and jump-in point (JP) closeness via boundary adjustment.

7.1

Filtering approaches for segmentation with

sliding window

Previous research has shown that the use of sliding windows to segment a speech transcript into segments that are close to the length of the target segments with further filtering of the overlap in the result list can achieve higher evaluation scores (MAP, mGAP, MASP, IPU-based metrics) as compared to runs based on lexical cohesion or turn of speakers (Wartena and Larson, 2011; Wartena, 2012; Akiba et al., 2011). This result is achieved when the target units are of approximate known length (e.g. where Rich Speech Retrieval relevant items were known to be no longer than 60 seconds, methods could be tuned to this value (Larson et al., 2011; Wartena and Larson, 2011)), therefore segmentation with overlapping windows into units of this tuned length enables creation of segments with higher internal precision, and recall of relevant content which increases the possibility of these segments being retrieved higher in the result list in a subsequent retrieval phase, as discussed in Chapter 6. However, in the case where the queries require varying amounts of information to be retrieved, as they di↵er in the level of specificity, this approach appears less useful, since it is impossible to define what size the segments and sliding window should be to achieve the same level of retrieval e↵ectiveness across a set of queries.

Figure 7.1: Segmentation adjustment per query within qrel after type RemSeg fil- tering approach (removal of overlapping segments at the lower ranks)

Figure 7.2: Segmentation adjustment per query within qrel after type CombSeg filtering (combination of overlapping segments into longer single ones)

When fixed length segments with a sliding window are used for collection segmen- tation, we increase the probability of creation of segments with a level of precision and recall of the relevant content within each segment being high enough to help the segment to be retrieved at high rank. At the same time as these segments are of the same length across the collection, they might be used more as anchors that define the region of potential relevance with its boundaries to be adjusted individually for each query. With this assumption we implement two filtering approaches that target either the creation of a result list that contains the regions of relevant information, but does not contain an abundance of long non-relevant content, or one that targets finding regions of relevance that are longer than the initial retrieval units, and hence may contain increased amounts of non-relevant content in addition to the relevant material. These filtering methods and their evaluation are described in the following sections.