Una interpretación a partir del testimonio de Pantaleo

With the boom of digital technologies, large collections of digital videos are increasingly available for people to download and use. The tremendous volume of digital videos, in turn, requires effective and efficient access to those videos. It has been demonstrated by a number of usability studies (Ding et al., 1999;

Yang et al., 2003;Wildemuth et al.,2002,2003) and some real video retrieval and search engines (e.g., Open Video, Internet Archive) that people can quickly make sense of videos by viewing the abbreviated video surrogates. Surrogates facilitate faster lookups of and access to video collections.

Yang et al. (2003) classified video surrogates into two modalities: textual andaudiovisual. In fact,audio andvisual can be further divided into two sep- arate modalities. Better understanding the role of the individual modalities and how to integrate multiple modalities so as to facilitate better video search- ing and browsing is crucial to the success of a digital video library. Hence, it is important to learn how people react to, perceive, conceive, and integrate these different modalities.

For the visual-based surrogates, people make sense of the videos through understanding the images. The previous sections have already discussed how people perceive images, and these models and theories also apply to how people make sense of the videos through visual surrogates.

2.1.2.1 Rephrasing the literal

Ding et al. (1999) designed an exploratory usability study to compare three types of video surrogates – visual (keyframes), verbal (keywords/phrases), and

visual and verbal combined.

After viewing the surrogates (i.e., keywords, key frames, or both), partic- ipants were asked to write 2-3 sentences that summarize what the video clip was about. It was found that participants tended to make up a sentence to include all keywords they saw in the surrogates or tried torephrase them. The participants also tended to usespecificterms with iconographic concepts such as names, location, and means to summarize the video. The same rephrasing pattern was also reported bySong and Marchionini (2007).

In addition, the summaries they wrote by viewing the video surrogates were often people oriented (Ding et al., 1999), consistent with Massey and Bender (1996). It seems to be easier to make a story about people, hence participants tended to make a story and put a particular person at the core of the story, which may or may not match the video content.

2.1.2.2 The Effects of Individual and Multiple Modalities

Surrogates of different modalities contribute unique values. Christel et al.

(1997) found the visual-based representations (i.e., poster frame) of the video documents led to far faster location of the relevant video than the text-based ones. Goodrum (2001) reported that visual-based surrogates support higher congruence in similarity judgments than do text-based ones, while for specific queries, the text-based surrogates force higher congruence in utility judgment than the visual-based ones. Yet many years of TREC video results in the past have demonstrated that linguistic data generally leads to better performance in video retrieval than visual features (Smeaton et al., 2004).

According to the research on information objects involving more than one modality (e.g., full motion videos), if the information from different modali- ties is well integrated, the modalities reinforce rather than interfere each other and may lead to increased usability and comprehension. The redundant infor- mation from different modalities provides cross-references to the target to be understood (Pryluck, 1976), and offsets weaknesses of one modality with the strengths of another. Ding et al.(1999) confirmed these benefits of combining multiple modalities (i.e., text and image), and found that redundant infor- mation simultaneously perceived through the two modalities actually sped up processing time.

The advantages of integrating multiple modalities in full video objects also apply to consuming condensed video surrogates. Ding et al.(1999) found that users viewed key frames and keywords sequentially and selectively: They may first look at key frames or keywords as a whole, then switch to the other modality. The textual surrogates such as keywords or short abstracts are often used by people to set up the baseline of the story. The visual surrogates such as key frames are often used to reinforce, confirm, and adjust the story. The study concluded that users strongly favor the combined surrogates, in which each modality (i.e., verbal, image) makes a unique contribution to the comprehension of a video, and in combination they reinforce each other.

The unique values and additive effects of the textual and visual surrogates were also confirmed by other studies (Wildemuth et al., 2002; Hughes et al.,

2003). Wildemuth et al. (2002) reported that participants used keywords to understand the content of the video, as advance organizers for viewing the

visual portion of the surrogate, and as a source of ideas for terms to use in future searches. They commented that textual video surrogates can facilitate the process of determining relevance, and non-textual video surrogates can effectively complement textual surrogates. Hughes et al. (2003) conducted an eye-tracking study and demonstrated that text dominates how people make sense of retrieval sets, while images add confirmatory value. Note that these studies concentrated on text and visual modalities, which both fall into the visual sensory modality. In a more recent multi-modal surrogate study, Song and Marchionini(2007) compared the effectiveness of three different surrogates – visual alone (a storyboard), audio alone (spoken description), and visual and audio combined (a storyboard augmented with spoken description) – for mak- ing sense of digital video, and showed that combined surrogates that employ both visual and audio modalities are more effective, strongly preferred, and do not penalize efficiency. According to the study participants, the audio and vi- sual reinforced each other. “With the two (modalities) together, the surrogate is more efficient, and understanding the surrogates becomes simpler than when they are apart.” In particular, the audio (spoken description) carries semantic information in video, “gives you a concrete outline of what is going on in the video”, while the visual (storyboard) anchors the content, aids memory, and motivates people to watch the video.