The Fischlar-Simpsons video retrieval application, as the name suggests, facilitates
browse and retrieval on Simpson’s content. In Chapter 3 we discussed object extraction explaining that animated content is the best choice for evaluation due to the nature of the content i.e. a limited number of specific colours with generally well defined object boundaries. As we explained in the previous section, “The Simpsons” is one of the most popular animated shows in the world with over 340 episodes (136 hours of content). For these reasons it was decided to select “The Simpsons” as the content for retrieval evaluation.
ER. performance evaluation requires a representative selection of content and in the case of the Fischlar-Simpsons experiment it needs a selection of representative episodes. The general procedure for creating an experimental corpus is to select a quantity of representative content and divide it into two segments, a development corpus and evaluation corpus. The development corpus is used to gauge the performance of the retrieval features and system during the development cycle, while the evaluation corpus is used for the actual experiments and results. The main reason for using a separate corpus for development and for evaluation is to avoid training on a specific corpus and producing artificially high results.
The first task for building a corpus is obtaining the content and in the case of the Simpsons a large quantity of video content is available on DVD and VHS making it easily available. Content on DVD is the medium of choice due to the availability of closed captions (subtitles) and the video’s high quality.
At the time writing, three seasons of content was available along with a number of special themed collections*. As we discussed in the previous section, the first season of the Simpson’s feature a different animation style to later seasons and for this reason it was not considered for inclusion in our corpus.
Table 5-1 shows all 12 episodes (4.8 hours, 6,525 shots) included in the development corpus, episode number is the order used by the system while broadcast number is the actual order of the episode.
The two main considerations for inclusion of content in the development corpus are that the episodes contain appropriate training examples for the object detection (discussed in Section 5.1.3) and secondly that the corpus contains an appropriate quantity of episodes for low-level feature and system evaluation.
Table 5-1: Development Corpus Content
Episode
Number
Episode Name Broadcast
Number
Season
1 Bart gets an F 14 2
2 Simpson’s and Delilah 15 2 3 Bart V's Thanksgiving 20 2
4 Bart the Daredevil 21 2
5 Itchy & Scratchy & Marge 22 2 6 Bart gets hit by a car 23 2 7 One fish, two fish, blowfish, blue fish 24 2
8 The way we was 25 2
9 Principal Charming 27 2
10 Brush with Greatness 31 2
11 Lisa's Pony 43 3
12 Separate Vocations 53 3
* The Themes featured four similar episodes from throughout the seasons and included collections like Bart at War, Halloween Specials and Simpsons go to Hollywood
Table 5-2 shows the 52 episodes (20.8 hours, 20,529 shots) that make up the complete evaluation corpus. While most of the content comes from the 2nd and 3rd seasons the episodes from 13 to 29 were obtained from special themed DVDs which came from seasons 4 to 12.
Table 5-2: Evaluation Corpus Content
Episode
Number
Episode Name Broadcast
Number
Season
1 Two cars in every garage and three eyes on every fish
17 2
2 Dancin Homer 18 2
3 Dead putting society 19 2
4 Tree House of horror 16 2
5 Homer Vs. Lisa and the 8th Commandment
26 2
6 Oh Brother, Where Art Thou? 28 2
7 Bart's Dog Gets an F 29 2
8 Old Money 30 2
9 The Substitute 32 2
10 The war of the Simpsons 33 2 11 Three men and a comic book 34 2
12 Blood Feud 35 2
13 Simpsons Halloween 5 86 5 14 Simpsons Halloween 6 109 6 15 Simpsons Halloween 7 134 7 16 Simpsons Halloween 12 249 12 17 When you dish upon a star 208 10
18 Fear of Flying 114 6
19 Krusty gets Kancelled 81 4
20 Tree House 9 182 9
21 The Cartridge Family 183 9 22 Natural Bom Kissers 203 9
24 Mayored To The Mob 212 10 25 The Secret War Of Lisa Simpson 178 8
26 Marge Be Not Proud 139 7
27 Homer To The Max 216 10
28 The Springfield Files 163 8 29 Lisa The Iconoclast 144 7
30 Homer B adman 112 6
31 Stark Raving Dad 36 3
32 Mr. Lisa Goes To Washington 37 3 33 When Flanders Failed 38 3
34 Bart The Murderer 39 3
35 Homer Defined 40 3
36 Like Father Like Clown 41 3 37 Treehouse Of Horror 2 42 3 38 Saturdays Of Thunder 44 3
39 Flaming Moe's 45 3
40 Bums Verkaufen Der Kraftwerk 46 3
41 I Married Marge 47 3
42 Radio Bart 48 3
43 Lisa The Greek 49 3
44 Homer Alone 50 3
45 Bart The Lover 51 3
46 Homer At The Bat 52 3
47 Dog Of Death 54 3
48 Colonel Homer 55 3
49 Black Widower 56 3
50 The Otto Show 57 3
51 Bart's Friend Falls In Love 58 3 52 Brother, Can You Spare Two Dimes 59 3
The Simpson’s episodes used in the corpus are stored on DVD in high quality MPEG-2 video format and this needs to be extracted and converted into MPEG-1 for image analysis, storage and playback.
The first step requires the extraction of the video data from the DVD onto the computer hard drive, from here the second step converts each episode to MPEG-1 using a transcoding application like Flaskmpeg [Flask].
Extraction of the closed captions presents a more difficult challenge, as the DVD subtitle text is actually stored as a series of images (white text on a black background). Therefore, optical character recognition (OCR) is required to convert the image text into machine-readable text.
Figure 5-1 shows a screenshot from “subrip” a DVD subtitle extraction application which takes the video data and converts the subtitles into a structured machine readable text file. It displays characters that it cannot identify with a red box, and as it is shown the characters it learns and quickly works automatically. The extracted subtitle text and timing information is displayed of at the bottom of the screen and this can be saved to a file.
■ ■ ■ ■
TteaB*ii¿OOjO)úl 23** (720*570)
—I Tkil*u>ti(U>uu Í
SSÍ2Í& Show Piet. P jy ai o._Yi o_.xa o y, a
0® §®ito© ©OFcam e ría Sym bol I n d o * : 4 0 0
Aboií
Skip Thu ino S k jp T h n S u b P ttu io £ h w g o Tfod Cofc* Fill t h i s ( t h e s e ] c h o r n c t e i j t ) . Stvaich l a a n oattirvg ihn
r
✓ flK ■ .Ú '.U V r Bujd p f~ Italic l~ Underfcw t i j EnglishChoi od« Map SubTiUo Index: 74
0O:OS:4O:Haybc t h a t ' s w h a t's wrong w ith me. ; | beul c h e m is try .
O O ; O S ; 4 2 :A ll my problem s and a n x i e t y can b e |r e d u c e d t o a ch em ical im b alan ce . 0 0 : 0 5 : 4 5 : . . . o r some k in d o f |m i s £ i r i n g s y n a p s e s .
0 0 :0 5 :4 7 :1 need t o g e t h e lp f o r t h a t . 0 0 :0 5 :4 9 :B u t I ' l l s t i l l be u g ly , th o u g h .