that may be useful for the searcher. For each result list item, the system gives a degree of relevance ranging from 0 to 1. For presentation purposes, the degree of relevance is divided
into four intervals of equal size and each interval is mapped to one colour. The colours assigned to the intervals are light yellow, yellow, light green and green.
The degree of relevance is indicated with the icons in the document table of contents. These icons are divided into two parts showing the colour as mentioned above in the upper part and and one to four orangish squares (for the four degree intervals) in the bottom part (see figures
8.1, 8.2, 8.3and 8.4).
For analysing this issue, the post-search questionnaires contained the following questions: • How useful was the paragraph highlighting feature in assisting you with the task? • What features of the interface were the most and least useful for this search task? The question was answered on a likert scale from 0 to 5 where 0 implied didn’t use the specific feature, 1-2 implied not at all, 3 implied somewhat and 4-5 implied extremely useful. Around 40% considered it as an extremely useful feature while 30% regarded it as somewhat useful, 28% voted 1 or 2 and 8% didn’t notice this feature.
Searchers commented only rarely on this feature; there are a few who commented negatively on this feature, like e. g. “ The paragraph highlighting did not do much for me. I prefer to
search the article myself and in this way find the relevant information.” or “The paragraph highlighting is useless as the highlighted passages are not the relevant passages. Often, only the external links section is highlighted. The interesting passages are not highlighted at all”.
We can conclude that paragraph highlighting for distinguishing between potentially relevant and irrelevant document parts is useful for most, but not for all participants. Therefore one should allow for switching this feature on/off.
Usefulness of resultlist presentation
The resultlist presentation in the element retrieval system uses the captions of document, sec- tion and subsections, whereas the passages retrieval uses a sentence-based query summariza- tion approach whenever needed. In order to investigate which strategy is preferred by the searchers, we analysed questionnaire data and the interaction logs.
After performing each of the tasks, the following two questions are posed about the result list • To what extent did you find the presentation format (interface) useful?
• What features of the interface were the most and least useful for this search task? The question was answered on a likert scale from 0 to 5 where 0 implied didn’t use the specific feature 1-2 implied not at all, 3 implied somewhat and 4-5 implied extremely useful. The analysis shows a slight preference for the passage retrieval system (see figure8.6).
Figure 8.6: Usefulness of Resultlist
Now we want to analyse the searcher’s interaction in the two systems to see whether the length of the surrogate influenced the searcher in viewing document details. The average length of the document/element surrogate in the passage retrieval system is 34 characters while the average length in the element retrieval is 18 characters. Comparing the lengths of relevant vs. irrelevant items, we get 38 vs. 30 for passages and 18 vs. 17 for elements so the surrogates for passages are not only longer than that of elements, there also is a length difference between relevant and irrelevant passages.
The analysis of the open-ended questions about the usefulness of result list presentation re- vealed that searchers found the entry point capability useful which allowed them to jump into the specific part of the document
We can conclude that captions are equally useful in both types of systems, they convey the information to determine the relevancy represented result item.
Query term highlighting
Query term highlighting has been identified as an important feature during the information seeking process [Tombros et al., 2005b], since it makes it easier to locate the interesting infor- mation. For validating this statement, the following two questions were asked:
• How useful was query term highlighting feature in assisting you with the task? • What features of the interface were the most and least useful for this search task? The question was answered on a likert scale from 0 to 5 where 0 implied didn’t use the specific feature 1-2 implied not at all, 3 implied somewhat and 4-5 implied extremely useful. Around 125 users considered it as an extremely useful feature, 105 searchers found it somewhat im- portant feature; only 64 users voted 1 or 2 and 42 searchers didn’t notice.
The content analysis of the open ended question showed that most of the searcher’s com- mented positively on the usefulness of query term highlighting. There were few searchers who suggested that other functions could be more useful than query term highlighting, like the pos- sibility to highlight terms other than the query terms: “highlighting the query in the docs was
of no use here (though being able to highlight other terms would have been).”
The other potentially useful function pointed out was the availability of search functions while examining the details of the fulltext: “No search within documents; Searching tools are re-
quired; lacking feature: no search function to search for terms within long paragraph”