La evaluación ex ante: la mirada de Neirotti

Without having the detailed knowledge of collection make-up and of the retrieval environment, most users find it difficult to formulate queries which are well designed for the purpose of retrieval. The observation of web search engines showed that users often make modification to their initial queries [Spink et al., 2002]. The first query should be considered as a mere guess [Efthimiadis, 1996].

There are a number of approaches that can help the users in such situations when their queries are imprecise. These include non-interactive and interactive methods for query expansion. We can contrast the two methods based on level of user involvement. Non-interactive methods work without the intervention of users and expand the query at the algorithm level, while in the other case, lists of terms are suggested to users and they can recognise and choose the terms deemed more relevant to their task at hand.

2.3.1 Related terms

Term relationships can be established from a number of different resources either at the global or local level. The global approach refers to the computation of a term-term relation con- sidering all the documents from the entire corpus while the local approach is restricted to the initial retrieved set of documents in response to the query [Attar and Fraenkel, 1977]. [Xu and Croft, 2000] incorporated the ideas from the global analysis into the local analysis approach.

Conventional approaches for term-term similarity are based on statistical measures such as e. g. co-occurrence frequencies, mutual information and chi square. There are a variety of ways to estimate the word occurrences in a text, by considering complete documents, passages, sentences or fixed-sized window [Terra and Clarke, 2003]. [Sanderson and Croft, 1999] ex- tracted terms and built the concept hierarchies from search results and used term co-occurrence to compute the term-term relationship.

New alternative approaches of term suggestion identify relevant query terms in collected logs of user queries [Beeferman and Berger, 2000].

2.3.2 Relevance feedback

Relevance feedback — explicit and implicit — has been shown to be an effective technique for improving retrieval results ([Salton and Buckley, 1990] [Harman, 1992] [Buckley et al., 1994] [White et al., 2004]).

Relevance feedback techniques require obtaining relevance information about the results re- trieved and presented to searchers. These techniques use feedback to re-weight the query terms for query modification.

Initially relevance feeback was thought of being user-directed where the user has to mark the documents that are found relevant to her information need at hand. Later, this notion is expanded to a bi-directional process where both the system and the user respond to one another in interactive IR [Spink and Losee, 1996].

Empirical studies showed that interactive IR systems users desire explicit relevance feedback features [Belkin et al., 2000]. However, much of the evidence indicated that relevance feed- back features are under-utilised [Belkin et al., 2001a].

The study [Koenemann and Belkin, 1996] showed that better retrieval results can be achieved when users have full control over the query modification process based on relevance feedback.

Implicit feedback

Implicit feedback techniques unobtrusively infer information needs from the search behaviour, and can be used to individuate system responses and build models of system users. As a major application area, implicit feedback techniques have been developed for recommender and filtering systems.

There are a number of behaviours that have been described in the literature as potential rele- vance feedback indicators. [Nichols, 1998] developed a classification scheme of observable behaviours as shown in figure 2.3.2, with a focus on its use in information filtering systems. He presented a list of potentially observable behaviours; adding purchase, assess, repeated use, refer, mark, glimpse, associate, and query to those mentioned above.

[Oard and Kim, 2001] extended the work, organising observable behaviours along two axes: The behaviour axis refers to the underlying purpose of behaviour. It is further sub-divided into four broad categories: examination, retention, reference and annotation

Examine is where a searcher studies a document, and examples of such behaviour are view

(e. g. reading time), listen and select.

Retain is where a searcher saves a document for later use and examples include bookmark,

save and print. Further examples of keeping behaviours on the Web, where information is retained for later re-use, Reference behaviours involve users linking all or part of a document

Figure 2.3: Classification of behaviours that can be used for implicit relevance feedback

to another document and examples include reply, link and cite.

Annotate are those behaviours that the searcher engages in to intentionally add personal value

to an information object, such as marking-up, rating and organising documents.

The horizontal axis: “Minimum Scope” refers to the smallest unit associated with the be- haviour. A Segment level includes operations whose minimum scope is a portion of an object (e. g. a paragraph is a segment of a document). Objects are self-contained items (e. g. docu- ments). A Class is a group of objects (e. g. a collection of index documents.)

This table continually evolves as new behaviours are added, with the most recent addition being the create behaviour added by [Kelly and Teevan, 2003]. Much of the current research is concentrating on the examine and retain categories.

InfoScope, a system for filtering Internet discussion groups (USENET), investigated the use of implicit and explicit feedback for modeling users [Stevens, 1993]. Three sources of implicit evidence were used: whether a message was read or ignored, whether it was saved or deleted, and whether or not a follow up message was posted. Stevens observed that implicit feedback was effective for tracking long-term interests.

back. Their results showed a strong positive correlation between reading time and ex- plicit relevance given. When treating messages as relevant that the user read for more than 20 seconds, this produced better recall and precision than with explicit rating by the user. [Konstan et al., 1997] repeated this study in a more natural setting. Their results indicated that recommendations based on reading time could be nearly as accurate as recommendations based on explicit feedback. They also suggested some additional observable behaviours as sources for implicit ratings namely printing, forwarding, and replying privately to a message. [Claypool et al., 2001] categorised a series of different interest indicators and proposed a set of observable behaviours that can be used as implicit measures of interest. The researchers found a strong positive correlation between time and scrolling behaviours and the explicit ratings assigned. However, since subjects were not engaged in a search task (just asked to browse a set of interesting documents), the applicability of the findings to information seeking scenarios is uncertain.

[Goecks and Shavlik, 2000] measured hyperlinks clicked, scrolling performed and processor cycles used to unobtrusively predict the interests of a searcher. They integrated these measures into an agent that employed a neural network and showed that it could predict user activity and build a model of their interests that could be used to search the Web on their behalf.

[Joachims et al., 2007] examined the reliability of implicit feedback generated from click- through data and query reformulations in World Wide Web (WWW) search. Results showed that clicks are informative but biased. It is difficult to interpret clicks as absolute relevance judgements. Relative preferences derived from clicks are reasonably accurate on average. They found that relative preferences are accurate not only between results from an individual query, but also across multiple sets of results within chains of query reformulations.