Phrases spring readily to mind. They are more specific than words and therefore less likely to be ambiguous (Lewis and Croft, 1990). For example, clustering is a general term that can be further focused by specifying the object being clustered: text clustering, image clustering, query clustering, and search result clustering; or the type of method it uses: hierarchical clustering, flat clustering and partitional clustering; or the characteristics of the method: incremental clustering and itera- tive clustering; and so on. Two texts that both mention the same phrase, such as query clustering, are more similar than those that only share the word clustering. Phrases also help to solve the ambiguity problem. For example, plane has several possible meanings, while plane ticket and Euclidean plane are unambiguous.
Phrases (including n-grams) have been extensively investigated to supplement or replace words in text categorization (Caropreso et al., 2001) and information retrieval (Zhai et al., 1997). In the context of clustering, utilization of phrases can be categorized into two types, as we discuss below.
Phrases as alternative features
The first type uses phrases as alternatives to words. Texts are represented by several models; for example, one with words and another with phrases. Similar- ity values computed from each model are combined to give the overall similarity between two texts. For example, Hatzivassiloglou et al. (2000) investigate noun phrase heads and proper names as features, and combine three similarities: one based on the former, another on the latter, and the third on traditional word vec- tors, all calculated using the cosine measure of vector similarity (see Section 2.2). Their experimental results show that considering linguistic features improves the overall quality of the resulting text clusters and benefits the subsequent task, which in their case is topic detection and tracking.
Furthermore, phrases provide more information about how similar the topics in two texts are. For example, texts tend to be more similar when they share longer phrases. Based on such considerations, Hammouda and Kamel (2004) specifically design a similarity measure for phrases, which combines four aspects with an ad hoc formula: the number and the lengths of matching phrases, their number of occurrences and their significance level in both texts (e.g., phrase matches in titles are more significant). The resulting similarity is then combined with the cosine similarity of the word vectors. Their experiments with the hierarchical agglomerative clustering algorithm (see Section 2.3) also suggest that including phrases benefits clustering: they achieve a 29% improvement over the bag-of- words model on a subset of the 20Newsgroup collection (see Section 3.1 for a description of the collection).
Phrases as cluster labels
Based on the observation that phrases make better cluster labels than words, the second approach for utilizing them prioritizes identifying phrases that are likely to be good labels and then groups texts around them (Zamir and Etzioni, 1998; Pantel and Lin, 2002; Hammouda and Kamel, 2004; Zeng et al., 2004; Stefanowski and Weiss, 2003). This is called descriptive clustering (Stefanowski and Weiss, 2003) or description-centric clustering (Carpineto et al., 2009), to highlight its underlying motivation: to generate high quality descriptions of the resulting text
2.1. TEXT REPRESENTATION
Figure 2.1: A clustered search results of the query clustering
clusters. Descriptive clustering dates back to at least Zamir and Etzioni (1998)’s work on suffix tree clustering, which uses the suffix tree data structure to index and organize texts by the phrases they mention.
Good labels are particularly important if the clusters are to be presented to users, and much research focuses on the task of clustering search results. Various techniques such as machine learning (Zeng et al., 2004) and matrix decomposition methods (Stefanowski and Weiss, 2003) have been explored for finding discrim- inative phrases in a given text collection, and have been shown to be effective. For example, Stefanowski and Weiss’ method has become the foundation of the
clustering search engine Carrot1 shown in Figure 2.1. The cluster labels—such as
cluster analysis and storage clustering in Figure 2.1—are identified as the most dominant topics mentioned in the returned search results, using the singular value decomposition technique.
Discussion
The effectiveness of phrases suggests that words are probably not the best features for describing a text’s topics. The main reason for preferring phrases over words is that they are semantically more specific and less likely to be ambiguous. Their success indicates that the ambiguity of words reduces the effectiveness of the orthodox bag-of-words paradigm in clustering.
However, phrases are by nature merely sequences of words (or text fragments for languages like Chinese and Arabic), so they can be ambiguous. For example, access point usually refers to the device to connect to a wireless network, yet it can also mean a rocky point on the Anvers Island of Antarctica. Unless external concept systems are consulted, they provide no semantic information such as the relations between two phrases. Furthermore, they are sparser than words: the number of matching phrases is usually much smaller than the number of matching words (Lewis and Croft, 1990). All these limitations restrict phrases from being a satisfactory solution to the problems in the bag-of-word model.