Our candidate concept selection employs the semantic similarity function for estimating the similarity between terms that are relatively semantically similar in order to reduce the noise. The semantics of keywords are identified through the relationships between keywords by performing semantic similarity on them [Fang et al. 2005] [Andrea et al. 2003] [Varelas et al. 2005] [Bonino et al. 2004] [Khan et al. 2006] [Sayed et al. 2007]. Experiment results show that all the similarity functions improve the retrieval performance, although the performance improvement varies for different functions. We find that the most effective way to utilize the information from WordNet is to compute the term similarity based on the overlap of synset definitions. Using this similarity function in query expansion can significantly improve the retrieval performance.
The WordNet semantic similarity function is used to calculate the semantic similarity between the original selected query terms with the expanded term. The lexical expansion module and the common sense reasoning module come up with the set of the expanded terms. These expanded terms will probably increase the recall but sometimes significantly decrease the precision of the system. Some of them are noises and it will derogate the retrieval performance. However, if the two terms have not that much in common then it will increase the recall at the expanse of precision. This is one of the drawbacks of the query expansion. We have tried to control the bottleneck of selecting some of the most similar concepts by utilizing the candidate concept selection module. The candidate concept selection uses the semantic similarity function to prune the less semantically similar expanded terms. The semantic similarity between the selected candidate terms and the set of lexical and conceptual expanded term is computed. The
Proposed Algorithm 3.2: Common Sense Reasoning Input: 𝑄 → ⋃𝑡 𝐾𝑖
𝑖 // List of selected candidate query terms
Output: 𝑄 → ⋃ 𝐾 𝑖 𝑡 𝑖 Method: i ← Length (Q‟)
threshold has been defined for selecting the list of candidate concepts. The threshold can be set by taking the average mean of all the semantic similarity value of the expanded terms. The expanded terms with the semantic similarity value below the threshold are prune and the rest of them are selected and passed to the retrieval model for retrieving the result according to the expanded query terms.
Semantic similarity can be measured in order to filter the concepts. This will significantly increase the precision of the system. The various semantic similarity measures are discussed below.
3.4.2.1
Semantic Similarity Calculation
Due to the subjectivity in the definition of the semantic word similarity, there is no unique way to compute the performance of the proposed measures. These measures are folded into two groups in [Mihalcea et al. 2006], corpus-based and knowledge-based similarity measures. The corpus-based measure attempts to recognize the similarity between two concepts exploiting the information exclusively derived from large corpora. The knowledge-based measures try to quantify the similarity using the information drawn from the semantic networks.
Knowledge-based Word Similarity Measures
The knowledge-based technique measures the similarity between two concepts employing the information drawn from the semantic networks. Most of these measures use WordNet [Miller et al. 1990] as the semantic network. The similarity between two concepts and two words is not same. Some words have different senses or different concepts. In order to compute the semantic similarity all the sense of the words are considered. The score are assigned to all the sense of words and then select the highest similarity score. Some of these similarity measures use information content (IC) which represents the amount of information belonging to a concept. It is described as
IC(c) = -log (P(c))
(3.3)
Where IC(c) is the information content of the concept c, and P(c) is the probability of encountering an instance of the concept c in a large corpus. Another used definition is the least common subsumer (LCS) of two concepts in taxonomy. LCS is the common ancestor of both concepts, which has the maximum information content.
Leacock & Chodorow Similarity
This similarity measure is introduced in [Leacock. et al. 1998]. The similarity between two concepts is defined as
Sim
lch(c
i, c
j) =log (
)
(3.4)
where ci, cj are the concepts, length(ci , cj) is the length of the shortest path between
concepts ci and cj using node counting, and D is the maximum depth of the taxonomy.
Lesk Similarity
In Lesk measure, [Lesk. et al. 1986] similarity of two concepts is defined as a function of overlap between the definitions of the concepts provided by a dictionary. It is described as
Sim
lesk(c
i, c
j) =
) )) )
(3.5)
Where def(c) represents the words in definition of concept c. This measure is not limited to semantic networks, it can be computed using any electronic dictionary that provides definitions of the concepts.
Wu & Palmer Similarity
This similarity metric [Wu. et al. 1994] measure the depth of two given concepts in the taxonomy, and the depth of the LCS of given concepts, and combines these figures into a similarity score
Sim
wnp(c
i, c
j) =
) ))
(3.6)
Where depth(c) is the depth of the concept c in the taxonomy, and LCS (ci, cj) is the LCS of
the concepts ci and cj .
Resnik Similarity
Resnik similarity measure [Resnik et al. 1995] is defined as the information content of the LCS of two concepts
Sim
res(c
i, c
j) = IC (LCS (c
i, c
j) )
(3.7)
Lin’s Similarity
The key idea in this measure is to find the maximum information shared by both concepts and normalize it. Lin’s similarity [Lin et al. 1998] is measured as the information content of LCS, which can be seen as a lower bound of the shared information between two concepts, and then normalized, with the sum of information contents of both concepts. The formulation is as below
Sim
lin(c
i, c
j) =
( ) )) )
(3.8)
Jiang & Conrath Similarity
This measure is introduced in [Jiang et al. 1997]. This measure also uses IC and LCS. It is defined as below
Sim
jnc(C
i, C
j) =
) ( ) ))(3.9)
Hirst & St-Onge Similarity
This measure is a path based measure, and classifies relations in WordNet as having direction. For example, is-a relations are upwards, while has-part relations are horizontal. It establishes the similarity between two concepts by trying to find a path between them that is
neither too long nor that changes direction too often. This similarity measure is represented with
Simhso. Detailed description of this method can be found in [Hirst et al. 1998].
The proposed candidate concept selection selects the candidate concepts from the list expanded terms in order to prune the noises. The candidate terms are selected on the basis of semantic similarity between the expanded terms and the original query terms are computed and then selected according to the threshold. The threshold can be computed by taking an average mean of the expanded terms and the query terms. The output of the algorithm is the list of selected candidate terms and these selected expanded terms are then used for further retrieval and ranking. The overall algorithm of proposed module is given below.
Input: 𝑄 → ⋃𝑡 𝐾𝑖
𝑖 // List of original query terms, Synset and ConceptNet
Output: 𝑄 → ⋃𝑡 𝐾𝑖
𝑖 // List of selected query terms
Method:
I ← Length (Q)
// Adding Semantic Similarity Q(i).keyword ← Q(j).keyword J ← Length (Q (i). synset)
Q (i).synSet (j).SS ← WordNet.SemSim (Q(i).SunSet(j).Word) Q (i).ConceptSet (j).SS ← WordNet.SemSim (Q(i).keyword, Q(i).ConceptSet(j).Word)
//Select candidate terms from the SynSet TH ← Q(i).SynSet.AvgMean()
K ← Length (Q(i).Synset) Q‟ ← Q(i).Keyword
If (Q (i).SynSet (k).SS ≥ th)
Q‟ ← Q‟ + Q(i).SynSet(k).keyword //Select candidate term from ConceptSet Th ← Q (i).ConceptSet.AvgMean () H ← Length (Q(i).ConceptSet) IF (Q (i). ConceptSet (h).Keyword