For a dataset of N images, each image is assigned an absolute score where the set of all the scores is given by s = s1, s2, ..., sN. This ranking is based on the judgements made by an-
notators using the ranking approach outlined in Chapter 3 Section 3.5. An intuitive method to utilize such rankings would be to obtain the nearest neighbour match for an image. Such an approach acts as a brute force method for the ranking problem and therefore does not rely on automatic recognition of attributes performed in Chapter 4. Images obtained after nearest neighbour matching can be compared with their ordering in rankings such as strong-5, weak-5 and all-10 that were obtained in Chapter 3 Section 3.6.5. And this comparison can be used to determine which image is ranked better to perform a comparison of aesthetics within images. In this work two leading approaches of BoV model and LDM are considered to evaluate ranking
Query Image Feature Extraction Feature Vectors Image Database Feature Extraction Feature Vectors Data Structure Nearest Neighbour Search Voting Sorting Database Post- processing Input Output
Figure 5.1: General process of achieving image matching.
based on matching [31, 93, 104, 114, 72]. A brief description of these is given in the following sections. This is done with reference to parameters and settings utilized to test the performance. The BoV and LDM approaches can be seen as variations of the basic algorithm shown in Figure 5.1. Images from the database are presented to the feature extractor and feature vectors are generated which are utilized to construct a data structure. Similar procedure is performed at the query end which results in the query feature vectors. Nearest neighbour search is performed with the aid of the data structure and matches are obtained after sorting the database.
5.1.1 Matching with Bag of Visual Words (BoV)
The query and database images are represented as a sparse vector of visual word occurrences and on search the similarity between the image vector and query vector is estimated [114]. The steps for achieving a BoV representation for describing visual content are shown in Figure 5.2 [62].
5.1. Background 63
The tf-idf weighting scheme has been shown to perform well and is used. If there is a vocabu- lary of w words given by a vector of length w, V = (v1, v2, ..., vw), then each element of this
vector is defined as in Equation 5.1 under the tf-idf scheme. The weighting is a product of two terms, the words frequency, nwd
nd , where nwdis the number of times the word w occurs in d, nd
is the total number of words in d and the inverse document frequency, lognN
w, where N is the
total number of images (or documents in the database) and nwis the number of occurrence of
word w in the complete database.
vw = nwd nd logN nw (5.1) Inverted file: In the inverted list, there is an entry for a visual word followed by the im- age indices of the images in which that visual word is present [114]. Using this technique avoids overheads and comparisons between high dimensional descriptors along with reducing the number of candidate images. This especially offers a speed-up for sparse queries since only database images containing the visual words need to be checked rather than checking all the images. Similarity is found by storing scores for all the relevant images.
Min-Hash algorithm: Efficient matching relies on the chosen representation of an image and the similarity metric used for comparison when retrieving. This makes hashing algorithms a powerful tool for use in large scale search applications. The min-Hash algorithm originates from the text retrieval community where it is used for returning duplicate documents [34]. It has been shown in literature to be effective for visual data due to several reasons. Firstly, it results in a compact image representation with a constant amount of data needed to be stored. It also allows fast search time that is proportional to the number of retrieved matches and can generate accurate results with reasonable probability.
The BoV model represents an image as a fixed length vector of length |V |, where |V | is the size of the vocabulary. Each element (or visual word) of this vector is a representation of the number of features that are encoded by that visual word. A set representation which is a weaker representation is used for min-Hash given by Ai ⊂ V . This uses only limited information of
whether a feature is present or not resulting in a binary vector. The distance between two images is computed using Jaccard measure for set similarity. This is defined by the ratio of the number of elements in intersection and union as shown in Equation 5.2:
sim(A1, A2) =
| A1∩ A2 |
similarity is determined when this measure exceeds a certain threshold. The min-Hash is a LSH function that selects an element from each set Ai of visual words for an image i. A number
of N independent hash functions fj are used and min-Hashes generated are recorded. Every
min-Hash function infers a certain ordering on the set and min-Hash of the set is then defined as where v ∈ Ai:
m(Ai, fj) = arg min fj(v) (5.3)
The method is based on probability of two sets having the same min-Hash being equal to the similarity of the two sets. This has been defined by the aforementioned Jaccard measure in Equation 5.4.
P (m(A1, fj) = m(A2, fj)) =
| A1∩ A2 |
| A1∪ A2 | = sim(A1, A2) (5.4) A single min-Hash being a visual word is no more distinctive than a single visual word. In order to increase the discriminativeness multiple min-Hashes are grouped into s-tuples called sketches. This results in increased precision, but also leads to a lower recall. To increase the recall this process is usually repeated n times. This allows the computation of n sketches with n × s independent hash functions. The probability of sketch collision and retrieval, where atleast one of the n sketches (hash tables) must collide are given by Equation 5.5 and Equation 5.6.
P (sketch collision) = sim(A1, A2)s (5.5)
P (retrieval) = 1 − (1 − sim(A1, A2)s)n (5.6)
When indexing, each sketch is inserted into a hash table such that the i-th sketch is inserted into the i-th hash table. For querying, min-Hashes and sketches are generated with the same hash functions used for indexing. The i-th hash table is queried to check if it contains the i-th query sketch. If this is the case then there is similarity between the images.
5.1.2 Matching with Local Descriptor Matching (LDM)
The LDM representation involves computation of the full set of descriptors from all images in the database. Then the most similar feature is found by employing k-nearest neighbours search over these descriptors. A voting criterion using the k-nearest descriptor distances found is finally used to output a list of matches.
5.1. Background 65
5.1.3 Global ranking based on matching
Matches obtained from different approaches can be used to establish a global ranking in which an image configuration is assigned a score. An approach that can be utilized to obtain such a ranking would be to view this process as a two-fold problem. In the first stage, every query is assigned to a unique output image after matching. Next from this output, correctly ranked pairs based on the annotated rankings can be used to achieve the final ranking.
The goal of finding a unique match for every test image can be viewed as a combinatorial optimization problem. A member of the set that optimizes an objective function in a solution space that is large but finite can be found. This can be considered as an assignment problem for which Munkres assignment algorithm can be used to solve it in polynomial time [70]. In this case, there will be a non-negative N2 ×N
2 matrix, where an element in the ith row and jth
column represents the cost of assigning the jth match to the ith query.
To achieve a matching a sorted list of all the images for a query is retrieved. A matrix is populated with cost assignments that vary in accordance with the matching method utilized. For example, it can be given by a function that is a representation of the distance between query and its match. It could also be a value derived from such a distance after a voting scheme has been applied. For a voting-based cost function, goal of the problem is altered and it is required to maximise the cost in order to find an assignment. That is why each cost is replaced by the maximum cost subtracted from the cost. After a complete matrix has been formed for all the queries, Munkres assignment algorithm is applied.
The comparisons from such a matching output can be used as an input to the Kemeny-Young method for ranking. Matches obtained from a pair of query images are compared to their ranking in the annotated lists. This is done to establish which image is ranked better as shown in Figure 5.3. Matches with their pairwise order serve as a comparison input to the Kemeny- Young method. If there are N/2 images in the database and N/2 query images, then in an ideal scenario Munkres algorithm would result in N/2 matches. This would result in a total number of (N42 − N2)/2 unique comparisons for the voting input. After a ranking is obtained, rank of images part of the same configuration can be averaged to obtain a configuration list similar to Chapter 3 Section 3.5.2. A configuration-based ranking can be used to perform comparisons of image configurations based on aesthetics. This averages the ranking scores of different images
that belong to the same configuration to give a configuration-based ranking and hence provides an averaged solution when comparing image configurations.
Figure 5.3: A pair of query images can be checked for their ranking in an annotated ranking. Matches for this query are also checked for their ranking in the ranking. If the rankings agree then that is used as a correct output in the voting file and if not it is included as an incorrect match. This voting file is input to the Kemeny-Young method to obtain an output ranking.