Hierarchical clustering is a kind of a so called agglomerative clustering. It uses similarity measure d(., .) to compare two vectors. At the beginning, each vector is considered as a single cluster. The clustering method uses a strategy to merge pairs of clusters. In each step of clustering, two clusters are merged together until a threshold is achieved.
The most famous strategies are single linkage, complete linkage, average linkage, and centroid linkage. A schematic illustration of these four strategies is demonstrated in Fig- ure 3.9. A brief description of these methods are presented below:
1. Single linkage: Two clusters Ci and Cj are merged, if they contain the pair of points
showing the strongest similarity. Two clusters Ci and Cj are merged by the single
linkage if they minimize the SL value given as:
SL= min{d(u, v) :u∈Ci, v ∈Cj} (3.27)
for all i and j with i6=j, i∈ {1, ..., k},j ∈ {1, ..., k} and k number of clusters. If the clusters are not well-separated, the so called chaining phenomenon can occur. This means two clusters are merged just because of a single pair of data points have
been close to each other. On the other hand, if the clusters are well-separated, this method is able to find clusters of arbitrary form and distribution. Even non-convex clusters, such as rings, can be found.
2. Complete linkage: For each pair of clusters, the most dissimilar pair of points is determined. Those two clusters Ci and Cj are merged that have the most similar of
all dissimilar pairs. The clusters Ci and Cj are merged by the complete linkage if
they minimize the value CL described as:
CL= max{d(u, v) :u∈Ci, v ∈Cj} (3.28)
for all i and j with i6=j, i∈ {1, ..., k},j ∈ {1, ..., k} and k number of clusters. Complete linkage tends to find compact clusters of approximately equal diameters. Thus, two small well-separated clusters might be merged together just because the diameter of the merged cluster is still less than another larger cluster.
3. Average linkage: Those two clustersCi andCj are merged that exhibit the minimum
of average pairwise distances. Two clusters Ci and Cj are merged, if they minimize
the average linkage:
AL= 1 |Ci| |Cj| X u∈Ci X v∈Cj d(u, v) (3.29) for all i and j with i6=j, i∈ {1, ..., k},j ∈ {1, ..., k} and k number of clusters. Average linkage is a compromise between single linkage and complete linkage meth- ods. Therefore, in the case of real data, it may work better.
4. Centroid linkage: In the first step, centroids µi = |C1i|
P
u∈Ciu of clusters Ci are
calculated. Two clusters are merged, if they have the smallest distance d(µi, µj)
between two centroids.
In hierarchical clustering, the whole dataset can be described by a dendrogram (see Figure 3.10). A dendrogram is a binary tree. It shows level of similarity at which two clusters are merged together. To calculate the dissimilarity matrix or array, at least n(n−
1)/2 calculations are required, where n is the number of data points. Therefore, the computation complexity as well as the memory usage of the hierarchical algorithms is approximated by O(n2). Due to the different linkage strategies, these algorithms are not based on a specific mixture model.
3.4 Unsupervised clustering methods without predefined number of clusters63
Figure 3.10: Denderogram of a simulated dataset with four clusters is depicted. The vertical axis gives at which dissimilarity index two clusters are merged.
In the case of average linkage method, the group average dissimilarityd(G, H) between two clusters Gand H is defined as:
d(G, H) = 1 NGNH X i∈G X i0∈H dii0, (3.30)
where NG and NH are the numbers of samples in each cluster. For N → ∞, Eqn. 3.30
becomes the following form:
Z Z
d(x, x0)pG(x)pH(x0)dxdx0. (3.31)
Eqn. 3.31 is a characteristic of the relationship between the densities pG(x) and pH(x0) of
samples in clusters Gand H [Hastie et al., 2005].
The average linkage method attempts to produce both relatively compact clusters and distant clusters [Kaufman and Rousseeuw, 1990]. Hierarchical clustering has some advan- tages over the widely used k-means clustering. Hierarchical clustering produces a hierarchi- cal representation in which clusters at each level of the hierarchy are created by merging clusters at the prior lower level. In k-means clustering, the choice of different numbers of clusters might lead to different assignments of data points to clusters. However, in hierarchical clustering, sets of clusters are nested into each other [Hastie et al., 2005].
Figure 3.11: The hierarchical clustering with four introduced linkage methods is performed on the same simulated dataset from section 3.3.2.
Different hierarchical linkage methods have been performed to the same simulated dataset used in section 3.3.2. The results are presented in Figure 3.11. None of the four linkage methods was able to recognize the original three clusters of the dataset. Nev- ertheless, all linkage methods can be quite useful in recognizing a specific kind of pattern in a dataset.
Hierarchical clustering methods are more flexible when looking for clusters in datasets sampled from non-Gaussian mixtures. The relationships among data points in a dataset are described through the given similarity measure. By selecting a linkage method, we decide what kind of ordering of the similarity matrix is desired. The number of clusters is not necessarily given as a parameter. The granularity of clustering can be controlled by the maximal dissimilarity allowed inside each cluster.