Multivariate Statistics
Chapter 6: Cluster Analysis
Pedro Galeano Departamento de Estad´ıstica Universidad Carlos III de Madrid
Course 2017/2018
Master in Mathematical Engineering
2 The clustering problem
3 Hierarchical clustering
4 Partition clustering
5 Model-based clustering
Introduction
The purpose of cluster analysis is to group objects in a multivariate data set into different homogeneous groups.
This is done by grouping individuals that are somehow similar according to some appropriate criterion.
Once the clusters are obtained, it is generally useful to describe each group using some descriptive tools to create a better understanding of the differences that exists among the formulated groups.
Cluster methods are also known asunsupervised classification methods.
These are different than thesupervised classification methods, or Classification Analysis, that will be presented in Chapter 7.
Clustering techniques are applicable whenever a data set needs to be grouped into meaningful groups.
In some situations we know that the data naturally fall into a certain number of groups, but usually thenumber of clustersis unknown.
Some clustering methods requires the user to specify the number of clusters a priori.
Thus, unless additional information exists about the number of clusters, it is reasonable to explore different values and looks at potential interpretation of the clustering results.
Introduction
Central to some clustering approaches is the notion of proximity of two random vectors.
We usually measure the degree of proximity of two multivariate observations by adistance measure.
The Euclidean distance is typically the first and also the most common distance one applies in Cluster Analysis.
Other distances such as those presented in Chapter 5 can be considered.
Some cluster procedures are based on using mixtures of distributions.
The underlying assumptions of these mixtures, i.e., that the data in the different parts are from a certain distribution, are not easy to verify and may not hold.
However, these methods have been shown to be powerful under general circum- stances.
Introduction
Cluster Analysis can be seen as an exploratory tool.
Different cluster solutions will appear if one considers different number of clus- ters, distance measures or mixture distribution.
These solutions might provide new understanding of the structure of the data set.
Therefore, if possible, the interpretation of cluster solutions should involve sub- ject experts.
There are a large vast amount of cluster procedures.
Here, we will focus on:
I Hierarchical clustering: start with single clusters (individual observations) and merges clusters or start with a single cluster (the whole data set) and split clusters.
I Partition clustering: starts from a given group definition and proceed by exchang- ing elements between groups until a certain criterion is optimized.
I Model-based clustering: the random vectors are modeled by mixtures of distri- butions leading to posterior probabilities of the observation memberships.
Before presenting these methods, we define the problem.
The clustering problem
Given a data matrix X of dimension n×p, we want to obtain a partition of the data set, C1, . . . ,CK, where Ck, fork = 1, . . . ,K, are sets containing the indices of the observations in each cluster.
Therefore,i∈Ck means that the observationxi·belongs to clusterk. Any partitionC1, . . . ,CK verifies the following two properties:
I Each observation belongs to at least one of theK clusters, i.e.,C1∪ · · · ∪CK= {1, . . . ,n}.
I No observation belongs to more than one cluster, i.e.,Ck∩Ck0 =∅, fork6=k0. The problem is to find an appropriate partition,C1, . . . ,CK, for our data set.
The key interpretative point of hierarchical and partition methods is that ele- ments within a Ck are much more similar to each other than to any element from a differentCk0.
This interpretation does not necessarily hold in model-based clustering, where similar observations can belong to different clusters.
Hierarchical clustering
There are two types ofhierarchical clustering methods:
1 Inagglomerative clustering, one starts with n single clusters and merges them into larger clusters.
2 Indivisive clustering, one starts with a single cluster and divides it into smaller clusters.
Most attention has been paid on agglomerative methods.
However, arguments have been made that divisive methods can provide more sophisticated and robust clusterings.
The end result of all hierarchical clustering methods is a graphical output called dendogram, where thek-th cluster solution is obtained by merging some of the clusters from the (k+ 1)-th cluster solution.
The result of hierarchical algorithms depend on the distance considered.
In particular, when the variables are in different units of measurement and the distance used do not take into account this fact, it is better to standardize the variables.
Hierarchical clustering
The algorithm for agglomerative hierarchical clustering (agglomerative nesting or agnes)is given next:
1 Initially, each observationxi·, fori = 1, . . . ,n, is a cluster.
2 ComputeD={dii0,i,i0= 1, . . . ,n}, the matrix that contains the distances be- tween thenobservations (clusters).
3 Find the smallest distance inD, say,dII0 and merge clustersI andI0 to form a new clusterII0.
4 Compute the distances,dII0,I00, between the new clusterII0 and all other clusters I006=II0 (detailed in the next slide).
5 Form a new distance matrix,D, by deleting rows and columnsI andI0and adding a new row and columnII0 with the distances computed from step 4.
6 Repeat steps 3, 4 and 5 a total ofn−1 times until all observations are merged together into a single cluster.
Computation of the distancesdII0,I00, between the new clusterII0 and all other clustersI006=II0 can be done using one of the followinglinkage methods:
I Single linkage: dII0,I00= min{dI,I00,dI0,I00}.
I Complete linkage: dII0,I00= max{dI,I00,dI0,I00}.
I Average linkage: dII0,I00 =P
i∈II0
P
i00∈II00di,i00/(nii0ni00), where nii0 andni00 are the number of items in clustersII0andI00, respectively.
I Ward linkage: dII0,I00is the squared Euclidean distance between the sample mean vector of the elements in both clusters.
Hierarchical clustering
Thedendogramis a graphical representation of the cluster solutions.
Particularly, the dendogram shows the distances at which clusters are combined together to form new clusters.
Similar clusters are combined at low distances, whereas dissimilar clusters are combined at high distances.
Consequently, the difference in distances defines how close clusters are of each other.
To obtain a partition of the data into a specified number of groups, we can cut the dendogram at an appropriate distance.
The number of vertical lines,K, cut by a horizontal line on the dendogram at a given distance identifies aK-cluster solution.
The items located at the end of all branches below the horizontal line constitute the members of the cluster.
Hierarchical clustering
To know whether or not the cluster solution is appropriate, we can use the Silhouette.
Let:
I a(xi·) be the average distance ofxi· with respect all other points in its cluster.
I b(xi·) be the lowest average distance ofxi· to any other cluster of whichxi· is not a member.
I s(xi·) be the silhouette ofxi·:
s(xi·) = a(xi·)−b(xi·) max{a(xi·),b(xi·)}
The silhouette s(xi·) ranges from −1 to 1, such that a positive value means that the object is well matched to its own cluster and a negative value means that the object is bad matched to its own cluster.
The average silhouette gives a global measure of the assignment, such that the more positive, the better the configuration.
We are going to apply the agnes algorithm to the states data set.
For that, we make use of the Euclidean distance after take logarithms of the first, third and eighth variables and after standardize all the variables.
The next slides shows dendograms for the solutions with the four linkage meth- ods (simple, complete, average and Ward), joint with scatterplot matrices, plots of the first two PCs and the silhouette are given.
To compare solutions, we focus onK = 3 although different linkage methods may provide with different suggestions on the number of clusters.
For K = 3, the silhouette suggests to consider the solution given with the complete linkage.
Illustrative example (I)
Alabama Louisiana Arkansas Kentucky Tennessee North Carolina Georgia Mississippi South Carolina West Virginia Florida Illinois Michigan Indiana Ohio Pennsylvania Missouri New York Oklahoma Virginia Colorado Idaho Iowa Minnesota Nebraska Kansas Wisconsin Montana Wyoming Utah Oregon Washington South Dakota Maine New Hampshire Vermont North Dakota Connecticut Maryland New Jersey Massachusetts Arizona Nevada California Texas New Mexico Delaware Rhode Island Hawaii Alaska
0.51.01.52.02.53.0
Single linkage
Agglomerative Coefficient = 0.6 X.s
Height
Log−Population
−2 −1 0 1 2 3 −2 −1 0 1 2 −2 −1 0 1 −3 −2 −1 0 1 2
−2012
−202
Income
Log−Illiteracy
−1.50.01.5
−2012
Life Exp
Murder
−1012
−201
HS Grad
Frost
−201
−2 −1 0 1 2
−3−11
−1.5 −0.5 0.00.51.01.52.0 −1 0 1 2 −2 −1 0 1
Log−Area Single linkage
Illustrative example (I)
−4 −3 −2 −1 0 1 2
−3−2−10123
CLUSPLOT( X.s )
Component 1
Component 2
These two components explain 62.5 % of the point variability.
Alabama
Alaska Arizona
Arkansas
California
Colorado Connecticut Delaware
Florida Georgia
Hawaii Idaho
Illinois
Indiana Iowa
Kansas Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota Mississippi
Missouri
Montana Nebraska
Nevada New Hampshire
New Jersey New Mexico
New York North Carolina
North Dakota
Ohio Oklahoma
Oregon Pennsylvania
Rhode Island
South Carolina South Dakota
Tennessee
Texas
Utah Vermont
Virginia
Washington West Virginia
WisconsinWyoming
Silhouette width si
−0.2 0.0 0.2 0.4 0.6 0.8 1.0
Silhouette for Agnes and Single
Average silhouette width : 0.2
n = 50 3 clusters Cj
j : nj | avei∈C
1 : 48 | 0.20
2 : 1 | 0.00 3 : 1 | 0.00
Illustrative example (I)
Alabama Louisiana Georgia Mississippi South Carolina Arkansas Kentucky Tennessee North Carolina West Virginia New Mexico Arizona Florida Texas California Illinois Michigan New York Virginia Indiana Ohio Pennsylvania Missouri Oklahoma Colorado Iowa Minnesota Wisconsin Kansas Nebraska Idaho Utah Oregon Washington Maine New Hampshire Vermont Montana Wyoming North Dakota South Dakota Connecticut Massachusetts Maryland New Jersey Delaware Rhode Island Hawaii Alaska Nevada
01234567
Complete linkage
Agglomerative Coefficient = 0.79 X.s
Height
Log−Population
−2 −1 0 1 2 3 −2 −1 0 1 2 −2 −1 0 1 −3 −2 −1 0 1 2
−2012
−202
Income
Log−Illiteracy
−1.50.01.5
−2012
Life Exp
Murder
−1012
−201
HS Grad
Frost
−201
−2 −1 0 1 2
−3−11
−1.5 −0.5 0.00.51.01.52.0 −1 0 1 2 −2 −1 0 1
Log−Area
Complete linkage
Illustrative example (I)
−4 −3 −2 −1 0 1 2
−3−2−10123
CLUSPLOT( X.s )
Component 1
Component 2
These two components explain 62.5 % of the point variability.
Alabama
Alaska Arizona
Arkansas
California
Colorado Connecticut Delaware
Florida Georgia
Hawaii Idaho
Illinois
Indiana Iowa
Kansas Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota Mississippi
Missouri
Montana Nebraska
Nevada New Hampshire
New Jersey New Mexico
New York North Carolina
North Dakota
Ohio Oklahoma
Oregon Pennsylvania
Rhode Island
South Carolina South Dakota
Tennessee
Texas
Utah Vermont
Virginia
Washington West Virginia
WisconsinWyoming
Silhouette width si
0.0 0.2 0.4 0.6 0.8 1.0
Silhouette for Agnes and Complete
Average silhouette width : 0.3
n = 50 3 clusters Cj
j : nj | avei∈C
1 : 24 | 0.31
2 : 2 | 0.31
3 : 24 | 0.28
Illustrative example (I)
Alabama Louisiana Georgia Mississippi South Carolina Arkansas Kentucky Tennessee North Carolina West Virginia New Mexico Arizona Florida Texas California Connecticut Massachusetts Maryland New Jersey Illinois Michigan New York Virginia Indiana Ohio Pennsylvania Missouri Oklahoma Colorado Iowa Minnesota Wisconsin Kansas Nebraska Idaho Utah North Dakota South Dakota Oregon Washington Maine New Hampshire Vermont Montana Wyoming Nevada Delaware Rhode Island Hawaii Alaska
012345
Average linkage
Agglomerative Coefficient = 0.74 X.s
Height
Log−Population
−2 −1 0 1 2 3 −2 −1 0 1 2 −2 −1 0 1 −3 −2 −1 0 1 2
−2012
−202
Income
Log−Illiteracy
−1.50.01.5
−2012
Life Exp
Murder
−1012
−201
HS Grad
Frost
−201
−2 −1 0 1 2
−3−11
−1.5 −0.5 0.00.51.01.52.0 −1 0 1 2 −2 −1 0 1
Log−Area Average linkage
Illustrative example (I)
−4 −3 −2 −1 0 1 2 3
−3−2−10123
CLUSPLOT( X.s )
Component 1
Component 2
These two components explain 62.5 % of the point variability.
Alabama
Alaska Arizona
Arkansas
California
Colorado Connecticut Delaware
Florida Georgia
Hawaii Idaho
Illinois
Indiana Iowa
Kansas Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota Mississippi
Missouri
Montana Nebraska
Nevada New Hampshire
New Jersey New Mexico
New York North Carolina
North Dakota
Ohio Oklahoma
Oregon Pennsylvania
Rhode Island
South Carolina South Dakota
Tennessee
Texas
Utah Vermont
Virginia
Washington West Virginia
WisconsinWyoming
Silhouette width si
−0.2 0.0 0.2 0.4 0.6 0.8 1.0
Silhouette for Agnes and Average
Average silhouette width : 0.29
n = 50 3 clusters Cj
j : nj | avei∈C
1 : 11 | 0.55
2 : 1 | 0.00
3 : 38 | 0.22
Illustrative example (I)
Alabama Louisiana Georgia Mississippi South Carolina Arkansas Kentucky Tennessee North Carolina West Virginia Alaska Montana Wyoming Nevada Colorado Iowa Minnesota Wisconsin Kansas Nebraska Idaho Utah North Dakota Oregon Washington Maine South Dakota New Hampshire Vermont Arizona Oklahoma New Mexico California Florida New York Virginia Texas Illinois Michigan Indiana Ohio Pennsylvania Missouri Connecticut Massachusetts Maryland New Jersey Delaware Rhode Island Hawaii
051015
Ward linkage
Agglomerative Coefficient = 0.9 X.s
Height
Log−Population
−2 −1 0 1 2 3 −2 −1 0 1 2 −2 −1 0 1 −3 −2 −1 0 1 2
−2012
−202
Income
Log−Illiteracy
−1.50.01.5
−2012
Life Exp
Murder
−1012
−201
HS Grad
Frost
−201
−2 −1 0 1 2
−3−11
−1.5 −0.5 0.00.51.01.52.0 −1 0 1 2 −2 −1 0 1
Log−Area
Ward linkage
Illustrative example (I)
−4 −3 −2 −1 0 1 2
−4−3−2−10123
CLUSPLOT( X.s )
Component 1
Component 2
These two components explain 62.5 % of the point variability.
Alabama
Alaska Arizona
Arkansas
California
Colorado Connecticut Delaware
Florida Georgia
Hawaii Idaho
Illinois
Indiana Iowa
Kansas Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota Mississippi
Missouri
Montana Nebraska
Nevada New Hampshire
New Jersey New Mexico
New York North Carolina
North Dakota
Ohio Oklahoma
Oregon Pennsylvania
Rhode Island
South Carolina South Dakota
Tennessee
Texas
Utah Vermont
Virginia
Washington West Virginia
WisconsinWyoming
Silhouette width si
−0.2 0.0 0.2 0.4 0.6 0.8 1.0
Silhouette for Agnes and Ward
Average silhouette width : 0.27
n = 50 3 clusters Cj
j : nj | avei∈C
1 : 10 | 0.55
2 : 19 | 0.28
3 : 21 | 0.12
Hierarchical clustering
None of the distance/linkage procedures is uniformly best for all clustering prob- lems.
Singe linkage often leads to long clusters, joined by singleton observations near each other, a result that does not have much appeal in practice.
Complete linkage tends to produce many small, compact clusters.
Average linkage is dependent upon the size of the clusters, while single and complete linkage do not.
Ward linkage also tends to produce many small, compact clusters.
Indivisive clustering (divisive analysis or diana), the idea is that at each step, the observations are divided into a “splinter” group (say cluster A) and the
“remainder” group (say cluster B).
The splinter group is initiated by extracting that observation that has the largest average distance from all other observations in the data set, and that observation is set up as cluster A.
Given the separation of the data into A and B, we next compute, for each observation in cluster B, the following quantities:
1 the average distance between that observation and all other observations in cluster B, and
2 the average distance between that observation and all observations in cluster A.
Hierarchical clustering
Then, we compute the difference between (1) and (2) above for each observation in B.
There are two possibilities:
1 If all the differences are negative, we stop the algorithm.
2 If any of these differences are positive, we take the observation in B with the largest positive difference, move it to A, and repeat the procedure.
This algorithm provides with a binary split of the data into two clusters A and B.
This same procedure can then be used to obtain binary splits of each of the clusters A and B separately.
We are going to apply the diana algorithm to the states data set.
The next slides shows dendograms for the solution, joint with scatterplot matri- ces, plots of the first two PCs and the silhouette with the optimal solution for K = 3.
It is not difficult to see that this algorithm points out the presence of special states.
Illustrative example (I)
Alabama Louisiana Georgia Mississippi South Carolina Arkansas Kentucky Tennessee North Carolina West Virginia Texas New Mexico Arizona Florida California Illinois Michigan New York Virginia Indiana Ohio Pennsylvania Missouri Oklahoma Maryland New Jersey Alaska Montana Wyoming Nevada Colorado Iowa Nebraska Kansas Minnesota Wisconsin Idaho Utah Oregon Washington Maine New Hampshire Vermont North Dakota South Dakota Connecticut Massachusetts Delaware Rhode Island Hawaii
01234567
Diana
Divisive Coefficient = 0.79 X.s
Height
Log−Population
−2 −1 0 1 2 3 −2 −1 0 1 2 −2 −1 0 1 −3 −2 −1 0 1 2
−2012
−202
Income
Log−Illiteracy
−1.50.01.5
−2012
Life Exp
Murder
−1012
−201
HS Grad
Frost
−201
−2 −1 0 1 2
−3−11
−1.5 −0.5 0.00.51.01.52.0 −1 0 1 2 −2 −1 0 1
Log−Area Diana
Illustrative example (I)
−4 −3 −2 −1 0 1 2
−3−2−10123
CLUSPLOT( X.s )
Component 1
Component 2
These two components explain 62.5 % of the point variability.
Alabama
Alaska Arizona
Arkansas
California
Colorado Connecticut Delaware
Florida Georgia
Hawaii Idaho
Illinois
Indiana Iowa
Kansas Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota Mississippi
Missouri
Montana Nebraska
Nevada New Hampshire
New Jersey New Mexico
New York North Carolina
North Dakota
Ohio Oklahoma
Oregon Pennsylvania
Rhode Island
South Carolina South Dakota
Tennessee
Texas
Utah Vermont
Virginia
Washington West Virginia
WisconsinWyoming
Silhouette width si
0.0 0.2 0.4 0.6 0.8 1.0
Silhouette for Diana
Average silhouette width : 0.27
n = 50 3 clusters Cj
j : nj | avei∈C
1 : 26 | 0.29
2 : 4 | 0.25
3 : 20 | 0.24
Partition clustering
Partition methodssimply split the data observations into a predetermined num- berK of groups or clusters, where there is no hierarchical relationship between theK-cluster solution and the (K + 1)-cluster solution.
GivenK, we seek to partition the data intoK clusters so that the observations within each cluster are similar to each other, whereas observations from different clusters are dissimilar.
Ideally, one can obtain all the possible partition of the data intoK clusters and selects the “best” partition using some optimizing criterion.
Clearly, for medium or large data sets such a method rapidly becomes infeasible, requiring incredible amount of computer time and storage.
As a result, all available partition methods are iterative and work on only a few possible partitions.
Thek-means algorithm is the most popular partition method.
As it is extremely efficient, it is often used for large-scale clustering projects.
The algorithm depends on the concept of centroid of a cluster, which is a representative point of the group (not necessarily an observation).
Usually, the centroid is taken as the sample mean vector of the observations in the cluster, although this is not always the choice.
Partition clustering
The algorithm is given next:
1 Letxi·, fori = 1, . . . ,nbe the set of observations in the data matrixX.
2 Do one of the following:
1 Form an initial random assignment of the observations intoK clusters and, for clusterk, compute its current centroid,kx.
2 Pre-specifyK cluster centroids,kx, fork= 1, . . . ,K.
3 Compute the squared Euclidean distance of each observation to its current cluster centroid and sum all of them:
SSE =
K
X
k=1
X
c(i)=k
(xi·−kx)0(xi·−kx)
wherekx is thek-th cluster centroid andc(i) is the cluster containingxi·.
4 Reassign each observation to its nearest cluster centroid so thatSSE is reduced in magnitude. Update the cluster centroids after each reassignment.
5 Repeat steps 3 and 4 until no further reassignment of observations takes place.
The solution (a configuration of observations intoK clusters) will typically not be unique.
This is because, the algorithm will only find a local minimum of theSSE.
It is recommended that the algorithm be run using different initial random assignments to the observations to theK clusters (or by randomly selecting K initial centroids) in order to find the lowest minimum of SSE and, hence, the best clustering solution based uponK clusters.
Illustrative example (I)
We are going to apply the k-means algorithm to the states data set.
As with the hierarchical algorithms, we use standardized variables, as the algo- rithm uses Euclidean distances.
The next slides show scatterplot matrices, plots of the first two PCs and the silhouette with the optimal solution forK = 3.
We run the algorithm 25 times, i.e., we form 25 initial random assignments of the observations into 3 clusters and run the algorithm.
The value of SSE attained by the algorithm is 203.2068.
Log−Population
−2 −1 0 1 2 3 −2 −1 0 1 2 −2 −1 0 1 −3 −2 −1 0 1 2
−2012
−202
Income
Log−Illiteracy
−1.50.01.5
−2012
Life Exp
Murder
−1012
−201
HS Grad
Frost
−201
−2 −1 0 1 2
−3−11
−1.5 −0.5 0.00.51.01.52.0 −1 0 1 2 −2 −1 0 1
Log−Area
k−means
Illustrative example (I)
−4 −3 −2 −1 0 1 2
−3−2−10123
CLUSPLOT( X.s )
Component 1
Component 2
These two components explain 62.5 % of the point variability.
Alabama
Alaska Arizona
Arkansas
California
Colorado Connecticut Delaware
Florida Georgia
Hawaii Idaho
Illinois
Indiana Iowa
Kansas Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota Mississippi
Missouri
Montana Nebraska
Nevada New Hampshire
New Jersey New Mexico
New York North Carolina
North Dakota
Ohio Oklahoma
Oregon Pennsylvania
Rhode Island
South Carolina South Dakota
Tennessee
Texas
Utah Vermont
Virginia
Washington West Virginia
WisconsinWyoming
Silhouette width si
0.0 0.2 0.4 0.6 0.8 1.0
Silhouette for k−means
Average silhouette width : 0.28
n = 50 3 clusters Cj
j : nj | avei∈C
1 : 18 | 0.28
2 : 12 | 0.46
3 : 20 | 0.16
Partition clustering
Partition around medoids (pam)is another partition algorithm.
Essentially, pam is a modification of the k-means algorithm.
This algorithm searches forK “representative objects” rather than the centroids among the observations in the data set.
Then, the method is expected to be more robust to data anomalies such as outliers.
A disadvantage of the pam algorithm is that, although it run well on small data sets, they are not efficient enough to use for clustering large data sets.
The algorithm is given next:
1 Letxi·, fori = 1, . . . ,nbe the set of observations in the data matrix.
2 ComputeD={dii0,i,i0= 1, . . . ,n}, the matrix that contains the distances be- tween thenobservations.
3 ChooseK observations as the medoids ofK initial clusters.
4 Assign every observation to its closest medoid using the matrixD.
5 For each cluster, search the observation, xi0·, of the cluster (if any) that gives the largest reduction in:
SSEmed=
K
X
k=1
X
c(i)=k
dii0
and select this observation as the medoid for this cluster (note thatSSEmed only considers distances from every observation in the cluster to the medoid).
6 Repeat steps 4 and 5 until no further reduction inSSEmed takes place.
Illustrative example (I)
We are going to apply the pam algorithm to the states data set.
As with the previous algorithms, we use standardized variables.
The next slides show the same information as in the previous methods.
For that we consider the case of 3 groups, as previously done.
The results do not appear to be very good.
Log−Population
−2 −1 0 1 2 3 −2 −1 0 1 2 −2 −1 0 1 −3 −2 −1 0 1 2
−2012
−202
Income
Log−Illiteracy
−1.50.01.5
−2012
Life Exp
Murder
−1012
−201
HS Grad
Frost
−201
−2 −1 0 1 2
−3−11
−1.5 −0.5 0.00.51.01.52.0 −1 0 1 2 −2 −1 0 1
Log−Area
pam
Illustrative example (I)
−4 −3 −2 −1 0 1 2
−3−2−10123
CLUSPLOT( X.s )
Component 1
Component 2
These two components explain 62.5 % of the point variability.
Alabama
Alaska Arizona
Arkansas
California
Colorado Connecticut Delaware
Florida Georgia
Hawaii Idaho
Illinois
Indiana Iowa
Kansas Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota Mississippi
Missouri
Montana Nebraska
Nevada New Hampshire
New Jersey New Mexico
New York North Carolina
North Dakota
Ohio Oklahoma
Oregon Pennsylvania
Rhode Island
South Carolina South Dakota
Tennessee
Texas
Utah Vermont
Virginia
Washington West Virginia
WisconsinWyoming
Silhouette width si
0.0 0.2 0.4 0.6 0.8 1.0
Silhouette for pam
Average silhouette width : 0.22
n = 50 3 clusters Cj
j : nj | avei∈C
1 : 15 | 0.29
2 : 22 | 0.13
3 : 13 | 0.27
Model-based clustering
Inmodel-based clustering, it is assumed that the data have been generated by a mixture ofK unknown distributions.
Maximum likelihood estimation can be carried out to estimate the parameters of the mixture model.
This is usually undertaken using theExpectation-Maximization (EM) algorithm.
Then, once the model parameters have been estimated, each observation is assigned to the mixture (cluster) with larger probability of having generated the observation.
Then, we assume that the data set have been generated from a mixture of distributions with pdf given by:
fx(x|θ) =
K
X
k=1
πkfx,k(x|θk)
whereθ is a vector with all the parameters of the model, including the weights πk and the parameters of the distributionsfx,k(·|θk), denoted byθk.
Model-based clustering
Then, for a data matrix,X, with observationsxi·= (xi1, . . . ,xip)0, the likelihood function is given by:
L(θ|X) =
n
Y
i=1
fx(xi·|θk) =
n
Y
i=1 K
X
k=1
πkfx,k(xi·|θk)
!
while the log-likelihood is given by:
`(θ|X) =
n
X
i=1
log
K
X
k=1
πkfx,k(xi·|θk)
!
Derivation of closed form expressions of the MLE of the mixture parameters is not possible, even in the case of the multivariate Gaussian distribution.
Moreover, although it is possible to apply a Newton-Raphson type algorithm to solve the equalities provided by the MLE method, the usual approach is to use the EM algorithm to obtain the MLEs (see the references).
Model-based clustering
Then, letbπ1, . . . ,bπG andbθ1, . . . ,θbG, be the MLE of the weights and the param- eters of the group distributions, respectively, obtained with the EM algorithm.
The estimated posterior probabilities that observationxi·belongs to population k are obtained by applying the Bayes Theorem:
Pr (k|xb i·) = πbkfx,k
xi·|bθk
PK
g=1bπgfx,g
xi·|bθg
The observations are assigned to the density (cluster) k with maximum value ofPr (kb |xi·).
In model-based clustering, it is possible to select the number of groups,K, from the data set.
The idea is to compare solutions with different values of K = 1,2, . . . and choosing the best result.
For that, we can rely onmodel selection criteriasuch as theAkaike Information Criterion (AIC)or theBayesian Information Criterion (BIC).
For instance, the BIC selects the number of clusters that minimizes:
BIC(k) =−2×`k
θ|Xb
+ log (n)×q
where`k θ|Xb
denotes the maximized log-likelihood assumingk groups andq is the number of parameters of the model.
Model-based clustering
M-clustis a popular method to perform model-based clustering.
M-clust assumes Gaussian densities and selects the optimal model according to BIC.
To reduce the number of parameters to fit, M-clust works with the spectral decomposition of the covariance matrices of the Gaussian densities, Σk, for k = 1, . . . ,K, given by:
Σk =λ1,kVkΛekVk0,
where λ1,k is the largest eigenvalue, Vk is the matrix that contains the eigen- vectors of Σk andeΛk is the diagonal matrix of eigenvalues divided byλ1,k.
The decompostion allows for different configurations:
1 spherical and equal volume,
2 spherical and unequal volume,
3 diagonal and equal volume and shape,
4 diagonal, varying volume and equal shape,
5 diagonal, equal volume and varying shape,
6 diagonal, varying volume and shape,
7 ellipsoidal, equal volume, shape, and orientation,
8 ellipsoidal, equal volume and equal shape,
9 ellipsoidal and equal shape, and
10 ellipsoidal, varying volume, shape, and orientation.
Here (i) spherical, diagonal and ellipsoidal are relative to the covariance matrices;
(ii) similar volume means that λ1,1 = · · · = λ1,K; (iii) equal shape means eΛ1=· · ·=eΛK; and (iv) equal orientation meansV1=· · ·=VK.
Illustrative example (I)
For the states data set, Mclust selects an ellipsoidal, equal shape and orientation (VEE) model with 3 components.
After estimating the model using the EM algorithm, the procedure compute the posterior probabilities for each country and population.
The results are shown in the next two slides.
The first one shows the scatterplot matrix with the assignments made by the algorithm.
The second one shows the first two principal components with the assignments made by the algorithm.
Note how close observations can be in different clusters.
Log−Population
3000 4000 5000 6000 68 69 70 71 72 73 40 45 50 55 60 65 7 8 9 1011 1213
678910
300045006000
Income
Log−Illiteracy
−0.50.5
687072
Life Exp
Murder
261014
405060
HS Grad
Frost
050150
6 7 8 9 10
791113
−0.5 0.0 0.5 1.0 24 6 8101214 0 50 100 150
Log−Area
M−clust solution
Illustrative example (I)
−4 −3 −2 −1 0 1 2 3
−3−2−10123
CLUSPLOT( X )
Component 1
Component 2
These two components explain 62.5 % of the point variability.
Alabama
Alaska Arizona
Arkansas
California
Colorado Connecticut Delaware
Florida Georgia
Hawaii Idaho
Illinois
Indiana Iowa
Kansas Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota Mississippi
Missouri
Montana Nebraska
Nevada New Hampshire
New Jersey New Mexico
New York North Carolina
North Dakota
Ohio Oklahoma
Oregon Pennsylvania
Rhode Island
South Carolina South Dakota
Tennessee
Texas
Utah Vermont
Virginia
Washington West Virginia
WisconsinWyoming
There are other alternatives procedures for model based clustering.
For instance, very appealing methodologies for estimating mixtures have been given from the Bayesian point of view.
These procedures include the number of groups as an additional parameter, and posterior probabilities are also provided for this number.
Also, procedures based on the use of projections (projection pursuit methods) are also very popular.
The idea is to project the data into different directions that separate the groups as much as possible and look for clusters in the univariate projected data.
Chapter outline
1 Introduction
2 The clustering problem
3 Hierarchical clustering
4 Partition clustering
5 Model-based clustering
