As mentioned in the previous section, the log-likelihood monotonically converges if the se- quence is bounded above. Unfortunately, examples of finite mixture models can be found, where the likelihood is unbounded, so that the maximum likelihood estimate does not ex- ist. Redner and Homer (1984) mentioned that “if one of the mixture means coincides with a sample observation and if the corresponding variance tends to zero [...] then the log- likelihood function increases without bound” (p.208). In particular, Kiefer and Wolfowitz (1956) offered an example with a mixture of two normal densities with unknown different means and unknown different variances, where no maximum likelihood estimate exists. Nevertheless, as suggested in McLachlan and Krishnan (1997), this problem can be cir- cumvented by adding constraints on the parameters, e.g. σj2 ≥ ε, j = 1, . . . , k which ensure the compactness of the parameter space and hence the boundedness of the likelihood func- tion. Other constraints are proposed by Day (1969), namely the equality of the variances, or by Hathaway (1983, 1985), who chose for k = 2 the constraint min(σ1/σ2)≥ ε > 0. In most practical situations these constraints are naturally satisfied, hence useful maximum likelihood solutions can be obtained, see Everitt (1984). However, since several maxima of the likelihood may occur, a global solution depends on the starting values, which is briefly discussed in the next section.
2.4 The Problem of Starting Values
As Böhning (1999) stated, the problem of finding the right starting values is scarcely dis- cussed in literature or not mentioned at all. Furthermore, Böhning et al. (1992) remarked that “there is no straightforward method to find starting values for the algorithm.” (p. 296). Some authors, for instance Hosmer (1973b), claimed that the initial values do not have much effect on the estimates. In contrast, others emphasize the need for a careful consideration of the starting values, since the likelihood equation often has multiple roots in finite mixtures, which may lead to different solutions.
Everitt (1984) compared several algorithms for obtaining a maximum likelihood estimate in three types of mixtures of two normal distributions. Thereby, he used in each case three different starting values to investigate the estimation results. In most data sets each al- gorithm converges to the same local maxima of the likelihood function from a variety of different starting values. However, he noted as well that in some situations different results or actually convergence to a singularity may occur. Hence, he suggested to make a choice between solutions on the basis of the value of L. In practice, one may choose different starting points, investigate the behavior of the likelihood and decide to accept a maximum or to move on.
2.4. The Problem of Starting Values 21
for the case of multivariate data. He proposed to use a two dimensional scatter plot to explore the data initially for the presence of clusters. “Any visual clustering of the data so obtained can be reflected in the initial values [..]” (p. 418). A similar procedure can by applied in the univariate case. With a kernel density estimation, one can obtain a first impression of the data. Based on such graphical considerations an initial guess may be verbalized, see example 2.1 below.
Strategies for the univariate case can be found in Karlis and Xekalaki (2003), where as well a good review of previous works on this research field is given. The authors compared several methods for choosing initial values for the EM algorithm. Among others, they ana- lyzed an approach of Finch et al. (1989), who proposed that for a two component Gaussian mixture only the mixture proportions need to be given as initial values. Their idea was that with given mixing proportions, the sample can be separated into two parts, one containing the smallest n· π1 observations assumed to belong to the first component of the mixture and one containing the remaining observations assumed to belong to the second component. The mean of the first part of the sample is then used as initial value for the first component mean while the mean obtained by the second part officiates as starting value for the second component mean. Indeed, this method provides good results. In contrast, using random initialization, where the starting points are generated randomly under adequate conditions, the EM algorithm failed in a much higher number of samples. However, the performance of the EM algorithm depends as well on the underlying mixture model. Mixtures with well separated components have been decomposed correctly in their study with almost all investigated initial values strategies. Similar results were reproduced in the present work and will be shown later.
To illustrate the importance of the choice of the right initial values in finite mixture models and their influence on the convergence of an algorithm and so on the estimation results, the following example is given.
Example 2.1 A mixture with parameter Ψ = (0.9, 0.1, 5, μ2, 1, 1) is chosen. The true value of μ2 is 9, however, this value is desired to be estimated. A density kernel estimation of a sample of size n = 100 is shown in figure 2.3. The corresponding likelihood function for μ2 is displayed in figure 2.4. As can be seen, this likelihood has two maxima.