While there are multitudinous analyses that could be performed across all of the combina- tions of results that we calculated, here, we use the collected results of our many runs to
Vocab Selection Coher. Perpl. Doc. Class. Purity
20
news
Document Frequency 2.57 1.20 0.29 0.35
Class Doc Frequency 1.36 1.19 0.23 0.53
Word Rank 1.47 1.24 0.32 0.42
Fixed List 1.91 1.28 0.25 0.56
Science
Document Frequency 2.86 0.88 0.58 0.50
Class Doc Frequency 1.80 0.72 0.41 0.45
Word Rank 2.35 0.76 0.45 0.44
Fixed List 2.34 0.72 0.36 0.45
Fiction
Document Frequency 0.86 1.16 0.53 0.42
Class Doc Frequency 0.53 0.89 0.45 0.39
Word Rank 0.51 1.38 0.53 0.52
Fixed List 0.89 1.02 0.50 0.43
Table 3.4: Averaged evaluation scores for each vocabulary selection method for each dataset.
answer five questions:
How does vocabulary selection method relate to the four evaluation metrics?
The vocabulary selection method depends on both the dataset and the metric that is being optimized (Table3.4). The Document Frequency filter leads to the most coherent set of topics in several cases, but also typically leads to high perplexity. This may be due to its ability to filter out less common words that are needed in order to achieve the best fit to new data, but by removing these words, the resulting topics appear more coherent. The Class Doc Frequency leads to the lowest average perplexity scores, meaning that it is a good choice when trying to build models that have the best statistic fit to unseen documents. The Word Rank filter gives good document classification performance in several cases, indicating that this method does a reasonable job selecting a set of features that are related to the original document classes. The Fixed List filter results in the best coherence score for the fiction dataset, which may be due to its ability to easily standardize large or less typical vocabularies, whereas the other filters will be influenced much more strongly by the words in the corpus.
How does lemmatization affect the model perplexity for each dataset?
In Figure 3.2, we can see that applying lemmatization only has a major impact when considering the 20 Newsgroups dataset. This may be due in part to the fact that lemmati- zation restricts the expressiveness of the model by reducing the vocabulary size, making it more difficult to achieve a good fit to new data. It may also be collapsing groups of words in a way that is unhelpful– for example, if the concept of a “run” (as a means of scoring a point) in a topic about the sport of baseball is a useful feature, but all other instances of
20news Science Fiction 0.8 1 1.2 1.4 T est Perple xity True False
Figure 3.2: Effect of lemmatization on average test perplexity.
Model Coher. Perpl. Doc. Class. Purity
20N LDA 4.83 0.00 0.39 0.89 MEM 5.15 1.36 0.63 0.93 Sci LDA 5.42 0.00 0.75 0.83 MEM 5.37 1.28 0.76 0.83 Fic LDA 5.28 0.00 0.43 0.66 MEM 4.90 0.63 0.70 0.90
Table 3.5: Top scores achieved on each dataset by any single model of each type.
“running”, “ran”, etc., are all mapped to “run”, it may be more difficult for the model to accurately distinguish documents that belong to the baseball topic. For the Science and Fiction datasets, the effect of lemmatization on the text perplexity is insignificant. These results suggest that it may not always be best to perform lemmatization or stemming over the entire corpus before topic modeling.
How does the best performing LDA model compare to the best performing MEM model for each metric, for each dataset?
Only considering the best model for each dimension, we can see that each model has its own advantages (Table3.5). LDA achieves a far superior (lower) perplexity score in every case due to its ability to assign probabilities to every word in the vocabulary for every topic, while the MEM, using a threshold to remove words with lower membership to a given topic, does not have the ability to explain all of the noise present in unseen documents. However, the MEM always provides the best classification and performance and cluster purity, suggesting that MEM features can be used to define groupings that better reflect the
20news Science Fiction 0.35 0.4 0.45 0.5 0.55 Cluster Purity 0.5 1.0 1.5 2.0 5.0
Figure 3.3: Average cluster purity for each dataset while varying the topic-class ration.
LDA MEM
count 2.23 1.96 bin 2.41 1.48
Table 3.6: Average topic coherence when using either count or bin input data for each model.
original class labels. In the case of coherence, MEM performs better in the Newsgroup data, similar to LDA in the Scientific data, and worse in the Fiction data.
Which topic-class ratio leads to the highest cluster purity for each dataset?
Not surprisingly, increasing the ratio of topics to number of underlying classes, the average cluster purity increases (Table 3.3). The likely reason is that as the number of clusters increases, the average cluster size is reduced, making it more likely that a higher proportion of documents in a given cluster have the same class label, even by chance. The only exception to the trend comes from selecting 1.5|C| = 15 topics for the fiction dataset. One hypothesis for this irregularity is that certain topic numbers may truly lend themselves to alignment with the underlying class distribution of the corpus, and it may be (relatively) easier to achieve a higher purity score when the number of topics matches the number of classes.
What is the effect on topic coherence when using count data for MEM, and binary indicator variables for LDA?
In an unexpected result, the most coherent topics for LDA come when using binary indicator variables for the input matrix, and the best results for the MEM come when us-
ing a count matrix as input (Table3.6). However, LDA typically uses count input while the MEM uses binary input. Based on this, it may be worth considering the type of input repre- sentation as a parameter to tune when fitting topic models with the end goal of maximizing topical coherence.