The dictionary characterizes the low-dimensional subspaces, and accordingly the type of errors to be extracted; hence, a good choice of D can improve the performance of low-rank representation [30].
The dictionary span the desired data lying in M as expressed in (7.1). In addition, the sub- dictionaries characterizing each senone class should be exclusive to each senone class as explained in [71]. For that reason, it was decided to apply algorithms to the training data to extract dictionaries that are supposed to span only the desired subspaces.
PCA algorithm (Section 4.1) was used to obtain the left singular vectors of M . The 95V-rank procedure (Section 5.1.1) is used to choose the number of left singular vectors. The procedure consists in iterative truncated SVD reconstruction to preserve 95% variability after discarding some of the smaller singular values. The value of 95% was fixed arbitrary at the beginning of the work to compute the rank, and it was also used to compute the first set of dictionaries from the training data. Figure 7.1 shows the relation between increase in MAP accuracy and decrease in matrix rank (or increase in reconstruction error after PCA). It is clear that the maximum accuracy is achieved around 82%, and this value will also be taken to be used for the creation of “82V-basis” instead of “95V-basis”.
Percentage of reconstruction kept
0.99 0.98 0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.9 0.89 0.88 0.87 0.86 0.85 0.84 0.83 0.82 0.81 0.8 0.7 0.6 0.5
MAP Accuracy achieved after PCA
0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95
MAP accuracy after PCA as function of percentage of reconstruction kept
Figure 7.1: MAP accuracy according to the level variability preserved after reconstruction.
Once those values were defined, it was possible to build the dictionaries D from the training data using a few left singular vectors, and compare them for LRR. The first comparison was performed between “82V-basis” and “95V-basis”. Based on a subset of the development set, a fair comparison was established by comparing the MAP accuracy achieved after applying LRR on each different senone class. The parameter λ was chosen the best overall the senone classes or the best for each senone class. For each senone class, at most 1000 representatives were used and all the results are averaged over all the senone classes. The results obtained are listed in Table 7.1, and show clearly that the “95V-basis” is better than the “82V-basis”. This might be due to the fact that the “82V-basis” are really restrictive, and a small variability of posterior probability is characterized. Hence, many components are regarded as sparse noise. This also suggests that if many representative posteriors are considered, the data itself could also be used as the dictionary for LRR [30]. Hence, another test was conducted to see if indeed the basis extracted from such an
Mean MAP accuracy 82V-basis 95V-basis
LRR on individual senones classes
Best λ 96.2% 91.9%
Best λ for each senone 96.2% 97.1%
LRR on aggregation of all senone classes
Best λ 54.6% 71.7%
Best λ for each senone 72.2% 76.9%
Table 7.1: MAP accuracy after applying LRR for both “82V/95V-basis” used as the dictionaries.
algorithm were better than just using the matrix M itself as dictionary.
For a computational reason, it was not possible to use directly the same setup as the previous test. The LRR implementation available was not able to handle 1000 representatives as input data and dictionaries. This is the reason why a smaller, but identical, test set was used to compare the procedure known as “data as dictionary” implying the use of input data as dictionaries and “95V-basis”. Ten representatives for each of the 557 senone classes were randomly extracted and used to form combinations of 10, 20, 50, 100 and 250 senone classes. The results are listed in Table 7.2, and represent MAP accuracy after applying LRR with the given dictionary and various
values of the regularization parameter λ.
Mean MAP accuracy Dictionary method chosen
Number of combinations Data “95V-basis” Aggregation of all “95V-basis”
10 71.2% 89.9% 53.8%
20 67.9% 86.4% 52.6%
50 61.5% 78.2% 51.1%
100 57.3% 68.9% 49.2%
250 52.1% 58.1% 50.8%
Table 7.2: MAP accuracy for “95V-basis” and data used as dictionary: MAP accuracy results after
applying LRR with different combination of basis are presented. Each MAP accuracy is computed for various combination of senone classes and averaged over all different combinations. The best choice of regularization parameter λ was considered.
The best MAP accuracy was picked for each senone class. It is clear that using the data itself as dictionary performs better than using the aggregated basis. Using senone class-specific basis is, of course, the best, but in real case scenario, it is impossible to find in advance which senone classes are present or not in a given utterance. For that reason, it was chosen to discard any use of dictionaries and rely only on the self-expressiveness property of the data lying on multiple low-dimensional subspaces [30]. As stated before the dictionary should ideally be exclusive to each senone class. Using data directly is a trade-off allowing to be restricted to the set of basis which spans only the desired space where M lies.