Probabilities derived from the ASR system are a widely-used source of confidence information. Recall from subsection 3.2 that the speech recognition process involves searching for the most likely path through a search space encoding all possible word strings known to the recognizer. The string that is output by the recognizer (i.e., the 1-best hypothesis)
is more likely than any other hypothesis, but the recognizer does not calculate its likelihood in absolute terms. Early work on ASR confidence scores [311] characterized the information derived from the ASR system with the remark, “We know which utterance is most likely, but we don’t really know how good of a match it is.”
The process of looking for the relatively most likely string corre- sponds to the simplified version of Bayes’ rule in which the denominator is dropped (cf. Equation 3.2 vs. Equation 3.3). During the recognition process, different word-string hypotheses are compared with respect to a given segment of the speech signal, that is, with respect to the same acoustic observations. For this reason, it makes sense for an ASR system to calculate only the relative likelihood of the hypotheses — the normalization introduced by the denominator is not necessary. The situation changes, however, when we want to generate a confidence score. A confidence score should reflect the overall probability of a word having been correctly recognized and not ignore dependencies on characteristics of the speech signal. The un-normalized scores generated by the ASR system are not suited for comparing word hypotheses that were generated by the recognizer as fitting different acoustics [294]. Instead, we need a normalized score, in other words a true poste- rior probability of the recognized words. The normalizing factor is the prior probability of the acoustic signal P (O), whose calculation is made tractable by applying a method for approximation.
Methods of generating confidence scores from speech recognizer output generally differ with respect to the approximation that they choose. In [311], normalization is accomplished by carrying out a second recognition of the speech signal using a phone-based recognizer and using the score derived in this way as an approximation for P (O). Such approaches are motivated by the following reasoning: P (O) can be cal- culated by summing over P (O|W ) for every possible hypothesis W . This sum can be approximated by determining P (O|W ) for the most closely competing hypotheses, in this case those of the second phone- based recognizer.
Other researchers have focused on approaches in which only one ASR system is necessary. A basic form of normalization can be accom-
plished using the n-best list. In [290], a word score is generated by summing the likelihood of all hypotheses containing the word and divid- ing by the total likelihood of all hypotheses. A related approach, used by [294], makes use of a posterior word probability calculated on a word lattice. Here, the probability for a word in the lattice, i.e., a single lattice arc, is calculated by combining a sum of all the proba- bilities of all possible histories of the arc and all possible futures of the arc, normalized with respect to the total probability mass in the lattice. Note that if the language model probability and the acous- tic probabilities are all set to one in the lattice, then this approach reduces to counting the number of paths through the lattice that pass through the word, normalized by the total number of paths through the lattice [294].
One of the challenges of lattice-based scores involves making a decision about which words will be considered to compete within the lattice. A word with few competitors should have a higher confidence than a word with many competitors. In the work of [142], a simple density-based approach is applied. The approach examines the speech signal spanned by each word in the 1-best ASR output. At each frame within the word, the number of competing links within the lattice is counted. This count is equivalent to the number of links that intersect a vertical line drawn through the lattice at a time point corresponding to a particular frame. Scores are calculated using various combinations of these statistics.
Another basic challenge is determining which hypothesized words should be treated as a realization of the same underlying spoken word [290]. In essence, the use of lattices in confidence score generation requires overcoming the same issues confronting the use of lattices for indexing, addressed in the previous subsection. In [294], word probabil- ity scores are accumulated for words in the lattice that are determined to correspond to the same word spoken in the signal.
The issue of confidence scores is addressed both in the literature on Large-Vocabulary Continuous Speech Recognition (LVCSR) and in the literature on STD. Recently [280, 281], a discriminative approach to estimating the confidence of words recognized by a STD system has
emerged. The discriminative approach is motivated by the conjecture that acoustic and language models perform poorly at modeling OOV terms and that generative approaches to confidence are sub-optimal in these cases. The strong performance delivered by the discriminative approach on OOV terms supports this conjecture.