Causas Raíz

EER?

Readers that are familiar with the language recognition literature and the NIST language recognition evaluations in particular, will have seen the use of DET-curves, minCAVG and EER applied to language recognition. See for example [26, 48]. We analysed DET-curves, minDCF and EER for the two- class problem at great length in chapter 7. Why are they being ignored here? The reason is that those tools, in the way they have been applied in the language recognition literature are not multi-class evaluation criteria, but are still the two-class versions. The multi-class language recognition scores are subjected to a non-invertible transformation to form what appear to be two- class scores, which are then plugged into the existing two-class machinery for computing DET-curves, minDCF and EER. The problem is that by so doing, these criteria lose most of their meaning. In particular, the threshold sweep that is implicit in all these criteria no longer plays the role of calibration optimization as explained in chapter 7.

The problem starts with the format of the scores. The NIST LRE evalua- tion plans to date9have asked for N detection scores for every speech segment, such that score k can be thresholded to accept or reject the proposition that the speech segment is in target language k. Given a log-likelihood vector, w = (w1, . . . , wN), a detection score can be computed for each target, k, for

example, as: sk= wk− log K X j=1 exp(wj) . (8.26)

If w is regarded as well-calibrated, then so is sk and it can be thresholded at

− log N to make the optimal Bayes decision, which minimizes CAVG. (Any other form for sk, which differs from (8.26) by a continuous, strictly monotonic

rising transformation could do the same job.) So far there is no problem. The first problem arises if the input to (8.26) is not well-calibrated. If we still assume that w is well-calibrated, but that the input to (8.26) is scaled and shifted, then the resultant detection score is

s0_k= αwk+ βk− log K

X

j=1

exp(αwj+ βj) . (8.27)

The problem is now that in general, there is no function f , such that sk =

f (s0_k). Re-calibrating the s0_k individually can improve accuracy, but not as much as re-calibrating the input w, as was demonstrated in figure 8.4.

The next problem occurs because we still have N score streams and we want to evaluate them with two-class tools that expect a single score stream. Now since all the scores have the same sense (larger, more positive, scores favour the target, smaller ,more negative, scores favour the non-target), it sounds like a good idea just to pool the score streams into a single score stream. Moreover, as we pointed out above, each of the pooled score streams has the same optimal threshold.

The problem is that the miss-calibrations in the different score streams are of opposing sense. If the recognizer is biased towards one class, the scores for that class would be shifted towards the positive side and all the other scores towards the negative. Now sweeping a single threshold cannot find a threshold that would be optimal for both senses. For these reasons, late calibration and score pooling, any two-class threshold-sweeping score analysis tool, like DET, minDCF and EER, cannot give a calibration-insensitive criterion like they do for a true two-class problem.

The argument has been voiced that the intention for using these two-class tools is not calibration insensitivity, but just to sweep a range of operating points for scores that are intended to be well-calibrated. Yes, the DET-curve does sweep operating points, but its (now broken) calibration compensation mechanism is still conflated with the operating point sweep. (We propose be- low how a pure operating point sweep can be achieved.) Moreover, if the inten- tion is for the evaluator to not optimize calibration, then what is the meaning of minCAVG, which goes through the motions of optimizing a threshold for a fixed operating point? Likewise, the meaning and utility of EER remains un- clear. It certainly does not give a calibration-insensitive, application-spanning summary as it does for speaker detection.

Finally, the argument has been voiced that in speaker detection, the scores of many speakers are pooled for analysis by DET, minDCF and EER. Why should scores not be pooled across languages? But speaker detection is a two- class problem—and nobody has ever suggested pooling the scores for the target and the non-target propositions. Specifically a speaker detection evaluation is structured in such a way that an equation of the form (8.26) does not apply. The speaker detection score for every trial is not computed via the likelihoods of a fixed set of speakers.

8.5.1

Proposal for sweeping the LRE operating point

Here we propose a way to sweep a parametrized CAVG-like evaluation criterion over a range of operating points, defined in such a way that all targets are rejected in the one extreme and all are accepted in the other extreme. This is done by introducing a variable target prior, say α, into (8.1), which then becomes:

πik = αδik+

(1 − α)(1 − δik)

(N − 1) . (8.28)

where α is varied from 0 to 1. The same derivation can then be followed as before, to find a version of C_lre∗ that is parametrized by α. A curve of Elre

(maybe normalized) against α, or maybe logit α, can then be plotted to show error-rate as a function of the target prior. In addition, E_lreopt may be added as a contrast to analyse calibration.

8.6

Summary

In this chapter we analysed LRE’s CAVG by viewing it as a mixture of applica- tions that exercises the recognizer’s output at a number of discrete operating points in the probability simplex. We showed that the multi-class logarith- mic cost function forms a continuous mixture of applications that exercises the recognizer’s output everywhere in the probability simplex. In this way logarithmic cost forms a representative evaluation criterion for a wide range of applications. Moreover, logarithmic cost forms a convenient discriminative training criterion, which reduces to logistic regression for linear models.

For multi-class, a direct analogue to the continuous application-sweeping plots of chapter 7 are problematic because of high dimensionality. Instead we showed how to form a scatter-plot of very many discrete operating points.

For multi-class, a generalization of the non-parametric PAV calibration analysis is not available. Instead we derived a simple, affine, parametric trans- form that can be optimized via logistic regression. We showed how it can be applied to both measure and improve calibration of a multi-class recognizer.

Conclusion

In this work we examined data-driven methods to evaluate the goodness of probabilistic pattern recognizer outputs in a very direct way, by effectively us- ing the outputs to make decisions and then recording how good those decisions are. There is a close connection between this form of evaluation and discrimi- native modelling, where the model parameters are discriminatively trained by optimizing the evaluation criterion.

This forms an interesting contrast with generative modelling for pattern recognition [1, 10]. Like discriminative modelling, generative modelling is also data-driven, but the concept of training differs. Ideally, a generative model should not be trained—all hidden model parameters should be integrated out in a fully Bayesian way. However, in practice one usually has to resort to mak- ing point estimates of some of the parameters in the model and this constitutes training of those parameters. This training is accomplished by optimization of an objective function, which makes it similar to discriminative training. However, the optimization objective is different.

In generative training, one optimizes the model to maximize the evidence, P (data, classes|model), while in discriminative training one maximizes1, or more generally optimizes, P (classes|data, model).

In the last decade in text-independent speaker recognition research, there has been an interesting tug-of-war between generative and discriminative train- ing techniques. Some landmark papers (by no means exhaustive), in chrono- logical order, are: [81], which represented the state-of-the-art in generative GMM-based speaker recognition; [82, 83], which established SVM as a dis- criminatively trained alternative to GMM (although the associated NAP- transformation was more generative than discriminative in nature); [21], an example of the power of discriminative system fusion; [84], which introduced the large-scale generatively trained JFA as a new monolithic state-of-the-art, which often outperformed the best heterogeneous fusions; [54], the 2008 JHU summer workshop, originally conceived to explore new discriminative training

1_{for the logarithmic cost function}

techniques, but which ended up instead improving on generative JFA, as well as seeding the new, generative, i-vector modelling [85, 86], which is currently an important part of the state-of-the-art.

This leaves the question whether there may be good generative alternatives to solving the problems addressed in this work. Thus far, good generative mod- els have been a partial answer to discriminative fusion, but as far as we know, no convincing generative alternative for implementing calibration transforma- tion has been demonstrated.

Ideally, if generative modelling works well enough, then both fusion and calibration transformation should become unnecessary. The raw scores of an ideal generative model should be accurate enough not to have to fuse with any other recognizer and moreover, should be naturally well-calibrated.

The problem that remains however, is how do we know a recognizer is well- calibrated? Can one ever get away with not explicitly testing calibration as we proposed here, or with some equivalent method? The problem with the methodology in our proposal is, as we pointed out, that it breaks down at extreme operating points, where the errors become too scarce to count. But is there any other way of judging the goodness of calibration that does not rely on large amounts of evaluation data?

A contrast exists between the evaluation methods proposed in this work and the field of forensic DNA, as represented by Balding in [87], where gener- ative models are used to compute likelihood-ratios, which are very similar in interpretation and function to the likelihood-ratios in speaker detection. The big difference is that the DNA is typically more discriminative than speaker recognition by orders of magnitude, so that DNA can be employed to literally find a single individual out of millions of candidates. The problem is that performing data-driven evaluation to test calibration at such extreme operat- ing points would be intractable. The result is that evaluation of the goodness of the calibration of DNA likelihood-ratios can only be performed by expert perusal of the generative modelling strategies that are used to compute them. Is there a good way to bridge this gap between the blind data-driven evalu- ation paradigm proposed in this work and the pure expert evaluation paradigm of DNA?

Functions: Surjective, injective,

bijective

This note explains the difference between an invertible function and a bijec- tion. Bijections are invertible, but there are invertible functions which are not bijections. See for example [88].

A.1

Domain, codomain, range

For a function that is defined as f : X 7→ Y, we have the following definitions: domain is the set X of input values which are allowed.

codomain is the set Y, in which this function may take values.

range which we write: range(f ) = {y ∈ Y : y = f (x) and x ∈ X } is the set of output values that can actually be reached with inputs from the domain. Note that in general1_{, range(f ) ⊆ Y.}

Consider for example the function that is defined as: f : R2 _{7→R, such that f(x, y) = x}2_{+ y}2_.

Here R2 _{is the domain, or the set of inputs (x, y) for which the function is}

defined. The codomain of this example is R, and the range is the subset of non-negative real numbers.