Modelo de densidades

4.2.6 Bayesian Methods

There are two main approaches for incorporating stochastic models into the statistical modelling: discriminative (conditional distribution model) and generative (joint prob- ability model). Firstly, the discriminative modelling does not make any assumption about the prior distribution and only includes the conditional probability. Therefore, discriminative modelling is also known as the frequentist approach, and linear clas- sifiers, like LR, are examples of it. On the other hand, Bayes modelling (Bayes and Price, 1763) methods are known as generative, and they include the prior (marginal) distribution of the evidence data to the discriminative model (Jordan,2002).

Under the frequentist approach, unknown parameters are considered to have fixed but unknown values (i.e. fixed priors). The unknown parameters might be calculated by maximisation of total marginal likelihood, or be estimated using methods, such as Maximum Likelihood Estimate (MLE) or numerical integral approximations like Laplace approximation. Moreover, the frequentist inference of the posterior probabil- ity distribution can be interpreted as procedures that guarantee long-run frequency. This inference is derived from procedures that guarantee probability within a random confidence interval (Berger et al.,2006,Gill,2014,Koller and Friedman,2009). In contrast, a Bayesian approach considers all parameters to be random variables (i.e. functions of the data), and the data is used to update the prior probabilities of these parameters. In this approach, the Bayesian inference of posterior probability distribu- tion is used for setting and updating beliefs. The computation methods of Bayesian priors can be categorised into three distinct groups (Berger et al.,2006,Gelman et al.,

2013,Gill,2014,Koller and Friedman,2009,Lunn et al.,2012,Press,2009):

• Subjective Bayesians: uses personal degrees of belief. It is applying informative priors, based on historical data or underlying theory.

• Objective Bayesians: uses non-informative priors (a.k.a. objective, diffuse, flat or reference priors). It is applying prior distributions that are formally express- ing ignorance (vague information); but, have well-defined posterior probability distributions2:

– Using conjugate priors to approximate uninformative priors.

– Using prior distributions that can span the range of likelihoods, such as flat prior or Gaussian prior.

2

It is especially useful for complex problems with many parameters that have very little amount of information about the data. However, for many problems, this approach can be misguided or have no clear choice of prior distributions and inference approximation.

4.2.6 Bayesian Methods 44 – Defining priors that are transformation invariant based on Jeffreys’ Prior

(Jeffreys,1946).

• Empirical Bayesian: Using data to estimate the prior.

Before doing the Bayesian inference, it is useful for comparison to ignore some infor- mation and do a crude estimation of the missing data. However, ultimately inference of missing data should be included as part of the model. The Bayesian inference ap- proximation can be categorised into two distinct groups: deterministic and stochastic. In below, some of the well-known inference approaches are highlighted (Barber,2012,

Gelman et al.,2013,Koller and Friedman,2009): • Deterministic approximation:

– Laplace approximation: finds the Gaussian approximation to a probability density, which is based on the second-order Taylor approximation of the log posterior around the Maximum-a-Posteriori (MAP) (Bishop and Nasrabadi,

2006).

– Expectation Propagation (EP): is an iterative approach for choosing the best approximation from within some tractable class of distributions (Minka,

2001c).

– Loopy belief propagation: is a dynamic programming approach, which cal- culates the marginal distributions for unobserved nodes, conditional on any observed nodes (Murphy et al.,1999).

– Expectation Maximisation (EM): is an iterative method for finding the max- imum likelihood orMAPestimates, where the model depends on unobserv- able variables (Bailey et al.,1994).

– Variational Bayesian methods: is an extension ofEMalgorithm fromMAP, and it finds a set of optimal parameter values based on a set of interlocked equations (Bernardo et al.,2003).

• Stochastic Approximation:

– Direct simulation: can be used for simulation of simple models, and it is often easy to draw from the posterior distribution (e.g. rejection sampling, univariate sampling, and multivariate sampling).

– Markov Chain (MC) simulation, a.k.a. Markov Chain Monte Carlo (MCMC): is a general method for drawing sequential samples, which distribution of

4.2.6 Bayesian Methods 45 sampled draws depends on the last value drawn. Gibbs sampler and Metropo- lis sampling are two examples of the Markov-based sampling algorithm (Gilks,2005,Koller and Friedman,2009).

Moreover, in Bayesian Network (BN) modelling, template-based representations are used to produce a single compact model that can represent properties of system dy- namics and to produce distribution over different trajectories (e.g. Dynamic Bayesian Network) or to produce a distribution over different worlds (e.g. Genetics networks). To be able to reason about non-static situations, Dynamic Bayesian Network (DBNs) (Dean and Kanazawa, 1989) are used to represent nodes with system states. The sys- tem states are either considered as stationary time-slices (homogeneous or invariant), like Markov Models, or they are regarded as the state observation model, like Hid- den Markov Models (HMMs). In state observation models, the states are variant and evolve on their own separately from the observations (Koller and Friedman,2009). Five principal methods have identified to model a Time-VaryingDBNand are presented in

Appendix A.1.5.

In the following subsection, Bayes Point Machines (BPM) is summarised, which is a generative approach for non-linear classification. Moreover, before the model develop- ment stage, a list of suitable Bayesian libraries are produced for the purpose of this research, that is presented inAppendix A.3.

4.2.6.1 Bayes Point Machine (BPM)

The Bayes Point Machine (BPM) (Herbrich et al., 2001, Minka, 2001b) is a type of nonlinear classification algorithm that identifies an average classifier known as a Bayes point in a version space. A version space can be defined as a set of hypotheses, each of which is an approximation of the main hypothesis class. Similar to SVMs, BPMs

are more geometrically motivated and are aimed to find a hyperplane with optimal margins between classes. In contrast, logistic regression maximises the probability of data by optimising the distance of each point to the decision boundary.

The soft margin SVM can be thought of as an approximation to the BPM (Herbrich et al.,2001). SVMs(Vapnik and Vapnik,1998) use a mapping to indirectly transform data into higher dimensional space using a kernel function. Then, they use quadratic programming to optimise the classification’s hyperplanes using support vectors and margins. However, SVMs are only efficient for a symmetric version space and its complexity is characterised by the number of support vectors.

4.2.6 Bayesian Methods 46 On the other hand, BPMs sample the Bayesian posterior (Eq. 4.5) for a nonlinear classification in a kernel space. Then, they approximate the centre of the version space, which is a set of consistent hypothesis, and the effective size is determined from the training sample. BPMs minimise the generalisation error over a set of hypotheses according to a prior probability, instead of maximising the classification boundary margin explicitly, asSVMsdo. The predictive distribution can be thought of as a linear discriminant function, which is assumed to have the following parametric density:

p(y|x, w) = p(y|s = wTx) (4.5) , where w is the weight or latent parameter vector, x is the fully observed feature vector, and s is the score function. BPMsuse the kernel trick (Hofmann et al.,2008) to find an optimised w. The centre mass of the version space is approximated using an average of the weight vectors while minimising the average generalisation error. The derived scores are subject to additive Gaussian noise (ε) to allow for measurement or labelling errors (Eq. 4.6).

p(y|s, ε) = (ys + ε > 0)1 , with p(ε) = N (ε|0, 1) ∧ 1(α > 0) =    1 if α > 0 0 if α ≤ 0 (4.6)

In this research, Microsoft’s Infer.Net library (Research,2016) was used to construct the BPM model (Figure 4.3). The applied algorithm uses the original version of the

BPM, with two main modifications. Firstly, it uses a mixture of Gamma-Gamma, a heavy-tailed prior probability distribution for the precision of weights and features. Secondly, it applies Expectation Propagation (EP) message passing to infer poste- rior probabilities, which has been demonstrated (Minka,2001a,b) in Gaussian Mixture problems to be better than approximation techniques.

Therefore, the applied BPMis invariant to parameter rescaling or shifting, unlike LR

orSVMmethods. Moreover, active Bayesian training can allow continuous updates of the model and account for changes in the prior probabilities. Furthermore, the BPM

can efficiently handle a relatively larger number of features.

For instance, (Tan et al., 2008) applied a BPM model in combination with a Hidden Markov Model (HMM) to analyse immunological data of Asthma patients from a hos- pital in the UK. The research has provided a basic proof of concept for the analysis of large-scale immunological datasets. But, the study provided very little detail about