CERTIFICADO DE TERMINACION DE LA PARTE SUSTANCIAL DE LA OBRA 1) Terminación de la parte sustancial de la Obra

One way to discover associations between variables in a dataset is by using information theory-based and probabilistic measures. Entropy, Kullback-Leibler Divergence, Mutual Information, Pearson correlation, Spearman rank-order, Phi and Point biserial are a few of them. In the rest of this subsection a description of Entropy, Kullback-Leibler Diver- gence, Mutual Information, Pearson correlation which will be used later in this manu- script.

Entropy, Kullback-Leibler Divergence and Mutual Information

Entropy is a formal quantification of uncertainty. It shows how even the probability distri- bution of a random variable is. In other words, entropy is the measure of information one can get, on average, from each value of the distributed variable in the domain. One of the interpretations of entropy can be calculated using Shannon’s formula:

𝐻(𝑋) = 𝐸𝑃(𝑋)[− log 𝑃(𝑋)] = − ∑ 𝑝(𝑥𝑖) ln 𝑝(𝑥𝑖) 𝑖

(15)

In which 𝐸𝑃(𝑋)[ ] is the expected value with respect to the distribution of random variable

X. This formula calculates the number of bits needed to describe the random variable X. Since the probability distribution of a random variable is non-negative, the value of en- tropy is non-negative too. The lower range of entropy value of a discrete variable can be zero and it happens when the discrete random variable has no uncertainty, i.e. the prob- ability of one of the values in the random variable is equal to 1 and for the rest of values it is equal to 0. This implies the situation that we are certain about the outcome of the random event. On the other hand, if the distribution of probabilities of a random variable is uniform, the value of entropy will grow to its maximum. This situation is called complete uncertainty in which the entropy value is a function of the number of states of the variable (Conrady & Jouffe, 2007).

In the case of a dataset with multiple random variables, another interpretation of entropy can be the measure of structuredness and regularities in the data (Yao, 2003). A more structured dataset tends to have lower entropy. For any two variables 𝑋 and 𝑌 in a joint probability distribution, entropy is defined as

𝐻(𝑋, 𝑌) = 𝐸𝑃(𝑋,𝑌)[− log 𝑃(𝑋, 𝑌)] = − ∑ ∑ 𝑝(𝑥𝑖, 𝑦𝑗) ln 𝑝(𝑥𝑖, 𝑦𝑗) 𝑗

𝑖

(16)

The degree of deviation of two probability distributions can be measured by calculating the relative entropy of two distributions. This measure is also known as Kullback-Leibler (KL) divergence or I-divergence and can be calculated as

𝐷(𝑃||𝑄) = 𝐸𝑃(𝑋)[ 𝑃(𝑋) 𝑄(𝑋)] = ∑ 𝑝(𝑥𝑖) ln 𝑝(𝑥𝑖) 𝑞(𝑥𝑖) 𝑖 (17)

In which 𝑃(𝑋) and 𝑄(𝑋) are probability distributions and 𝑃 is absolutely continuous with respect to 𝑄, i.e. 𝑃(𝑥) → 0 if 𝑄(𝑥) → 0. KL divergence is a non-negative with a minimum value of 0 in case 𝑃(𝑋) = 𝑄(𝑋). The maximum value is obviously for the case that 𝑃(𝑋) is maximum (equals to 1) while 𝑄(𝑋) has its lowest value. The other attribute of this measure is that it is not symmetric, meaning 𝐷(𝑃||𝑄) ≠ 𝐷(𝑄||𝑃).

Observation of other predictive random variables can increase the amount of information and consequently the entropy value increases. The entropy of a random variable, 𝑋,

given the observations of another random variable, 𝑌, is called conditional entropy and can be calculated as 𝐻(𝑋|𝑌) = − ∑ 𝑝(𝑥𝑖, 𝑦𝑗) log 𝑝(𝑥𝑖, 𝑦𝑗) 𝑝(𝑦𝑗) 𝑖,𝑗 = − ∑ 𝑝(𝑥𝑖, 𝑦𝑗) log 𝑝(𝑥𝑖|𝑦𝑗) 𝑖,𝑗 (18)

The value of conditional entropy is non-negative and non-symmetric, which the later means 𝐻(𝑋; 𝑌) ≠ 𝐻(𝑌, 𝑋). It can also be expressed as

𝐻(𝑋|𝑌) = 𝐻(𝑋, 𝑌) − 𝐻(𝑌) (19)

The difference between the marginal entropy of a variable of choice, 𝑋, and the condi- tional entropy of the same variable given the observations of another random variable, 𝑌, is called entropy reduction or mutual information between 𝑥 and 𝑦. Mutual information can show us what will be the benefit of observing a particular random variable in predict- ing the variable of choice. In this way, we can find out which variable has the most pre- dictive importance. 𝐼(𝑋|𝑌) = 𝐻(𝑋) − 𝐻(𝑋|𝑌) = ∑ ∑ 𝑃(𝑥𝑖)𝑃(𝑥𝑖|𝑦𝑗) log2( 𝑃(𝑥𝑖|𝑦𝑗) 𝑃(𝑥𝑖) ) 𝑗 𝑖 (20)

It can also be expressed using conditional entropy and entropy of 𝑋 and 𝑌

𝐼(𝑋; 𝑌) = 𝐻(𝑋) + 𝐻(𝑌) − 𝐻(𝑋, 𝑌) (21)

The other interpretation of mutual information can be obtained with KL divergence and the degree of independence of two variables. Mutual information can be described as KL divergence of the joint probability distribution of 𝑋 and 𝑌, i.e. 𝑃(𝑋, 𝑌), with a probability distribution if random variables 𝑋 and 𝑌 are independent and their joint probability distri- bution is obtained by multiplying the marginal distribution of 𝑋 and 𝑌, meaning 𝑄(𝑋, 𝑌) = 𝑃(𝑋) × 𝑃(𝑌). In this way, the real probability distribution of 𝑋 and 𝑌 is compared with a situation with an assumption of independence of 𝑋 and 𝑌.

𝐷(𝑃(𝑋, 𝑌)||𝑄(𝑋, 𝑌)) = 𝐷(𝑃(𝑋, 𝑌)||𝑃(𝑋) × 𝑃(𝑌)) = 𝐸𝑃(𝑋,𝑌)[ 𝑃(𝑋, 𝑌) 𝑃(𝑋) × 𝑃(𝑌)] = ∑ ∑ 𝑃(𝑥𝑖, 𝑦𝑗) log2 𝑃(𝑥𝑖, 𝑦𝑗) 𝑃(𝑥𝑖) × 𝑃(𝑦𝑗) 𝑗 𝑖 = ∑ ∑ 𝑃(𝑥𝑖)𝑃(𝑥𝑖|𝑦𝑗) log2( 𝑃(𝑥𝑖|𝑦𝑗) 𝑃(𝑥𝑖) ) 𝑗 𝑖 (22)

Mutual information in a non-negative and symmetric value.

According to Yao (2003), conditional entropy and mutual information can be used to de- termine one-way associations between variables. If two variables 𝑋 and 𝑌 have a functional association, i.e. they have a deterministic relationship with each other that implies 𝑃(𝑋|𝑌) is either 1 or 0, these equations will hold:

𝐻(𝑋|𝑌) = 0 (23)

𝐻(𝑋, 𝑌) = 𝐻(𝑌) (24)

𝐼(𝑋; 𝑌) = 𝐻(𝑋) (25)

A functional dependency is the strongest one-way association between variables. The value of mutual information is in its maximum and the conditional entropy value is mini- mum. On the contrary, probabilistic independence between two variables 𝑋 and 𝑌 implies these equalities:

𝐻(𝑋|𝑌) = 𝐻(𝑋) (26)

𝐻(𝑌|𝑋) = 𝐻(𝑌) (27)

𝐻(𝑋, 𝑌) = 𝐻(𝑋) + 𝐻(𝑌) (28)

𝐼(𝑋; 𝑌) = 0 (29)

Two random variables are associated if they are not independent. For two independent variables, the value of mutual information is minimum, and the condition entropy reaches its maximum. Moreover, the joint uncertainty about 𝑋 and 𝑌 is the sum of the uncertainty of each of them.

Pearson correlation shows how linear is the relation between the variables. Pearson cor- relation coefficient is normally shown by 𝑟 and the formula for two variables is

𝑟 = ∑ (𝑥𝑖− 𝑥̅ 𝑠𝑥 )( 𝑦_𝑖− 𝑦̅ 𝑠𝑦 ) 𝑛 𝑖=1 𝑛 − 1 (30)

In which n is the sample size, 𝑥̅ (or 𝑥̅) is the sample mean for variable 𝑥 (or 𝑦) 𝑠𝑥 (or 𝑠𝑦)

is the sample standard deviation for variable 𝑥 (or 𝑦) and can be calculated as

𝑠𝑥 = √ 1 𝑛 − 1∑ 𝑥𝑖− 𝑥̅ 𝑛 𝑖=1 (31)

Pearson correlation can have values between -1 and 1, where values close to zero show a weak linear relationship. Values close to 1 show a strong forward correlation and val- ues close to -1 shows a strong reverse correlation between variables under study (Boslaugh & Watters, 2008).