Almacenamiento de líquidos de la subcIase A2

distribution pi(z) = (1 − αi)δ0(z) + αiN(0,_αi1, z). Let A be a t × k matrix and let

S = AZk be a t dimensional linearly correlated signal. Suppose O = S + ˜N is the

observation vector, where ˜N is a t × 1 zero mean Gaussian measurement noise with

a covariance matrix ˜Σ. We denote by ηi(x) = E(Si|O = x) the MMSE estimator of

Si, the i-th component of the signal, given a t × 1 observation vector O = x. We will

compute η1(x). The other estimators can be computed similarly.

It is easy to check that one can represent Zi, i ∈ [k] by ΘiNi, where Θk are

independent binary random variables with P(Θi= 1) = αi and Nk are independent

zero mean Gaussian variables with variance 1

αi. Assume that Σ is the covariance

matrix of Nk _{with diagonal elements Σ}

first row of A and assume that for a given binary sequence θk and for an arbitrary

n× k matrix B, B(θk_{) denotes an n × k matrix whose i-th column is the i-th column}

of B provided θi= 1 and zero otherwise.

Using the conditioning on Θk, we have

η1(x) = X

θk_∈{0,1}k

E_{(S1|O = x, Θ}k_{= θ}k_)P(θk

|O = x). Conditioned on θk_{, S}

1 = a1(θk)N is a zero mean Gaussian with variance a1(θk)Σ a1(θk)∗.

The observation vector is also Gaussian with a zero mean and a covariance matrix

A(θk)Σ A(θk)∗ _{+ ˜Σ, thus the estimation of S1 is reduced to a Gaussian estima-}

tion problem where the estimator is known to be a linear function of observa- tion. Let ˆS1(θk, x) = a(θk)ΣA(θk)(A(θk)ΣA(θk)∗+ ˜Σ)−1x. It is easy to check that

E(S_{1|O = x, Θ}k = θk) = ˆS₁(θk_{, x}). Therefore, one obtains

η1(x) =X θk ˆS1(θk, x)P(θk|O = x) = 1 po(x) X θk ˆS1(θk, x)P(θk)po(x|θk) = Pθk ˆS1(θk, x)P(θk)N(0, A(θk)ΣA(θk)∗+ ˜Σ, x) P θkP(θk)N(0, A(θk)ΣA(θk)∗+ ˜Σ, x) , where N(µ, C, x) = _√ 1 (2π)ndet(C)exp(− 1

2(x − µ)∗C−1(x − µ)) denotes the Gaussian

Entropy Power Inequality for

Integer-valued Random

Variables

A

In Chapter 2, we observed that in order to prove the absorption phenomenon for integer-valued random variables, it is sufficient to find a lower bound for the gap between the conditional entropy of sum and the individual conditional entropy of a pair of random variables in terms of their individual conditional entropy1_{. To be}

more precise, suppose that (X, Y ) are integer-valued random variables with a given conditional entropy H(X|Y ) = c for some c ∈ R+ and let (X0, Y0) be an independent

copy of (X, Y ). We needed to show that there is a universal function g : R+→ R+

such that for any pair of integer-valued random variables (X, Y ) and its independent copy (X0_{, Y}0_),

H(X + X0_{|Y, Y}0) − H(X|Y ) ≥ g(H(X|Y )). (A.1)

We also required that g be increasing and strictly positive except in the origin where g(0) = 0. It is important to emphasize that for the proof of the absorption phenomenon, g must be a universal function not depending on a specific pair (X, Y ) but their conditional entropy H(X|Y ).

It is interesting to know that finding universal functions like g as in Equation (A.1) that establish universal bounds or inequalities between information measures is vastly studied in information theory, with the first result proposed by Shannon himself generally known as Entropy Power Inequality (EPI) [29]. EPI yields lower bounds on the differential entropy of the sum of two independent real-valued random variables in terms of the individual entropies. Versions of the EPI for discrete random variables have been obtained for special families of distributions with the differential entropy replaced by the discrete entropy, but no universal inequality is known (beyond trivial ones). More recently, the sumset theory for the entropy function provides a sharp inequality H(X + X0_{) − H(X) ≥} 1

2 − o(1) when X, X0

are i.i.d. with high entropy [72]. We strengthen this result by finding a universal lower bounds which holds for all range of values of H(X) not necessarily large ones. Moreover, we extend the result to non-identically distributed random variables and to conditional entropies.

1_{This chapter is the result of collaboration with Emmanuel Abbe and Emre Telatar. I am also} grateful to Christophe Vignat for helpful discussions on this problem.

The structure of this appendix is as follows. In Section A.1, we overview a history of the EPI and recent results and extensions. In Section A.2, we state the main results that we obtained and give further intuitions. Section A.3, explains the proof techniques. Finally, in Section A.4, we explain some open problems and state a conjecture under which one can further strengthen the proven lower bounds.

A.1 History and Introduction

For a continuous random variable2 _{X on R}n, let h(X) be the differential entropy

of X and let N(X) = 22nh(X) denote the entropy power of X. If Y is another

continuous Rn-valued random variable independent of X, the EPI states that

N(X + Y ) ≥ N(X) + N(Y ), (A.2)

with equality if and only if X and Y are Gaussian with proportional covariance matrices. A weaker statement of the EPI, yet of key use in applications, is the following inequality stated here for n = 1,

h(X + X0) − h(X) ≥ 1

2, (A.3)

where X, X0 _{are i.i.d., and where equality holds if and only if X is Gaussian.}

The EPI was first proposed by Shannon [29] who used a variational argument to show that Gaussian random variables X and Y with proportional covariance matrices and specified differential entropies constitute a stationary point for h(X+Y ). However, this does not exclude saddle points and local minima. The first rigorous proof of the EPI was given by Stam [73] in 1959, using the De Bruijin’s identity which connects the derivative of the entropy with Gaussian perturbation to the Fisher information. This proof was further simplified by Blachman [74]. Another proof was given by Lieb[75] based on an extension of Young’s inequality.

While there is a wide variety of entropic inequalities, the EPI is the only general inequality in information theory giving a tight lower bound on the entropy of a sum of independent random variables by means of the individual entropies. It is used as a key ingredient to prove converse results in coding theorems [76–80].

There have been numerous extensions and simplifications of the EPI over the reals [81–91]. There have also been several attempts to obtain discrete versions of the EPI, using Shannon entropy. Of course, it is not clear what is meant by a discrete version of the EPI, since (A.2), (A.3) clearly do not hold verbatim for Shannon entropy.

Several extensions have yet been developed. First, there have been some extensions using finite field additions, for example, the so-called Mrs. Gerber’s Lemma (MGL) proved in [92] by Wyner and Ziv for binary alphabets. The MGL was further extended by Witsenhausen [93] to non-binary alphabets, who also provided counter-examples for the case of general alphabets. More recently, [94] obtained EPI and MGL results for abelian groups of order 2n. Second, concerning discrete random variables and

addition over the reals, Harremoes and Vignat [95] proved that the discrete EPI holds for binomial random variables with parameter 1

2, which later on was generalized by 2_{All continuous random variables are assumed to have well-defined differential entropies.}