In this section I briefly review the basic concepts of the classical and quantum infor- mation theory which are based on the definition of entropy.
2.10.1 Shannon Entropy
Shannon entropyis the key concept of classical information theory. For a classical vari- able X with values x occurring with probability px, the Shannon entropy measures
how much information one gains after learning the value of X. There is another complementary view of entropy as a measure of one’s uncertainty before learning the value ofX [41].
The entropy of a random variable is characterized as a function of the probabilities of the different possible values that the random variable can take. The Shannon entropy related to these probabilities is defined as [41]:
H(X)≡ H(p1, ...,pn)≡ −
∑
xpxlogpx. (2.71)
Logarithms are taken to base two. The main reason for this definition of entropy is its ability to quantify the physical resources which are required to store the information [41].
§2.10 Information Theory and Entropy 23
2.10.2 Relative Entropy
The relative entropy is a measure of closeness of two probability distributions, p(x) andq(x), on the same index set,x. It is defined as [41] :
H(p(x)||q(x))≡
∑
x
p(x)log p(x)
q(x) ≡ −H(X)−
∑
x p(x)logq(x). (2.72) It is assumed that−0 log 0≡0 and−p(x)log 0≡+∞if p(x)>0 [41].2.10.3 Shannon Entropy of Continuous Random Variable
Shannon’s entropy can be extended to continuous random variables. AssumeXbe a continuous random variable with the probability density function defined byp(x)on I, where I = (−∞,∞). Then the Shannon’s entropy for continuous random variables is given by [42] :
H(X) =−
Z
I p(x)logp(x)dx, (2.73)
Although it has many properties of Shannon’s entropy of discrete variables, un- like that it can become infinitely large or negative. In addition, the Shannon’s entropy for continuous random variables does not necessary remain invariant under a change of variable, while Shannon’s entropy of discrete variables remains invariant [42]. 2.10.4 Joint Entropy
The joint entropy quantifies one’s total uncertainty about the pair of random variables (X,Y). It is naturally defined as [41]:
H(X,Y)≡ −
∑
x,y
p(x,y)logp(x,y). (2.74)
2.10.5 Conditional Entropy
If we have a pair of random variables (X,Y), by performing a measurement on the random variableYwe can learn its value and acquireH(Y)bits of information about the pair. The conditional entropy quantifies our lack of knowledge about the pair (X,Y), on average, given the fact that we know the value of Y. It is simply defined as [41]:
H(X|Y)≡ H(X,Y)−H(Y). (2.75) Another way to express It, is the lack of knowledge ofXwhen the state ofYis in the ythstate, weighted by the probability forythoutcome as [44] :
H(X|Y) =−
∑
y
whereyis the outcome of the measurement performed on subsystemY. Conditional entropy can be shown schematically using the ’entropy Venn diagram’ depicted in figure 2.5.
H(X)
H(Y)
H(X:Y)
H(X|Y)
H(Y|X)
Figure 2.5: Entropy Venn diagram, showing the relationship between different en- tropies [41].
2.10.6 Mutual Information
The mutual information quantifies how much information two random variables (X,Y)have in common [41]. It is defined as [30]:
H(X :Y) =H(X)−H(X|Y) (2.77) This can be seen in ’entropy Venn diagram’depicted in figure 2.5. The mutual infor- mation quantifies the average decrease of entropy ofXwhen the value ofYis known [30]. Using the ’Bayes’ rule, which defines the conditional probability for classical variables as px|y= pxy/py [43], equation (2.77) can be written equivalently as [30]:
H(X:Y)≡ H(X) +H(Y)−H(X,Y). (2.78) Further information can be found in ref [30, 41, 43]
2.10.7 von Neumann Entropy
In classical information theory we calculate the uncertainty of the classical probability distribution using Shannon entropy. Quantum state can be described similarly with density operatorsρ, and Shannon entropy can be generalized for quantum states to Von Neumann entropy [41].
The Von Neumann entropy measures the information of a quantum state by finding the entropy of the probability distribution resulted from the state ρ by a projective measurement onto the state’s eigenvectors [29]. It is defined as:
§2.10 Information Theory and Entropy 25
S(ρ) =−Tr(ρlogρ) =H(λ) (2.79) Whereλ= λi are the eignevalues of the state. Here also the logarithms are taken to
base two and 0 log 0≡0.
2.10.8 Quantum Mutual Information and Conditional Entropy
Classical mutual information described by equation (2.78) can be easily generalized to quantum mutual information by simply replacing the Shannon entropy by Von Neumann entropy and the classical probability distributions by density matricesρas follows [30]:
I(ρXY) =S(ρX) +S(ρY)−S(ρXY) (2.80)
However, equation (2.77) cannot be easily generalized to quantum states, as to define the conditional entropy H(X|Y) one needs to define the state of X given the result of the measurement performed on the state ofY. This is more obvious by looking at the measurement-based version of conditional entropy defined by the relation 2.76. Such statement is ambiguous in quantum mechanics until the set of the measure- ment which will be performed on the stateY is selected [30]. In order to define the quantum analogue of the measurement-based conditional entropy a set of (POVM) with elements{Ey= M†yMy}is assumed to perform on subsystemYwithyto be the
outcome of the measurement (see subsection 2.7.1 for description on POVM). The probability of obtaining the outcome y is py = tr(ρxyEy) and subsystem Xis left in
the conditional state of ρX|y = trY(ρXYEy)/py. This allows us to write the quantum
version of measurement- based conditional entropy as S{Ey}(X|Y) ≡ ∑y pyS(ρX|y). Hence, the quantum analogue of mutual information defined by equation (2.77) can be written as [43]:
max{Ey}J(X|Y)≡S(X)−S{Ey}(X|Y) (2.81) The quantity max{Ey}J(X|Y)which is maximized over all possible POVMs is called
one way classical correlations [29].
2.10.9 Holevo Bound
Considering the fact that quantum states are generally nonorthogonal, a nontrivial concern to address is the maximum information that can be extracted from it using a quantum measurement. This quantity is called the accessible information of the ensemble. TheHolevo boundwhich is an extremely useful tool in quantum informa- tion theory, provides an upper bound on the amount of accessible information. It is
defined as follows [41] :
H(X:Y)≤S(ρ)−
∑
x
pxS(ρx), (2.82)
Whereρ =∑xpxρx. This refers to the situation when Alice prepares a stateρX, while
X =0, ...,nwith probability of occurring asp0, ...,pn, and Bob operates a POVM mea-
surement on the state with elements{Ey}={E0, ...,Em}, andybeing the outcome of
the measurement [41]. (See ref [41] for more details).