On an average we require H(X) bits of information to specify one input symbol.

**MUTUAL
INFORMATION**

On
an average we require ** H(X)** bits of information to specify
one input symbol. However, if we are allowed to observe the output symbol
produced by that input, we require, then, only

Notice
that in spite of the variations in the source probabilities, ** p (x_{k})**
(may be due to noise in the channel), certain probabilistic information
regarding the state of the input is available, once the conditional probability

This
is the definition with which we started our discussion on information theory.
Accordingly *I*** (x_{k}) **is also referred to
as

*Eq
(4.22) simply means that “the Mutual information ‟ is symmetrical with respect
to its arguments.i.e.*

*I (x _{k},
y_{j}) = I (y_{j}, x_{k})*

Averaging
Eq. (4.21*b*) over all admissible characters ** x_{k}**
and

*I***( X,
Y) = E {I (x_{k}, y_{j})}**

Or *I*(*X*,*Y*) =*H*(*X*)
+*H*(*Y*)**– H(X,**

Further, we conclude
that, ― even though for a particular received symbol, yj, H(X) –H(X | Yj) may
be negative, when all the admissible output symbols are covered, the average
mutual information is always non- negative‖. That is to say, we cannot loose information
on an average by observing the output of
a channel. An easy method, of remembering the various relationships, is given
in Fig 4.2.Althogh the diagram resembles a Venn-diagram, it is not, and the
diagram is only a tool to remember the relationships. That is all. You cannot
use this diagram for proving any result.

The
entropy of ** X** is represented by the circle on the left and that of

*H***( X
| Y) = H(X) –I(X, Y) and
H (Y| X) = H(Y) –I(X, Y)**

The
joint entropy ** H(X,Y)** is the sum of

*H***( X,
Y) = H(X) + H(Y) - I(X, Y)**

Also observe *H*(*X*,*Y*) =*H*(*X*) +*H***( Y|X)**

**= H(Y) + H(X
|Y)**

For the **JPM**
given by I**( X,**

Study Material, Lecturing Notes, Assignment, Reference, Wiki description explanation, brief detail

Analog and Digital Communication : Mutual Information |

**Related Topics **

Privacy Policy, Terms and Conditions, DMCA Policy and Compliant

Copyright © 2018-2023 BrainKart.com; All Rights Reserved. Developed by Therithal info, Chennai.