Information Theory

Fundamental Properties of Information

Learned about this while trying to understand why Information Theory uses log to characterize entropy.

This is also talked about in wikipedia here: https://en.wikipedia.org/wiki/Entropy_(information_theory)#Characterization

Suppose you want a function $I(p)$ that tells you how much “information” is gained when observing an event with probability ppp.

Shannon proved that if you want this function to satisfy the following properties:

1. Non-negativity: $I(p)\ge 0$, and $I(1)=0$ (no surprise for certain events). Events that always occur do not communicate information.
2. Monotonicity: Rarer events carry more information: $p_1<p_2⟹I(p_1)>I(p_2)$
3. Additivity: The information of independent events multiplies probabilities, i.e. should add information: $I(p_1\cdot p_2)=I(p_1)+I(p_2)$
   • see Log Laws

A function that satisfies all these is: $I(p)=\log \left(\frac{1}{p}\right)=-\log p$

Intuition behind additivity:

• Imagine a coin flip. Every time you get the result of coin flip, there’s more entropy to the system. So these coin flips should add information

Related

• Entropy