Shannon's Entropy for Images

Image Entropy turned out to be really interesting. I wanted to find the maximum entropy for a given image size and resolution and started with the assumption that it would occur with all pixel values equally probable. For most images we use 8 bits per RGB channel so its easy to figure out the maximum but what about real numbers that have infinite resolution? What about small images when we dont have enough pixels to represent the 256 values for 8 bit?

Image Entropy

To calculate the Shannon’s entropy for data \(x\) we use,

\[\begin{align} H = -\sum_i {\bf{p}}(x_i)\log_2 {\bf{p}}(x_i), \end{align}\]

for probabilities \({\bf{p}}(x_i)\) and entropy calculated in bits.

For an image we need to determine the pixel depth \(p\), often \(p=8\) for 8 bits or 256 levels per pixel or \(p=\infty\) for real numbers, and then bin the pixel values to determine their probability.

Maximum Entropy value for Images

To find the maximum possible entropy for a specific image size \(m\times n\) and pixel depth \(p\) we observe that entropy is maximized when all of the probabilities are equal. So for a \(16\times 16\) image with \(p=8\) we can define 4 pixels per pixel value (\(n\times m / 2^p\)).

For small image sizes there may be insufficient pixels to support all pixel values. For example a \(7\times 7\) image can only support a pixel depth of \(\log_2 7^2\) or 5.614.

Our maximum entropy, for an \(n\times m\) image with pixel depth \(p\) is,

\[\begin{align} H_{\mathrm{max}}^p(m,n) &= -\frac{mn}{2^{2\hat p}}\log_2 \left(\frac{mn}{2^{2\hat p}}\right)\times 2^{\hat p}, \\ \label{eqn:maxent} &= -\frac{1}{2^{\hat p}}nm\left(log_2\left( nm\right) - 2\hat p\right), \end{align}\]

where \(\hat p= \min(p, \log_2( nm))\).

We note that for the common case for arrays of float values, where the pixel depth is infinite, the ``effective’’ pixel depth is \(\log_2( nm))\) and the maximum entropy Equation above can be simplified to,

\[\begin{align} H_{\mathrm{max}}^{\infty}(m,n) &= \log_2 \left(mn\right). \end{align}\]

If you want to know what a maximum entropy image looks like here is such a thing, Maximum Entropy Image