Shannon's Entropy for Images

Image Entropy turned out to be really interesting. I wanted to find the maximum entropy for a given image size and resolution and started with the assumption that it would occur with all pixel values equally probable. For most images we use 8 bits per RGB channel so its easy to figure out the maximum but what about real numbers that have infinite resolution? What about small images when we dont have enough pixels to represent the 256 values for 8 bit?

Image Entropy

To calculate the Shannon’s entropy for data \(x\) we use,

\[\begin{align} H = -\sum_i {\bf{p}}(x_i)\log_2 {\bf{p}}(x_i), \end{align}\]

for probabilities \({\bf{p}}(x_i)\) and entropy calculated in bits.

For an image we need to determine the pixel depth \(p\), often \(p=8\) for 8 bits or 256 levels per pixel or \(p=\infty\) for real numbers, and then bin the pixel values to determine their probability.

Maximum Entropy value for Images

To find the maximum possible entropy for a specific image size \(m\times n\) and pixel depth \(p\) we observe that entropy is maximized when all of the probabilities are equal. So for a \(16\times 16\) image with \(p=8\) we can define 4 pixels per pixel value (\(n\times m / 2^p\)).

For small image sizes there may be insufficient pixels to support all pixel values. For example a \(7\times 7\) image can only support a pixel depth of \(\log_2 7^2\) or 5.614.

Our maximum entropy, for an \(n\times m\) image with pixel depth \(p\) is,

\[\begin{align} H_{\mathrm{max}}^p(m,n) &= -\frac{mn}{2^{2\hat p}}\log_2 \left(\frac{mn}{2^{2\hat p}}\right)\times 2^{\hat p}, \\ \label{eqn:maxent} &= -\frac{1}{2^{\hat p}}nm\left(log_2\left( nm\right) - 2\hat p\right), \end{align}\]

where \(\hat p= \min(p, \log_2( nm))\).

We note that for the common case for arrays of float values, where the pixel depth is infinite, the ``effective’’ pixel depth is \(\log_2( nm))\) and the maximum entropy Equation above can be simplified to,

\[\begin{align} H_{\mathrm{max}}^{\infty}(m,n) &= \log_2 \left(mn\right). \end{align}\]

If you want to know what a maximum entropy image looks like here is such a thing, Maximum Entropy Image


Scanning the Taj Mahal

The Taj Mahal, completed in 1643, was the only Mughal mausoleum that was built by a ruler for his own purposes during his own lifetime. This may serve as one of the reasons that the sheer scale, delicacy of material usage, and overall harmonic aesthetics are more prevalent in this structure than any other Mughal mausoleum. Built to honor his wife, Mumtaz Mahal, Shah Jahan built not only one, but two major mausoleums during his lifetime, one for his wife and another for his father Jahangir. It was under the rule of Shah Jahan, that it is said that Mughal architecture entered its classical phase.

In 2015 we got to 3D scan the Taj Mahal! This is a picture of me on the roof! Chris on the roof of the Taj Mahal

As part of our work with DHARMA Digital Historic Architectural Research and Material Analysis group we have been documenting various heritage sites to generate 3D scans for future preservation of the monuments. It was an amazing experience and ouside the scope of work for most researchers/software developers.

We have also visited the Roman Forum many times, the following image is made up of many scans we did from different viewpoints. Scan of the roman Forum