Open in Colab

Image histogram and equalizations techniques#

In this tutorial we are going to learn how using kornia components and Matplotlib we can visualize image histograms and later use kornia functionality to equalize images in batch and using the gpu.

Install Kornia#

%%capture
!pip install kornia

Prepare the data#

The low contrast color image used in this tutorial can be downloaded here (By Biem (Own work) [Public domain], via Wikimedia Commons)

%%capture
!wget https://upload.wikimedia.org/wikipedia/commons/f/fd/Crayfish_low_contrast.JPG
!wget https://upload.wikimedia.org/wikipedia/commons/1/16/Women%27s_Soccer_-_USA_vs_Japan_%281%29.jpg
!wget https://imagemagick.org/image/mountains.jpg
!wget https://p0.piqsels.com/preview/817/405/123/santa-maria-del-mar-la-catedral-del-mar-church-barcelona.jpg
from typing import List, Tuple

from matplotlib import pyplot as plt
import cv2
import numpy as np
import torch
import kornia as K
import torchvision
/home/docs/checkouts/readthedocs.org/user_builds/kornia-tutorials/envs/latest/lib/python3.7/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Image show functionality

def imshow(input: torch.Tensor, height: int, width: int):
    out: torch.Tensor = torchvision.utils.make_grid(input, nrow=2)
    out_np: np.array = K.utils.tensor_to_image(out)
    plt.figure(figsize=(height, width))
    plt.imshow(out_np); plt.axis('off');

Image read functionality

def imread(data_path: str) -> torch.Tensor:
    """Utility function that load an image an convert to torch."""
    # open image using OpenCV (HxWxC)
    img: np.ndarray = cv2.imread(data_path, cv2.IMREAD_COLOR)

    # cast image to torch tensor and convert to RGB
    img_t: torch.Tensor = K.utils.image_to_tensor(img, keepdim=False)  # BxCxHxW
    img_t = img_t.float() / 255.

    img_t = K.color.bgr_to_rgb(img_t)
    img_t = K.geometry.resize(img_t, 1200, side='long', align_corners=True)[..., :600, :]

    return img_t

Image and histogram plot functionality

def histogram_img(img_t: torch.Tensor, size: Tuple[int, int] = (16, 4)):
    CH, H, W = img_t.shape
    img = K.utils.tensor_to_image(img_t.mul(255.).byte())

    plt.figure(figsize=size)
    ax1 = plt.subplot(1, 2, 1)
    ax2 = plt.subplot(1, 2, 2)

    colors = ('b','g','r')
    kwargs = dict(histtype='stepfilled', alpha=0.3, density=True, ec="k")

    for i in range(CH):
        img_vec = img[..., i].flatten()
        ax2.hist(img_vec, range=(0, 255), bins=256, color=colors[i], **kwargs)

    ax1.imshow(img, cmap=(None if CH > 1 else 'gray'))
    ax1.grid(False)
    ax1.axis('off')

    plt.show()

Load the images in batch using OpenCV and show them as a grid.

img_rgb_list: List[torch.Tensor] = []
img_rgb_list.append(imread("Crayfish_low_contrast.JPG"))
img_rgb_list.append(imread("Women's_Soccer_-_USA_vs_Japan_(1).jpg"))
img_rgb_list.append(imread("santa-maria-del-mar-la-catedral-del-mar-church-barcelona.jpg"))
img_rgb_list.append(imread("mountains.jpg"))

# cast to torch.Tensor
img_rgb: torch.Tensor = torch.cat(img_rgb_list, dim=0)
print(f"Image tensor shape: {img_rgb.shape}")

# Disable the line below to make everything happen in the GPU !
# img_rbg = img_rbg.cuda()

imshow(img_rgb, 10, 10)  # plot grid !
[ WARN:0@1.789] global /io/opencv/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('santa-maria-del-mar-la-catedral-del-mar-church-barcelona.jpg'): can't open/read file: check file path/integrity
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_1027/725079409.py in <module>
      2 img_rgb_list.append(imread("Crayfish_low_contrast.JPG"))
      3 img_rgb_list.append(imread("Women's_Soccer_-_USA_vs_Japan_(1).jpg"))
----> 4 img_rgb_list.append(imread("santa-maria-del-mar-la-catedral-del-mar-church-barcelona.jpg"))
      5 img_rgb_list.append(imread("mountains.jpg"))
      6 

/tmp/ipykernel_1027/469960124.py in imread(data_path)
      5 
      6     # cast image to torch tensor and convert to RGB
----> 7     img_t: torch.Tensor = K.utils.image_to_tensor(img, keepdim=False)  # BxCxHxW
      8     img_t = img_t.float() / 255.
      9 

~/checkouts/readthedocs.org/user_builds/kornia-tutorials/envs/latest/lib/python3.7/site-packages/kornia/utils/image.py in image_to_tensor(image, keepdim)
     35         torch.Size([1, 3, 4, 4])
     36     """
---> 37     if len(image.shape) > 4 or len(image.shape) < 2:
     38         raise ValueError("Input size must be a two, three or four dimensional array")
     39 

AttributeError: 'NoneType' object has no attribute 'shape'

Image Histogram#

Definition - An image histogram is a type of histogram that acts as a graphical representation of the tonal distribution in a digital image.[1] It plots the number of pixels for each tonal value. By looking at the histogram for a specific image a viewer will be able to judge the entire tonal distribution at a glance Read more - Wikipedia.

In short - An image histogram is:

  • It is a graphical representation of the intensity distribution of an image.

  • It quantifies the number of pixels for each intensity value considered.

See also OpenCV tutorial: https://docs.opencv.org/master/d4/d1b/tutorial_histogram_equalization.html

Lightness with Kornia#

We first will compute the histogram of the lightness of the image. To do so, we compute first the color space Lab and take the first component known as luminance that reflects the lightness of the scene.

Notice that kornia Lab representation is in the range of [0, 100] and for convenience to plot the histogram we will normalize the image between [0, 1].

Note: kornia computes in batch, for convenience we show only one image result. That’s it - modify below the plot_indices variable to explore the results of the batch.

plot_indices: int = 0  # try: [0, 1, 2, 3]

Tip: replace K.color.rgb_to_lab by K.color.rgb_to_grayscale to see the pixel distribution in the grayscale color space. Explore also kornia.color for more exotic color spaces.

img_lab: torch.Tensor = K.color.rgb_to_lab(img_rgb)
lightness: torch.Tensor = img_lab[..., :1, :, :] / 100. # L in lab is in range [0, 100]

histogram_img(lightness[plot_indices])
_images/74786d13f775741cd22a664697fe728b54308c9a7ad184d760ea6ee9d77cb354.png

RGB histogram with Kornia#

Similar to above - you can just visualize the three (red, green, blue) channels pixel distribution.

Tip - Use as follows to visualize a single channel (green) : histogram_img(img_rgb[plot_indices, 1:2])

histogram_img(img_rgb[plot_indices])
_images/e26526d9eba5bc173785fd3fec8ac6b0b113d2ec7d7540b4d5d2c1f7f4f13462.png

Histogram stretching#

Sometimes our images have a pixel distribution that is not suitable for our application, being biased to a certain range depending on the illumination of the scene.

In the next sections, we are going to show a couple of techniques to solve those issues. We will start with a basic technique to normalize the image by its minimum and maximum values with the objective to strecth the image histrogram.

on the lightness with Kornia#

We use kornina.enhance.normalize_min_max to normalize the image Luminance. Note: compare the histrograms with the original image.

Tip - play with the other functions from kornia.enhance to modify the intensity values of the image and thus its histograms.

lightness_stretched = K.enhance.normalize_min_max(lightness)
histogram_img(lightness_stretched[plot_indices])
_images/70b865b01a922624ecb8e2942d7e8ad45b49f1a41e040e59fcee1a9f2cc22ff9.png

In order to properly visualize the effect of the normalization in the color histogram, we take the normalized Luminance and use it to cast back to RGB.

img_rgb_new = K.color.lab_to_rgb(torch.cat([lightness_stretched * 100., img_lab[:, 1:]], dim=1))
histogram_img(img_rgb_new[plot_indices])
_images/2a6fac995da1acca166f05d84c884b98a4050046c7c23f47d798b83633d6f5f8.png

on the RGB with Kornia#

In this case we normalize each channel independently where we can see that resulting image is not as clear as the one only stretching the Luminance.

rgb_stretched = K.enhance.normalize_min_max(img_rgb)
histogram_img(rgb_stretched[plot_indices])
_images/e22052b4622a1c09d05d68398fcaf690c0b5e9f564faae35f79a59bc3e6c2a53.png

Histogram Equalization#

A more advanced technique to improve the pixel distribution is the so called Histogram equalization - a method in image processing of contrast adjustment using the image’s histogram [Read more - Wikipedia].

In kornia we have implemented in terms of torch tensor to equalize the images in batch and the gpu very easily.

on the lightness with Kornia#

lightness_equalized = K.enhance.equalize(lightness)
histogram_img(lightness_equalized[plot_indices])
/usr/local/lib/python3.7/dist-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  /pytorch/aten/src/ATen/native/BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)
_images/cacb562b30d05c5c397e5b84afc337b8535e36afdac606bb94b30a828f86b00f.png

We convert back from Lab to RGB using the equalized Luminance and visualize the histogram of the RGB.

img_rgb_new = K.color.lab_to_rgb(torch.cat([lightness_equalized * 100., img_lab[:, 1:]], dim=1))
histogram_img(img_rgb_new[plot_indices])
_images/ac61072e9a5edb2b387afefd38e683749d32f9ce81a8080c321708be0e4acbbb.png

on the RGB with Kornia#

rgb_equalized = K.enhance.equalize(img_rgb)
histogram_img(rgb_equalized[plot_indices])
_images/f6e005d6776eeb0fd4c9b9cc554b1b4171658b3d436b9a874fe1a97bbeb5dfa5.png

on the RGB with OpenCV#

Just to compare against OpenCV - close results :)

rgb_equalized_cv = []
for img in img_rgb:
    equ00 = torch.tensor(cv2.equalizeHist(K.utils.tensor_to_image(img[0].mul(255).clamp(0, 255).byte())))
    equ01 = torch.tensor(cv2.equalizeHist(K.utils.tensor_to_image(img[1].mul(255).clamp(0, 255).byte())))
    equ02 = torch.tensor(cv2.equalizeHist(K.utils.tensor_to_image(img[2].mul(255).clamp(0, 255).byte())))
    rgb_equalized_cv.append(torch.stack([equ00, equ01, equ02]))
rgb_equalized_cv = torch.stack(rgb_equalized_cv)

histogram_img(rgb_equalized_cv[plot_indices] / 255.)
_images/2ee6ebe7c48b8b1c0eeecf77dd5dcadda3c768e73c98ed62c079ee5f1ffaf2b9.png

Adaptive Histogram Equalization#

Adaptive histogram equalization (AHE) is a computer image processing technique used to improve contrast in images. It differs from ordinary histogram equalization in the respect that the adaptive method computes several histograms, each corresponding to a distinct section of the image, and uses them to redistribute the lightness values of the image. It is therefore suitable for improving the local contrast and enhancing the definitions of edges in each region of an image [Read more - Wikipedia].

on the lightness with Kornia#

We will use kornia.enhance.equalize_clahe and by playing with the clip_limit and grid_size variables to produce different effects to the image.

lightness_equalized = K.enhance.equalize_clahe(lightness, clip_limit=0.)
histogram_img(lightness_equalized[plot_indices])
_images/192248111fa08428b089dec1fd3baba4d9417f39689132646ecf197f0ca4f6c4.png

We convert back from Lab to RGB using the equalized Luminance and visualize the histogram of the RGB.

img_rgb_new = K.color.lab_to_rgb(torch.cat([lightness_equalized * 100., img_lab[:, 1:]], dim=1))
histogram_img(img_rgb_new[plot_indices])
_images/653d8cc5f7f2a824e2314d24c6819cb6227d264cb2e0b259799da9220e88e151.png

on the RGB with Kornia#

rgb_equalized = K.enhance.equalize_clahe(img_rgb, clip_limit=0.)
histogram_img(rgb_equalized[plot_indices])
_images/085059d06e35feb9c5ff37cde2743a0f9204d0d53839dab6a0736cfefc60e1a5.png

Contrast Limited Adaptive Histogram Equalization (CLAHE)#

An improvement of the algorithm is CLAHE that divides the image into small blocks and controlled by the variable grid_size. This means, that the equalization is performed locally in each of the NxM sublocks to obtain a better distribution of the pixel values.

on the lightness with Kornia#

lightness_equalized = K.enhance.equalize_clahe(lightness, clip_limit=20., grid_size=(8,8))
histogram_img(lightness_equalized[plot_indices])
_images/7b3b2fad6fa0b2afe3bda5c0a53158f6450b5f7416542d7196f5f3c337a15b3b.png

We convert back from Lab to RGB using the equalized Luminance and visualize the histogram of the RGB.

img_rgb_new = K.color.lab_to_rgb(torch.cat([lightness_equalized * 100., img_lab[:, 1:]], dim=1))
histogram_img(img_rgb_new[plot_indices])
_images/19a61f794dd91fd98c1b767f73f2e5a9a06ebb54474ad948a35e84c414d7bc3a.png

on the RGB with Kornia#

We directly equalize all the RGB channels at once

rgb_equalized = K.enhance.equalize_clahe(img_rgb, clip_limit=20., grid_size=(8,8))
histogram_img(rgb_equalized[plot_indices])
_images/384bf27cd4cf1cfeb41445b53bdc96e6de20969c131ac58e4923694444ebb1a8.png

on the RGB with OpenCV#

imgs = []
clahe = cv2.createCLAHE(clipLimit=20.0, tileGridSize=(8, 8))
for im in img_rgb:
  # equalize channels independently as gray scale images
  equ00 = torch.tensor(clahe.apply(K.utils.tensor_to_image(im[0].mul(255).clamp(0, 255).byte())))
  equ01 = torch.tensor(clahe.apply(K.utils.tensor_to_image(im[1].mul(255).clamp(0, 255).byte())))
  equ02 = torch.tensor(clahe.apply(K.utils.tensor_to_image(im[2].mul(255).clamp(0, 255).byte())))
  imgs.append(torch.stack([equ00, equ01, equ02]))
imgs = torch.stack(imgs)

histogram_img(imgs[plot_indices] / 255.)
_images/12ca849d286ffae1b6198440b84396b022545f68e0361f752ad2b27f2056950c.png