Image matching example with LoFTR¶
First, we will install everything needed:
fresh version of OpenCV for MAGSAC++ geometry estimation
kornia_moons for the conversions and visualization
%%capture
!pip install git+https://github.com/kornia/kornia
!pip install kornia_moons
!pip install opencv-python --upgrade
Now let’s download an image pair
%%capture
!wget https://github.com/kornia/data/raw/main/matching/kn_church-2.jpg
!wget https://github.com/kornia/data/raw/main/matching/kn_church-8.jpg
First, we will define image matching pipeline with OpenCV SIFT features. We will also use kornia for the state-of-the-art match filtering – Lowe ratio + mutual nearest neighbor check and Pydegensac as RANSAC
import matplotlib.pyplot as plt
import cv2
import kornia as K
import kornia.feature as KF
import numpy as np
import torch
from kornia_moons.feature import *
def load_torch_image(fname):
img = K.image_to_tensor(cv2.imread(fname), False).float() /255.
img = K.color.bgr_to_rgb(img)
return img
fname1 = 'kn_church-2.jpg'
fname2 = 'kn_church-8.jpg'
img1 = load_torch_image(fname1)
img2 = load_torch_image(fname2)
matcher = KF.LoFTR(pretrained='outdoor')
input_dict = {"image0": K.color.rgb_to_grayscale(img1), # LofTR works on grayscale images only
"image1": K.color.rgb_to_grayscale(img2)}
with torch.no_grad():
correspondences = matcher(input_dict)
/home/docs/checkouts/readthedocs.org/user_builds/kornia-tutorials/envs/latest/lib/python3.7/site-packages/kornia/feature/loftr/utils/coarse_matching.py:255: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
[i_ids % data['hw0_c'][1], i_ids // data['hw0_c'][1]],
/home/docs/checkouts/readthedocs.org/user_builds/kornia-tutorials/envs/latest/lib/python3.7/site-packages/kornia/feature/loftr/utils/coarse_matching.py:258: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
[j_ids % data['hw1_c'][1], j_ids // data['hw1_c'][1]],
/home/docs/checkouts/readthedocs.org/user_builds/kornia-tutorials/envs/latest/lib/python3.7/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
for k,v in correspondences.items():
print (k)
keypoints0
keypoints1
confidence
batch_indexes
Now let’s clean-up the correspondences with modern RANSAC and estimate fundamental matrix between two images
mkpts0 = correspondences['keypoints0'].cpu().numpy()
mkpts1 = correspondences['keypoints1'].cpu().numpy()
H, inliers = cv2.findFundamentalMat(mkpts0, mkpts1, cv2.USAC_MAGSAC, 0.5, 0.999, 100000)
inliers = inliers > 0
Finally, let’s draw the matches with a function from kornia_moons. The correct matches are in green and imprecise matches - in blue
draw_LAF_matches(
KF.laf_from_center_scale_ori(torch.from_numpy(mkpts0).view(1,-1, 2),
torch.ones(mkpts0.shape[0]).view(1,-1, 1, 1),
torch.ones(mkpts0.shape[0]).view(1,-1, 1)),
KF.laf_from_center_scale_ori(torch.from_numpy(mkpts1).view(1,-1, 2),
torch.ones(mkpts1.shape[0]).view(1,-1, 1, 1),
torch.ones(mkpts1.shape[0]).view(1,-1, 1)),
torch.arange(mkpts0.shape[0]).view(-1,1).repeat(1,2),
K.tensor_to_image(img1),
K.tensor_to_image(img2),
inliers,
draw_dict={'inlier_color': (0.2, 1, 0.2),
'tentative_color': None,
'feature_color': (0.2, 0.5, 1), 'vertical': False})
