# PyTorch crop images differentiablly

## Intro

PyTorch provides a variety of means to crop images. For example, torchvision.transforms provides several functions to crop PIL images; PyTorch Forum provides an answer of how to crop image in a differentiable way (differentiable with respect to the image). However, sometimes we need a fully differentiable approach for the cropping action itself. How shall we implement that?

## Theory: Affine transformation

Before reaching the answer, we need first to learn about the image coordinate system in PyTorch. It is a left-handed Cartesian system origined at the middle of an image. The coordinate has been normalized to range $[-1,1]$, where $(-1,-1)$ indicates the top-left corner, and $(1,1)$ indicates the bottom-right corner, as pointed out by the doc.

Let $(x,y)$ be the top-left corner of the cropped image with respect to the coordinate of the original image; likewise, we denote $(x',y')$ as the bottom-right corner of the cropped image. It’s clear that $(x,y)$ corresponds to $(-1,-1)$ with respect to the cropped image coordinate system, and $(x',y')$ corresponds to $(1,1)$. We’d like a function $f$ that maps from the cropped image system to the original image system for every point in the cropped image. Since only scaling and translation are involved, the function $f$ can be parameterized by an affine transformation matrix $\Theta$ such that

$\Theta = \begin{pmatrix} \theta_{11} & 0 & \theta_{13}\\ 0 & \theta_{22} & \theta_{23}\\ 0 & 0 & 1\\ \end{pmatrix}$

where $\theta_{12}=\theta_{21}=0$ since skewing is not involved. Denote $\mathbf{u}_H$ as the homogeneous coordinate of $\mathbf{u}=\begin{pmatrix}u & v\\ \end{pmatrix}^T$ such that $\mathbf{u}_H=\begin{pmatrix}\mathbf{u}^T&1\end{pmatrix}^T$, $\Theta$ maps $\mathbf{u}_H$ with respect to the cropped image system to $\mathbf{x}_H$ with respect to the original image system, i.e. $\mathbf{x}_H = \Theta \mathbf{u}_H$. Thus,

$\begin{pmatrix} x & x'\\ y & y'\\ 1 & 1 \end{pmatrix} = \begin{pmatrix} \theta_{11} & 0 & \theta_{13}\\ 0 & \theta_{22} & \theta_{23}\\ 0 & 0 & 1\\ \end{pmatrix} \begin{pmatrix} -1 & 1\\ -1 & 1\\ 1 & 1\\ \end{pmatrix}$

Solving the equations,

$\Theta = \begin{pmatrix} \frac{x'-x}{2} & 0 & \frac{x'+x}{2}\\ 0 & \frac{y'-y}{2} & \frac{y'+y}{2}\\ 0 & 0 & 1\\ \end{pmatrix}$

where $x'\ge x, y' \ge y$.

## Coding time

We’ll need two functions:

1. torch.nn.functional.affine_grid to convert the $\Theta$ parameterization to $f$
2. torch.nn.functional.grid_sample to find the corresponding original image coordinate from each cropped image coordinate
import torch
import torch.nn.functional as F

B, C, H, W = 16, 3, 224, 224  # batch size, input channels
# original image height and width
# Let I be our original image
I = torch.rand(B, C, H, W)
# Set the (x,y) and (x',y') to define the rectangular region to crop
x, y = -0.5, -0.3  # some examplary random coordinates;
x_, y_ = 0.7, 0.8  # in practice, (x,y,x_,y_) might be predicted
# as a tensor in the computation graph
# Set the affine parameters
theta = torch.tensor([
[(x_-x)/2,       0, (x_+x)/2],
[       0,(y_-y)/2, (y_+y)/2],
]).unsqueeze_(0).expand(B, -1, -1)
# compute the flow field;
# where size is the output size (scaling involved)
# align_corners option must be the same throughout the code
f = F.affine_grid(theta, size=(B, C, H//2, W//2), align_corners=False)
I_cropped = F.grid_sample(I, f, align_corners=False) 01-08 2353

#### [Pytorch]Pytorch中图像的基本操作(TenCrop)

01-04 1万+

#### PyTorch源码解析--torchvision.transforms（数据预处理、数据增强）

12-14 8333

#### pytorch中T.RandomResizedCrop()等图像操作转换的效果

01-17 8297

#### PyTorch学习之数据增强（image transformations）

11-30 13万+

#### 基于PyTorch的深度学习入门教程（一）——PyTorch安装和配置

05-26 4020

#### pytorch之Resize()函数

11-25 6377

#### Pytorch LSTM 时间序列预测

06-30 5655

#### pytorch中图片数据增强的方法

12-06 7万+

#### Ubuntu16.04环境下PyTorch简易安装教程

02-01 1051

#### 【从零学习PyTorch】 transfrom之五种Crop裁剪的方法

12-26 1万+

#### PyTorch 学习笔记（三）：transforms的二十二个方法

11-02 510

#### Pytorch--RandomScaleCrop操作

02-29 1万+

#### 玩PyTorch？你不得不看的PyTorch资源大列表

12-01 1万+

#### 什么是PyTorch，为何要使用PyTorch

12-19 9356

#### pytorch之ImageFolder

05-06 1万+

#### PyTorch 1.0：PyTorch 1.0的简介、安装、使用方法详细攻略

02-03 6783

#### PyTorch 1.0 中文官方教程：使用 PyTorch C++ 前端

06-25 1573

#### PyTorch学习笔记(16)——编写你自己的PyTorch kernel(基于PyTorch1.2.0)

03-29 1万+

#### pytorch: PyTorch中 使用 Tensorboard

06-03 2149

#### windows下pytorch安装

04-30 1万+

#### DL框架之PyTorch：PyTorch的简介、安装、使用方法之详细攻略

11-04 2625

#### PyTorch—整体框架

03-19 82万+

#### 大学四年自学走来，这些私藏的实用工具/学习网站我贡献出来了

04-14 60万+

#### 在中国程序员是青春饭吗？

02-28 1万+

#### springboot+jwt实现token登陆权限认证

03-01 14万+

#### 技术大佬：我去，你写的 switch 语句也太老土了吧

03-08 7万+

#### 女程序员，为什么比男程序员少？？？

04-25 7万+

#### 总结了 150 余个神奇网站，你不来瞅瞅吗？

03-10 13万+

#### 副业收入是我做程序媛的3倍，工作外的B面人生是怎样的？ ©️2019 CSDN 皮肤主题: 大白 设计师: CSDN官方博客 点击重新获取   扫码支付