Panorama Motion Stacking , Part 5

Panorama is a popular entertainment for people in rural areas. Stories are not yet known cases that the sausage photo would be interesting to someone other than its author, but it is impossible not to mention it - for many, stacking began with this.

The first useful way to use a panorama is to get a higher resolution photo than the camera matrix allows by gluing together several frames. Photographers have long used different software for the so-called super-resolution photos - when slightly shifted photos seem to complement each other between the pixels. This way you can get even hundreds of gigapixels , which is very useful if you need to print it on a house-sized advertising poster.

Another, more interesting approach is Pixel Shifting. Some mirrors like Sony and Olympus started to support him since 2014, but they still forced me to glue the results. Typical innovations of large cameras.

Smartphones have succeeded here for a ridiculous reason - when you take a photo, your hands are shaking. This seemingly problem formed the basis for the implementation of native super-resolution on smartphones.

To understand how it works, we need to remember how the matrix of any camera is arranged. Each pixel (photodiode) is able to capture only the intensity of light - i.e. the number of photons flown. However, a pixel cannot measure its color (wavelength). To get an RGB picture, I had to pile up crutches here - to cover the entire matrix with a grid of multi-colored glass windows. Its most popular implementation is called the Bayer filter and is used today in most matrices. Looks like the picture below.

It turns out that each pixel of the matrix catches only the R, G or B component, because the rest of the photons are mercilessly reflected by the Bayer filter. He recognizes the missing components by stupid averaging the values ​​of the neighboring pixels.

There are more green cells in the Bayer filter - this is done by analogy with the human eye. It turns out that out of 50 million pixels on the matrix, green will pick up 25 million, red and blue - 12.5 million each. The rest will be averaged - this process is called debayer or demosaic, and this is such a fat, funny crutch on which everything keeps.

In fact, each matrix has its own smart-ass proprietary demozaic algorithm, but in the context of this story, we don’t give a shit.

Other types of matrices (such as Foveon ) have not yet taken root at all well. Although some manufacturers try to use the matrix without a Bayer filter to improve sharpness and dynamic range.

When the light is small or the details of the object are completely tiny, we lose a lot of information because the Bayer filter brazenly cuts off photons with an objectionable wavelength. Therefore, they came up with Pixel Shifting - move the matrix 1 pixel up, down, left and right to catch them all. The photo does not work out 4 times as much as it may seem, just the processor uses this data to more accurately record the value of each pixel. It averages not by neighbors, so to speak, but by the four values ​​of itself.

Shaking our hands when taking photos on the phone makes this process a natural consequence. In the latest versions of Google Pixel, this thing is implemented and turned on whenever you use the zoom on the phone - called Super Res Zoom (yes, I also like their merciless naming). The Chinese also copied it into their liaophones, although it turned out a little worse.

Overlapping slightly biased photos allows you to collect more information about the color of each pixel, and therefore reduce noise, increase sharpness and increase the resolution without increasing the physical number of matrix megapixels. Modern Android flagships do this automatically, until their users even think about it…
cont…