IOS Dev Intro - Image Resizing

http://nshipster.com/image-resizing/

Image Resizing Techniques

Written by Mattt Thompson — September 15^th, 2014 (revised)

Since time immemorial, iOS developers have been perplexed by a singular question: “How do you resize an image?”. It is a question of beguiling clarity, spurred on by a mutual mistrust of developer and platform. A thousand code samples litter web search results, each claiming to be the One True Solution, and all the others false prophets.

It’s embarrassing, really.

This week’s article endeavors to provide a clear explanation of the various approaches to image resizing on iOS (and OS X, making the appropriate UIImage → NSImage conversions), using empirical evidence to offer insights into the performance characteristics of each approach, rather than simply prescribing any one way for all situations.

Before reading any further, please note the following:

When setting a UIImage on a UIImageView, manual resizing is unnecessary for the vast majority of use cases. Instead, one can simply set the contentMode property to either .ScaleAspectFit to ensure that the entire image is visible within the frame of the image view, or .ScaleAspectFill to have the entire image view filled by the image, cropping as necessary from the center.

imageView.contentMode = .ScaleAspectFit
imageView.image = image

---

Determining Scaled Size

Before doing any image resizing, one must first determine the target size to scale to.

Scaling by Factor

The simplest way to scale an image is by a constant factor. Generally, this involves dividing by a whole number to reduce the original size (rather than multiplying by a whole number to magnify).

A new CGSize can be computed by scaling the width and height components individually:

let size = CGSize(width: image.size.width / 2, height: image.size.height / 2)

…or by applying a CGAffineTransform:

let size = CGSizeApplyAffineTransform(image.size, CGAffineTransformMakeScale(0.5, 0.5))

Scaling by Aspect Ratio

It’s often useful to scale the original size in such a way that fits within a rectangle without changing the original aspect ratio. AVMakeRectWithAspectRatioInsideRect is a useful function found in the AVFoundation framework that takes care of that calculation for you:

import AVFoundation
let rect = AVMakeRectWithAspectRatioInsideRect(image.size, imageView.bounds)

Resizing Images

There are a number of different approaches to resizing an image, each with different capabilities and performance characteristics.

UIGraphicsBeginImageContextWithOptions & UIImage -drawInRect:

The highest-level APIs for image resizing can be found in the UIKit framework. Given a UIImage, a temporary graphics context can be used to render a scaled version, usingUIGraphicsBeginImageContextWithOptions() and UIGraphicsGetImageFromCurrentImageContext():

let image = UIImage(contentsOfFile: self.URL.absoluteString!)

let size = CGSizeApplyAffineTransform(image.size, CGAffineTransformMakeScale(0.5, 0.5))
let hasAlpha = false
let scale: CGFloat = 0.0 // Automatically use scale factor of main screen

UIGraphicsBeginImageContextWithOptions(size, !hasAlpha, scale)
image.drawInRect(CGRect(origin: CGPointZero, size: size))

let scaledImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()

UIGraphicsBeginImageContextWithOptions() creates a temporary rendering context into which the original is drawn. The first argument, size, is the target size of the scaled image. The second argument, isOpaque is used to determine whether an alpha channel is rendered. Setting this to falsefor images without transparency (i.e. an alpha channel) may result in an image with a pink hue. The third argument scale is the display scale factor. When set to 0.0, the scale factor of the main screen is used, which for Retina displays is 2.0 or higher (3.0 on the iPhone 6 Plus).

CGBitmapContextCreate & CGContextDrawImage

Core Graphics / Quartz 2D offers a lower-level set of APIs that allow for more advanced configuration. Given a CGImage, a temporary bitmap context is used to render the scaled image, using CGBitmapContextCreate() and CGBitmapContextCreateImage():

let cgImage = UIImage(contentsOfFile: self.URL.absoluteString!).CGImage

let width = CGImageGetWidth(cgImage) / 2
let height = CGImageGetHeight(cgImage) / 2
let bitsPerComponent = CGImageGetBitsPerComponent(cgImage)
let bytesPerRow = CGImageGetBytesPerRow(cgImage)
let colorSpace = CGImageGetColorSpace(cgImage)
let bitmapInfo = CGImageGetBitmapInfo(cgImage)

let context = CGBitmapContextCreate(nil, width, height, bitsPerComponent, bytesPerRow, colorSpace, bitmapInfo.rawValue)

CGContextSetInterpolationQuality(context, kCGInterpolationHigh)

CGContextDrawImage(context, CGRect(origin: CGPointZero, size: CGSize(width: CGFloat(width), height: CGFloat(height))), cgImage)

let scaledImage = CGBitmapContextCreateImage(context).flatMap { UIImage(CGImage: $0) }

CGBitmapContextCreate takes several arguments to construct a context with desired dimensions and amount of memory for each channel within a given colorspace. In the example, these values are fetched from the CGImage. Next, CGContextSetInterpolationQuality allows for the context to interpolate pixels at various levels of fidelity. In this case, kCGInterpolationHigh is passed for best results. CGContextDrawImage allows for the image to be drawn at a given size and position, allowing for the image to be cropped on a particular edge or to fit a set of image features, such as faces. Finally, CGBitmapContextCreateImage creates a CGImage from the context.

CGImageSourceCreateThumbnailAtIndex

Image I/O is a powerful, yet lesser-known framework for working with images. Independent of Core Graphics, it can read and write between many different formats, access photo metadata, and perform common image processing operations. The framework offers the fastest image encoders and decoders on the platform, with advanced caching mechanisms and even the ability to load images incrementally.

CGImageSourceCreateThumbnailAtIndex offers a concise API with different options than found in equivalent Core Graphics calls:

import ImageIO

if let imageSource = CGImageSourceCreateWithURL(self.URL, nil) {
    let options: [NSString: NSObject] = [
        kCGImageSourceThumbnailMaxPixelSize: max(size.width, size.height) / 2.0,
        kCGImageSourceCreateThumbnailFromImageAlways: true
    ]

    let scaledImage = CGImageSourceCreateThumbnailAtIndex(imageSource, 0, options).flatMap { UIImage(CGImage: $0) }
}

Given a CGImageSource and set of options, CGImageSourceCreateThumbnailAtIndex creates a thumbnail image. Resizing is accomplished by the kCGImageSourceThumbnailMaxPixelSize. Specifying the maximum dimension divided by a constant factor scales the image while maintaining the original aspect ratio. By specifying either kCGImageSourceCreateThumbnailFromImageIfAbsent orkCGImageSourceCreateThumbnailFromImageAlways, Image I/O will automatically cache the scaled result for subsequent calls.

Lanczos Resampling with Core Image

Core Image provides a built-in Lanczos resampling functionality with the CILanczosScaleTransformfilter. Although arguably a higher-level API than UIKit, the pervasive use of key-value coding in Core Image makes it unwieldy.

That said, at least the pattern is consistent. The process of creating a transform filter, configuring it, and rendering an output image is just like any other Core Image workflow:

let image = CIImage(contentsOfURL: self.URL)

let filter = CIFilter(name: "CILanczosScaleTransform")!
filter.setValue(image, forKey: "inputImage")
filter.setValue(0.5, forKey: "inputScale")
filter.setValue(1.0, forKey: "inputAspectRatio")
let outputImage = filter.valueForKey("outputImage") as! CIImage

let context = CIContext(options: [kCIContextUseSoftwareRenderer: false])
let scaledImage = UIImage(CGImage: self.context.createCGImage(outputImage, fromRect: outputImage.extent()))

CILanczosScaleTransform accepts an inputImage, inputScale, and inputAspectRatio, all of which are pretty self-explanatory. A CIContext is used to create a UIImage by way of a CGImageRefintermediary representation, since UIImage(CIImage:) doesn’t often work as expected.

Creating a CIContext is an expensive operation, so a cached context should always be used for repeated resizing. A CIContext can be created using either the GPU or the CPU (much slower) for rendering—use the kCIContextUseSoftwareRenderer key in the options dictionary to specify which.

vImage in Accelerate

The Accelerate framework includes a suite of vImage image-processing functions, with a set of functions that scale an image buffer. These lower-level APIs promise high performance with low power consumption, but at the cost of managing the buffers yourself. The following is a Swift version of a method suggested by Nyx0uf on GitHub:

let cgImage = UIImage(contentsOfFile: self.URL.absoluteString!).CGImage

// create a source buffer
var format = vImage_CGImageFormat(bitsPerComponent: 8, bitsPerPixel: 32, colorSpace: nil, 
    bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.First.rawValue), 
    version: 0, decode: nil, renderingIntent: CGColorRenderingIntent.RenderingIntentDefault)
var sourceBuffer = vImage_Buffer()
defer {
    sourceBuffer.data.dealloc(Int(sourceBuffer.height) * Int(sourceBuffer.height) * 4)
}

var error = vImageBuffer_InitWithCGImage(&sourceBuffer, &format, nil, cgImage, numericCast(kvImageNoFlags))
guard error == kvImageNoError else { return nil }

// create a destination buffer
let scale = UIScreen.mainScreen().scale
let destWidth = Int(image.size.width * 0.5 * scale)
let destHeight = Int(image.size.height * 0.5 * scale)
let bytesPerPixel = CGImageGetBitsPerPixel(image.CGImage) / 8
let destBytesPerRow = destWidth * bytesPerPixel
let destData = UnsafeMutablePointer<UInt8>.alloc(destHeight * destBytesPerRow)
defer {
    destData.dealloc(destHeight * destBytesPerRow)
}
var destBuffer = vImage_Buffer(data: destData, height: vImagePixelCount(destHeight), width: vImagePixelCount(destWidth), rowBytes: destBytesPerRow)

// scale the image
error = vImageScale_ARGB8888(&sourceBuffer, &destBuffer, nil, numericCast(kvImageHighQualityResampling))
guard error == kvImageNoError else { return nil }

// create a CGImage from vImage_Buffer
let destCGImage = vImageCreateCGImageFromBuffer(&destBuffer, &format, nil, nil, numericCast(kvImageNoFlags), &error)?.takeRetainedValue()
guard error == kvImageNoError else { return nil }

// create a UIImage
let scaledImage = destCGImage.flatMap { UIImage(CGImage: $0, scale: 0.0, orientation: image.imageOrientation) }

The Accelerate APIs used here clearly operate at a lower-level than the other resizing methods. To use this method, you first create a source buffer from your CGImage using a vImage_CGImageFormatwith vImageBuffer_InitWithCGImage(). The destination buffer is allocated at the desired image resolution, then vImageScale_ARGB8888 does the actual work of resizing the image. Managing your own buffers when operating on images larger than your app’s memory limit is left as an exercise for the reader.

---

Performance Benchmarks

So how do these various approaches stack up to one another?

Here are the results of a set of performance benchmarks done on an iPhone 6 running iOS 8.4, viathis project:

JPEG

Loading, scaling, and displaying a large, high-resolution (12000 ⨉ 12000 px 20 MB JPEG) source image from NASA Visible Earth at 1/10^th the size:

Operation	Time (sec)	σ
UIKit	0.612	14%
Core Graphics 1	0.266	3%
Image I/O	0.255	2%
Core Image 2	3.703	33%
vImage 3	–	–

PNG

Loading, scaling, and displaying a reasonably large (1024 ⨉ 1024 px 1MB PNG) rendering of thePostgres.app Icon at 1/10^th the size:

Operation	Time (sec)	σ
UIKit	0.044	30%
Core Graphics 4	0.036	10%
Image I/O	0.038	11%
Core Image 5	0.053	68%
vImage	0.050	25%

^{1, 4} Results were consistent across different values of CGInterpolationQuality, with negligible differences in performance benchmarks.

³ The size of the NASA Visible Earth image was larger than could be processed in a single pass on the device.

^{2, 5} Setting kCIContextUseSoftwareRenderer to true on the options passed on CIContext creation yielded results an order of magnitude slower than base results.

Conclusions

• UIKit, Core Graphics, and Image I/O all perform well for scaling operations on most images.
• Core Image is outperformed for image scaling operations. In fact, it is specifically recommended in the Performance Best Practices section of the Core Image Programming Guide to use Core Graphics or Image I/O functions to crop or downsample images beforehand.
• For general image scaling without any additional functionality,UIGraphicsBeginImageContextWithOptions is probably the best option.
• If image quality is a consideration, consider using CGBitmapContextCreate in combination with CGContextSetInterpolationQuality.
• When scaling images with the intent purpose of displaying thumbnails,CGImageSourceCreateThumbnailAtIndex offers a compelling solution for both rendering and caching.
• Unless you’re already working with vImage, the extra work to use the low-level Accelerate framework for resizing doesn’t pay off.

nsmutablehipster

Questions? Corrections? Issues and pull requests are always welcome — NSHipster is made better by readers like you.

This article uses Swift version 2.0 and was last reviewed on September 30, 2015. Find status information for all articles on the status page.

September 15 ^th, 2014

Original publication.

September 30 ^th, 2015

Revised for Swift 2.0, vImage method added.