ios swift_用于iOS部署的Tensorflow Lite 2 x Swift或C API

最新推荐文章于 2021-06-30 14:35:18 发布

weixin_26718993

最新推荐文章于 2021-06-30 14:35:18 发布

阅读量706

点赞数

文章标签： ios

原文链接：https://medium.com/kamil-rzechowski/tensorflow-lite-2-x-swift-or-c-api-for-ios-deployment-fd5bee29d99c

版权

ios swift

TensorFlow Lite 2.x API allows us to deploy neural network models on iOS devices. TensorFlow Lite comes with three different programming languages support: SWIFT, Objective-C, and C. However only the two first are recommended on the official TensorFlow website. In the following article, I describe differences and reasons why I prefer using TensorFlow lite C API, instead of recommending SWIFT or objective-C languages. Moreover, I provide code examples for SWIFT and C API models implementation.

TensorFlow Lite 2.x API允许我们在iOS设备上部署神经网络模型。 TensorFlow Lite带有三种不同的编程语言支持：SWIFT，Objective-C和C。但是，在TensorFlow官方网站上仅推荐前两种语言。 在下面的文章中，我将介绍为什么我更喜欢使用TensorFlow lite C API而不是推荐SWIFT或Objective-C语言的差异和原因。 此外，我提供了SWIFT和C API模型实现的代码示例。

Xcode环境中的TensorFlow Lite 2.x部署 (TensorFlow Lite 2.x deployment in the Xcode environment)

Deployment instruction includes the TensorFlow Lite and OpenCV framework installation. OpenCV will be used for image pre-processing while using C API.

部署说明包括TensorFlow Lite和OpenCV框架安装。使用C API时，OpenCV将用于图像预处理。

Pre-stepIt is required to have CocoaPods installed on the Mac. It can be installed by following the instruction on the official website.

预先步骤需要在Mac上安装CocoaPods。可以按照官方网站上的说明进行安装。

Installation

安装

Create a new project in Xcode.
在Xcode中创建一个新项目。
Close the Xcode window.
关闭Xcode窗口。
Open a terminal window in the project directory.
在项目目录中打开一个终端窗口。
Run ‘pod init‘. A pod file will be generated in the project directory.
运行“ pod init”。 在项目目录中将生成一个pod文件。
In the generated file add the following line for Swift version:
在生成的文件中，为Swift版本添加以下行：

pod 'TensorFlowLiteSwift'

or following lines for C version of TensorFlow lite:
或针对TensorFlow lite的C版本的以下行：

pod 'TensorFlowLiteC'
pod 'OpenCV2'

Image for post — View of pod file after modification.

A comparison between the usage and the performance of both languages is presented below in the article.

下文在本文中介绍了两种语言的用法和性能之间的比较。

Save the file and run the command ‘pod install’. If you made a mistake and correction to the pod file is necessary run ‘pod upgrade’ instead of ‘pod install’. After successfully running the command, a *.xcworkspace file is created. Open the project by double click on the *.xcworkspace file.
保存文件并运行命令“ pod install ”。如果您输入有误，则需要更正pod文件，请运行'pod upgrade'而不是'pod install' 。成功运行命令后，将创建一个* .xcworkspace文件。双击* .xcworkspace文件打开项目。

将图像输入TensorFlow Lite模型-SWIFT。 (Feeding image into the TensorFlow Lite model — SWIFT.)

Usually, images on iOS come in two different types: UIImage or CVPixelBuffer. The input to the TensorFlow Lite model is a float array, where each array value represents a next pixel value. Moreover, Apple uses 4 channels for image coding: BGRA or RGBA, where the first 3 channels correspond to colors and the last Alpha channel is responsible for opacity. In order to insert the image into the TensorFlow Lite model, it is necessary to convert the Apple image type into a float array.

通常，iOS上的图像有两种不同的类型：UIImage或CVPixelBuffer。 TensorFlow Lite模型的输入是一个float数组，其中每个数组值代表下一个像素值。此外，Apple使用4个通道进行图像编码：BGRA或RGBA，其中前3个通道对应于颜色，而最后一个Alpha通道负责不透明度。为了将图像插入TensorFlow Lite模型，必须将Apple图像类型转换为float数组。

UIImage to CVPixelBuffer

UIImage到CVPixelBuffer

Usually, the CVPixelBuffer is already available and there is no need for UIImage to CVPixelBuffer conversion. CVPixelBuffer is a standard output type from the iPhone camera. However if only the UIImage is available, conversion to CVPixelBuffer is necessary.

通常，CVPixelBuffer已经可用，并且不需要将UIImage转换为CVPixelBuffer。 CVPixelBuffer是iPhone相机的标准输出类型。但是，如果只有UIImage可用，则必须转换为CVPixelBuffer。

extension UIImage {
    public convenience init?(pixelBuffer: CVPixelBuffer) {
        var cgImage: CGImage?
        VTCreateCGImageFromCVPixelBuffer(pixelBuffer, options: nil, imageOut: &cgImage)


        guard let myCgImage = cgImage else {
            return nil
        }


        self.init(cgImage: myCgImage)
    }
}

Create the UIImage class extension and then use code below to convert UIImage to CVPixelBuffer.

创建UIImage类扩展，然后使用下面的代码将UIImage转换为CVPixelBuffer。

UIImage(ciImage: CIImage(cvPixelBuffer: imageBuffer))

CVPixelBuffer to TensorFlow Lite model input

CVPixelBuffer到TensorFlow Lite模型输入

The official TensorFlow recommendation is to convert CVPixelBuffer to the input ‘tensor’ using their function presented below. The function copies the CVPixelBuffer to the float array, skipping the alpha channel in the CVPixelBuffer.

TensorFlow的官方建议是使用下面介绍的函数将CVPixelBuffer转换为输入“张量”。该函数将CVPixelBuffer复制到float数组，跳过CVPixelBuffer中的alpha通道。

/// Returns the RGB data representation of the given image buffer with the specified `byteCount`.
  ///
  /// - Parameters
  ///   - buffer: The pixel buffer to convert to RGB data.
  ///   - byteCount: The expected byte count for the RGB data calculated using the values that the
  ///       model was trained on: `batchSize * imageWidth * imageHeight * componentsCount`.
  ///   - isModelQuantized: Whether the model is quantized (i.e. fixed point values rather than
  ///       floating point values).
  /// - Returns: The RGB data representation of the image buffer or `nil` if the buffer could not be
  ///     converted.
private func rgbDataFromBuffer(
        _ buffer: CVPixelBuffer,
        byteCount: Int,
        isModelQuantized: Bool
    ) -> Data? {
        CVPixelBufferLockBaseAddress(buffer, .readOnly)
        defer {
            CVPixelBufferUnlockBaseAddress(buffer, .readOnly)
        }
        guard let sourceData = CVPixelBufferGetBaseAddress(buffer) else {
            return nil
        }
    
        let width = CVPixelBufferGetWidth(buffer)
        let height = CVPixelBufferGetHeight(buffer)
        let sourceBytesPerRow = CVPixelBufferGetBytesPerRow(buffer)
        let destinationChannelCount = 3
        let destinationBytesPerRow = destinationChannelCount * width
    
        var sourceBuffer = vImage_Buffer(data: sourceData,
                                     height: vImagePixelCount(height),
                                     width: vImagePixelCount(width),
                                     rowBytes: sourceBytesPerRow)
    
        guard let destinationData = malloc(height * destinationBytesPerRow) else {
            print("Error: out of memory")
            return nil
        }
    
        defer {
            free(destinationData)
        }


        var destinationBuffer = vImage_Buffer(data: destinationData,
                                          height: vImagePixelCount(height),
                                          width: vImagePixelCount(width),
                                          rowBytes: destinationBytesPerRow)


        let pixelBufferFormat = CVPixelBufferGetPixelFormatType(buffer)


        switch (pixelBufferFormat) {
            case kCVPixelFormatType_32BGRA:
                vImageConvert_BGRA8888toRGB888(&sourceBuffer, &destinationBuffer, UInt32(kvImageNoFlags))
            case kCVPixelFormatType_32ARGB:
                vImageConvert_ARGB8888toRGB888(&sourceBuffer, &destinationBuffer, UInt32(kvImageNoFlags))
            case kCVPixelFormatType_32RGBA:
                vImageConvert_RGBA8888toRGB888(&sourceBuffer, &destinationBuffer, UInt32(kvImageNoFlags))
            default:
                // Unknown pixel format.
                return nil
        }


        let byteData = Data(bytes: destinationBuffer.data, count: destinationBuffer.rowBytes * height)
        if isModelQuantized {
            return byteData
        }


        // Not quantized, convert to floats
        let bytes = Array<UInt8>(unsafeData: byteData)!
        var floats = [Float]()
        for i in 0..<bytes.count {
            floats.append(Float(bytes[i])) // / 255.0)
            //print(Float(bytes[i]))
        }
        
        return Data(copyingBufferOf: floats)
    }

After creating a Data buffer (which is a float array), the input can be fed into the model and inference can be performed.

创建数据缓冲区(它是一个浮点数组)之后，可以将输入输入模型并执行推断。

guard let rgbData = rgbDataFromBuffer(
  thumbnailPixelBuffer,
  byteCount: batchSize * inputWidth * inputHeight * inputChannels,
  isModelQuantized: inputTensor.dataType == .uInt8
) else {
  print("Failed to convert the image buffer to RGB data.")
  return nil
}


// Copy the RGB data to the input `Tensor`.
try interpreter.copy(rgbData, toInputAt: 0)

将图像馈入TensorFlow Lite模型-C.(Feeding image into the TensorFlow Lite model — C.)

Previous paragraph presented necessary steps for running a TensorFlow Lite model on an image using SWIFT. The same result (but faster) can be obtained using TensorFlow Lite C API. For the purpose of running inference using C language, we will use OpenCV for image pre-processing. The OpenCV framework was already installed in the installation step.

上一段介绍了使用SWIFT在图像上运行TensorFlow Lite模型的必要步骤。使用TensorFlow Lite C API可以获得相同的结果(但更快)。为了使用C语言运行推理，我们将使用OpenCV进行图像预处理。 OpenCV框架已在安装步骤中安装。

In order to perform inference on an image using TensorFlow Lite C API, it is necessary to create a float array from an image. The easiest way to do it is by creating cv::Mat from the UIImage or CVPixelBuffer. As the OpenCV library is available only in C, we will use Objective-C and C++ code.

为了使用TensorFlow Lite C API对图像进行推理，有必要从图像创建一个float数组。最简单的方法是从UIImage或CVPixelBuffer创建cv :: Mat。由于OpenCV库仅在C语言中可用，因此我们将使用Objective-C和C ++代码。

UIImage to cv::Mat

UIImage到cv :: Mat

The UIImage can be converted to cv::Mat using standard OpenCV function UIImageToMat().

可以使用标准OpenCV函数UIImageToMat()将UIImage转换为cv :: Mat。

- (bool)hasAlpha : (UIImage*) img
{
    CGImageAlphaInfo alpha = CGImageGetAlphaInfo(img.CGImage);
    return (
            alpha == kCGImageAlphaFirst ||
            alpha == kCGImageAlphaLast ||
            alpha == kCGImageAlphaPremultipliedFirst ||
            alpha == kCGImageAlphaPremultipliedLast
            );


}


- (void) UIImage2Mat:(UIImage *) image{


    cv::Mat opencvImage;
    bool alpha = [self hasAlpha:image];
    // convert uiimage to mat
    UIImageToMat(image, opencvImage, alpha);
    cv::cvtColor(opencvImage, opencvImage, cv::COLOR_RGBA2RGB);
    
    // use opencvImage here
}

CVPixelBuffer to cv::Mat

CVPixelBuffer到cv :: Mat

cv::Mat can be created from CVPixelBuffer using the function presented below.

可以使用以下功能从CVPixelBuffer创建cv :: Mat。

- (void) CVPixelBuffer2Mat:(CVImageBufferRef) buffer {
    
    CVPixelBufferLockBaseAddress(buffer, 0);


    void *address = CVPixelBufferGetBaseAddress(buffer);
    int width = (int) CVPixelBufferGetWidth(buffer);
    int height = (int) CVPixelBufferGetHeight(buffer);


    cv::Mat opencvImage;
    cv::Mat tmp = cv::Mat(height, width, CV_8UC4, address, 0);
    cv::cvtColor(tmp, opencvImage, cv::COLOR_BGRA2RGB);


    CVPixelBufferUnlockBaseAddress(buffer, 0);
    
    // use convertedColorSpaceImage here
}

cv::Mat stores the image already in one place in the memory. The only necessary action is creating a float pointer pointing to the first element of the Mat (first pixel) and using TensorFlow function to copy the whole frame into the model input. Voila! The inference can be performed.

cv :: Mat已将图像存储在内存中的一个位置。唯一必要的操作是创建一个指向Mat的第一个元素(第一个像素)的浮动指针，并使用TensorFlow函数将整个框架复制到模型输入中。瞧！可以执行推断。

opencvImage.convertTo(opencvImage, CV_32FC3);


float* in0 = opencvImage.ptr<float>(0);
TfLiteTensorCopyFromBuffer(input_model, in0, opencvImage.rows* opencvImage.cols * 3 * sizeof(float));

为什么我更喜欢TensorFlow Lite C API，而不是TensorFlow SWIFT API？(Why do I prefer TensorFlow Lite C API, rather than TensorFlow SWIFT API?)

So far we saw 2 different approaches to run the inference on images using TensorFlow Lite API. One using C API and second using SWIFT API. Now let's evaluate both programming languages in terms of processing time.

到目前为止，我们看到了两种使用TensorFlow Lite API在图像上进行推理的方法。一种使用C API，第二种使用SWIFT API。现在让我们根据处理时间评估两种编程语言。

As it is shown in the table above, creating an input tensor for the TensorFlow model is x62 faster in the case of TensorFlow Lite C API. Further model inference time is similar, but the pre-processing time is way slower in the case of SWIFT.

如上表所示，在TensorFlow Lite C API的情况下，为TensorFlow模型创建输入张量的速度快了x62。 进一步的模型推断时间相似，但是在SWIFT情况下，预处理时间要慢得多。

Apart from that, having a cv::Mat allows us to perform advanced image pre-processing using OpenCV library. OpenCV library in many cases performs better, then SWIFT library for CVPixelBuffer handling.

除此之外，使用cv :: Mat可以使我们使用OpenCV库执行高级图像预处理。在许多情况下，OpenCV库的性能要好一些，然后再使用SWIFT库处理CVPixelBuffer。

TensorFlow Lite与CoreML框架 (TensorFlow Lite vs CoreML framework)

One can ask whether to use TensorFlow Lite if the CoreML framework is available. The CoreML framework was specially designed for Apple architecture. There might be a couple of reasons:

如果CoreML框架可用，则可以询问是否使用TensorFlow Lite。 CoreML框架是专门为Apple架构设计的。可能有两个原因：

Avoiding model conversion from the TensorFlow model to the CoreML model. In some rare cases, the conversion might not be possible due to a lack of special types of layers or connections.
避免将模型从TensorFlow模型转换为CoreML模型。在极少数情况下，由于缺少特殊类型的图层或连接，可能无法进行转换。
Using CoreML requires to deal with MLMultiArrays. It is a CoreML version of a tensor. However, operations on MLMultiArrays are slow, especially if the array is large. Thus for image processing, Apple decided to use CVPixelBuffer as the input to the network. To accept CVPixelBuffer as the input, the model needs to be specially converted. CVPixelBuffer as the model input doesn’t support however custom inputs. For example images with 2 channels or 5 channels. CoreML supports only standard image shapes of shape BGRA. Thus TensorFlow Lite might perform better in particular situations.
使用CoreML需要处理MLMultiArrays 。它是张量的CoreML版本。但是，对MLMultiArrays的操作很慢，尤其是在数组较大的情况下。因此，对于图像处理，Apple决定使用CVPixelBuffer作为网络的输入。要接受CVPixelBuffer作为输入，需要对模型进行特殊转换。 CVPixelBuffer作为模型输入不支持自定义输入。例如具有2个通道或5个通道的图像。 CoreML仅支持形状为BGRA的标准图像形状。因此，TensorFlow Lite在特定情况下可能会表现更好。
code generalization — code written using TensorFlowC can be easily moved to the Android implementation (and vice versa).
代码泛化-使用TensorFlowC编写的代码可以轻松移至Android实现(反之亦然)。

In general, the CoreML should perform better and should be used if possible. On the other hand, TensorFlow has Metal API support (Apple GPU support) and inference speed using TensorFlow can be comparable to CoreML. Each case is unique and should be judged on its own merits.

通常，CoreML应该表现得更好，应该尽可能使用。另一方面，TensorFlow具有Metal API支持(Apple GPU支持)，并且使用TensorFlow的推理速度可以与CoreML相提并论。每种情况都是独特的，应根据自身情况进行判断。

翻译自: https://medium.com/kamil-rzechowski/tensorflow-lite-2-x-swift-or-c-api-for-ios-deployment-fd5bee29d99c

ios swift

weixin_26718993

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
ios swift_用于iOS部署的Tensorflow Lite 2 x Swift或C API

ios swiftTensorFlow Lite 2.x API allows us to deploy neural network models on iOS devices. TensorFlow Lite comes with three different programming languages support: SWIFT, Objective-C, and C. However ...
复制链接

扫一扫