fritz 使用手册_如何在带有Fritz的React Native中使用样式传输API

fritz 使用手册

Fritz is a platform that’s intended to make it easy for developers to power their mobile apps with machine learning features. Currently, it has an SDK for both Android and iOS. The SDK contains ready-to-use APIs for the following features:

Fritz是一个平台，旨在使开发人员可以轻松使用机器学习功能为其移动应用程序提供动力。 当前，它具有适用于Android和iOS的SDK。 SDK包含具有以下功能的即用型API：

1. Object Detection

   物体检测

2. Image Labeling

   图像标注

3. Style Transfer

   样式转移

4. Image Segmentation

   图像分割

5. Pose Estimation

   姿势估计

Today, we’ll explore how to use the Style Transfer API in React Native.

今天，我们将探讨如何在React Native中使用Style Transfer API。

I was only able to develop and test in Android (no Macs here!) and got a working application.

我只能在Android(这里没有Mac！)上进行开发和测试，并获得了可运行的应用程序。

The Style Transfer API styles images or video according to real art masterpieces. There are 11 pre-trained artwork styles, including Van Gogh’s Starry Night and Munch’s Scream, among others.

Style Transfer API根据实际杰作来为图像或视频设置样式。 有11种经过预先训练的艺术品风格，包括梵高的《星夜》和蒙克的《尖叫》等。

The app we’ll be developing allows the user to take a picture and convert it into a styled image. It will also allow the user to pick the artwork style they wish to use on the image.

我们将开发的应用程序允许用户拍照并将其转换为样式图像。 它还将允许用户选择他们希望在图像上使用的图稿样式。

The app will contain a Home page, where the user can pick the art style. It will also include a separate Camera View, where the user captures the image.

该应用程序将包含一个主页，用户可以在其中选择艺术风格。 它还将包括一个单独的“相机视图”，用户可以在其中捕获图像。

Note: The following tutorial is for the Android platform only.

注意：以下教程仅适用于Android平台。

先决条件 (Prerequisites)

1. React Native CLI: run npm i -g react-native-cli to globally install the CLI

   React Native CLI：运行npm i -g react-native-cli全局安装CLI

Since there is no default React Native module for Fritz, we’ll need to write our own. Writing a native module means writing real native code to use on one or both platforms.

由于Fritz没有默认的React Native模块，因此我们需要编写自己的模块。 编写本机模块意味着编写实际的本机代码以在一个或两个平台上使用。

第1步-创建RN应用并安装模块 (Step 1 — Creating the RN app and install modules)

To create the app, run the following command in the terminal:

要创建该应用，请在终端中运行以下命令：

react-native init <appname>

Move into the root of the folder to begin configuration.

移动到文件夹的根目录开始配置。

For navigation, we’ll be using React Navigation and React Native Camera for the Camera View.

对于导航，我们将对相机视图使用React Navigation和React Native Camera 。

To install both dependencies, run the following command in the terminal:

要安装两个依赖项，请在终端中运行以下命令：

npm i --save react-navigation react-native-camera

Follow the instructions here to configure React Navigation for the app. We’ll need to install react-native-gesture-handler as well, as it’s a dependency of React Navigation.

请按照此处的说明为应用程序配置React Navigation。 我们还需要安装react-native-gesture-handler ，因为它是React Navigation的依赖项。

Follow the instructions here to configure the React Native Camera for the app. We can stop at step 6, as for this example we will not be using text, face, or barcode recognition.

请按照此处的说明为应用程序配置React Native Camera。 我们可以在步骤6停止，因为对于本示例，我们将不使用文本，面部或条形码识别。

第2步-在应用程序中包含Fritz SDK (Step 2 — Including Fritz SDK in the app)

First, we need to create a Fritz account and a new project.

首先，我们需要创建一个Fritz帐户和一个新项目。

From the Project overview, click on Add to Android to include the SDK for the Android platform. We’ll need to include an App Name and the Application ID. The Application ID can be found in android/app/build.gradle, inside the tag defaultConfig.

在“项目”概述中，单击“添加到Android”以包含适用于Android平台的SDK。 我们需要包括一个应用名称和应用ID。 可以在android/app/build.gradle的标签defaultConfig找到应用程序ID。

Upon registering the app, we need to add the following lines in android/build.gradle:

注册应用程序后，我们需要在android/build.gradle添加以下几行：

allprojects {    
	.....    
    repositories {        
    	.....        
        maven { url "https://raw.github.com/fritzlabs/fritz-repository/master" } //add this line    
    }
}

Afterward, include the dependency in the android/app/build.gradle:

之后，在android/app/build.gradle添加依赖android/app/build.gradle ：

dependencies {    
	implementation 'ai.fritz:core:3.0.2'
}

We’ll need to update the AndroidManifest.xml file to give the app permission to use the Internet and register the Fritz service:

我们需要更新AndroidManifest.xml文件，以授予应用程序使用Internet的权限并注册Fritz服务：

<manifest xmlns:android="http://schemas.android.com/apk/res/android">
	.....    
	<uses-permission android:name="android.permission.INTERNET" />    
    <application>        
    	.....        
        <service            
        	android:name="ai.fritz.core.FritzCustomModelService"            
            android:exported="true"            
            android:permission="android.permission.BIND_JOB_SERVICE" />    
    </application>
</manifest>

We then need to include the following method within the MainActivity.java:

然后，我们需要在MainActivity.java包含以下方法：

import ai.fritz.core.Fritz;
import android.os.Bundle; //import these two as well

public class MainActivity extends ReactActivity {    
	.....    
    @Override    
    protected void onCreate(Bundle savedInstanceState) {        
    	// Initialize Fritz        
        Fritz.configure(this, "<api-key>");    
    }
}

步骤3 —创建本机模块 (Step 3 — Create the Native Module)

Since the SDK only supports iOS and Android, we’ll need to make the native module. To get a better understanding of this, refer to the docs here:

由于SDK仅支持iOS和Android，因此我们需要制作本机模块。 为了更好地理解这一点，请参考此处的文档：

Native Modules · React NativeSometimes an app needs access to a platform API that React Native doesn't have a corresponding module for yet. Maybe…facebook.github.io

本机模块·React Native 有时，应用程序需要访问一个平台API，而React Native尚没有相应的模块。 也许… facebook.github.io

To make an Android Native module, we’ll need to make two new files. They will be within the root package of the Android source folder.

要制作一个Android Native模块，我们需要制作两个新文件。 它们将位于Android源文件夹的根包中。

1. FritzStyleModule: This contains the code that will return the styled image

   FritzStyleModule ：这包含将返回样式化图像的代码

2. FritzStylePackage: This registers the module so that it can be used by the JavaScript side of the app.

   FritzStylePackage ：这将注册模块，以便应用程序JavaScript端可以使用它。

FritzStyleModule (FritzStyleModule)

package com.fritzexample;

import com.facebook.react.bridge.Callback;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.bridge.ReactContextBaseJavaModule;
import com.facebook.react.bridge.ReactMethod;

import java.util.*;
import java.io.*;
import android.graphics.*;

import ai.fritz.fritzvisionstylepaintings.PaintingStyles;
import ai.fritz.vision.styletransfer.*;
import ai.fritz.core.FritzOnDeviceModel;
import ai.fritz.vision.*;

public class FritzStyleModule extends ReactContextBaseJavaModule {
    private final ReactApplicationContext reactContext;

    public FritzStyleModule(ReactApplicationContext reactContext) {
        super(reactContext);
        this.reactContext = reactContext;
    }

    @Override
    public String getName() {
        return "FritzStyle";
    }

    @ReactMethod
    public void getNewImage(String image, String filter, Callback errorCallback, Callback successCallback) {

        try {

            // Get the style of painting the user wishes to convert the image into.

            FritzOnDeviceModel styleOnDeviceModel;

            switch (filter) {
            case "STARRY_NIGHT":
                styleOnDeviceModel = PaintingStyles.STARRY_NIGHT;
                break;
            case "BICENTENNIAL_PRINT":
                styleOnDeviceModel = PaintingStyles.BICENTENNIAL_PRINT;
                break;
            case "FEMMES":
                styleOnDeviceModel = PaintingStyles.FEMMES;
                break;
            case "HEAD_OF_CLOWN":
                styleOnDeviceModel = PaintingStyles.HEAD_OF_CLOWN;
                break;
            case "HORSES_ON_SEASHORE":
                styleOnDeviceModel = PaintingStyles.HORSES_ON_SEASHORE;
                break;
            case "KALEIDOSCOPE":
                styleOnDeviceModel = PaintingStyles.KALEIDOSCOPE;
                break;
            case "PINK_BLUE_RHOMBUS":
                styleOnDeviceModel = PaintingStyles.PINK_BLUE_RHOMBUS;
                break;
            case "POPPY_FIELD":
                styleOnDeviceModel = PaintingStyles.POPPY_FIELD;
                break;
            case "RITMO_PLASTICO":
                styleOnDeviceModel = PaintingStyles.RITMO_PLASTICO;
                break;
            case "THE_SCREAM":
                styleOnDeviceModel = PaintingStyles.THE_SCREAM;
                break;
            case "THE_TRAIL":
                styleOnDeviceModel = PaintingStyles.THE_TRAIL;
                break;
            default:
                styleOnDeviceModel = PaintingStyles.THE_TRAIL;
                break;
            }

            // Initialize the style Predictor with the selected artwork style.
            FritzVisionStylePredictor stylePredictor = FritzVision.StyleTransfer.getPredictor(styleOnDeviceModel);

            // Get the Base 64 encoder and decoder.
            Base64.Decoder decoder = Base64.getDecoder();
            Base64.Encoder encoder = Base64.getEncoder();

            // Decode the base 64 image into an array of bytes.
            byte[] decodedString = decoder.decode(image);

            // Convert the byte array into an Bitmap image from the beginning (0) to the end
            // (decodedString.length) of the array.
            Bitmap bitmap = BitmapFactory.decodeByteArray(decodedString, 0, decodedString.length);

            // A standard input class for the style Predictor.
            FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);

            // Convert the normal image into a styled image according to the selected
            // artwork style.
            FritzVisionStyleResult styleResult = stylePredictor.predict(visionImage);

            // Get a Bitmap image from the styled Result.
            Bitmap styledBitmap = styleResult.getResultBitmap();

            ByteArrayOutputStream baos = new ByteArrayOutputStream();

            // Compress the Bitmap image into a .png image and add it to the output stream
            // baos.
            styledBitmap.compress(Bitmap.CompressFormat.PNG, 0, baos);

            // Convert the output stream into a byte array.
            byte[] b = baos.toByteArray();

            // Encode the byte array into a base 64 image.
            String newImage = encoder.encodeToString(b);

            // Send the styled images' base 64 string through the success callback to the
            // Javascript side.
            successCallback.invoke(newImage);

        } catch (Exception e) {

            errorCallback.invoke(e.getMessage());

        }

    }

}

The React method being used has a success and error callback. The chosen artwork style and a base64 of the original image are sent to the method. The error callback is invoked when an Exception is thrown and returns the error. The success callback returns a base64 encoded string of the converted image. On a high-level, the above code does the following:

正在使用的React方法具有成功和错误回调。 所选图稿样式和原始图像的base64将发送到该方法。 当引发Exception并返回错误时，将调用错误回调。 成功回调返回转换后图像的base64编码字符串。 在较高级别，以上代码执行以下操作：

1. Initializes the style predictor with the user’s choice of artwork.
   用用户选择的图稿初始化样式预测器。

2. Converts the original base64 image into a Bitmap.

   将原始base64图像转换为Bitmap 。

3. Creates a FritzVisionImage, which is the input of the style predictor.

   创建FritzVisionImage ，它是样式预测变量的输入。

4. Converts the FritzVisionImage into a styled FritzVisionStyleResult, which is the converted image.

   将FritzVisionImage转换为样式为FritzVisionStyleResult的图像，即转换后的图像。

5. Gets a Bitmap out of the FritzVisionStyleResult.

   从FritzVisionStyleResult获取一个Bitmap 。

6. Converts the Bitmap into a base64 to be sent back to the JavaScript side of the app.

   将Bitmap转换为base64，然后发送回应用程序JavaScript端。

FritzStylePackage (FritzStylePackage)

package com.fritzexample;

import com.facebook.react.ReactPackage;
import com.facebook.react.bridge.NativeModule;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.react.uimanager.ViewManager;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public class FritzStylePackage implements ReactPackage {

    @Override
    public List<NativeModule> createNativeModules(ReactApplicationContext reactContext) {
        List<NativeModule> modules = new ArrayList<>();

        // Append the DataUsage Module to the list of Native module list, that is
        // reffered by the React-Native code
        modules.add(new FritzStyleModule(reactContext));
        return modules;
    }

    @Override
    public List<ViewManager> createViewManagers(ReactApplicationContext reactContext) {
        return Collections.emptyList();
    }
}

This class is used to register the package so it can be called in the JavaScript side of the app.

此类用于注册包，因此可以在应用程序JavaScript端调用它。

This class is also initialized in the getPackages() of MainApplication.java:

此类也可以在MainApplication.java的getPackages()中MainApplication.java ：

@Override
protected List<ReactPackage> getPackages() {  
	return Arrays.<ReactPackage>asList(    
    	new MainReactPackage(),    
        ......,     
        new FritzStylePackage() //Add this line and import it on top  
        );
}

Now on to the JavaScript side of the application.

现在到应用程序JavaScript端。

第4步-创建UI (Step 4 — Creating the UI)

To do this, we’ll be creating/updating the following pages:

为此，我们将创建/更新以下页面：

1. Home.js — Display the picker of artwork styles and the final result.
   Home.js —显示艺术品样式的选择器和最终结果。
2. CameraContainer.js — Display the camera view to capture an image.
   CameraContainer.js —显示摄像机视图以捕获图像。
3. FritzModule.js — Export the above-created Native module to the JavaScript side.
   FritzModule.js —将上面创建的本机模块导出到JavaScript端。
4. App.js — Root of the app which includes the navigation stack.
   App.js —包含导航堆栈的应用程序的根目录。

Home.js (Home.js)

import React, { Component } from 'react';
import { StyleSheet, Text, View, Button, Image, Picker } from 'react-native';
import { ScrollView } from 'react-native-gesture-handler';

export default class Home extends Component {

    // Hide the header
    static navigationOptions = {
        header: null,
    }

    constructor(props) {
        super(props);

        // initialize the picker with the first value
        this.state = {
            filter: "BICENTENNIAL_PRINT"
        }
    }

    render() {

        // Get the following parameters from navigation props, if they have a value.
        const { navigation } = this.props;
        const oldImage = navigation.getParam('oldImage');
        const newImage = navigation.getParam('newImage');

        return (
            <View style={styles.container}>
                <ScrollView>
                    <View style={styles.innerContainer}>
                        <Text style={styles.welcome}>React Native Fritz Example!</Text>
                        <Text style={{ fontSize: 18 }}>Style Transfer</Text>
                        <Picker style={{ width: "100%" }} selectedValue={this.state.filter} mode="dropdown" onValueChange={(value) => this.setState({ filter: value })}>
                            <Picker.Item value="BICENTENNIAL_PRINT" label="Bicentennial Print" />
                            <Picker.Item value="FEMMES" label="Femmes" />
                            <Picker.Item value="HEAD_OF_CLOWN" label="Head of Clown" />
                            <Picker.Item value="HORSES_ON_SEASHORE" label="Horses on Seashore" />
                            <Picker.Item value="KALEIDOSCOPE" label="Kaleidoscope" />
                            <Picker.Item value="PINK_BLUE_RHOMBUS" label="Pink Blue Rhombus" />
                            <Picker.Item value="POPPY_FIELD" label="Poppy Field" />
                            <Picker.Item value="RITMO_PLASTICO" label="Ritmo Plastico" />
                            <Picker.Item value="STARRY_NIGHT" label="Starry Night" />
                            <Picker.Item value="THE_SCREAM" label="The Scream" />
                            <Picker.Item value="THE_TRAIL" label="The Trail" />
                        </Picker>
                        <Button title="Take Picture" onPress={() => this.props.navigation.navigate('Camera', { filter: this.state.filter })} />
                        {/* Display the images, only if the values are not undefined or empty strings */}
                        {oldImage && <Image style={styles.imageStyle} source={{ uri: 'data:image/png;base64,' + oldImage }} />}
                        {newImage && <Image style={styles.imageStyle} source={{ uri: 'data:image/png;base64,' + newImage }} />}
                    </View>
                </ScrollView>
            </View>
        );
    }
}

const styles = StyleSheet.create({
    container: {
        flex: 1,
        flexDirection: 'column',
        backgroundColor: '#F5FCFF',
    },
    innerContainer: {
        flex: 1,
        flexDirection: 'column',
        justifyContent: "center",
        alignItems: "center",
        padding: 20
    },
    welcome: {
        fontSize: 20,
        textAlign: 'center',
        margin: 10,
    },
    imageStyle: {
        width: 250,
        height: 250,
        marginVertical: 5
    }
});

This page contains:

此页面包含：

1. Text to display the app description.
   用于显示应用说明的文字。
2. Picker to allow the user to select the artwork style of the converted image.
   选择器允许用户选择转换后图像的图稿样式。
3. Button to redirect the user to the Camera page. It will pass the selected artwork style to the CameraContainer.
   用于将用户重定向到“相机”页面的按钮。 它将选定的图稿样式传递给CameraContainer。
4. If the navigation prop contains the original and converted image, it will be displayed.
   如果导航道具包含原始图像和转换后的图像，则将显示该图像。

The page currently looks like this;

当前页面如下：

CameraContainer.js (CameraContainer.js)

import React, { Component } from 'react';
import { RNCamera } from 'react-native-camera';
import { View, StyleSheet, Button, Alert, ActivityIndicator } from 'react-native';
import FritzStyle from "./FritzModule";

const styles = StyleSheet.create({
    container: {
        flex: 1,
        flexDirection: 'column',
        backgroundColor: "#000",
        position: 'absolute',
        height: '100%',
        width: '100%'
    },
    preview: {
        flex: 1,
        justifyContent: 'flex-end',
        alignItems: 'center',
    },
    cameraButton: {
        position: "absolute",
        bottom: 0,
        width: "100%",
        backgroundColor: "#000",
        alignItems: "center",
        justifyContent: "center",
        paddingVertical: 10
    },
});

class CameraContainer extends Component {

    // Hide the header
    static navigationOptions = {
        header: null,
    }

    constructor(props) {
        super(props);

        // Initialize below properties
        this.state = {
            oldImage: '',
            newImage: '',
            loading: false
        };
    }

    render() {

        return (
            <View style={styles.container}>
                <RNCamera
                    ref={ref => {
                        this.camera = ref;
                    }}
                    style={styles.preview}
                    type={RNCamera.Constants.Type.back}
                    captureAudio={false}
                >
                    {/* Display the button to take picture only if camera permission is given */}
                    {({ camera, status }) => {
                        if ((status !== 'NOT_AUTHORIZED')) {
                            return (
                                <View style={styles.cameraButton}>
                                    {/* Display spinner if loading, if not display button */}
                                    {this.state.loading ? <ActivityIndicator size="large" color="#FFF" /> : <Button onPress={this.takePicture.bind(this)} title={"Take Pic"} />}
                                </View>
                            );
                        }
                    }}
                </RNCamera>
            </View>

        );
    }

    takePicture = async function () {

        // set loading to true on button click, to show user and action is taking place.
        this.setState({ loading: true });

        // Get the chosen artwork filter picked byt user.
        const { navigation } = this.props;
        const filter = navigation.getParam('filter');

        // If the reference to the camera exists.
        if (this.camera) {

            // Take a base64 image with the following options.
            const options = { quality: 0.75, base64: true, maxWidth: 500, maxHeight: 500, fixOrientation: true };
            const data = await this.camera.takePictureAsync(options);

            // Set the old image as the one captured above.
            this.setState({
                oldImage: data.base64
            },
                () => {

                    // Call the native module method and pass the base64 of the original image and name of selected artwork style.
                    FritzStyle.getNewImage(data.base64, filter,
                        // Error Callback
                        (error) => {
                            // Display an alert to tell user an Arror was encountered.
                            console.log(error);
                            Alert.alert("Alert", "An Error has occured.");
                        },
                        //Success Callback
                        (newData) => {

                            // Set the new image as the one sent from the success callback.
                            this.setState({
                                newImage: newData
                            },
                                () => {

                                    // Navigate to the Home page, while passing the old and converted image.
                                    this.props.navigation.navigate("Home", {
                                        oldImage: this.state.oldImage,
                                        newImage: this.state.newImage
                                    });
                                });
                        });
                }
            );
        }
    }
}

export default CameraContainer;

The CameraContainer page displays a full page CameraView. It includes a button to take the picture at the bottom of the page. Upon clicking it, a spinner will be displayed to convey to the user that an action is taking place.

CameraContainer页面显示整页CameraView。 它在页面底部包括一个用于拍照的按钮。 单击它后，将显示一个微调框，以向用户传达正在进行的操作。

The image is first captured using the react-native-camera method takePictureAsync(). The original image is then saved into the state of the page. The setState method is asynchronous and thus has a success callback that runs after the state is set.

首先使用react-native-camera方法takePictureAsync()捕获图像。 然后将原始图像保存到页面状态。 setState方法是异步的，因此具有在状态设置后运行的成功回调。

The getNewImage method from the FritzModule is run within this success callback. The original image and the filter (artwork style) picked from the Home Page is passed to the method. On the error callback, an alert is displayed to the user to convey that an error has occurred. On the success callback, the new styled image is saved into the state. On this second setState methods’ success callback, the user is redirected to the Home page with both the original and styled images.

该getNewImage从方法FritzModule是这个成功回调中运行。 从主页中选取的原始图像和滤镜(图稿样式)将传递给该方法。 在错误回调上，将向用户显示警报，以表明发生了错误。 在成功回调中，新样式的图像将保存到状态中。 在第二个setState方法的成功回调中，将使用原始图像和样式图像将用户重定向到主页。

FritzModule.js (FritzModule.js)

import { NativeModules } from 'react-native';
export default NativeModules.FritzStyle;

This page exposes the Native module, FritzStyle. This allows the JavaScript side to make calls to the method getNewImage.

该页面公开了Native模块FritzStyle 。 这允许JavaScript端调用方法getNewImage 。

App.js (App.js)

import React, { Component } from 'react';
import Home from './src/Home';
import CameraContainer from './src/CameraContainer';
import { createStackNavigator, createAppContainer } from 'react-navigation';

const AppNavigator = createStackNavigator({  
	Home: { screen: Home },  
    Camera: { screen: CameraContainer }
});

const AppContainer = createAppContainer(AppNavigator);

export default class App extends Component {
	render() {    
    	return (
        	<AppContainer />
        );  
    }
}

First, we create the Stack navigator with the Home Page and Camera View. The key ‘Home’ is used when navigating to the Home Page, and the key ‘Camera’ when navigating to the CameraContainer.

首先，我们使用主页和相机视图创建堆栈导航器。 导航到主页时使用键“主页”，而导航到CameraContainer时使用键“摄像机”。

The AppContainer becomes the root component of the App. It’s also the component that manages the app’s state.

AppContainer成为应用程序的根组件。 它也是管理应用程序状态的组件。

Now to see the entire app in function;

现在查看整个应用程序的功能；

回顾一下，我们有； (To recap, we have;)

1. Created a React Native app,
   创建了一个React Native应用
2. Included the Fritz SDK in it,
   包含Fritz SDK，
3. Created a Native Module that makes use of the Style Transfer API, and
   创建了一个使用样式传输API的本机模块，并
4. Designed a UI to display the styled image.
   设计了一个UI以显示样式图像。

Find the code repo, here.

在此处找到代码仓库。

For native iOS or Android implementations of Fritz’s Style Transfer API, check out the following tutorials:

对于Fritz的Style Transfer API的本地iOS或Android实现，请查看以下教程：

Real-Time Style Transfer for Android — Transform your photos and videos into masterpiecesStyle Transfer allows you to take inspiration from artists like Picasso and Van Gogh and transform ordinary images to…heartbeat.fritz.aiReal-Time Style Transfer for iOS— Transform your photos and videos into masterpiecesheartbeat.fritz.ai

适用于Android的实时样式转换-将您的照片和视频转换成杰作通过“ 样式转换”，您可以从毕加索和梵高等艺术家 那里汲取 灵感，并将普通图像转换为... heartbeat.fritz.ai 适用于iOS的实时样式转换-转换您的照片和视频成为杰作 heartbeat.fritz.ai

翻译自: https://www.freecodecamp.org/news/how-to-use-the-style-transfer-api-in-react-native-with-fritz-e90bc609fb17/

fritz 使用手册