fritz 使用手册_Fritz对象检测指南：使用机器学习在Android中构建宠物监控应用

fritz 使用手册

by Eric Hsiao

萧敬轩

Fritz对象检测指南：使用机器学习在Android中构建宠物监控应用 (A guide to Object Detection with Fritz: Build a pet monitoring app in Android with machine learning)

Whether it is detecting plant damage for farmers, tracking vehicles on the road, or monitoring your pets — the applications for object detection are endless. With the rise of mobile frameworks like TensorFlow Lite and Core ML, more and more mobile apps leverage the power of machine learning to create features that leave us in awe.

无论是为农民发现植物受损 ，在路上追踪车辆还是监视宠物-用于物体检测的应用层出不穷。 随着TensorFlow Lite和Core ML等移动框架的兴起，越来越多的移动应用利用机器学习的力量来创建令我们敬畏的功能。

那么对象检测到底是什么呢？ (So what exactly is object detection?)

In plain English, object detection identifies and locates specific items in an image or live video with a bounding box.

用简单的英语来说，物体检测可以识别和定位带有边框的图像或实时视频中的特定项目。

But creating features powered by machine learning isn’t easy. Many engineering teams cannot justify the time and resources. You need the right in-house expertise to collect data, train a model, and iterate on the performance and accuracy. Understandably, with pressure from product teams to deliver value quickly for end-users, potential features are tossed aside in a backlog abyss.

但是创建由机器学习支持的功能并不容易。 许多工程团队无法证明时间和资源的合理性。 您需要合适的内部专家来收集数据，训练模型并迭代性能和准确性。 可以理解的是，在产品团队的压力下，他们需要Swift为最终用户交付价值，潜在的功能被积压的深渊抛弃了。

In this post, I’ll show you how any Android developer can use real-time object detection to create an app that detects and recognizes pets — all in less than 30 minutes. To do this, I’ll use the Fritz SDK (full disclosure, I work at Fritz) which makes it easier to leverage machine learning capabilities without any prior experience.

在这篇文章中，我将向您展示任何Android开发人员如何使用实时对象检测来创建可检测和识别宠物的应用程序-只需不到30分钟即可完成。 为此，我将使用Fritz SDK (完整的披露内容，我在Fritz工作)，这使得在没有任何先验经验的情况下更容易利用机器学习功能。

入门 (Getting started)

To start using the Fritz SDK, we’ll go through adding the necessary dependencies in a sample app that we’ve created.

要开始使用Fritz SDK，我们将在我们创建的示例应用程序中添加必要的依赖项。

1.首先，创建一个Fritz帐户 (1. First, create a Fritz account)

Sign up here and follow the get started directions.

在这里注册并按照入门指南进行操作 。

2.克隆示例相机应用 (2. Clone the sample camera app)

Set up a skeleton app that includes a video feed and camera code. In this tutorial, we won’t go depth into the Camera 2 API, but if you have any questions, please leave a comment.

设置一个包含视频供稿和摄像机代码的骨架应用程序。 在本教程中，我们不会深入研究Camera 2 API ，但是如果您有任何疑问，请发表评论。

git clone https://github.com/fritzlabs/camera-sample-app

3.在webapp中注册Android应用 (3. Register the Android app in the webapp)

You need to register your app with Fritz in order to use ML-features. When you’re adding the app to Fritz, use the same applicationId (ai.fritz.camera) as the app/build.gradle.

您需要向Fritz注册应用才能使用ML功能。 将应用程序添加到Fritz时，请使用与app / build.gradle相同的applicationId(ai.fritz.camera) 。

Make sure you note the API Key for Step 5. If you need to access it again, you can go to Project Settings > Your App (Pet Monitor)> Show API Key (in the options menu).

确保注意步骤5的API密钥。如果需要再次访问它，则可以转到“项目设置”>“您的应用程序(宠物监控器)”>“显示API密钥”(在选项菜单中)。

4.在app / build.gradle中添加FritzCore +依赖项 (4. Add FritzCore + dependencies in app/build.gradle)

Make sure to add the Fritz repository. This will allow you to download the necessary dependencies:

确保添加Fritz存储库。 这将允许您下载必要的依赖项：

In the dependencies section, add these 2 libraries:

在“依赖关系”部分中，添加以下两个库：

dependencies {    implementation "ai.fritz:core:2.0.0"    implementation "ai.fritz:vision-object:2.0.0"}

5.配置SDK (5. Configure the SDK)

Call Fritz.configure in the Application or MainActivity onCreate lifecycle method with the API key you got in Step 3.

使用在步骤3中获得的API密钥，在Application或MainActivity onCreate生命周期方法中调用Fritz.configure 。

With that, you’re ready to use object detection in your app.

这样，您就可以在应用程序中使用对象检测了。

在实时视频中检测猫狗 (Detecting dogs & cats on live video)

Now let’s get to the fun stuff. We’ll jump into the MainActivity and use the object detection predictor on each frame passed in on the video stream.

现在让我们来看看有趣的东西。 我们将跳入MainActivity，并在视频流上传递的每个帧上使用对象检测预测值。

1.获取FritzVisionObjectPredictor的实例 (1. Get an instance of FritzVisionObjectPredictor)

The predictor takes in a FritzVisionImage and returns a list of FritzVisionObjects detected.

预测器接收FritzVisionImage并返回检测到的FritzVisionObjects列表。

2.将每一帧转换为FritzVisionImage对象 (2. Convert each frame to a FritzVisionImage object)

Use either fromBitmap or fromMediaImage static methods to create an object from a Bitmap or media.Image object. For the sample app, use fromMediaImage, which also takes in the rotation applied on the image from the camera.

使用fromBitmap或fromMediaImage静态方法从Bitmap或media.Image对象创建对象。 对于示例应用程序，请使用fromMediaImage ， 这也吸收了应用在相机图像上的旋转。

The rotation depends on the device rotation and the camera orientation sensor. The cameraId identifies the active camera being used on the device (front, back, etc.), and you can get the rotation angle with the following helper method.

旋转取决于设备旋转和相机方向传感器。 cameraId标识设备上正在使用的活动摄像机(正面，背面等)，您可以使用以下辅助方法获取旋转角度。

int rotation = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);

Finally, create a FritzVisionImage object with the Image and rotation value.

最后，使用Image和rotation值创建一个FritzVisionImage对象。

3.运行预测 (3. Run prediction)

Pass the image into the predictor to detect different objects in the image.

将图像传递到预测变量中以检测图像中的不同对象。

4.过滤结果并显示边界框 (4. Filter the result and display bounding boxes)

Each FritzVisionObject comes with a label, a confidence score, and a bounding box that shows where it’s located on the original image. In this case, we only care about pets (specifically cats and dogs), so we can filter out the other items.

每个FritzVisionObject都有一个标签，一个置信度分数和一个边界框，用于显示其在原始图像上的位置。 在这种情况下，我们只关心宠物(特别是猫和狗)，因此我们可以过滤掉其他物品。

Finally, display the bounding boxes around your pets. FritzVisionObject has a convenient method called drawOnCanvas which makes it easy to display the detected objects.

最后，显示宠物周围的边界框。 FritzVisionObject有一个称为drawOnCanvas的便捷方法，可轻松显示检测到的对象。

Here’s the complete code after the render callback:

这是渲染回调之后的完整代码：

Notice the scale factor on the boxes. This is because the media.Image object we used to create the FritzVisionImage object is the same size as the preview viewport. In the camera sample app, it’s 1280 x 960. The bounding boxes will have coordinates associated with the preview size. Since we want to show this on the full screen, we need to scale the result to the phone’s viewport.

注意包装盒上的比例因子。 这是因为我们用来创建FritzVisionImage对象media.Image对象是大小预览视窗相同。 在相机示例应用中，尺寸为1280 x960。边界框将具有与预览尺寸相关联的坐标。 由于我们要在全屏上显示此内容，因此需要将结果缩放到手机的视口。

Here’s the final result:

这是最终结果：

For the finished code, take a look at the GitHub repo.

有关完成的代码，请查看GitHub repo 。

为什么这很有用 (Why this is useful)

With the machine learning feature behind this basic app, there are ton of different features you can create (both practical and goofy):

有了这个基本应用程序背后的机器学习功能，您可以创建许多不同的功能(实用的和愚蠢的)：

• Alert the owners with a text message if the dog walker hasn’t returned.
  如果walk狗没有回来，请通过短信提醒主人。
• Record a message telling your dog to “Sit down!” when they’re running around the room with no one around. I bet you could capture funny photos of your dog in this moment, too.
  记录一条消息，告诉您的狗“坐下！” 当他们在房间里跑来跑去时，没人在附近。 我敢打赌，您也可以在这一刻捕捉狗的有趣照片。
• Show the user a message when a cat / dog is detected (take a look at the completed code for an example)
  在检测到猫/狗时向用户显示一条消息(以完整代码为例)
• Sound an alarm when the camera detects cats (I’m allergic).
  相机检测到猫时发出警报声(我过敏)。

Of course, not many people have a spare Android tablet / phone that they can use as an expensive pet monitoring camera, but this is just a simple example among many different possibilities for how you might create an app with object detection using Fritz. I can’t wait to see what all the creative developers of the world build using object detection.

当然，没有多少人有备用的Android平板电脑/手机可以用作昂贵的宠物监控摄像头，但这只是您使用Fritz创建带有对象检测的应用程序的许多不同可能性中的一个简单示例。 我迫不及待地想看看世界上所有的创意开发人员都使用对象检测构建了什么。

Got an idea? Leave a comment!

有想法吗？ 发表评论！

I’m a lead developer at Fritz specializing in mobile machine learning. If you’re looking to create features powered by AI/ML, we offer prebuilt APIs (image segmentation, image labeling, style transfer) and custom model support.

我是Fritz的首席开发人员，专门研究移动机器学习。 如果您要创建由AI / ML提供支持的功能，我们提供预构建的API( 图像分割 ， 图像标签 ， 样式转换 )和自定义模型支持。

翻译自: https://www.freecodecamp.org/news/a-guide-to-object-detection-with-fritz-build-a-pet-monitoring-app-in-android-with-machine-learning-a8ed500978e5/

fritz 使用手册