arkit 检测坐标_在ARKit中使用视觉框架对象检测

arkit 检测坐标

In this short tutorial we’ll use Vision Framework to add object detection and classification capabilities to a bare-bones ARKit project. We’ll use an open source Core ML model to detect a remote control, get its bounding box center, transform its 2D image coordinates to 3D and then create an anchor which can be used for placing objects in an AR scene.

在这个简短的教程中，我们将使用Vision Framework将对象检测和分类功能添加到一个简单的ARKit项目中。 我们将使用开源的Core ML模型来检测遥控器，获取其边界框中心，将其2D图像坐标转换为3D，然后创建可用于在AR场景中放置对象的锚点。

Here’s a preview of what we’ll create:

这是我们将创建的内容的预览：

To get started you’ll need to create a new Augmented Reality App in Xcode: File > New > Project … and then choose “Augmented Reality App”.

首先，您需要在Xcode中创建一个新的增强现实应用程序：File> New> Project…，然后选择“ Augmented Reality App”。

Replace the code in ViewController.swift with the code below so we can get a clean start:

用下面的代码替换ViewController.swift中的代码，这样我们就可以轻松开始：

We’ll use a freely available open source Core ML model called YOLO which stands for “You Only Look Once”. It is a state-of-the-art, real-time object detection system. which can locate and classify 80 different types of objects.

我们将使用一个免费的开源核心ML模型，称为YOLO，它代表“您只看一次”。 它是最先进的实时对象检测系统。 可以定位和分类80种不同类型的对象。

The Core ML model can be downloaded from Apple’s Developer website: https://developer.apple.com/machine-learning/models/. Scroll down to “YOLOv3-Tiny”, click “View Models” and then download the file “YOLOv3TinyInt8LUT.mlmodel”.

可以从Apple的开发人员网站上下载Core ML模型： https : //developer.apple.com/machine-learning/models/ 。 向下滚动到“ YOLOv3-Tiny”，单击“查看模型”，然后下载文件“ YOLOv3TinyInt8LUT.mlmodel”。

Once downloaded, drag the .mlmodel file from Finder into Xcode and make sure it is added to the target.

下载后，将.mlmodel文件从Finder拖到Xcode中，并确保将其添加到目标中。

Object detection needs a camera image so we’ll hook into SCNSceneRendererDelegate’s renderer(_:willRenderScene:atTime:) method to query for an image and start the object detection process if the image is available.

对象检测需要相机图像，因此我们将使用SCNSceneRendererDelegate的renderer(_:willRenderScene:atTime:)方法来查询图像，并在图像可用时启动对象检测过程。

Using ARKit’s captured image we’ll create an image request and make it perform an object detection request:

使用ARKit捕获的图像，我们将创建一个图像请求，并使其执行对象检测请求：

Here the image imageRequestHandler performs an object detection request called objectDetectionRequest . This request needs to be created only once and can be defined in a lazy variable. Here, we create an instance of the YOLO model and create a CoreML request.

在这里，图像imageRequestHandler执行一个称为objectDetectionRequest的对象检测请求。 该请求仅需要创建一次，并且可以在惰性变量中定义。 在这里，我们创建YOLO模型的实例并创建CoreML请求。

Here, processDetections is VNCoreMLRequest’s completion handler. This is where we’ll get the recognized remote control object and its bounding box and then do all the necessary conversions to get 3D world coordinates which we can use to create an ARKit anchor.

在这里， processDetections是VNCoreMLRequest的完成处理程序。 在这里，我们将获得公认的远程控制对象及其边界框，然后进行所有必要的转换，以获取可用于创建ARKit锚点的3D世界坐标。

First we need to go through all the observations, check the classification string to see if a remote control is detected and then filter out low confidence observations:

首先，我们需要检查所有观察值，检查分类字符串以查看是否检测到遥控器，然后过滤掉低置信度观察值：

Now that we are confident we’ve detected a remote control we can get its bounding box. The bounding box’s coordinates are normalized image coordinates. A few conversions have to be done to get the view coordinates:

现在，我们有信心检测到一个遥控器，可以得到它的边界框。 边界框的坐标是归一化的图像坐标。 必须进行一些转换才能获取视图坐标：

We’ll now use the bounding box center as the coordinate we’ll want to convert to 3D space. It might not be the actual center of the detected ‘real world’ object but that goes beyond the scope of this tutorial.

现在，我们将边界框中心用作要转换为3D空间的坐标。 它可能不是检测到的“真实世界”对象的实际中心，但是超出了本教程的范围。

To get the 3D world coordinate we can use the view-space center point to perform a hit test. If we specify featurePoint as the hit test result type, ARKit finds the feature point nearest to the hit-test ray. If we get a result we can use its worldTransfrom property to create an ARKit anchor and add it to the session:

要获取3D世界坐标，我们可以使用视图空间中心点进行点击测试。 如果我们将featurePoint指定为命中测试结果类型，则ARKit会找到最接近命中测试射线的特征点。 如果得到结果，则可以使用其worldTransfrom属性创建一个ARKit锚并将其添加到会话中：

Adding this anchor to the session will invoke ARSCNViewDelegate ‘s renderer(_,didAdd:,for:) function in which we can add 3D content to the scene. In this case we’ll add a simple red sphere and attach it to the anchor:

将这个锚点添加到会话中将调用ARSCNViewDelegate的renderer(_,didAdd:,for:)函数，我们可以在其中向场景添加3D内容。 在这种情况下，我们将添加一个简单的红色球体并将其附加到锚点：

The red sphere will now be placed on the detected remote control and will persist like any other ARKit object.

现在，红色球体将放置在检测到的遥控器上，并且将像其他任何ARKit对象一样持续存在。

Again, this might not be the most accurate solution to place virtual content over real world content. However, it shows a few really useful techniques that might come in handy in your own ARKit projects.

同样，这可能不是将虚拟内容放置在现实世界内容上的最准确解决方案。 但是，它显示了一些非常有用的技术，这些技术可能会在您自己的ARKit项目中派上用场。

The Xcode project can be found here: https://github.com/MasDennis/ARKitVisionObjectDetection

可以在这里找到Xcode项目： https : //github.com/MasDennis/ARKitVisionObjectDetection

翻译自: https://medium.com/@Rozengain/using-vision-framework-object-detection-in-arkit-c0b5366f465d

arkit 检测坐标