Wechat miniapp Integrate third-party AR plug-ins

AR
Augmented reality, also known as Mixed reality. It uses computer technology to apply virtual information to the real world. World, the real environment and virtual objects are superimposed on the same screen in real time Or space exists at the same time.

AR equipment: AR is a combination of real and virtual. All need a camera, based on the picture taken by the camera, combined with virtual Screens for display and interaction, such as GOOGLE GLASS (in fact Strictly speaking, IPAD, mobile phones and other only products with cameras are available. For AR, you only need to install AR software. )

development process:
Select APP, webpage, small program, desktop program, server and so on as the carrier
Select the application scenario of AR
Select a 3D interactive scene
Select a usage scenario for the AI

working principle:
Feature points are extracted from each frame of the camera image: no matter whether the image is rotated or scaled, the direction and size of the feature points will not change, which is called rotation invariance and scaling invariance. Through the feature detection algorithm, the edges and corners are searched as feature points.
Match the feature points of the camera image with the recognition image: use the violent matching or nearest neighbor search algorithm to find the matching result between the feature points extracted in the first step and the recognition image. There are 4 matching results: matched (TP), can not match (TN), the match did not match (FN), should not match when the match (FP). The outliers of the matching results were deleted by cross-matching filtering, ratio testing and homography estimation.
Calculate the camera and image transformation: according to the matching results of feature points obtained in the second step, the internal camera parameters obtained by the camera calibration, calculate the projection matrix of the camera and image transformation.

WeChat provides basic components about AR

摄像头组件（用户授权下，可获取视频帧数据）：https://developers.weixin.qq.com/miniprogram/dev/api/media/camera/CameraContext.html
-WebGL画布：https://developers.weixin.qq.com/miniprogram/dev/component/canvas.html
-完整示例代码：https://developers.weixin.qq.com/s/ElC24AmF729W
-传感器：https://developers.weixin.qq.com/miniprogram/dev/api/device/compass/wx.onCompassChange.html#accuracy%20%E5%9C%A8%20iOS/Android%20%E7%9A%84%E5%B7%AE%E5%BC%82
-陀螺仪：https://developers.weixin.qq.com/miniprogram/dev/api/device/gyroscope/wx.startGyroscope.htmlhttps://developers.weixin.qq.com/miniprogram/dev/api/wx.onGyroscopeChange.html

code sample:

<!-- 相机组件，放在UI最底层显示相机内容 -->
<camera frame-size="medium"
        bindinitdone="onCameraInit"
        mode="normal"
        device-position="back"
        resolution="high"
        flash="off"/>
<!-- canvas组件，用于进行图片压缩，位置在屏幕外，不可见 -->
<canvas type="2d"
        id="mycanvas"
        style="width:1px; height: 1px;"/>

Third Party Plugins-AR service
Introduction：WeChat provides the basis of the canvas, camera function, AR engine can use the following mentioned third party plugins or use yourself AR engine development, not to use technology without AR scene, such as image recognition, gesture recognition only to image recognition services, glasses try, lipstick test and AR scene need to involve the face recognition, Face tracking and other technical support. Different AR services are integrated into their own small programs to achieve different AR functions. The known third-party plug-ins are listed below, and the specific access process and development process can be developed by referring to the technical documents of the company of the third-party plug-ins.

（一） EasyAR小程序插件

Support cloud recognition, image recognition function.
Description of functions: Cloud Recognition is used to detect and recognize planar objects with textures in daily life, which needs to be used together with Easyar CRS. The Recognition process is carried out in the Cloud, and the Target data of the object to be recognized is also stored in the Cloud. Cloud Recognition client obtains the Recognition result from the Cloud by sending the Recognition request.
Easyar has four subscription models: Personal (free), Pro (monthly), Classic (one-time), and Enterprise (business). You can choose which model you want based on the number of times you drop your app.

1. wxml文件中添加camera和canvas组件。camera组件用于展示相机内容和获取相机帧，canvas组件用于图片压缩。:

<!-- 相机组件，放在UI最底层展示相机内容 -->
<camera frame-size="medium"
        bindinitdone="onCameraInit"
        mode="normal"
        device-position="back"
        resolution="high"
        flash="off"/>
<!-- canvas组件，用于图片压缩，位置在屏幕外，不可见 -->
<canvas type="2d"
        id="capture"
        style="width: 1px; height: 1px;"/>

2. 初始化camera和canvas

const query = wx.createSelectorQuery();
query.select('#capture')
    .fields({node: true})
    .exec((res) => {
        const canvas = res[0].node;
        //设置canvas内部尺寸为480*640，frame-size="medium"的设置下相机帧大多是480*640
        canvas.width = 480;
        canvas.height = 640;
        this.canvas = canvas;
    });
 
let cameraContext = wx.createCameraContext();
let listener = cameraContext.onCameraFrame(frame => {
    if (!this.canvas) return;
    let canvas = this.canvas;
    //如果尺寸不匹配，就修改canvas尺寸以适应相机帧
    if (canvas.width !== frame.width || canvas.height !== frame.height) {
        canvas.width = frame.width;
        canvas.height = frame.height;
    }
 
    //TODO 在这里保存frame对象，以便在需要的时候进行下一步压缩图片、发起CRS请求。不要在onCameraFrame回调中直接处理。
});
listener.start();
  

3. 压缩图片为JPEG（在需要发起一次CRS请求时进行第3、4步，比如用户主动触发一次识别时，或者已经开始识别循环时。不要在onCameraFrame回调中直接发起）

let context = this.canvas.getContext(‘2d’);
let ctxImageData = context.createImageData(frame.width, frame.height);
ctxImageData.data.set(new Uint8ClampedArray(frame.data)); //复制相机帧内容到imageData
context.putImageData(ctxImageData, 0, 0); //将imageData画到canvas上
let dataUrl = this.canvas.toDataURL(“image/jpeg”, 0.7); //使用toDataURL方法将相机帧压缩为JPEG，质量70%
let base64 = dataUrl.substr(23); //去除dataURL头，留下文件内容的base
4. 组装请求参数，发起CRS请求

const params = {
    image: base64,
    notracking: "true",
    appId: "开发前准备中的 Cloud Key",
};
 
return new Promise((resolve, reject) => {
    wx.request({
        url: "https://cn1-crs.easyar.com:8443/search/",
        method: 'post',
        data: params,
        header: {
            'Authorization': "开发前准备中的 Token",
            'content-type': 'application/json'
        },
        success: res => resolve(res.data), //处理方法见下
        fail: err => reject(err),
    });
});

5. 处理请求结果

queryImage(frame).then(response => { //response为上面的res.data
    let result = response && response.result;
    if (!result) return;
 
    if (result.target) {
        console.log("识别成功", result.target.targetId);
        //小程序可以根据targetId获知识别到了哪一张图，进而进行判断或触发事件等，也可以对每张图配置meta信息，根据meta信息来进行操作
 
        if (result.target.meta) { //读取识别图meta的base64(meta在上传识别图时配置)
            console.log("meta base64:", result.target.meta);
        }
    } else {
        console.log("识别失败", result.message);
    }
});

（二） beautyAR

It supports the realization of AR scenes in small programs, such as glasses try-on, clothes try-on, lipstick try-on, image recognition, e-commerce shopping guide, fun filter and other complex AR functions. However, at present, this third-party service requires cooperation and communication, and the purchase of relevant services. Development, testing and access are not available yet.

（三） 元元AR服务

Support small programs to try on and wear, identifying objects shopping guide, fun stickers (based on facial tracking) and other AR scenes. Support cloud recognition, gesture recognition, gesture recognition, holographic display and other functions.

Access process :
Introducing plug-ins and adding meta-cloud plug-ins through small program background search
Obtain the secret key, and obtain the secret key through email after signing the authorization agreement
Code integration, access AR plug-in to applet according to plug-in demo
Start AR service, call meta-cloud AR service, and realize various AR scenes
Consulting business is needed, and development access is not supported for the time being
Some problems about small program loading 3D model
First, need to introduce the three.js plug-in into the miniapp

npm install three

Load the 3D model by gltf-lodeust.js

The texture image for the 3D model is set to 512 by 512, not too large

The 3D model is black after loading, possibly because no light source is set

var point = new THREE.PointLight(0xffffff);
   point.position.set(400, 200, 300);
   scene.add(point);