2019CHI Paper Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction

Teo T, Lawrence L, Lee G A, et al. Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction[C]//Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 2019: 201.

 

ABSTRACT

Remote Collaboration using Virtual Reality (VR) and Augmented Reality (AR) has recently become a popular way for people from different places to work together. Local workers can collaborate with remote helpers by sharing 360-degree live video or 3D virtual reconstruction of their surroundings. However, each of these techniques has benefts and drawbacks. In this paper we explore mixing 360 video and 3D reconstruction together for remote collaboration, by preserving benefts of both systems while reducing drawbacks of each. We developed a hybrid prototype and conducted user study to compare benefts and problems of using 360 or 3D alone to clarify the needs for mixing the two, and also to evaluate the prototype system. We found participants performed signifcantly better on collaborative search tasks in 360 and felt higher social presence, yet 3D also showed potential to complement. Participant feedback collected after trying our hybrid system provided directions for improvement. 

INTRODUCTION
主要说清楚这两个点Motivation and Contributions

传统的远程协同（video call）---------缺点：

However, video calls have many limitations such as sharing a small feld of view (FOV), limited resolution, or fxing the view of the remote guest user to that of the local host user.

To overcome such limitations:

VR 可以提供更好的视角和用户体验（宽广的视野和沉浸感）

To overcome such limitations, it could be more efcient if the remote guest user could immerse themselves in a view of the local host user’s environment. Virtual Reality (VR) technology enables this by having the user wear a Head Mounted Display (HMD) that provides an immersive viewing experience with a wider FOV compared to a standard phone or monitor.

VR 的优势:全景和三维重建

采用VR时360全景和三维重建场景可以更好的呈现，但两者各有优缺点，所以需要结合

Sharing 360 panorama views can provide a high quality view without consuming a large amount of bandwidth but it is a 2D presentation that provides limited depth perception. In contrast, sharing a 3D reconstruction supports depth perception as well as the ability to navigate through the 3D model. However, the quality of 3D reconstruction and the amount of bandwidth required to transfer it are directly proportional to each other. So a high-quality 3D reconstruction of the user’s environment would require a signifcant amount of bandwidth and is difcult to update in real time. 

Contributions:

In this paper, we present a novel Mixed Reality (MR) remote collaboration system (see Figure 1) that combines 360 and 3D reconstructions into one. This creates a system that aims to merge the advantages of the individual approaches while minimizing the limitations stated above.

Compared to prior work, this paper makes a number of novel and signifcant contributions: 
(1) A novel MR Remote Collaboration technique that merges 360-views and 3D Reconstruction. (2) The frst user study that compares 360 live panorama and 3D reconstruction based MR remote collaboration systems. (3) The frst user study that explores the benefts and implications of combining 360-view and 3D reconstructed scene into a hybrid MR remote collaboration system. 

RELATED WORK

360 Video Sharing Remote Collaboration

Collaboration using 3D Scene Reconstruction
Visual Cue in Remote Collaboration

Compared to prior work in MR remote collaboration systems, our work propose combining 360 video and 3D reconstruction to complement each other. In this paper, we describe the system design and implementation of our novel MR remote collaboration system that merges live 360 video and 3D reconstruction, and report on a user study that investigates how merging the two techniques would be useful for enhancing remote collaboration. 

SYSTEM OVERVIEW
 

 

Part A - 360 vs. 3D
 

The aim of the frst part of the study was to compare the two remote collaboration mediums: 360 live panorama video and 3D reconstructed scene. The main research questions in this part were: 
(1) How does the remote collaboration medium affect the task performance of collaborative object searching task? 
(2) How does the remote collaboration medium affect the user experience including usability, sense of being together, and motion sickness?
 (3) Would a certain type of medium (360 or 3D) be a dominant choice for remote collaboration? Or would there be room for complementing each other? 

Part B - Hybrid (360 + 3D) System
 

In the second part of the study (Part B), we focused on evaluating the proposed hybrid system that combines 360 and 3D modes. We designed this session as an explorative study where the participants tried using the hybrid system on both the remote and local user’s side, and provided subjective feedback through a questionnaire. 

RESULTS
 

Part A - 360 vs. 3D

Part B - Hybrid (360 + 3D) System

 

DISCUSSION
 

对结论的分析写的很细致，可以借鉴。