展望混合现实的未来–第一部分

In this 3-part series we analyze areas we believe should be addressed for mixed reality to become mainstream, and not only to improve existing solutions, but reinvent and influence the future and mass adoption of immersive technologies including augmented reality.

在这个由三部分组成的系列文章中，我们分析了混合现实成为主流的领域，我们认为应该解决这些领域，不仅要改善现有解决方案，而且要重新发明和影响包括增强现实在内的沉浸式技术的未来和大规模采用。

The Mixed Reality Research Group is a recently created team I lead within Unity Labs. The team was brought together to study the impact and future of mixed/augmented reality technologies in the years to come and formulate a strategy to empower Unity developers to be at the forefront of how games and applications will be created, deployed and consumed in what is expected to be a new era of computing. The group conducts advanced research, hypothesizes about the long term future, and predicts scenarios.  Our mission is to assist R&D in properly supporting our developers and get them ready for the long term future.

混合现实研究小组是我在Unity Labs中领导的最近创建的团队。 该团队汇聚一堂，研究未来几年混合/增强现实技术的影响和未来，并制定战略，使Unity开发人员能够在如何创建，部署和使用游戏和应用程序方面处于最前沿。有望成为计算的新时代。 该小组进行了高级研究，对长期的未来进行了假设，并预测了情况。 我们的使命是协助研发为开发人员提供适当的支持，并使他们为长期发展做好准备。

Why did we create this group?  The reasons are many, but at its core, the purpose of the group is to get ready for nascent technologies that will become mainstream in the long term future. Additionally, another part of our mission is to explore new tech in order to inspire devs and studios by showing them what will be possible in the future; to go beyond “what is proven” by experimenting with new scenarios.

我们为什么创建这个小组？ 原因很多，但从根本上讲，该小组的目的是为新兴技术做好准备，这些新兴技术将在长期内成为主流。 此外，我们任务的另一部分是探索新技术，以向开发人员和工作室展示未来的发展潜力，从而激发他们的灵感。 通过尝试新的场景来超越“已证明的内容”。

Why focus on Mixed Reality (MR)?  Let us clarify AR, VR, MR and other terminology first. Milgram’s 1994 paper “Augmented Reality: A class of displays on the reality-virtuality continuum” regards  experiences in the purely physical world and experiences in the purely synthetic world of virtual reality as two opposite ends of a spectrum named the Reality-Virtuality Continuum. Within this spectrum, anything that combines (at varying degrees) real and virtual elements on the same display is said to be a Mixed Reality (MR) experience.  Mixed Reality is itself a continuum, and includes Augmented Reality (AR).

为什么要关注混合现实(MR) ？   首先让我们阐明AR，VR，MR和其他术语。 米尔格拉姆(Milgram)在1994年发表的论文 “增强现实：一类关于现实-虚拟连续性的展示” 将纯粹的物理世界中的经验和纯粹的虚拟现实的合成世界中的经验视为 现实-虚拟连续性 光谱的两个相对两端。 在此频谱范围内，将(在不同程度上)在同一显示器上组合真实和虚拟元素的任何事物称为混合现实(MR)体验。 混合现实本身就是一个连续体，包括增强现实(AR)。

The reality-virtuality continuum concept was introduced in Milgram’s 1994 paper “Augmented Reality: A class of displays on the reality-virtuality continuum.”  Image credit: Matteo Valoriani, Etna dev 2016 – Introduction to Mixed Reality with HoloLens

现实-虚拟连续性概念在Milgram 1994年的论文“增强现实：关于现实-虚拟连续性的一类展示”中引入。 图片来源：Matteo Valoriani，Etna dev 2016 – HoloLens的混合现实介绍

The original definition of virtuality continuum only took into account visual stimuli. In 2009, Jeon and Choi extended it to include the sense of touch.  In this 2-axes continuum, mixed reality encompasses any experience that combines real and virtual visual or haptic elements.  As an example, our research prototype of Carte Blanche (CB) is an Oculus Rift application in which the virtual creation desk is aligned with a physical one in order to facilitate longer authoring sessions due to the added comfort. Additionally, we make use of the Touch controllers API to simulate haptic feedback through vibration.  These elements make CB a mixed reality experience, specifically falling in the visual virtuality – haptic mixed reality (vV-hMR) category.

虚拟连续体的原始定义仅考虑了视觉刺激。 在2009年，Jeon和Choi 将其扩展到包括触摸感 。 在这个2轴连续体中，混合现实包含将真实和虚拟视觉或触觉元素结合在一起的任何体验。 例如，我们的 Carte Blanche (CB) 研究原型 是Oculus Rift应用程序，其中虚拟创建台与物理创建台对齐，以增加舒适度，从而有助于更长的创作时间。 此外，我们利用触摸控制器API通过振动模拟触觉反馈。 这些元素使CB成为混合现实体验，特别是视觉虚拟-触觉混合现实(vV-hMR)类别。

Researchers Jeon and Choi added the touch dimension to the reality-virtuality continuum, resulting in a 2-axis spectrum where MR consists of experiences that combine real and virtual visual or haptic elements (gray areas in the figure).

研究人员Jeon和Choi将触摸维度添加到了现实与虚拟的连续性中，形成了一个2轴频谱，其中MR包含了结合了真实和虚拟视觉或触觉元素(图中的灰色区域)的体验。

演示地址

Our Carte Blanche research prototype is a mixed reality experience that combines real and simulated haptic feedback.  The virtual creation desk is aligned with a physical one, and we also simulate haptic feedback through the Touch controllers API.
我们的Carte Blanche研究原型是结合了真实和模拟触觉反馈的混合现实体验。 虚拟创建台与物理创建台对齐，我们还通过触摸控制器API模拟触觉反馈。

Jeon and Choi’s “composite visuo-haptic reality-virtuality continuum” covers just two of the traditionally recognized human senses. What about the others? Work on multisensory immersive experiences has been ongoing for quite some time; a taste simulator that includes haptic feedback was created as far back as 2003 at the University of Tsukuba, and a more recent example of a  taste simulator came from  the National University of Singapore.  Work on systems for generating smells during virtual sessions has been advancing as well, just this year a Japanese startup showed off one of the latest odor emitting attachments for VR headsets.  So, should we add more dimensions to the figure until all five classic human senses are included? Or should we go with other approaches that define a multidimensional space of criteria for senses?  Another important question with respect to the definition of the reality-virtuality continuum is: Why are some senses, vision in particular, considered necessary while others are not?

Jeon和Choi的“视觉-触觉现实-虚拟综合连续体”仅涵盖了两种传统上公认的人类感官。 那其他人呢？ 关于多感觉沉浸式体验的工作已经进行了相当长的时间。 早在2003年，筑波大学就创建了包含触觉反馈 的 味觉模拟器 ， 而新加坡国立大学也提供了 一个 味觉模拟器的 最新示例 。 在虚拟会议中产生气味的系统的工作也在不断发展，就在今年， 一家日本初创公司 展示了最新的VR头戴式发臭附件。 那么，在把所有五种经典的人类感觉都包括在内之前，我们是否应该在图形上增加更多的尺寸？ 还是应该采用其他方法来定义 感官标准的多维空间 ？ 关于现实-虚拟连续性定义的另一个重要问题是：为什么某些感觉，特别是视觉被认为是必需的，而其他感觉则没有？

Regardless of the number of dimensions we add to the definition, the reason our group uses the concept of mixed reality continuum is so that we don’t box ourselves into a limited vision or particular type of experience (like augmented reality is).  Virtual reality is one extreme of the virtuality continuum, where experiences happen in a controlled and purely synthetic environment. In that way, it is like TV, radio, or even a book: you experience it for a while and then return to reality. It’s a way to escape reality and experience non-existing or remote worlds, and thus it provides rich entertainment/education and is essential for designing new scenarios. But we believe hybrid virtual+physical immersive technologies will dominate for day-to-day, continuous uses.  Mixed reality will be used widely because it will be experienced as a consistent overlay in everyday life — Carte Blanche is one example. A second example is providing a persistent layer on top of reality that will ultimately allow AR to replace your smartphone, TV and other day-to-day devices.  The current wave of immersive technologies started with consumer VR headsets. Devices that are able to switch to-and-from VR/AR then become the ultimate step in this evolution. MR will become the dominant technology for everyday life, and we are here to help ensure Unity empowers creators to transfer their imaginations into experiences.

无论我们在定义中添加多少维度，我们小组使用混合现实连续体概念的原因都是为了使我们不会将自己置于有限的视野或特定类型的体验中(就像增强现实一样)。 虚拟现实是虚拟连续性的一种极端，在这种环境中，经验是在受控的纯合成环境中发生的。 这样，就好像电视，广播甚至一本书一样：您体验了一段时间后又回到了现实。 这是一种逃避现实并体验不存在或遥远的世界的方法，因此它提供了丰富的娱乐/教育，对于设计新的场景至关重要。 但是我们相信虚拟+物理沉浸式混合技术将在日常连续使用中占主导地位。 混合现实将被广泛使用，因为它会作为经验丰富的在日常生活中一贯的叠加- 全权委托 I S 一个例子 。 第二个示例是在现实之上提供一个持久层，最终将允许AR取代您的 智能手机，电视和其他日常设备。 当前的沉浸式技术浪潮始于消费者VR头盔。 能够在VR / AR之间来回切换的设备成为这一演进的最终步骤。 MR将成为日常生活中的主导技术，我们在这里帮助确保Unity使创作者能够将他们的想象力转化为体验。

预测 (Forecasting )

We envision that within the next five years a big leap in display resolution will push mixed reality towards more constant usage making it possible for people to start working directly in MR devices for long periods of time. The readability of text will be a major driving factor for the adoption of this usage.  Another factor for enabling constant usage will be the ability to easily switch between VR and AR.

我们预计，在未来五年内，显示分辨率的巨大飞跃将推动混合现实向更稳定的使用方向发展，使人们有可能长时间直接在MR设备中直接工作。 文本的可读性将是采用这种用法的主要驱动因素。 实现持续使用的另一个因素是能够 在VR和AR之间 轻松 切换 的能力 。

MR is poised to become more social, and since sensor occlusion will still be an issue when multiple people are in a room, we can expect more mixed positional tracking with sensors; for example we could see static LiDAR on a chip for point cloud generation combined with inside-out tracking from MR devices.

MR有望变得更加社交化，并且当多个人在一个房间中时，由于传感器的遮挡仍将是一个问题，因此我们可以期待更多与传感器混合的位置跟踪； 例如，我们可以在芯片上看到静态LiDAR，以生成点云并结合MR设备的内外跟踪。

Wireless systems and eye tracking will be the norm, as well as articulated hand tracking through untethered sensors, leading people to become unglued from their PCs. Instead, they will sit around a living room table playing virtual board games with friends.  And with a proliferation of virtual screens, they could sit on a couch watching e-sports, or lie on a bed playing video games. That means we have to change our mindset regarding MR app design to fit these new norms.

无线系统和眼睛跟踪将成为常态，同时通过不受限制的传感器进行关节式手跟踪将成为常态，这将使人们与PC脱钩。 相反，他们将坐在客厅的桌子旁，与朋友一起玩虚拟棋盘游戏。 随着虚拟屏幕的普及，他们可以坐在沙发上看电子竞技，也可以躺在床上玩电子游戏。 这意味着我们必须改变我们对MR应用程序设计的观念，以适应这些新规范。

Autonomous digital actors (holobots, avatars) will populate mixed reality layers. For example,  voice assistants will probably have an emotive voice indistinguishable from a real human voice and have a virtual body to more directly interact with the user (maybe to sit next to them on a chair, walk ahead to show directions…etc.)

自主数字角色(全息机器人，化身)将填充混合现实层。 例如，语音助手可能会具有与真实人的声音无法区分的情感声音，并具有虚拟的身体以与用户进行更直接的交互(例如，坐在他们旁边的椅子上，向前走以显示方向等)。

MR applications such as holobots will become ubiquitous in our daily lives.

全息机器人等MR应用程序将在我们的日常生活中无处不在。

What will our group be doing in 2017-2018?  We’ll be posting on a regular basis to let you know, but as a bit of a teaser, here are some of the interesting areas of MR that we believe should be improved and that we briefly touched on our presentation at SIGGRAPH 2017:

我们团队在2017-2018年将做什么？ 我们会定期发布信息，以通知您，但作为一个预告片，以下是一些我们认为有待改进的有趣的MR领域，我们 在SIGGRAPH 2017 上简要介绍了我们的 演讲 ：

I. Applications are invisible, invisible things are forgettable

一，应用是看不见的，看不见的东西是可以忘记的

As we alluded to above, MR devices could eventually probably replace smartphones.  And with Apple’s ARKit available in all iOS devices, and Google’s ARCore in Android devices, the transition from smartphone to full fledged MR device has already begun.  Along with devices, the mobile application market has reached a maturity which precipitates a saturation problem. Typically, people use applications only once  after download , and over time they slowly forget about them.

如上所述，MR设备最终可能会取代智能手机。 随着苹果的ARKit可在所有iOS设备中使用，以及谷歌的ARCore在Android设备中使用，从智能手机到成熟的MR设备的过渡已经开始。 随着设备的发展，移动应用市场已经 趋于成熟 ，这引发了饱和问题。 通常，人们在下载后仅使用一次应用程序，随着时间的流逝，他们会逐渐忘记它们。

“Gaming, the app category formerly known as “the darling of the mobile industry,” saw time-spent decline by 4% year-over-year. […]

“游戏，以前被称为“移动行业的宠儿”的应用类别，所花费的时间同比下降了4％。 […]

After 10 Years, Mobile Reaches Moment of Truth.  As the iPhone celebrates its first decade, the mobile industry has grown into a dog-eat-dog world. The decelerating rate of growth could signal market maturity, saturation or simply the end of the app gold rush” (On Their Tenth Anniversary, Mobile Apps Start Eating Their Own)

10年后，Mobile达到了关键时刻。 随着iPhone庆祝成立十周年，移动行业已经发展成为一个“狗吞狗”的世界。 增长速度的下降可能预示着市场的成熟，饱和或仅仅是应用程序淘金热的结束” ( 在其成立十周年之际，移动应用程序开始吃饱了自己的东西 )

The “1 content / service = 1 mobile application” mode of business is no longer possible in a world that has reached saturation and reusing the principle of an AR application to replace a flat application will not be the solution. All brands / companies / startups will try to find other vectors to interact with consumers. We’re looking into ways of making MR apps more engaging and less of a one-off occurrence.  As our colleague Mauricio Vergara points out, “With the adoption of immersive computing at scale, real time 3D graphics will become the way we interact with the world.”

在一个已经达到饱和的世界中，不再可能使用“ 1内容/服务= 1移动应用程序”的业务模式，而重用AR应用程序的原理代替平面应用程序将不是解决方案。 所有品牌/公司/初创公司都将尝试寻找其他媒介与消费者互动。 我们正在研究使MR应用程序更具吸引力，减少一次性事件的方法。 正如我们的同事Mauricio Vergara 指出的那样 ：“随着大规模沉浸式计算的采用，实时3D图形将成为我们与世界互动的方式。”

II. Expanding mixed reality beyond current perceptions

二。 将混合现实扩展到当前的观念之外

Until now, MR applications were primarily designed as solitary and single-use experiments. Because of that, many of today’s user scenarios are somewhat limited and seem to lean towards being novelties rather than engaging solutions.

到目前为止，MR应用程序主要设计为单独的和一次性的实验。 因此，当今的许多用户场景都受到一定程度的限制，并且似乎倾向于新颖性而不是吸引人的解决方案。

“The most thrilling mixed reality experience involves real-time, 3D mapping of the environment, which enables virtual objects to interact with surfaces and objects in the real world. For example, a computer-generated creature that can stand on a table — or hide behind it.” (Why Google and Apple will rule mixed reality By Mike Elgan)

“最激动人心的混合现实体验涉及环境的实时3D映射，它使虚拟对象能够与现实世界中的表面和对象进行交互。 例如，一个计算机生成的生物可以站在桌子上或藏在桌子后面。” ( 为什么Google和Apple将统治混合现实，作者Mike Elgan )

To create the uses of tomorrow we must change perspective and ask ourselves interesting questions: does reality disappear when we close our eyes? In the future, devices could potentially no longer be used to generate a sandboxed reality, but rather devices will be a tool that connects users to a persistent layer of reality. For example, museums could create interactive captions around masterpieces, sports stadiums could display real time virtual FX above players in motion on the field, restaurants could display interactive menus on tables…etc. The possibilities are endless, and in this future, users just need to be in the right place to take advantage of the augmented layer.  Platforms like Google’s Visual Positioning Service (VPS) which brings quick and accurate indoor location mapping and understanding, will help provide seamless augmented experiences anywhere. The MR Research Group will look into creating tools that will help users and devs align their thinking with this new perspective.

为了创造明天的生活，我们必须改变看法，问自己一个有趣的问题：当我们闭上眼睛，现实会消失吗？ 将来，设备可能不再用于生成沙盒现实，而设备将成为将用户连接到持久现实层的工具。 例如，博物馆可以在杰作周围创建交互式字幕，运动场可以在运动中的运动员上方显示实时虚拟FX，餐厅可以在桌上显示交互式菜单等。 可能性是无止境的，在未来，用户只需要在正确的位置使用扩展层即可。 诸如Google的 视觉定位服务 (VPS)之类的平台可带来快速，准确的室内位置地图和理解，将有助于在任何地方提供无缝的增强体验。 MR研究小组将研究创建工具，以帮助用户和开发人员将其思想与这一新观点保持一致。

III.  Future MR experiences should be more social

三， 未来的MR体验应该更具社交性

Many current MR experiences do not have a strong enough social component and end up being a lonely affair for both the user and observer; we want to help change that paradigm. There are some notable social VR apps that have sprung up this past year, but social augmented reality experiences are more difficult to create and are just in their infancy.  Often, even if an observer is in the same room as the participant, it is difficult for this observer to imagine the user’s experience because they typically can’t see anything.

当前的许多MR经验都没有足够强大的社交成分，最终对于用户和观察者都是孤独的事情。 我们想帮助改变这种范例。 去年有一些引人注目的社交VR应用程序出现，但是社交增强现实体验更难以创建，而且还处于起步阶段。 通常，即使观察者与参与者在同一个房间，该观察者也很难想象用户的体验，因为他们通常看不到任何东西。

Over the past few months, we have observed many augmented reality users during events; our conclusion is that they get bored, no longer interact with the wearer, and ignore them as if they were no longer part of their reality (here are some example videos). In a similar way, it is important to keep in mind that the popularity of VR does not come from people who try the VR HMD themselves, but rather from the people who demo it on YouTube with a “Mixed Reality View”. Seeing both the content and the user’s reaction is what generates the interest for the observer.  

在过去的几个月中，我们在活动期间观察到许多增强现实用户。 我们的结论是，他们觉得无聊，不再相互作用与佩戴者，而忽略他们，如果他们自己的现实不再一部分 ( 这里 有一些例子，视频 )。 以类似的方式，请记住，VR的流行不是来自亲自尝试VR HMD的人，而是来自通过“ 混合现实视图 ” 在YouTube上进行演示的人 。 既看到内容又看到用户的React，就会引起观察者的兴趣。

Mark Billinghurst tells us augmented reality can bring empathic computing: “Systems that allow us to share what we are seeing, hearing and feeling with others”.  It is an amazing idea, like obtaining new superpowers, however, it comes at a cost. Increasing the social aspect naturally raises questions regarding privacy and freedom, which will be big issues for the future of immersive technologies regardless of the specifics of the implementation. Privacy and freedom must be considered by everyone involved: engineers, artists, designers, and policymakers alike. Since MR tech could be ubiquitous within a decade, discussions need to start now, and we will be contributing.

马克·比林赫斯特(Mark Billinghurst)告诉我们， 增强现实可以带来 移情计算 ：“使我们能够与他人共享我们所看到，听到和感觉到的东西的系统”。 这是一个了不起的想法，就像获得新的超级大国一样，但这是有代价的。 社会方面的增加自然会引发有关隐私和自由的问题，这将成为沉浸式技术未来的重大问题，无论其实现的具体细节如何。 涉及到的每个人都必须考虑隐私和自由：工程师，艺术家，设计师和政策制定者都应如此。 由于MR技术可能在十年内无处不在，因此讨论必须从现在开始，我们将做出贡献。

Social aspects of MR will be important for both personal and business uses.

MR的社会方面对于个人和企业用途都将是重要的。

IV. Game Authoring Evolution

IV。 游戏创作演进

The future of gaming and game authoring has never been so bright. MR brings with it a whole new era filled with new types of games and new ways to play them. Being Unity, we are in a unique position to affect that future.  Our group will help study how games can and will evolve in this new era of mixed reality.  How will games be played? When will they be played? How persistent will they be in regard to location and participation? How will they be created?  These are important questions, and mixed reality provides a sea of possibilities to answer them.  Part of our work will hopefully help guide Unity into that future.  That being said, Unity’s core strength is its developers community and our group’s work will help influence the creation of a concrete plan for our developers to be successful in this new world.  We will be involved with other groups within Unity on the creation of new and improved MR focused development tools that will help drive our user’s success.

游戏和游戏创作的未来从未如此光明。 MR带来了一个全新的时代，充满了新类型的游戏和新的玩法。 作为团结，我们处于影响这一未来的独特位置。 我们的小组将帮助研究游戏如何在新的混合现实时代中发展。 游戏将如何进行？ 他们什么时候播放？ 他们将在位置和参与方面保持多久？ 如何创建它们？ 这些是重要的问题，混合现实为回答这些问题提供了无限可能。 我们的部分工作有望帮助指导Unity迈向未来。 话虽这么说，Unity的核心优势是其开发人员社区，而我们团队的工作将有助于影响为我们的开发人员在新世界取得成功而制定具体计划的过程。 我们将与Unity内的其他小组合作，共同创建新的和经过改进的，针对MR的开发工具，这将有助于推动用户的成功。

In part 2 and 3 of this series we will explore the implications of the above issues and will dive into the design challenges we face with this new medium. Part 2 has been published and can be seen here. Part 3 can be found here.

在本系列的第2部分和第3部分中，我们将探讨上述问题的含义，并将深入探讨这种新媒介所面临的设计挑战。 第2部分已经出版，可以在这里看到。 第三部分可以在这里找到。

Article contributors: Greg Madison, Lead UX/IxD Designer and futurist; Colin Alleyne, Senior Technical Writer; and Sylvio Drouin, VP – Unity Labs.  

文章贡献者：UX / IxD首席设计师和未来主义者Greg Madison； Colin Alleyne，高级技术作家； 和Unity Labs副总裁Sylvio Drouin。

翻译自: https://blogs.unity3d.com/2017/09/05/looking-to-the-future-of-mixed-reality-part-i/