关于眼动交互的笔记

眼动问题

Eye tracking research studies investigate the impact of an intervention on the eye movements of a participant. Typically, the gaze samples or fixations, i.e., the periods for which the eye is relatively still, are used to approximate the human visual attention, and are mapped to areas of interest (AOIs) for analysis.

Fixation->AOIs

Typical metrics for the error of gaze estimation include spatial accuracy and spatial precision [21]. Spatial accuracy is commonly computed as the mean angular deviation of fixations to the actual position, and spatial precision as the root mean square error or standard deviation of individual gaze samples from their centroid [21,22].

spatial accuracy and spatial precision
注视到实际位置的平均角度偏差
单个凝视样本与其质心的均方根误差或标准偏差

原生API

MRTK 中的眼动跟踪配置通过 IMixedRealityEyeGazeProvider 接口进行配置。 使用 CoreServices.InputSystem.EyeGazeProvider 在运行时提供在工具包中注册的默认注视提供程序实现。

在 Unity 脚本中访问眼动跟踪数据 — MRTK2：https://learn.microsoft.com/zh-cn/windows/mixed-reality/mrtk-unity/mrtk2/features/input/eye-tracking/eye-tracking-eye-gaze-provider?view=mrtkunity-2022-05

The MRTK primarily focuses on enabling gaze-based interaction via an easy-to-use API for developers. It does not offer recordings for research purposes, nor does it guarantee a fixed sampling rate which is tied to the Unity3D update rate. Hence, gaze samples might be missed. Our system is based on the API for the UWP which provides unsmoothed data, more stable data rates, and a higher level of control. Further, a high precision timestamp in the system-relative QueryPerformanceCounter (QPC) time domain with a precision of 100 ns is provided for each data point.

MRTK不提供用于研究目的的录音，也不保证与Unity3D更新率相关的固定采样率。因此，凝视样本可能会被遗漏。我们的系统基于UWP的API，它提供了无平滑的数据、更稳定的数据速率和更高级别的控制。此外，为每个数据点提供精度为100ns的系统相关

AR-眼动追踪工具包

https://github.com/AR-Eye-Tracking-Toolkit/ARETT
我们的目标是以固定的数据速率、无延迟、尽可能高的空间精度和准确度稳健地提供原始凝视数据。

相关文章：
https://www.mdpi.com/1424-8220/21/6/2234#B13-sensors-21-02234

Each gaze sample includes the origin of the gaze point, its direction vector, and a timestamp. All gaze samples received by the access layer are queued in the data provider and processed in the next frame update in the Unity 3D main thread. For each gaze sample, we cast a ray and check for hits with collider objects in the scene. If the option spatial mapping of the MRTK is enabled for the application, this includes the real environment that is scanned by the depth sensors of the HoloLens 2. If a collider is hit, we extend the gaze sample by the intersection coordinates in the world coordinate system, the object’s name, position, rotation and scale, the intersection point in the object’s local coordinate system, and the gaze point projection to the 2D eye displays. In addition, we support AOI colliders for real-time gaze-to-AOI mapping with support for dynamic AOIs. AOI collider objects can be placed at any position of a Unity 3D scene or attached to virtual objects in the scene. AOIs must be defined during the application development phase. Real-time and gaze-based adaptations can be realized using custom scripts. Synchronized recordings of the gaze signal and the front-facing camera can be used to define further AOIs post-hoc. We separately cast gaze rays to check for hits with AOI colliders. In addition, we offer an option to store the position, rotation and scaling of game objects in the scene in our gaze sample. This can be used to simulate or visualize sequences of interest post-hoc. For each processed sample, we raise an event that can be subscribed by other components, such as the data logger.

每个注视样本包括注视点的原点、其方向向量和时间戳。
对于每个凝视样本，我们投射一条光线，并检查场景中碰撞器对象的命中情况。如果为应用程序启用了MRTK的选项空间映射，则这包括HoloLens 2的深度传感器扫描的真实环境。如果碰撞器被击中，我们将视线样本扩展为世界坐标系中的交点坐标、对象的名称、位置、旋转和比例、对象局部坐标系统中的交点以及到2D眼睛显示器的视线点投影。
此外，我们支持AOI对撞机实时凝视AOI映射，并支持动态AOI。AOI对撞机对象可以放置在Unity 3D场景的任何位置，也可以附着在场景中的虚拟对象上。
必须在应用程序开发阶段定义AOI。可以使用自定义脚本实现实时和基于凝视的改编。凝视信号和前置摄像头的同步记录可用于定义后续AOI。
我们分别投射凝视射线，以检查AOI对撞机是否命中。
此外，我们还提供了一个选项，可以在我们的凝视样本中存储场景中游戏对象的位置、旋转和缩放。这可以用于事后模拟或可视化感兴趣的序列。对于每个处理过的样本，我们都会引发一个事件，其他组件（如数据记录器）可以订阅该事件。

通常，头戴式眼动仪使用世界相机从自我中心的角度记录环境，并将佩戴者的瞳孔位置映射到相应的视频帧。然而，微软HoloLens 2的集成眼动仪将瞳孔位置映射到设备的3D坐标系中的凝视光线。 我们的工具包为3D场景添加了一个虚拟摄像头，该摄像头与集成前置摄像头的位置、投影和分辨率相匹配。这使得能够将3D凝视位置投影到虚拟2D摄像机图像，并因此投影到网络摄像机图像。2D凝视信号通过数据提供者的凝视样本事件报告，并记录在gazePointWebcam列中。

（如果仅需要用于演示目的的视频流或捕获，则可以使用HoloLens 2的混合现实捕获（MRC）模块。我们的工具包通过将一个小球体附加到当前凝视位置来支持该模块中的凝视可视化，该位置在捕捉中可见，但对用户不可见。然而，这种方法在计算上要求很高，它将所有应用程序的帧速率限制在每秒30帧，并且对实时交互式应用程序有负面影响，这将其用于演示目的。）

R语言包

We provide a new R package that supports this data paradigm. It offers offline fixation detection with corresponding pre- and post-processing routines. The R package and detailed documentation is published on GitHub (https://github.com/AR-Eye-Tracking-Toolkit/ARETT-R-Package, accessed on 22 March 2021) under the MIT open-source license.

gap fill and noise reduction

We implement two functions for pre-processing the raw gaze data, gap fill and noise reduction, similar to Reference [31]. The gap fill function linearly interpolates between valid gaze points with small gaps in between, e.g., due to loss of tracking. The noise reduction function applies a mean or median filter to the gaze data with a given window size.

The Tobii I-VT Fixation
Filter
Algorithm description
March 20, 2012
Anneli Olsen
http://www.vinis.co.kr/ivt_filter.pdf

The gap fill function :linearly interpolates between valid gaze points with small gaps in between, e.g., due to loss of tracking. The noise reduction function applies a mean or median filter to the gaze data with a given window size.
间隙填充函数在有效注视点之间进行线性插值。

Three methods from the literature for offline fixation detection are implemented. This includes I-VT using a velocity threshold similar to Reference [31], I-DT for VR as described by Llanes-Jurado et al. [32] using a dispersion threshold, and I-AOI proposed by Salvucci and Goldberg [33] based on detected areas of interest. Our implementation of I-VT follows the description by Olsen [31]. It reproduces a similar behavior based on the data recorded using our toolkit. We calculate a velocity for each gaze point over a specified duration and categorize the points by comparing the velocities to a specified threshold. I-DT follows the implementation by Llanes-Jurado et al. [32]. It computes the angular dispersion distance over a window of a specific size in terms of its duration. If the initial window exceeds this threshold it is moved forward until it does not exceeded the threshold. Then, the window is extended to the right until the dispersion threshold is exceeded. All samples in the window, excluding the last sample, are classified as belonging to a fixation. Afterwards, a new window is initialized at the position of the last gaze sample. These steps are repeated until all samples are classified. The I-AOI method for fixation detection is based on Salvucci and Goldberg [33]. It differs from the other methods as it classifies fixations based on predefined areas of interest. First, all gaze points within an AOI are classified as belonging to a fixation. Next, groups of fixation samples are identified as a fixation event using a minimum duration threshold. Short events are discarded.

①速度阈值的I-VT
②I-DT，使用色散阈值
③基于检测到的感兴趣区域提出的I-AOI。

In addition, we provide two functions for post-processing of detected fixations: merging adjacent fixations and discarding short fixations. The merge adjacent fixations function merges subsequent fixations if the gap is smaller than a defined maximum duration and, depending on the detection algorithm used, a maximum angle between them (I-VT) or a maximum dispersion distance (I-DT). For I-AOI, the two fixations must belong to the same AOI. The discard short fixations function removes short fixations based on a minimum fixation duration and is mainly interesting for the I-VT method because both other methods inherently contain a minimum fixation duration.

合并相邻的注视merging adjacent fixations 和丢弃短注视discarding short fixations：如果间隙小于定义的最大持续时间，并且根据所使用的检测算法，它们之间的最大角度（I-VT）或最大分散距离（I-DT），则合并相邻注视功能合并后续注视。对于I-AOI，两个固定必须属于同一AOI。丢弃短固定功能基于最小固定持续时间去除短固定，并且对于I-VT方法来说主要是感兴趣的，因为两种其他方法固有地包含最小固定持续期。

However, for the integrated eye tracker of the Microsoft HoloLens 2 we only find limited information about **spatial accuracy and no information about spatial precision有限的准确度没有 ** [26]. We conduct a user study to analyze the accuracy and precision of gaze data from the HoloLens 2 that is recorded using our toolkit. We ask participants to fixate a set of targets, which have a static position with respect to the participant’s head, at different distances. We record the gaze signal while the participants are seated (setting I) or walking (setting II). Further, we ask them to fixate a target with a static world position while moving around (setting III). The results can serve as a reference for researchers when designing eye tracking studies, e.g., to decide whether the accuracy is sufficient, or to influence the position and size of AOIs. In addition, our results can guide interaction designers that develop gaze-based AR applications, for example to improve gaze-based selection [34].

评估设计

当参与者就座（设置I）或行走（设置II）时，我们记录凝视信号。此外，我们要求他们在移动时固定一个具有静态世界位置的目标（设置III）。研究结果可作为研究人员设计眼动追踪研究的参考，例如，确定准确度是否足够，或影响AOI的位置和大小。我们要求参与者从左上角开始，从左上到右和从上到下的方向固定所有目标三秒钟。

每个任务单独记录，导致设置 I 和 II 的每个距离记录一个记录，以及设置 III 的单个记录。对于设置I和II，我们执行手动注视检测并删除属于扫视事件的凝视样本。我们对凝视信号进行了手动注释，以比使用自动算法更准确地提取注视点，自动算法尤其存在移动眼动追踪信号中的事件检测问题[35]。注视样本被标记为属于注视点，除非注视位置远离注视中心，即当变成在下一个注视中心结束的扫视时。标签基于一位专家的目视检查。

论文 Been There, Seen That: Visualization of Movement and 3D Eye Tracking Data from Real-World Environments

Vision Pro

开发手册：
https://developer.apple.com/documentation/visionos/
看文档，当用户看向一个按钮并使用手势🤌的时候，会出发 click 事件。感觉像是封装了，只提供事件信息？
开发者论坛：
https://developer.apple.com/forums/thread/732552
看起来有一个问题 目前vision os没有开放eye position数据的访问权限，可能是出于数据安全的原因（存疑？