Extended Reality期末复习(PolyU)

Individual Assignment

1. Summary

• Personalized avatars significantly increase body ownership, presence, and dominance compared to their generic counterparts, even if the latter were generated by the same photogrammetry process and hence could be valued as equal in terms of the degree of realism and graphical quality.
• The degree of immersion significantly increases the body ownership, agency, as well as the feeling of presence.

2. The first hypothesis proposed by the authors in this journal article is “H1. Increased personalization increases the strengths of the target effects.” What is the null hypothesis of H1? Do you think H1 is a well proposed hypothesis? Give your reasons. (2 points)

• Null hypothesis: Increased personalization does not increase the strengths of the target effects.
• The authors used target effects to represent three potential effects, including E1 virtual body ownership, E2 presence, and E3 emotional response. Thus, H1 can be broken down into three hypotheses.
• It would be better if the authors could indicate the measures for each dependent variable (i.e., E1, E2, and E3).
  pp. 44 – 46 of L6

3. The experimenter gave the participants audio-instructions during the initial trial, such as “Lift your right arm and wave to your mirror image in a relaxed way” and “Now wave with your other hand” (p. 1649). Can you explain the purposes of such instructions?

• For enhancing the participants’ sense of agency (or sense of embodiment).
• A good way for the participants to get familiar with the virtual environment
• Sensorimotor contingency and rubber hand illusion.
  refer to Q3(b) of the in-class test

4. In Section 4.2 and 4.3, the authors gave very detailed specifications of the hardware used for the user study. Do you think it is necessary? Give your reasons. Also, two visualisation systems (L-shape projection and HMD) are used for this user study. Can you explain the major difference between the two systems from the perspective of experimental design and their potential impacts to the results of the study?

• It is necessary for the purpose of reproducibility.
• The two systems are for manipulating immersion, which is an independent variable of this study.

5. The authors of this paper used photogrammetry techniques to create “personalised avatars” from 3D scans of participants. In many commercial VR applications, avatars are created without 3D scans, but an avatar customisation process will be introduced when users first use these applications. During this process, users can pick their avatars’ appearances, of which the outputs are often referred to as “customised avatars”.
   Do you think customised avatars can also affect users’ sense of presence, virtual body ownership, and emotional response? Use prior publications to support your argument and design a laboratory experiment to test your argument.

• Customised avatars can also affect users’ sense of presence, virtual body ownership, and emotional response.
  pp. 66 – 67 of L2
• Lab experiment design
  • Participants – e.g., inclusion and exclusion criteria.
  • Experimental design – objective: the confounding effects can be mitigated by the design.
  • Measures – ways to measure the dependent variables (i.e., presence, virtual body ownership, and emotional response). The simplest way to answer this part is to use exactly the same set of measures used in the original paper.
  • Data analysis – proper employment of data analysis methods given the experimental design.

Mid-term test

1. Statements of Extended Reality (XR) and XR research
   1. One of the shortcomings of photogrammetry is that the generated 3D geometries may be too complex to be used directly in XR applications even for today’s hardware.
   2. Postural instability theory can not be applied to explain cybersickness. p49 of L2
   3. Immersion is an objective description of the affordances of the VR system. Immersion is multi-dimensional. p82 of L2
   4. The sense of self-location under the umbrella of the sense of embodiment means the sense of being in a particular virtual world one’s spatial experience of being inside a body. It does not refer to the spatial experience of being inside a world (with or without a body) – in contrast to sense of presence (or more precisely “place illusion”) p62 of L2
2. Defining Virtual Reality
   1. Vividness and interactivity mean?
      1. Vividness: representational richness of a mediated environment as defined by its formal features, that is, the way in which an environment presents information to the senses.
      2. Interactivity: the extent to which users can participate in modifying the form and content of a mediated environment in real time.
   2. Explain how this figure can help define virtual reality by considering virtual reality as a medium.
      • This figure classifies a wide range of media technologies, both real and fictional, in terms of vividness and interactivity.
      • Some media technologies may share a similar level of vividness, such as films and broadcast HDTV. Some media technologies may share a similar level of interactivity, such as email and fax.
      • VR vary in vividness as well as interactivity. However, it is obvious that VR should be somewhere on the top right area of the graph since it should have higher vividness and interactivity than many conventional media.
3. Where do you think should Metaverse be placed on this two-dimensional plane?
   The ultimate form of the Metaverse is a virtual world which is in parallel with the physical world. Thus, it should have the ultimate vividness and interactivity, just as the physical world we live in. From this perspective, the Metaverse can be place on the top right corner on this 2D plane.
4. Will the location of Metaverse change on this two-dimension plane over time?
   However, the Metaverse and its enabling technologies are constantly evolving. At the moment, we haven’t reached the technological achievements that are needed to create such parallel world. Thus, we can safely place the Metaverse should somewhere to the left and below Star Trek Holodeck. Meanwhile, we also expect it to evolve thus gradually move to the top right corner on this 2D plane
5. Rubber hand illusion
   1. Use a theory we discussed during the lecture to explain why the subjects can only experience the rubber hand as their own when the two hands were “stroked synchronously”.
      1. Sensorimotor contingency
      2. Sensorimotor contingency can be summaries as what we just sensed matches with what we expected moments ago
      3. What we expect to sense (as we see the stroking) should match with we actually sense (as we feel the stroking). Otherwise, what we expect to sense does not match with we actually sense
6. In many research studies, participants were instructed to move their bodies (e.g., raise up hands) when facing virtual mirrors in the virtual environments. The goal is to induce the sense of embodiment. Use your own words to explain why this procedure can help induce the sense of embodiment among the participants based on the rubber hand illusion
   1. This is also related to sensorimotor contingency.
   2. When the participant is instructed to raise up hands, he/she expects that his/her virtual hands will also raise up in the virtual environment.
   3. The image reflected by the virtual mirror will confirm the one-to-one mapping of the participant’s motion to the virtual body’s motion.
   4. The participant believes that he/she has the total control of the motion of the virtual body, which corresponds to the sense of agency under the umbrella of sense of embodiment.
7. Isomorphic and Non-isomorphic p13 of L3
8. Isomorphic
   1. Pro: Natural thus zero learning cost for users
   2. Con: Not as efficient as non-isomorphic
9. Non-isomorphic
   1. Pro: More efficient
   2. Con: Less natural, thus users need some practice to master the techniques
10. UML State Diagram
    Use a UML state diagram to model the spatial user interface for controlling spaceships in No Man’s Sky VR mode. Players use their left hands to control the speed of their spaceships. Specifically, pushing the left-hand control interface forwards can accelerate their spaceships and pulling the interface backwards can deaccelerate. Their right hands are for controlling the directions where their spaceships are heading to (i.e., pitch and yaw controlling). These two interfaces can be operated concurrently.
    [图片]
11. From the perspectives of human sensation and perception as well as the enabling technologies, explain how such experience can be delivered.
12. Stereoscopic vision is very important for us to perceive depth. Stereoscopic vision can provide depth cues via convergence and binocular disparity (i.e., two physiological depth cues).
13. However, there are many other ways to provide depth cues. For example, we also perceive depth via other psychological cues, such as perspective, light and shadow, occlusion, details, known size, and relative motion, etc. Slide 21 to 33 of L2
14. The artist used many perceptual illusions or tricks to deliver the experience, including:
    1. The virtual frame, which gives viewers a strong sense of depth due to occlusion.
    2. The correct perspective when standing in the right front of the installation.
    3. The simulated light sources in the rendering scene generally match with the light sources in the physical space, which again tricks viewers brain to think the frame exists and is casting real shadows to the moving elements of the artwork.
15. The rendering equation Slide 53 to 61 of L4
16. Use your own words to explain how the rendering equation simulates the physical properties of light.
17. Path tracing is one of the well-known rendering techniques based on the rendering equation. Use your own words to summarise path tracing algorithm and explain how path tracing implements but also simplifies the rendering equation.

Lecture 1

1. Goal
2. Good = Comfortable + Adequate
   1. Comfortable: enjoyable, less cybersickness, less cognitive overload, accessibility
   2. Adequate: Adequate functions and details to the rendering, to achieve intended outcomes.
   1. Technologically flawless(less bug)、fidelity(vividness) and interactivity、Documentation and support(tutorial)
3. XR Design
4. Definition of VR

• Inducing targeted behavior in an organism by using artificial sensory stimulation, while the organism has little or no awareness of the interference通过人工感官刺激诱导生物体的目标行为，而生物体很少或根本没有意识到这种干扰

4. Boundary and Presence
5. Presence: sense of being there
   1. Vividness: breadth and depth
   • representational richness of a mediated environment as defined by its formal features, that is, the way in which an environment presents information to the senses.

2. Interactivity: Speed, range and mapping
  - the extent to which users can participate in modifying the form and content of a mediated environment in real time.  

2. Boundary of VR (Vividness and Interactivity)
   2. Explain how this figure can help define virtual reality by considering virtual reality as a medium.
   • This figure classifies a wide range of media technologies, both real and fictional, in terms of vividness and interactivity.
   • Some media technologies may share a similar level of vividness, such as films and broadcast HDTV. Some media technologies may share a similar level of interactivity, such as email and fax.
   • VR vary in vividness as well as interactivity. However, it is obvious that VR should be somewhere on the top right area of the graph since it should have higher vividness and interactivity than many conventional media.

3. Where do you think should Metaverse be placed on this two-dimensional plane? 
  The ultimate form of the Metaverse is a virtual world which is in parallel with the physical world. Thus, it should have the ultimate vividness and interactivity, just as the physical world we live in. From this perspective, the Metaverse can be place on the top right corner on this 2D plane.
4. Will the location of Metaverse change on this two-dimension plane over time?
  However, the Metaverse and its enabling technologies are constantly evolving. At the moment, we haven’t reached the technological achievements that are needed to create such parallel world. Thus, we can safely place the Metaverse should somewhere to the left and below Star Trek Holodeck. Meanwhile, we also expect it to evolve thus gradually move to the top right corner on this 2D plane

5. The difference between Presence and Immersion【p82 of L2，p28 of L1】
6. 3D projection
7. Transmission of the images (multiplexing)
   1. Time-based multiplexing (画图)
   2. Spatial-based multiplexing (画图)
8. Separation of the images
   1. Passive 3D (polarized glasses)
   2. Active 3D (shutter glasses)
9. CAVE: back projections
10. Tracking
11. Controller and HMD tracking
    1. Inside-out: sensors are located on the device
    2. Outside-in: installed in stationary in env
12. Full-body tracking
    1. Optical-based tracking: similar to HMD’s outside-in.
    2. Inertial-based tracking: similar to HMD’s inside-out.
13. Facial Expression Tracking、Hand/Finger Tracking
    1. RGB/RGBD based ML、ML-based
    2. Optical-based tracking、Data gloves
14. Photogrammetry and 3D Scanning
15. Photogrammetry
    1. Pros: price, accessiblity(no radar)、many scales/sizes
    2. Cons: complexity
16. 3D Scanning
    1. Laser
    2. Light

Lecture 2

1. Perception and Sensation
2. Deifnition
   1. Sensation: how sensory info is detected by sensor receptors.
   2. Perception: how sensory info is organised, interpreted, consciously experienced by bain.
3. Process
   1. Stimuli、Sensation、Perception、(process the data)Exposure、Attention(eye,ear)、Interpretation(make sense)、Adaptation
4. Senses: Vision(color[electromagnetic and visible] and depth)、Hearing、Touch、Balance、Taste/Smell
5. RGB, HUE(continuous), HSB(HUE+Saturation and Brightness)
6. Dithering Setting抖动设置
   1. Due to a limited colour palette, use Floyd-Steinberg dithering algorithm
   2. 画图，伪代码 (7, 3, 5, 1)
7. High dynamic range (HDR)强光照对比
8. Depth cues (related to stereoscopic vision)
9. Psychological cues心理 from images
   1. All psychological cues are monocular(单眼)
   2. Consist: Perspective、Sizes of known objects、Details、Occlusion、Light & shadows、Relative motion
10. Physiological cues生理
    1. Some physiological cues are monocular, some are binocular.
    2. Consist
    1. Binocular disparity双眼视差
    2. Convergence汇聚(binocular)
    3. Acoomodation聚焦在特定的深度(monocular)
11. Other Sense
12. Hearing：Software: Attenuation、Direct sound path、In direct sound path反射路径(like raytracing)
13. Touch
    1. Force feedback
    2. Textural(viboratactile, electrotactile, ultrasound) feedback: Sensory/perceptual illusions
    3. Thermal feedback
14. Haptic Rendering and Resolution: different body areas - different s and t resolving capacity of haptics
15. Balance(Vestibular sense)
16. Cybersickness and motion sickness
17. Cybersickness: Mismatch between visual and vestibular senses
    1. Symptoms: Sweating, headache, dizziness.
    2. Vestibular: stationary(do not detect any motion, conflict happens) VS Sight: Moving
    3. Postural instability can explain motion sickness but no cybersickness.
    4. Eye-movement theory - most argued
    5. How to mitigate cybersickness!
    1. Less locomotion required(elevator > stairs)
    2. Avoid artificial locamotioni(teleportation is better)
    3. Use visual tech(adjust FoV)
    4. Use special equipment for locomotion(6-DoF motion platform)
18. Motion sickness: eg: one may be sensitive to roller coasters.
19. Sensorimotor Contingency - a theory support rubber hand(visual and tactile illusion)
20. What we expect to sense (as we see the stroking) should match with what we actually sense (as we feel the stroking). Otherwise, what we expect to sense does not match what we actually sense.
21. Why moving body when facing virtual mirrors in the virtual environments? The goal is to induce the sense of embodiment
    - The image reflected by the virtual mirror will confirm the one-to-one mapping of the participant’s motion to the virtual body’s motion.
    - The participant believes that he/she has total control of the motion of the virtual body, which corresponds to the sense of agency under the umbrella of sense of embodiment.
    • Sense of embodiment
      • Self-location: spatial experience of being inside a body. Place illusion(in contrast to presence)
      • Agency: having global motor control
      • Body ownership: self-attribution of a body

• How did rubber hand illusion enable haptic retargeting?
  • There are some programs to change the direction of the virtual hand. People would not realise it when not significant.

6. Place illusion and Plausibility illusion
7. PI: being in a place【提到地点】
8. Psi: is actually occurring
9. Avatar(by real human) and Agent(by computer)
10. Avatar realism
    1. Behaviour Realism
    2. Appearance Realism
    1. Realism: how close to a human being
    2. Resemblance: how close to an individual
    3. Personalisation: Customised avatars can also affect users’ sense of presence, virtual body ownership, and emotional response.
    4. Uncanny Valley Effect
11. Immersion(objective) vs presence(subjective)
12. Immersion is an objective description of the affordances of the VR system.
13. Immersion is multi-dimensional
14. The level of immersion only focuses on technical specifications of a VR system.

Lecture 3

1. UI and state machine
2. User interfaces’ behavior画图
3. UML State Diagram 画图
4. Isomorphic vs Nonisomorphic
5. Isomorphic
   1. Pro: Natural thus zero learning cost for users
   2. Con: Not as efficient as non-isomorphic
6. Non-isomorphic
   1. Pro: More efficient
   2. Con: Less natural, thus users need some practice to master the techniques
7. Virtual Locomotion and Wayfinding
8. Locomotion in XR (users explore virtual spaces that are larger than physical spaces)
   1. Artificial locomotion: use joystick or trackpad, uncomfortable
   2. Teleportation: aim at a point and instantly teleport to it, easier to get lost.
   1. 画图
9. Wayfinding and Cognitive Map
   1. Spatial information is represented by
   1. Egocentric frame: location of individual.
   2. Allocentric frame: positions of objects relative to each other.
10. UX: how people interact with a product、evaluation: Value、Function、Usability、General impression
11. User-centred Design and Co-design
12. Research->Concept->Design->Testing

Lecture 4

1. Rasterization Rendering Pipepline画图
2. Rotation
3. 2d rotation公式
4. 3d rotation公式
5. Others
6. Gimbal Lock Problem: A special state when two of the three rotation plans are driven into a parallel configuration. The 3D rotation losses on DoF.
   1. To avoid gimbal lock: yzx, zxy
7. View transformations
   1. Perspective透视、Orthographical(keep parallel)、Isometric(ISO)
8. Stereoscopic rendering(2 virtual cameras render to the same physical screen [off-axis frustum])
9. Cornell Box: a test aimed at determining acc of rendering. Comparing rendered img and actual img.
10. Light system
11. How the lights are baked in Unity?
12. Why do we need to do lightmap/probes?
    1. Why lightmap? It is the process of pre-calculating the brightness of surfaces in the scene. Stored as 2d images(textures).
    2. Why probes? Reflection probes allow visual env to be sampled at strategic points in the scene. Store the visual env as a cubemap, which can be baked in advance.
13. How to deal with moving objects if lightmaps are all baked? Light probes.
14. How to deal with moving light sources? Light Source Tracking、Shadow Mapping
15. Visual rendering
16. Rasterization(geometry(polygon) level) vs Raytracing / Path tracing(pixel level) the latter can enhance the former.
17. The Rendering Equation----------------------------------------------------------------------
    1. Use your own words to explain how the rendering equation simulates the physical properties of light.
    2. Path tracing is one of the well-known rendering techniques based on the rendering equation. Use your own words to summarise path tracing algorithm and explain how path tracing implements but also simplifies the rendering equation.
18. Limitations of path tracing algorithm: Shoot too many rays, not efficient. Recursive process. Need to maximize the usefulness and effectiveness of each ray.
19. Others: BSDF, denoising
20. Multisensory and Multimodal XR
21. Multisensory: simulation(Force/textual/Tactile feedback)
    1. Shortcomings: Distraction and adaptation、Cognitive load
22. Multimodal: tracking ability(talk to agents)
23. 画图

Lecture 5

1. Application basis
2. Application Domains: workplace, department, organisation, business
3. Application Development画图
4. Cooperation Picture重点画图，设计XR系统可能要画这个图
5. Kinds of Models
6. Usage model: how functionality of system is represented at the interface and how it can be accessed by end-users. Presenting prototypes(low/high fidelity) is an effective way for usage modeling.
7. Design Model: Tools like UML diagrams(class/object/state diagrams)
8. Implementation Model: pseudo code/ flow chart
9. Usability(is the system easy to use?):
10. effectiveness(acc and completeness)
11. efficiency(resources)
12. satisfaction(comfort and acceptability)
13. User Acceptance Testing(UAT)
14. Objective: verify if meet the needs of end-users or not.
15. Cover the following broad areas
    1. Test cases to validate end-to-end business processes
    2. Comparison of actual test results against expected results
    3. Meeting forum

Lecture 6

1. Cognitive load
2. Intrinsic load: difficult task (train a dentist)
3. Extraneous load: info is presented (UI/Interation tech)
4. Steps to avoid it
   1. Offload tasks: shift unnecessary tasks
   2. Avoid visual clutter: present info
   3. Build on existing mental models (users are familiar)
5. Universal Simulation Principle
6. Any interaction mechanism from the real world can be simulated in VR
7. Realism may not always lead to better interactions
8. Many interactions have no counterpart in physical world
9. Accessible Interface
10. For low vision: provide a brightness len, edge enhancement, text color, depth tool, magnifying glass
11. For motor disabilities: speak, text, recognition of cv.
12. Evaluation
13. Formative (during development) vs Summative (after development)
14. Qualitative (subjective) vs Quantitative (objective)
15. Commonly used instruments in XR evaluations
    1. Presence(IP/SUS/WS Questionnaire)
    2. Social Presence: Social Presence Inventory(SPI)
    3. Cybersickness: Simulator Sickness Quesionnaire(SSQ)
    4. System Usability: System Usability Scale(SUS)
16. Scientific Method
17. Learn related literature and experience
18. Design and implement a new method (a baseline)
19. Formulate the hypothesis (null hypothesis)
20. Design and conduct experiment
    - Experimental design – objective: the confounding effects can be mitigated by the design.
    • Control not of interest、manipulate IV、observe DV
    • Two basic types
      • Between-subjects Design(并联)
      • Within-subjects Design(串联)
    • Participants recruitments – e.g., inclusion and exclusion criteria. Number, sampling.
    • Content and Apparatus
    • Measures and instruments – ways to measure the dependent variables (i.e., presence, virtual body ownership, and emotional response).
    • Data analysis
21. Analyze data and draw conclusions
22. Promote new method
23. Data analysis
24. Statistical Methods
    1. Descriptive statistics(summerize)
    2. Inferential statistics(draw conclusions and make inferences)
25. Independent Sample t-test 公式(signal / noise)，越大越可信
26. Draw conclusion
    1. Correlation: 相关但不导致
    2. Causation: 相关且导致