3D Interaction in XR
If visual perception is distorted in VR, how would this affect visually guided movements in the virtual environment?
To test this, we used the two visual stream hypothesis, which postulates that optical stimulation at the retina propagates to cognitive systems via two distinct neural streams that are specialized in different types of processing.
The ventral stream (vision-for-perception, or “what”) is commonly activated for object recognition and other static, image-based tasks, whereas the dorsal stream (vision-for-action or “where”) is dominant for spatial processing, action planning, and other motion-related tasks.
 |  |
|---|---|
| Dorsal Stream (Vision for Action) | Ventral Stream (Vision for Perception) |
Harris et al. (2019) suggested that observers/actors inside a virtual environment conveyed through an HMD or other viewing devices rely more on the ventral stream to process visual information for planning and executing movements.
To test this postulate, I used the Müller-Lyer illusion, where the perceived length of the stem varies as a function of the fin configuration:
This illusion’s effect only applies to tasks related to perceptual judgment (i.e., the ventral stream), but not to action planning and execution (i.e., the dorsal stream), such as manually grasping and aiming.
Therefore, I developed three identical experiments in the physical environment, HMD VR (HTC VIVE Pro Eye), and optical see-through (OST) AR (Microsoft HoloLens 2), where the participants were asked to reach to the tip of a Müller-Lyer stimulus, either with visual feedback (dorsal stream) or without (ventral stream).
I showed that 1) AR produced a similar performance as UR that reflected the effect of visual illusion, and 2) VR elicited additional distortions to the movement accuracy independent of the illusion and visual feedback. Moreover, the participants adapted to this distortion and exhibited an adaptation aftereffect in a manual reaching task in the physical environment after exiting VR.
This finding suggests that VR intrinsically engages different neuropathways and elicits different perceptuomotor processes that could fundamentally affect how the users interact with the virtual environment.
Stay tuned for the preprint version of this study to learn more.
Related Publications
Wang, X.M., Southwick, D., Robinson, I., Shou, X., Nitsche. M., Resch, G., Mazalek, A., & Welsh, T.N. (in preparation). Odd One Out: Visual Illusions Reveal Differences in Per-ceptual-Action Processing in Unmediated, Virtual, and Augmented Reality.