An In-Depth Analysis of Visual Saliency in Rendered Virtual Reality Environments

Abstract

In virtual reality (VR), visual saliency has been exploited in order to solve different problems in the literature. However, the studies to reveal the underlying mechanism and the potential of visual saliency in VR are still inadequate. In this study, we give a comprehensive analysis of visual saliency in VR displayed by modern head-mounted-displays (HMDs) by comparing various heuristic-based and learning-based saliency prediction approaches in different virtual environments. Our work consists of two main parts which zoom in on the role of the real-timeliness and depth information. To this end, in the first part of our work, we give an analysis of seven heuristic-based real-time saliency prediction methods (5 RGB, 2 RGB-D) in three different virtual environments in desktop (2D) and VR (3D) displaying situations. Our results showed that the methods that use 2D cues based on the relation between objects and their backgrounds have superiority over other methods in the real-time saliency prediction of virtual environments. In the second part, we analyze 5 RGB-D image saliency prediction methods (2 heuristic-based, 3 learning-based) in the same virtual environments under 2D and 3D displaying situations. Moreover, we deepen our investigation and include a heuristic-based RGB video saliency prediction approach with its modified version obtained by injecting depth information into it. Our results in this part showed that the prediction achievements of the learning-based saliency approaches have a strong domination against other methods and the depth cue is a crucial factor in the saliency prediction of virtually created scenes by learning-based approaches. Additionally, we investigate the viewing behaviors of the users under 2D and 3D viewing conditions and reveal the important differences between them by using the data collected throughout a user study. Our findings point out a center bias in both viewing conditions however, it is found to be more apparent in 3D viewing.